You can use the Cloud Firestore managed export and import service to recover from accidental deletion of data and to export data for offline processing. You can export all documents or just specific collections. Likewise, you can import all data from an export or only specific collections. Data exported from one Cloud Firestore database can be imported into another Cloud Firestore database. You can also load Cloud Firestore exports into BigQuery.
This page describes how to export and import Cloud Firestore documents using
the managed export and import service and Cloud Storage. The
Cloud Firestore managed export and import service is available through
command-line tool and the Cloud Firestore
API (REST, RPC).
Before you begin
Before you can use the managed export and import service, you must complete the following tasks:
- Enable billing for your Google Cloud Platform project. Only GCP projects with billing enabled can use the export and import functionality.
- Create a Cloud Storage bucket for your project in a location near your Cloud Firestore database location. You cannot use a Requester Pays bucket for export and import operations.
Make sure your account has the necessary permissions for Cloud Firestore and Cloud Storage. If you are the project owner, your account has the required permissions. Otherwise, the following roles grant the necessary permissions for export and import operations and for access to Cloud Storage:
gcloud for your project
Set up the
gcloud command-line tool and connect to your project in one of the
gcloudfrom the Google Cloud Platform console using Cloud Shell.
gcloudis configured for the correct project:
gcloud config set project [PROJECT_ID]
If not already installed, add the
component which contains the Cloud Firestore export and import commands:
gcloud components install beta
An export operation copies documents in your database to a set of files in a Cloud Storage bucket. Note that an export is not an exact database snapshot taken at the export start time. An export may include changes made while the operation was running.
Export all documents
firestore export command to export all
the documents in your database, replacing
[BUCKET_NAME] with the name of your
Cloud Storage bucket. Add the
--async flag to prevent the
gcloud tool from
waiting for the operation to complete.
gcloud beta firestore export gs://[BUCKET_NAME]
You can organize your exports by adding a file prefix after
the bucket name, for example,
you do not provide a file prefix, the managed export service
creates one based on the current timestamp.
Once you start an export operation, closing the terminal does not cancel the operation, see cancel an operation.
Export specific collections
To export specific collections, use the
gcloud beta firestore export gs://[BUCKET_NAME] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2]
Once you have export files in Cloud Storage, you can import documents in those files back into your project or to another project. Note the following points about import operations:
When you import data, the required indexes are updated using your database's current index definitions. An export does not contain index definitions.
Imports do not assign new document IDs. Imports use the IDs captured at the time of the export. As a document is being imported, its ID is reserved to prevent ID collisions. If a document with the same ID already exists, the import overwrites the existing document.
If a document in your database is not affected by an import, it will remain in your database after the import.
Import all documents from an export
firestore import command to import documents from a
previous export operation.
gcloud beta firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/
[EXPORT_PREFIX] point to the location of your
export files. For example:
gcloud beta firestore import gs://exports-bucket/2017-05-25T23:54:39_76544/
You can confirm the location of your export files in the Cloud Storage browser in the Google Cloud Platform Console:
Once you start an import operation, closing the terminal does not cancel the operation, see cancel an operation.
Import specific collections
To import specific collections from a set of export files, use the
--collection-ids flag. You can import specific
collections only from an export of specific collections. You cannot import
specific collections from an export of all documents.
gcloud beta firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/ --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2]
Managing export and import operations
After you start an export or import operation, Cloud Firestore assigns the operation a unique name. You can use the operation name to delete, cancel, or status check the operation.
Operation names are prefixed with
However, you can leave out the prefix when specifying an operation name for
List all export and import operations
operations list command to see all running and
recently completed export and import operations:
gcloud beta firestore operations list
Check operation status
operations describe command to show the status of an export
or import operation.
gcloud beta firestore operations describe [OPERATION_NAME]
Estimate the completion time
A request for the status of a long-running operation returns the metrics
workCompleted. Each of these metrics is returned in both
number of bytes and number of entities:
workEstimatedshows the estimated total number of bytes and documents an operation will process.
workCompletedshows the number of bytes and documents processed so far. After the operation completes, the value shows the total number of bytes and documents that were actually processed, which might be larger than the value of
workEstimated for a rough progress estimate. This
estimate might be inaccurate, because it depends on delayed statistics
Cancel an operation
operations cancel command to stop an operation in progress:
gcloud beta firestore operations cancel [OPERATION_NAME]
Cancelling a running operation does not undo the operation. A cancelled export operation will leave documents already exported in Cloud Storage, and a cancelled import operation will leave in place updates already made to your database. You cannot import a partially completed export.
Delete an operation
operations delete command to remove an operation from the
operations list. This command will not delete export files from
gcloud beta firestore operations delete [OPERATION_NAME]
Billing and pricing for export and import operations
You are required to enable billing for your Google Cloud Platform project before you use the managed export and import service. Export and import operations are charged for document reads and writes at the rates listed in Cloud Firestore pricing.
The costs of export and import operations do not count towards your spending limit. Export or import operations will not trigger your Google Cloud Platform budget alerts until after completion. Similarly, reads and writes performed during an export or import operation are applied to your daily quota after the operation is complete.
Export to BigQuery
You can load data from a Cloud Firestore export into BigQuery,
but only if you specified a
collection-ids filter. See
Loading data from Cloud Firestore exports.