Google is committed to advancing racial equity for Black communities. See how.

Export and import data

You can use the Cloud Firestore managed export and import service to recover from accidental deletion of data and to export data for offline processing. You can export all documents or just specific collections. Likewise, you can import all data from an export or only specific collections. Data exported from one Cloud Firestore database can be imported into another Cloud Firestore database. You can also load Cloud Firestore exports into BigQuery.

This page describes how to export and import Cloud Firestore documents using the managed export and import service and Cloud Storage. The Cloud Firestore managed export and import service is available through the gcloud command-line tool and the Cloud Firestore API (REST, RPC).

Before you begin

Before you can use the managed export and import service, you must complete the following tasks:

  1. Enable billing for your Google Cloud Platform project. Only GCP projects with billing enabled can use the export and import functionality.
  2. Create a Cloud Storage bucket for your project in a location near your Cloud Firestore database location. You cannot use a Requester Pays bucket for export and import operations.
  3. Make sure your account has the necessary permissions for Cloud Firestore and Cloud Storage. If you are the project owner, your account has the required permissions. Otherwise, the following roles grant the necessary permissions for export and import operations and for access to Cloud Storage:

    • Cloud Firestore roles: Owner, Cloud Datastore Owner, or Cloud Datastore Import Export Admin
    • Cloud Storage roles: Owner or Storage Admin

Default service account permissions

Each Google Cloud Platform project automatically creates a default service account named Export and import operations use this service account to authorize Cloud Storage operations.

Your project's default service account requires access to the Cloud Storage bucket used in an export or import operation. If your Cloud Storage bucket is in the same project as your Cloud Firestore database, then the default service account has access to the bucket by default.

If the Cloud Storage bucket is in another project, then you must give the default service account access to the Cloud Storage bucket.

The service account needs the Storage Admin role for the Cloud Storage bucket to be used for the export or import operation.

If you disable or delete your App Engine default service account, your App Engine app will lose access to your Cloud Firestore database. If you disabled your App Engine service account, you can re-enable it, see enabling a service account. If you deleted your App Engine service account within the last 30 days, you can restore your service account, see undeleting a service account.

Set up gcloud for your project

Set up the gcloud command-line tool and connect to your project in one of the following ways:

Export data

An export operation copies documents in your database to a set of files in a Cloud Storage bucket. Note that an export is not an exact database snapshot taken at the export start time. An export may include changes made while the operation was running.

Export all documents

Use the firestore export command to export all the documents in your database, replacing [BUCKET_NAME] with the name of your Cloud Storage bucket. Add the --async flag to prevent the gcloud tool from waiting for the operation to complete.

gcloud firestore export gs://[BUCKET_NAME]

You can organize your exports by adding a file prefix after the bucket name, for example, BUCKET_NAME/my-exports-folder/export-name. If you do not provide a file prefix, the managed export service creates one based on the current timestamp.

Once you start an export operation, closing the terminal does not cancel the operation, see cancel an operation.

Export specific collections

To export specific collection groups, use the --collection-ids flag. The operation exports only the collection groups with the given collection IDs. The collection group includes all collections and subcollections (at any path) with the specified collection ID.

gcloud firestore export gs://[BUCKET_NAME] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2]

Import data

Once you have export files in Cloud Storage, you can import documents in those files back into your project or to another project. Note the following points about import operations:

  • When you import data, the required indexes are updated using your database's current index definitions. An export does not contain index definitions.

  • Imports do not assign new document IDs. Imports use the IDs captured at the time of the export. As a document is being imported, its ID is reserved to prevent ID collisions. If a document with the same ID already exists, the import overwrites the existing document.

  • If a document in your database is not affected by an import, it will remain in your database after the import.

  • Import operations do not trigger Cloud Functions. Snapshot listeners do receive updates related to import operations.

Import all documents from an export

Use the firestore import command to import documents from a previous export operation.

gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/

where [BUCKET_NAME] and [EXPORT_PREFIX] point to the location of your export files. For example:

gcloud firestore import gs://exports-bucket/2017-05-25T23:54:39_76544/

You can confirm the location of your export files in the Cloud Storage browser in the Google Cloud Platform Console:

Open Cloud Storage browser

Once you start an import operation, closing the terminal does not cancel the operation, see cancel an operation.

Import specific collections

To import specific collection groups from a set of export files, use the --collection-ids flag. The operation imports only the collection groups with the given collection IDs. The collection group includes all collections and subcollections (at any path) with the specified collection ID.

Only an export of specific collection groups supports an import of specific collection groups. You cannot import specific collections from an export of all documents.

gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/ --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2]

Managing export and import operations

After you start an export or import operation, Cloud Firestore assigns the operation a unique name. You can use the operation name to delete, cancel, or status check the operation.

Operation names are prefixed with projects/[PROJECT_ID]/databases/(default)/operations/, for example:


However, you can leave out the prefix when specifying an operation name for the describe, cancel, and deletecommands.

List all export and import operations

Use the operations list command to see all running and recently completed export and import operations:

gcloud firestore operations list

Check operation status

Use the operations describe command to show the status of an export or import operation.

gcloud firestore operations describe [OPERATION_NAME]

Estimate the completion time

A request for the status of a long-running operation returns the metrics workEstimated and workCompleted. Each of these metrics is returned in both number of bytes and number of entities:

  • workEstimated shows the estimated total number of bytes and documents an operation will process.

  • workCompleted shows the number of bytes and documents processed so far. After the operation completes, the value shows the total number of bytes and documents that were actually processed, which might be larger than the value of workEstimated.

Divide workCompleted by workEstimated for a rough progress estimate. This estimate might be inaccurate, because it depends on delayed statistics collection.

Cancel an operation

Use the operations cancel command to stop an operation in progress:

gcloud firestore operations cancel [OPERATION_NAME]

Cancelling a running operation does not undo the operation. A cancelled export operation will leave documents already exported in Cloud Storage, and a cancelled import operation will leave in place updates already made to your database. You cannot import a partially completed export.

Delete an operation

Use the operations delete command to remove an operation from the output of operations list. This command will not delete export files from Cloud Storage.

gcloud firestore operations delete [OPERATION_NAME]

Billing and pricing for export and import operations

You are required to enable billing for your Google Cloud Platform project before you use the managed export and import service. Export and import operations are charged for document reads and writes at the rates listed in Cloud Firestore pricing.

The costs of export and import operations do not count towards your spending limit. Export or import operations will not trigger your Google Cloud Platform budget alerts until after completion. Similarly, reads and writes performed during an export or import operation are applied to your daily quota after the operation is complete. Export and import operations will not affect the usage shown in the usage section of the console.

Viewing export and import costs

Export and import operations apply the goog-firestoremanaged:exportimport label to billed operations. In the Cloud Billing reports page, you can use this label to view costs related to import and export operations:

Access the goog-firestoremanaged label from the filters menu.

Audit logs

Cloud Firestore writes Admin Activity audit logs for Cloud Audit Logs. Admin Activity audit logs include export operations, import operations, and indexing operations. To view Admin Activity audit logs for your Cloud Firestore database, see Viewing audit logs.

Cloud Firestore Admin Activity audit logs appear under the Cloud Datastore Database and Cloud Datastore Index resource types. Although the resource types use the Cloud Datastore name, individual log entries contain a Cloud Firestore identifier. Cloud Firestore logs the following operations:

Audit logs category Cloud Firestore operations
Admin Activity FirestoreAdmin.CreateIndex

Export to BigQuery

You can load data from a Cloud Firestore export into BigQuery, but only if you specified a collection-ids filter. See Loading data from Cloud Firestore exports.

BigQuery column limit

BigQuery imposes a limit of 10,000 columns per table. Cloud Firestore export operations generate a BigQuery table schema for each collection group. In this schema, each unique field name within a collection group becomes a schema column.

If a collection group's BigQuery schema surpasses 10,000 columns, the Cloud Firestore export operation attempts to stay under the column limit by treating map fields as bytes. If this conversion brings the number of columns below 10,000, you can load the data into BigQuery, but you cannot query the subfields within the map fields. If the number of columns still exceeds 10,000, the export operation does not generate a BigQuery schema for the collection group and you cannot load its data into BigQuery.