Move data between projects

This page describes how to use the managed import and export features to move Cloud Firestore data from one project to another. This can be useful for setting up a development environment or as part of permanently migrating an app to another project. The example on this page demonstrates how to export data from a source project and then import that data into a destination project. Moving data between projects involves the following steps:

  1. Create a Cloud Storage bucket to hold the data from your source project.
  2. Export the data from your source project to the bucket.
  3. Give your destination project permission to read from the bucket.
  4. Import the data from the bucket into your destination project.

Before you begin

Before you can use the managed export and import service, you must complete the following tasks:

  1. Enable billing for both your source project and destination project. Only Google Cloud projects with billing enabled can use the export and import functionality.

  2. Make sure your account has the necessary Cloud IAM permissions in your source project and destination project. If you are a project owner for both projects, your account has the required permissions. Otherwise, the following Cloud IAM roles grant the necessary permissions for Cloud Firestore export and import operations:

    Owner, Cloud Datastore Owner, or Cloud Datastore Import Export Admin

    A project owner can grant one of these roles by following the steps in Grant access.

  3. Set up the gcloud command-line tool and connect to your project in one of the following ways:

  4. Set up indexes in your new project. The composite indexes should match between the source and destination projects. Indexes should be set up first to avoid having to process each document multiple times.

Export data from the source project

Export your data by creating a Cloud Storage bucket for your Cloud Firestore export files and starting an export operation.

Create a Cloud Storage bucket

Create a Cloud Storage bucket in the same location as your Cloud Firestore database. To view your database location, see your project location setting. You cannot use a Requester Pays bucket for export and import operations.

If your Cloud Storage bucket is not in your source project, you must give the source project's default service account access to the bucket. Each Google Cloud project has an automatically created default service account with the name PROJECT_ID@appspot.gserviceaccount.com. Cloud Firestore export operations use this default service account to authorize Cloud Storage bucket operations. To give the default service account access to your source bucket, grant it the Storage Admin role.

You can grant this role with the gsutil tool available in Cloud Shell:

Start Cloud Shell 

gsutil iam ch serviceAccount:[service-PROJECT_NUMBER]@gcp-sa-firestore.iam.gserviceaccount.com :roles/storage.admin\
gs://[BUCKET_NAME]@

You can also grant this role in the Google Cloud console.

Disable write operations (optional)

If your app continues to write to your database while you perform an export operation, you might not capture all of those writes in your export files. To export data from a consistent state, disable writes to your database by updating your security rules and halting any Admin SDK operations.

  1. Update security rules

    In the Cloud Firestore Rules tab of the console, update your source project security rules to deny all writes. For example:

      // Deny write access to all users under any conditions
  service cloud.firestore {
    match /databases/{database}/documents {
      match /{document=**} {
        allow write: if false;
      }
      // Reads do not affect export operations
      // Add your read rules here
    }
  }

  2. Halt writes from Admin SDKs

    Security rules do not stop writes coming from privileged server environments created using a Firebase Admin SDK or a Google Cloud Server Client Library. Make sure to halt write operations from your admin servers by shutting down or updating your servers.

Begin an export operation

Use the gcloud firestore export command to export data from your source project. You can export all your data or only specific collections. Replace [SOURCE_BUCKET] with the name of your Cloud Storage bucket:

Export all data
gcloud firestore export gs://[SOURCE_BUCKET] --async
Export specific collections
gcloud firestore export gs://[SOURCE_BUCKET] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async

Take note of your export operation's outputURIPrefix as you will use this later on. By default, Cloud Firestore adds a pre-fix to your export files based on a timestamp:

outputUriPrefix: gs://[SOURCE_BUCKET]/2019-03-05T20:58:23_56418

As the export operation runs, you can use the firestore operations list command to view your operation's progress:

gcloud firestore operations list

Import data into the destination project

Next, give the destination project access to your Cloud Firestore data files and start an import operation.

Give the destination project access to your data files

Before you can begin an import operation, you must make sure your destination project can access your Cloud Firestore data files.

Move data files to a local bucket

If your source bucket location is different from the Cloud Firestore location of your destination project, you must move your data files to a Cloud Storage bucket in the same location as your destination project.

Move your data files to another Cloud Storage bucket by following the steps in Moving and Renaming Buckets. For all the following steps, use this new bucket as the [SOURCE_BUCKET].

Give the project service account access to your source bucket

If your source bucket is not in your destination project, then you must give the destination project's default service account access to your source bucket. The default service account is named [DESTINATION_PROJECT_ID]@appspot.gserviceaccount.com. To give the default service account access to your source bucket, grant it the proper permissions to access the bucket.

You can grant the necessary roles with the gsutil tool available in Cloud Shell:

Start Cloud Shell 

gsutil iam ch serviceAccount:[DESTINATION_PROJECT_ID]@appspot.gserviceaccount.com:legacyBucketReader,legacyObjectReader \
gs://[SOURCE_BUCKET]

You can also grant this role in the Google Cloud console.

Begin an import operation

Before starting the import operation, make sure gcloud is configured for the correct project:

gcloud config set project [DESTINATION_PROJECT_ID]

Use the gcloud firestore import command to import the data in your source bucket into your destination project:

gcloud firestore import gs://[SOURCE_BUCKET]/[EXPORT_PREFIX] --async

Where [EXPORT_PREFIX] matches the pre-fix in your export operation's outputUriPrefix. For example:

gcloud firestore import gs://[SOURCE_BUCKET]/2019-03-05T20:58:23_56418 --async

As the export operation runs, you can use the firestore operations list command to view your operation's progress:

gcloud firestore operations list