Schedule data exports

This page describes how to schedule exports of your Cloud Firestore data. To run exports on a schedule, we recommend deploying an App Engine service that calls the Cloud Firestore managed export feature. Once deployed, you can schedule calls to this service using the App Engine Cron Service.

Before you begin

Before you schedule data exports with the managed export feature, you must complete the following tasks:

Enable billing for your Google Cloud Platform project. Only GCP projects with billing enabled can use the export and import feature.
Note: Firebase projects must be on the Blaze plan to use the managed export and import feature. Enabling billing for the Google Cloud Platform automatically upgrades your Firebase project to the Blaze plan.
Create a Cloud Storage bucket for your project in a location near your Cloud Firestore database location. You cannot use a Requester Pays bucket for export and import operations.
Note: You can use your project's default bucket, gs://YOUR_PROJECT_ID.appspot.com. The default bucket includes 5GB of free storage.
Install the Google Cloud SDK to grant access permissions and deploy the application.

Configure access permissions

The app uses the App Engine default service account to authenticate and authorize its export operations. When you create a project, the default service account is created for you with the following format:

YOUR_PROJECT_ID@appspot.gserviceaccount.com

The service account requires permission to start an export operation and to write to your Cloud Storage bucket. To grant these permissions, assign the following IAM roles to the default service account:

Cloud Datastore Import Export Admin
Note: This Cloud Datastore role also grants permissions for Cloud Firestore.
Owner or Storage Admin role on the bucket

You can use the gcloud and gsutil command-line tools from the Google Cloud SDK to assign these roles:

Assign the Cloud Datastore Import Export Admin role:

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member serviceAccount:YOUR_PROJECT_ID@appspot.gserviceaccount.com \
    --role roles/datastore.importExportAdmin

Assign the Storage Admin role on your bucket:

gsutil iam ch serviceAccount:YOUR_PROJECT_ID@appspot.gserviceaccount.com:storage.admin \
    gs://BUCKET_NAME

Application files

In a new folder, create the following application files using the code below:

app.yaml: Configures the App Engine runtime. The app uses the standard environment Node.js runtime.
app.js: The main app code. This app sets up a web service at https://YOUR_PROJECT_ID.appspot.com that starts export operations.
package.json: Includes information about the app and its dependencies.
cron.yaml: Configures a cron job that calls the web service.

app.yaml

runtime: nodejs8app.yaml

The code above assumes this app is the default application. If it is not, add the following line:

target: cloud-firestore-admin

app.js

const axios = require('axios');
const dateformat = require('dateformat');
const express = require('express');
const { google } = require('googleapis');

const app = express();

// Trigger a backup
app.get('/cloud-firestore-export', async (req, res) => {
  const auth = await google.auth.getClient({
    scopes: ['https://www.googleapis.com/auth/datastore']
  });

  const accessTokenResponse = await auth.getAccessToken();
  const accessToken = accessTokenResponse.token;

  const headers = {
    'Content-Type': 'application/json',
    Authorization: 'Bearer ' + accessToken
  };

  const outputUriPrefix = req.param('outputUriPrefix');
  if (!(outputUriPrefix && outputUriPrefix.indexOf('gs://') == 0)) {
    res.status(500).send(`Malformed outputUriPrefix: ${outputUriPrefix}`);
  }

  // Construct a backup path folder based on the timestamp
  const timestamp = dateformat(Date.now(), 'yyyy-mm-dd-HH-MM-ss');
  let path = outputUriPrefix;
  if (path.endsWith('/')) {
    path += timestamp;
  } else {
    path += '/' + timestamp;
  }

  const body = {
    outputUriPrefix: path
  };

  // If specified, mark specific collections for backup
  const collectionParam = req.param('collections');
  if (collectionParam) {
    body.collectionIds = collectionParam.split(',');
  }

  const projectId = process.env.GOOGLE_CLOUD_PROJECT;
  const url = `https://firestore.googleapis.com/v1beta1/projects/${projectId}/databases/(default):exportDocuments`;

  try {
    const response = await axios.post(url, body, { headers: headers });
    res
      .status(200)
      .send(response.data)
      .end();
  } catch (e) {
    if (e.response) {
      console.warn(e.response.data);
    }

    res
      .status(500)
      .send('Could not start backup: ' + e)
      .end();
  }
});

// Index page, just to make it easy to see if the app is working.
app.get('/', (req, res) => {
  res
    .status(200)
    .send('[scheduled-backups]: Hello, world!')
    .end();
});

// Start the server
const PORT = process.env.PORT || 6060;
app.listen(PORT, () => {
  console.log(`App listening on port ${PORT}`);
  console.log('Press Ctrl+C to quit.');
});
app.js

package.json

{
  "name": "solution-scheduled-backups",
  "version": "1.0.0",
  "description": "Scheduled Cloud Firestore backups via AppEngine cron",
  "main": "app.js",
  "engines": {
    "node": "8.x.x"
  },
  "scripts": {
    "deploy": "gcloud app deploy --quiet app.yaml cron.yaml",
    "start": "node app.js"
  },
  "author": "Google, Inc.",
  "license": "Apache-2.0",
  "dependencies": {
    "axios": "^0.18.0",
    "dateformat": "^3.0.3",
    "express": "^4.16.4",
    "googleapis": "^38.0.0"
  },
  "devDependencies": {
    "prettier": "^1.16.4"
  }
}
package.json

cron.yaml

cron:
- description: "Daily Cloud Firestore Export"
  url: /cloud-firestore-export?outputUriPrefix=gs://BUCKET_NAME[/PATH]&collections=test1,test2
  target: cloud-firestore-admin
  schedule: every 24 hourscron.yaml

Modify the url line to configure the export operation. The app sets up a service at https://YOUR_PROJECT_ID.appspot.com/cloud-firestore-export that accepts the following URL parameters:

outputUriPrefix: the location of you Cloud Storage bucket in the format of gs://BUCKET_NAME.
collections: A comma separated list of collection IDs to export. If not specified, the operation exports all collections.

For example, to export all collections with collection ID Songs or Albums , you would use the following:

url: /cloud-firestore-export?outputUriPrefix=gs://BUCKET_NAME&collections=Songs,Albums

The example cron.yaml runs an export every 24 hours. For different schedule options, see the schedule format.

Deploy the app and cron job

Using gcloud, deploy the app and the cron job:

gcloud app deploy app.yaml cron.yaml

Test your cron job

You can test your deployed cron job by starting it in the Cron Jobs page of the Google Cloud Platform Console.

Open the Cron Jobs page in the GCP console.
Open the Cron Jobs page
For the cron job with a description of Daily Cloud Firestore Export, click Run now.
After the job completes, see the status message under Status. Click View to see the job log. The status message and job log will provide information on job success or failure.

View your exports

After an export operation completes, you can view the exports in your Cloud Storage bucket:

Open the Cloud Storage browser in the GCP Console.
Open the Cloud Storage browser