Manage data retention with TTL policies

This page describes how to use the Google Cloud Platform Console and the Google Cloud CLI to configure time-to-live (TTL) policies. Before you read this page, you should understand the Cloud Firestore data model.

Time-to-live overview

Use time-to-live (TTL) policies to automatically remove stale data from your databases. A TTL policy designates a given field as the expiration time for documents in a given collection group. With TTL, you can decrease storage costs by cleaning out obsolete data. Data is typically deleted within 24 hours after its expiration date.

Pricing

TTL delete operations count towards your document delete costs. For pricing of delete operations, see Cloud Firestore pricing.

Limits and constraints

  • Only one field per collection group can be marked as a TTL field.
  • A total of 200 field level configurations are allowed. One field configuration can contain multiple configurations for the same field. For example, a single-field indexing exemption and a TTL policy on the same field count as one field configuration towards the limit.
  • For Firestore in Datastore mode customers, TTL cannot be used with a concurrency mode of Optimistic With Entity Groups. Consider changing the concurrency mode following this.

TTL deletion

Note the following key behaviors of TTL-driven deletion:

  • Deletion through TTL is not an instantaneous process. Expired documents continue to appear in queries and lookup requests until the TTL process actually deletes them. TTL trades deletion timeliness for the benefit of reduced total cost of ownership for deletions. Data is typically deleted within 24 hours after its expiration date.

  • Deleting a document through TTL does not delete subcollections under that document.

  • Applying a TTL policy on an existing collection group results in a bulk deletion of all data that's expired according to the new TTL policy. Note that this bulk deletion is also not instantaneous and depends on how much data exists for that collection group.

  • If a document has an expiration time in the past and you add a new ttl policy to the collection, the document will be deleted within 24 hours of the ttl policy finishing setup and becoming active.

  • TTL does not necessarily delete documents in the same order as their expiration timestamps.

  • Deletions are not done transactionally. Documents with the same expiration time are not necessarily deleted at the same time. If you require this behavior, perform the deletions using a client library.

  • Cloud Firestore will always honor the latest TTL field to determine the expiration. For example, if an expired but not-yet-deleted document has its TTL field updated to a later date, the document will not be expired and the new date will be used.

  • TTL is designed to minimize impact on other database activities. Deletions driven by TTL are treated with a lower priority. Other strategies are also in place to smooth out traffic spikes from TTL-driven deletes.

  • Deletion through TTL calls all active snapshot listeners and triggers Cloud Functions Cloud Firestore triggers.

TTL fields and indexes

A TTL field can be indexed or un-indexed. However, because a TTL field is a timestamp, indexing the field can affect performance at higher traffic rates. Indexing a timestamp field can create hotspots which is against best practices. Hotspots are high read, write, and delete rates to a narrow document range.

By default, Cloud Firestore creates a single-field index for all fields. You can create a single-field index exemption to disable indexes on a TTL field.

Permissions

The principal configuring a TTL policy requires the following permission in the project:

  • Viewing TTL policies requires the datastore.indexes.list and datastore.indexes.get permissions.
  • Modifying TTL policies requires the datastore.indexes.update permission.
  • Checking the status of TTL operations requires datastore.operations.list and datastore.operations.get.

For roles that assign these permissions, see Cloud Firestore Identity and Access Management roles.

Before you begin

Before you use the gcloud CLI to manage TTL policies, use the gcloud components update command to update components to the latest available version:

gcloud components update

Create a TTL policy

When you create a TTL policy, you designate a document field as the expiration time for documents in a collection group.

TTL uses a specified field to identify documents that are eligible for deletion. This TTL field must be of type Date and time. You can select a field that already exists or you can designate a field that you plan to add later.

Consider the following before you set the TTL field value:

  • The TTL field value can be a time in the future, now, or in the past. If the value is a time in the past, the document is immediately eligible for deletion. For example, you might create a TTL policy with the field expireAt, which you then add to existing documents.

  • Using any other data type or not setting the TTL field value will disable the TTL for the individual document.

Follow the steps below to create a TTL policy:

Google Cloud Console

  1. In the Google Cloud Platform Console, go to the Databases page.

    Go to Databases

  2. Select the required database from the list of databases.

  3. In the navigation menu, click Time-to-live.

  4. Click Create Policy.

  5. Enter a collection group name and a timestamp field name.

  6. Click Create.

The console returns to the Time-to-live page. If the operation successfully starts, the page adds an entry to the TTL policies table. On failure, the page displays an error message.

gcloud

Use the firestore fields ttls update command to configure a TTL policy. Add the --async flag to prevent the gcloud CLI from waiting for the operation to complete.

 gcloud firestore fields ttls update
  ttl_field --collection-group=collection_group_name
  --enable-ttl 

Even on an empty database, it can take ten minutes or more to enable a TTL policy. Once you start an operation, closing the terminal does not cancel the operation.

View TTL policies

Follow the steps below to view TTL policies and their statuses.

Google Cloud Console

  1. In the Google Cloud Platform Console, go to the Databases page.

    Go to Databases

  2. Select the required database from the list of databases.

  3. In the navigation menu, click Time-to-live.

The console lists TTL policies for your database and includes each policy's status.

gcloud

Use the firestore fields ttls list command to configure a TTL policy. The following command lists all TTL policies.

   gcloud firestore fields ttls list
   

To list TTL policies under a specific collection group, use the following:

   gcloud firestore fields ttls list  --collection-group=collection_group_name
   

View operation details

You can use the gcloud CLI to view more details about a TTL policy that is in the CREATING state.

Use the operations list command to see all running and recently completed operations:

gcloud firestore operations list

The response includes an estimate of the operation's progress.

Disable a TTL policy

Follow the steps below to disable a TTL policy.

Google Cloud Console

  1. In the Google Cloud Platform Console, go to the Databases page.

    Go to Databases

  2. Select the required database from the list of databases.

  3. In the navigation menu, click Time-to-live.

  4. In the TTL policy table, find the row for the TTL policy. Within this table row, click the Delete (trashcan) button.

  5. Confirm by clicking Delete.

The console returns to the Time-to-live page. On success, Cloud Firestore removes the TTL policy from the table.

gcloud

1. Use the firestore fields ttls update command to configure a TTL policy. Add the --async flag to prevent the gcloud CLI from waiting for the operation to complete.

   gcloud firestore fields ttls update ttl_field --collection-group=collection_group_name --disable-ttl
   

Monitor TTL deletions

You can use Cloud Monitoring to view metrics about TTL-driven deletions. Cloud Firestore provides the following metrics for TTL:

firestore.googleapis.com/document/ttl_deletion_count Time-to-live deletion count

Total count of documents deleted by Time-to-live (TTL) policies.

firestore.googleapis.com/document/ttl_expiration_to_deletion_delays Time-to-live expiration to deletion delays

Time elapsed between when a document expired under a Time-to-live (TTL) policy and when it was actually deleted.

To set up a dashboard with Cloud Firestore metrics, see manage custom dashboard and add dashboard widgets.