Firestore Pipelines operations overview

The Firestore Pipeline operations is a central feature of the Firestore Enterprise edition, built on a new query engine to significantly expand the range of possible queries. Firestore Pipeline operations employ a flexible query syntax and a distinct indexing method where indexes are optional and not automatically created, enabling advanced data retrieval operations for applications.

Features of Firestore Pipeline operations

The Firestore Enterprise edition and the new Pipeline operations utilizes a new, advanced query engine that removes many existing limitations of the Firestore Standard edition. Firestore Pipeline operations provides 120+ new query features. The Firestore Pipeline operations has the following capabilities:

Stage-Based Composable Syntax

Pipeline queries are constructed by defining a series of sequential stages that are executed in order. This allows for complex operations, such as filtering on the result of an aggregation, which was not previously possible.

The following example shows a pipeline query that finds the number of unique product IDs viewed in the last month:

guard let cutoffDate = Calendar.current.date(byAdding: .month, value: -1, to: Date()) else {
  return
}
let snapshot = try await db.pipeline()
  .collection("productViews")
  .where(Field("viewedAt").greaterThan(cutoffDate.timeIntervalSince1970))
  .aggregate([Field("productId").countDistinct().as("uniqueProductViews")])
  .execute()

Expanded Capabilities

The Pipelines query introduces a vast number of new capabilities, including:

• Aggregations: Support for new aggregation functions (like sum(...), min(...), and count_distinct(...)) combined with arbitrary grouping fields.
• Complex Filtering: Support for 120+ new functions to express arbitrarily complex where(...) statements, including regex_match(...), add(...) and str_contains(...), all without hard index requirements.
• Partial Reads / Projections: Retrieve dynamic subsets of documents using select(...), remove_fields(...), and many other document manipulation stages.

Realtime and Offline Support

To utilize Realtime and Offline, developers can utilize the Firestore Core operations on Firestore Enterprise edition.

Client and Tooling Integration

The Enterprise edition includes specialized features to interact with and manage Pipeline Queries:

• Query explaining and profiling: You can use Query Explain results to understand how many read or write units a query consumes and analyze its execution.
• Query insights: The Enterprise edition supports Query Insights which helps you determine where indexes could be created to improve performance and cost by providing visibility into query performance characteristics.
• New index types: You can create specialized indexes for Enterprise edition, including sparse indexes. It also supports creating and editing vector search indexes for Enterprise databases.

Differences between Firestore Standard and Firestore Enterprise

The major operational difference between the Core and Pipeline operations lies in the management of indexing, which directly affects performance and cost.