Google은 흑인 공동체를 위한 인종적 평등을 추구하기 위해 노력하고 있습니다. 자세히 알아보기
컬렉션을 사용해 정리하기 내 환경설정을 기준으로 콘텐츠를 저장하고 분류하세요.

BigQuery로 컬렉션 스트리밍

Firebase
Made by Firebase

지정된 Cloud Firestore 컬렉션에서 BigQuery로 실시간 증분 업데이트를 전송합니다.

이 확장 프로그램의 작동 방식

Use this extension to export the documents in a Cloud Firestore collection to BigQuery. Exports are realtime and incremental, so the data in BigQuery is a mirror of your content in Cloud Firestore.

The extension creates and updates a dataset containing the following two BigQuery resources:

  • A table of raw data that stores a full change history of the documents within your collection. This table includes a number of metadata fields so that BigQuery can display the current state of your data. The principle metadata fields are timestamp , document_name , and the operation for the document change.
  • A view which represents the current state of the data within your collection. It also shows a log of the latest operation for each document ( CREATE , UPDATE , or IMPORT ).

If you create, update, delete, or import a document in the specified collection, this extension sends that update to BigQuery. You can then run queries on this mirrored dataset.

Note that this extension only listens for document changes in the collection, but not changes in any subcollection . You can, though, install additional instances of this extension to specifically listen to a subcollection or other collections in your database. Or if you have the same subcollection across documents in a given collection, you can use {wildcard} notation to listen to all those subcollections (for example: chats/{chatid}/posts ).

Enabling wildcard references will provide an additional STRING based column. The resulting JSON field value references any wildcards that are included in ${param:COLLECTION_PATH}. You can extract them using JSON_EXTRACT_SCALAR.

Partition settings cannot be updated on a pre-existing table, if these options are required then a new table must be created.

Clustering will not need to create or modify a table when adding clustering options, this will be updated automatically.

Additional setup

Before installing this extension, you'll need to:

Transform function

Prior to sending the document change to BigQuery, you have an opportunity to transform the data with an HTTP function. The payload will contain the following:

{ 
  data: [{
    insertId: int;
    json: {
      timestamp: int;
      event_id: int;
      document_name: string;
      document_id: int;
      operation: ChangeType;
      data: string;
    },
  }]
}

The response should be indentical in structure.

Backfill your BigQuery dataset

This extension only sends the content of documents that have been changed -- it does not export your full dataset of existing documents into BigQuery. So, to backfill your BigQuery dataset with all the documents in your collection, you can run the import script provided by this extension.

Important: Run the import script over the entire collection after installing this extension, otherwise all writes to your database during the import might be lost.

Generate schema views

After your data is in BigQuery, you can run the schema-views script (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views.

Billing

To install an extension, your project must be on the Blaze (pay as you go) plan

  • You will be charged a small amount (typically around $0.01/month) for the Firebase resources required by this extension (even if it is not used).
  • This extension uses other Firebase and Google Cloud Platform services, which have associated charges if you exceed the service’s no-cost tier:
  • BigQuery (this extension writes to BigQuery with streaming inserts )
  • Cloud Firestore
  • Cloud Functions (Node.js 10+ runtime. See FAQs )

확장 프로그램 설치 방법

Firebase Console 사용

Firebase Console을 사용해 확장 프로그램을 설치하고 관리할 수 있습니다.

Console을 사용해 설치하기

Firebase CLI 사용

Firebase CLI를 사용해 확장 프로그램을 설치하고 관리할 수도 있습니다.

1단계: 다음 npm 명령어를 실행하여 CLI를 설치하거나 최신 CLI 버전으로 업데이트합니다.

npm install -g firebase-tools
작동하지 않는 경우 Firebase CLI 참조 를 살펴보거나 npm 권한 을 변경하세요.

2단계: 새 Firebase 프로젝트 디렉터리를 설정 하거나 기존 디렉터리로 이동합니다.

3단계: 다음을 실행하여 이 확장 프로그램을 확장 프로그램 매니페스트 에 추가합니다.

firebase ext:install firebase/firestore-bigquery-export --local --project=projectId_or_alias

Step 4 (Optional): Test this extension locally with the Firebase Emulator Suite

firebase emulators:start

5단계: 매니페스트의 확장 프로그램을 프로젝트에 배포 합니다.

firebase deploy --only extensions --project=projectId_or_alias
지원
Cloud Firestore
작성자
라이선스
Apache-2.0
버전
0.1.25