Gate Firestore backup update on a successful BigQuery write#56
Merged
Conversation
Copilot
AI
changed the title
[WIP] Upgrade backup to use BigQuery Storage Write API
Migrate BigQuery write path to Storage Write API (managedwriter)
Jun 13, 2026
…eeded on Cloud Run
Copilot
AI
changed the title
Migrate BigQuery write path to Storage Write API (managedwriter)
Resolve BigQuery destination project from table metadata instead of an explicit project-id lookup
Jun 13, 2026
Copilot
AI
changed the title
Resolve BigQuery destination project from table metadata instead of an explicit project-id lookup
Add BigQueryStreamWriter for Storage Write API ingestion
Jun 13, 2026
Copilot
AI
changed the title
Add BigQueryStreamWriter for Storage Write API ingestion
Integrate BigQueryStreamWriter into backup.ts
Jun 13, 2026
Copilot
AI
changed the title
Integrate BigQueryStreamWriter into backup.ts
Gate Firestore backup update on a successful BigQuery write
Jun 13, 2026
ernysans
marked this pull request as ready for review
June 13, 2026 21:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new production-grade BigQuery streaming writer using the BigQuery Storage Write API, updates the Firestore-to-BigQuery backup logic to use this new writer, and updates documentation and exports to reflect these changes. The most important changes are as follows:
BigQuery Streaming and Backup Logic
BigQueryStreamWriterclass inlib/bigquery-stream-writer.jsthat wraps the BigQuery Storage Write API (@google-cloud/bigquery-storage) for high-throughput, at-least-once streaming writes, with batching, backpressure, connection reuse, and type coercion.lib/backup.jsto useBigQueryStreamWriterfor streaming Firestore documents into BigQuery, replacing the legacytabledata.insertAllmethod. The backup function now buffers and flushes rows via the Storage Write API and handles errors and resource cleanup more robustly. [1] [2]Exports and API Surface
BigQueryStreamWriterfromlib/index.js, making it available as a public API. [1] [2]Documentation and Instructions
README.MDand.github/copilot-instructions.mdto document the new@google-cloud/bigquery-storagedependency, and to instruct on mocking it in tests. [1] [2] [3] [4]Dependency and Versioning
1.0.1inpackage.jsonto reflect these new features and changes.