docs: document adopting existing destination tables - #3325
Closed
jwhartley wants to merge 2 commits into
Closed
Conversation
Attaching a materialization to a table that already exists was undocumented, so there was nothing to find for either of the two cases people hit: pre-creating a table to control DDL that Estuary does not manage, and migrating off another pipeline without dropping the destination. - New guide, Migrate an Existing Pipeline to Estuary. Splits the empty-table case (one feature flag) from the populated-table case (also needs Exclude Flow Document and a cutover filter), since the trigger is rows existing rather than the table existing. - BigQuery: split Table Partitioning into partitioning a new table, which is the previously undocumented pre-create path, and repartitioning an existing one. Partitioning cannot be altered after creation, so pre-creating is the only way to avoid the copy-and-rename conversion. - feature-flags: note that allow_existing_tables_for_new_bindings is not sufficient once the table holds rows, and describe what the connector actually does to an adopted table (adds missing columns as nullable, drops NOT NULL on unselected columns, never recreates the table). - customize-materialization-fields: give Exclude Flow Document its migration motivation, not just storage savings. Two things the guide is careful about, both from reading the source rather than from support threads: - notBefore filters on collection publish time, not on when the row changed in the source, so a timestamp taken from the outgoing tool's last sync filters nothing once the capture backfills. Only Changes mode, or notBefore set to the backfill's completion, are the approaches that work. - Exclude Flow Document takes effect on an existing binding's next transaction, so it does not require recreating the binding.
The feature-flags note said the table is "never dropped or recreated, so partitioning, clustering, and other table-level DDL survive". That is true of the reconciliation the connector performs when a binding attaches, and false of the table's lifetime: a backfill truncates by default, and drops and recreates when a selected field's type changes incompatibly, when the collection key changes, or when always_drop_tables_on_backfill is set (materialize-boilerplate/materializer.go:594-626). The default onIncompatibleSchemaChange is backfill, so an incompatible schema change reaches that path without anyone asking for a backfill, which is the case the original wording most needed to warn about. Scoped the claim to the reconciliation and cross-linked the backfill guide, which already documents the triggers accurately. Also corrects the migration guide's warning, which had truncate and drop-and-recreate losing partitioning equally. TRUNCATE TABLE preserves it (materialize-bigquery/client.go:155); only the drop path loses it. And says "stored flow_document fields" rather than "stored documents" where the storage savings are described, since that is the specific thing a backfill clears.
Contributor
Author
|
Scrapping this as generally we recommend backfilling in Estuary rather than trying to in-place migrate a dataflow |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attaching a materialization to a destination table that already exists was undocumented, so there was nothing to find for either case people actually hit: pre-creating a table to control DDL that Estuary does not manage, and migrating off another pipeline without dropping the destination.
Changes
/guides/migrate-to-estuary/. Splits the empty-table case (one feature flag) from the populated-table case (also needs Exclude Flow Document and a cutover filter), since the trigger is rows existing rather than the table existing.allow_existing_tables_for_new_bindingsis not sufficient once the table holds rows, and describes what the connector actually does to an adopted table (adds missing columns as nullable, dropsNOT NULLon unselected columns, never recreates the table).Two claims worth a second opinion
Both come from reading the source rather than from support threads, and both contradict guidance that has circulated internally:
notBeforefilters on collection publish time, not on when the row changed in the source (crates/models/src/source.rs). So a timestamp copied from the outgoing tool's last sync filters nothing once the capture backfills, and the whole backfill lands in the existing table. The guide recommends Only Changes mode, ornotBeforeset to the backfill's completion, instead.materialize-bigquery/transactor.go).Notes for review
slug:frontmatter rather than file paths, since several targets (time-travel,backfilling-data,materialization) sit at directories that do not match their URLs.