From b1d5ab6efc99f31c1a8b6a25f4606cfe7394ddf1 Mon Sep 17 00:00:00 2001 From: James Date: Fri, 7 Aug 2026 17:23:23 +0700 Subject: [PATCH 1/2] docs: document adopting existing destination tables Attaching a materialization to a table that already exists was undocumented, so there was nothing to find for either of the two cases people hit: pre-creating a table to control DDL that Estuary does not manage, and migrating off another pipeline without dropping the destination. - New guide, Migrate an Existing Pipeline to Estuary. Splits the empty-table case (one feature flag) from the populated-table case (also needs Exclude Flow Document and a cutover filter), since the trigger is rows existing rather than the table existing. - BigQuery: split Table Partitioning into partitioning a new table, which is the previously undocumented pre-create path, and repartitioning an existing one. Partitioning cannot be altered after creation, so pre-creating is the only way to avoid the copy-and-rename conversion. - feature-flags: note that allow_existing_tables_for_new_bindings is not sufficient once the table holds rows, and describe what the connector actually does to an adopted table (adds missing columns as nullable, drops NOT NULL on unselected columns, never recreates the table). - customize-materialization-fields: give Exclude Flow Document its migration motivation, not just storage savings. Two things the guide is careful about, both from reading the source rather than from support threads: - notBefore filters on collection publish time, not on when the row changed in the source, so a timestamp taken from the outgoing tool's last sync filters nothing once the capture backfills. Only Changes mode, or notBefore set to the backfill's completion, are the approaches that work. - Exclude Flow Document takes effect on an existing binding's next transaction, so it does not require recreating the binding. --- .../guides/advanced-usage/feature-flags.md | 13 ++- .../customize-materialization-fields.md | 8 +- site/docs/guides/migrate-to-estuary.md | 108 ++++++++++++++++++ .../materialization-connectors/BigQuery.md | 29 ++++- 4 files changed, 155 insertions(+), 3 deletions(-) create mode 100644 site/docs/guides/migrate-to-estuary.md diff --git a/site/docs/guides/advanced-usage/feature-flags.md b/site/docs/guides/advanced-usage/feature-flags.md index 54b32184f56..c9b12bb9968 100644 --- a/site/docs/guides/advanced-usage/feature-flags.md +++ b/site/docs/guides/advanced-usage/feature-flags.md @@ -115,13 +115,24 @@ Generates synthetic row identifiers for tables without primary keys. Allows materializations to write to tables that already exist in the destination, even for newly added bindings. - **Default:** Disabled. New bindings fail if the target table already exists. -- **Use case:** Migrating data from another system into Estuary-managed tables, or re-creating a materialization that was previously deleted. +- **Use case:** Migrating data from another system into Estuary-managed tables, re-creating a materialization that was previously deleted, or pre-creating a table so you control its DDL, for example [BigQuery partitioning](/reference/Connectors/materialization-connectors/BigQuery/#table-partitioning), which cannot be set on a table after it is created. - **Caveats:** - Enabling this flag makes the connector load existing keys from the destination before merging, instead of skipping that lookup for keys it expects to be new. It is slightly slower but ensures merges and updates work correctly against rows that already exist in the table. - The connector cannot verify that existing table schemas are compatible. - This flag alone does **not** prevent backfill of the source collection. To avoid backfilling data into the existing table, also configure [`notBefore`](/reference/time-travel) or use "Only Changes" mode on the binding. + - If the table already contains rows that Estuary did not write, this flag is not sufficient on its own. See [Migrate an existing pipeline to Estuary](/guides/migrate-to-estuary/). - **Applies to:** All SQL and warehouse materialization connectors (PostgreSQL, MySQL, Snowflake, BigQuery, Redshift, etc.) +#### What the connector does to the existing table + +On the first publication after attaching, the connector reconciles the table against the binding's field selection: + +- Columns in the field selection that the table does not have are added with `ALTER TABLE ... ADD COLUMN`, as nullable. This includes Estuary's own `flow_document`, `flow_published_at`, and `_meta/op` columns. +- Columns the table has that are **not** in the field selection are altered to drop `NOT NULL`, if they had it. Keep this in mind when you pre-create a table from DDL you care about: a `NOT NULL` constraint on a column Estuary does not materialize will be relaxed. +- The table itself is never dropped or recreated, so partitioning, clustering, and other table-level DDL survive. + +Because missing columns are added for you, you can pre-create a table with only the columns you need to control (the partition column and the collection key, for instance) and let the connector fill in the rest. + ### retain_existing_data_on_backfill Skips truncating destination tables when a backfill is triggered. diff --git a/site/docs/guides/customize-dataflows/customize-materialization-fields.md b/site/docs/guides/customize-dataflows/customize-materialization-fields.md index 9016c59d2f3..bb857f2876d 100644 --- a/site/docs/guides/customize-dataflows/customize-materialization-fields.md +++ b/site/docs/guides/customize-dataflows/customize-materialization-fields.md @@ -366,9 +366,15 @@ Since Estuary no longer writes to it, you can manually drop the leftover column A new binding, or a backfill that takes the [drop-and-recreate path](/reference/backfilling-data/#schema-changes-during-backfill), creates the table without the `flow_document` column entirely. :::note -The main motivation for this option is saving destination storage. For an existing table, those savings are only realized after a backfill clears the stored documents. +For an existing Estuary-managed table, the storage savings are only realized after a backfill clears the stored documents. ::: +### Migrating into a table another tool populated + +Saving storage is one reason to exclude `flow_document`. The other is migration: this option is **required** when a new binding attaches to a table that already holds rows Estuary did not write, because the connector cannot populate a stored document for rows it never wrote and reads a null instead. Reconstructing from the top-level columns is what makes those rows mergeable. + +That case has its own guide, including the rest of the configuration it needs: [Migrate an existing pipeline to Estuary](/guides/migrate-to-estuary/). + ## Pruned fields Estuary's schema inference tracks unique field locations. To prevent excessively large schemas, there's a complexity limit: diff --git a/site/docs/guides/migrate-to-estuary.md b/site/docs/guides/migrate-to-estuary.md new file mode 100644 index 00000000000..13c9c9cbcab --- /dev/null +++ b/site/docs/guides/migrate-to-estuary.md @@ -0,0 +1,108 @@ +--- +description: Move an existing pipeline to Estuary without dropping your destination tables, using allow_existing_tables_for_new_bindings, Exclude Flow Document, and a cutover filter. +slug: /guides/migrate-to-estuary/ +--- + +# Migrate an Existing Pipeline to Estuary + +When you replace another pipeline with Estuary, the destination tables it wrote are usually worth keeping. They hold history that is expensive to re-extract, they may be partitioned by hand, and downstream models already point at them. + +Estuary can attach to those tables and keep writing to them. By default that is blocked: + +``` +validating binding: table [warehouse schema orders] already exists for new binding +"acmeCo/production/orders". You must drop this table to continue. +``` + +The default is right whenever the destination holds nothing you cannot regenerate: drop the table and let Estuary rebuild it. This guide is for when it does hold something. + +## Which case are you in + +| Your table | What you need | +| --- | --- | +| Exists but is **empty**, pre-created to control its DDL | [`allow_existing_tables_for_new_bindings`](/guides/advanced-usage/feature-flags/#allow_existing_tables_for_new_bindings) only | +| Exists and **holds rows Estuary did not write** | That flag, plus [Exclude Flow Document](#why-exclude-flow-document-is-required) and a cutover filter | + +The trigger is rows existing, not the table existing. The first case is not really a migration: it is how you apply table-level DDL that Estuary does not manage, such as [BigQuery partitioning](/reference/Connectors/materialization-connectors/BigQuery/#partitioning-a-new-table), which cannot be set after a table is created. The rest of this page is the second case. + +## Cutover + +### 1. Run the capture alongside your existing pipeline + +Create the capture and let it run. Data lands in collections and nothing reaches your destination yet. Captures are read-only against the source, so both tools can read it concurrently. For CDC sources each tool needs its own replication slot or equivalent, so check your source's limits before adding a second. + +### 2. Decide how to avoid re-loading history + +[`notBefore`](/reference/time-travel) filters on when a document was **published into the collection**, not on when the row changed in your source. A timestamp copied from your outgoing tool's last sync therefore does not work: if the capture backfills, every document is published today, nothing is filtered, and the whole backfill lands in your table. + +Two approaches that do work: + +- Capture with **Only Changes** backfill mode, so the collection only ever holds post-cutover events. +- Let the capture backfill, then set `notBefore` to when that backfill completed. Rows that changed between the old tool's last sync and the capture's start are then picked up only when CDC next touches them. + +Cleanest of all is to stop the old pipeline's writes before you publish the binding, and use that moment as the cutover point. + +### 3. Check the table against the collection + +Do this before cutting over, not after. See [Pre-cutover checks](#pre-cutover-checks). + +### 4. Create the materialization + +```yaml +materializations: + acmeCo/production/materialize-bigquery: + endpoint: + connector: + image: ghcr.io/estuary/materialize-bigquery:dev + config: + # ...other endpoint configuration... + advanced: + feature_flags: "allow_existing_tables_for_new_bindings" + no_flow_document: true + bindings: + - source: acmeCo/production/orders + resource: { table: orders, dataset: warehouse } + notBefore: 2026-01-01T00:00:00Z + onIncompatibleSchemaChange: abort +``` + +- `allow_existing_tables_for_new_bindings` lets the binding attach instead of failing. For what the connector then does to the table, see [What the connector does to the existing table](/guides/advanced-usage/feature-flags/#what-the-connector-does-to-the-existing-table). +- **Exclude Flow Document** (`no_flow_document`) is required once the table holds rows. See [why](#why-exclude-flow-document-is-required). +- [`onIncompatibleSchemaChange: abort`](/concepts/advanced/evolutions/) replaces the default of `backfill`, which responds to an incompatible schema change by refreshing the table from the collection. Against history the collection does not have, that is data loss. + +:::warning +Do not use the **Backfill** button on this materialization. A backfill truncates the destination table, and paired with an incompatible schema change it drops and recreates it. Either way the pre-existing rows are gone, along with any partitioning. See [Schema changes during backfill](/reference/backfilling-data/#schema-changes-during-backfill). +::: + +### 5. Overlap, then decommission + +Let both pipelines run long enough to compare row counts and spot-check recently changed rows, then disable the old one. + +## Why Exclude Flow Document is required + +Under [standard updates](/concepts/materialization/#delta-updates) the connector keeps each complete document in a `flow_document` column and reads it back to merge updates into. + +When a binding attaches to an existing table, the connector adds that column but cannot populate it for the rows already there. The load query selects it anyway and reads a null, and the task fails. The exact error varies by connector, but it is always a task failure rather than silent corruption: the pipeline stops instead of writing wrong rows. + +[Exclude Flow Document](/guides/customize-materialization-fields/#excluding-flow_document-with-standard-updates) reconstructs the document from the table's own top-level columns instead, and those do hold values for the pre-existing rows. + +Set it when you create the binding. Enabling it on a binding that already exists also works, and takes effect on that binding's next transaction, but the task will have been failing in the meantime. + +## Pre-cutover checks + +The connector cannot verify that an existing table is compatible with the collection. In rough order of how much damage they do when missed: + +- **Row uniqueness must match the collection key.** The connector merges on the collection key. If the existing rows are not uniquely identified by that key, updates will not find them and will insert instead, leaving duplicates and no error. This is the most common and least visible migration failure. If the old tool generated a surrogate key of its own, use [`groupBy`](/guides/customize-materialization-fields/#group-by-keys) to merge on the real one. +- **Column names and types.** Other tools lowercase names or add their own metadata, and type mappings differ for the same source column, particularly numeric precision and timestamps. Estuary uses the collection's field names and the mapping documented on your destination connector's page. +- **The old tool's metadata columns go stale.** They stay in the table but stop being updated, since Estuary does not write them. Check that no downstream model or incremental filter depends on one. Estuary's equivalent is `flow_published_at`. +- **Deletion behavior.** Confirm whether the table used hard deletes or a soft-delete marker, and configure the materialization to match. + +If you can, run the materialization against a copy of the table in a scratch schema first and compare. It catches all of the above at once. + +## Related + +- [Feature flags](/guides/advanced-usage/feature-flags/) for `allow_existing_tables_for_new_bindings`, and for `retain_existing_data_on_backfill` +- [Customize materialized fields](/guides/customize-materialization-fields/) for field selection, Exclude Flow Document, and `groupBy` +- [Time travel](/reference/time-travel) for `notBefore` +- [Backfilling data](/reference/backfilling-data/) for what a backfill does to a destination table +- [Schema evolution](/guides/schema-evolution/) for `onIncompatibleSchemaChange` diff --git a/site/docs/reference/Connectors/materialization-connectors/BigQuery.md b/site/docs/reference/Connectors/materialization-connectors/BigQuery.md index da8eda0ba1f..ee97eb0160e 100644 --- a/site/docs/reference/Connectors/materialization-connectors/BigQuery.md +++ b/site/docs/reference/Connectors/materialization-connectors/BigQuery.md @@ -180,7 +180,34 @@ Tables are automatically created with [clustering](https://cloud.google.com/bigquery/docs/clustered-tables) based on the Estuary collection primary keys. Tables are not created with any other [partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables), but pre-existing partitioned tables can be materialized to. -It isn't possible to alter the partitioning of an existing table, but you can convert an existing table to one with partitioning by creating a new table and copying the data from the existing table into it. This can be done to tables that the connector is materializing to, as long as the materializing task is temporarily disabled while doing the conversion. +BigQuery does not allow the partitioning of a table to be altered after it is created, so there are two ways to get a partitioned destination table. Which one you need depends on whether the table exists yet. + +### Partitioning a new table + +If the binding does not exist yet, create the table yourself with the partitioning you want, then attach a binding to it. This avoids the copy-and-rename conversion below entirely, which matters if you are adding many tables. + +1. Create the table in BigQuery with your partitioning. You only need the columns you want to control, such as the partition column and the collection key: +```sql +create table . ( + timestamp, + string +) +partition by ; +``` +Partitioning on `flow_published_at` is a common choice, since Estuary writes that column on every row and incremental transformations often filter on it. + +2. Enable the [`allow_existing_tables_for_new_bindings`](/guides/advanced-usage/feature-flags/#allow_existing_tables_for_new_bindings) feature flag on the materialization, under **Endpoint Config → Advanced Options**. Without it, a new binding whose table already exists fails validation. + +3. Add the binding. On the first publication the connector adds the remaining columns from the binding's field selection with `ALTER TABLE ... ADD COLUMN`, including its own `flow_document`, `flow_published_at`, and `_meta/op` columns. The table is not recreated, so your partitioning is preserved. + +Two things to note: + +* A `NOT NULL` constraint on a column that Estuary does not materialize will be dropped when the connector reconciles the table. +* This procedure assumes the table you create is **empty**. If it already holds rows that Estuary did not write, more configuration is needed: see [Migrate an existing pipeline to Estuary](/guides/migrate-to-estuary/). + +### Repartitioning an existing table + +If the table is already being materialized to, you can convert it by creating a new table and copying the data from the existing table into it. This can be done to tables that the connector is materializing to, as long as the materializing task is temporarily disabled while doing the conversion. To convert an existing materialized table to one with different partitioning: 1. Pause your materialization by disabling it from the [UI](../../../concepts/web-app.md) or editing the task specification with the [CLI](../../../guides/flowctl/edit-specification-locally.md). From 4792d0b7c871027891acd0d0a9631aa3a23c70e8 Mon Sep 17 00:00:00 2001 From: James Date: Fri, 7 Aug 2026 17:39:43 +0700 Subject: [PATCH 2/2] docs: correct the claim that an adopted table is never recreated The feature-flags note said the table is "never dropped or recreated, so partitioning, clustering, and other table-level DDL survive". That is true of the reconciliation the connector performs when a binding attaches, and false of the table's lifetime: a backfill truncates by default, and drops and recreates when a selected field's type changes incompatibly, when the collection key changes, or when always_drop_tables_on_backfill is set (materialize-boilerplate/materializer.go:594-626). The default onIncompatibleSchemaChange is backfill, so an incompatible schema change reaches that path without anyone asking for a backfill, which is the case the original wording most needed to warn about. Scoped the claim to the reconciliation and cross-linked the backfill guide, which already documents the triggers accurately. Also corrects the migration guide's warning, which had truncate and drop-and-recreate losing partitioning equally. TRUNCATE TABLE preserves it (materialize-bigquery/client.go:155); only the drop path loses it. And says "stored flow_document fields" rather than "stored documents" where the storage savings are described, since that is the specific thing a backfill clears. --- site/docs/guides/advanced-usage/feature-flags.md | 4 +++- .../customize-dataflows/customize-materialization-fields.md | 2 +- site/docs/guides/migrate-to-estuary.md | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/site/docs/guides/advanced-usage/feature-flags.md b/site/docs/guides/advanced-usage/feature-flags.md index c9b12bb9968..2f0af2f0ea5 100644 --- a/site/docs/guides/advanced-usage/feature-flags.md +++ b/site/docs/guides/advanced-usage/feature-flags.md @@ -129,10 +129,12 @@ On the first publication after attaching, the connector reconciles the table aga - Columns in the field selection that the table does not have are added with `ALTER TABLE ... ADD COLUMN`, as nullable. This includes Estuary's own `flow_document`, `flow_published_at`, and `_meta/op` columns. - Columns the table has that are **not** in the field selection are altered to drop `NOT NULL`, if they had it. Keep this in mind when you pre-create a table from DDL you care about: a `NOT NULL` constraint on a column Estuary does not materialize will be relaxed. -- The table itself is never dropped or recreated, so partitioning, clustering, and other table-level DDL survive. +- The table itself is not dropped or recreated by this reconciliation, so partitioning, clustering, and other table-level DDL survive it. Because missing columns are added for you, you can pre-create a table with only the columns you need to control (the partition column and the collection key, for instance) and let the connector fill in the rest. +Attaching is not the only thing that touches the table, though. A [backfill](/reference/backfilling-data/#schema-changes-during-backfill) truncates it by default, which preserves table-level DDL, but drops and recreates it when a selected field's type changes incompatibly, when the collection key changes, or when `always_drop_tables_on_backfill` is set. Since the default [`onIncompatibleSchemaChange`](/concepts/advanced/evolutions/) is `backfill`, an incompatible schema change can reach that path without anyone asking for a backfill. Set it to `abort` if the table's DDL or its pre-existing rows matter. + ### retain_existing_data_on_backfill Skips truncating destination tables when a backfill is triggered. diff --git a/site/docs/guides/customize-dataflows/customize-materialization-fields.md b/site/docs/guides/customize-dataflows/customize-materialization-fields.md index bb857f2876d..2787feb0b56 100644 --- a/site/docs/guides/customize-dataflows/customize-materialization-fields.md +++ b/site/docs/guides/customize-dataflows/customize-materialization-fields.md @@ -366,7 +366,7 @@ Since Estuary no longer writes to it, you can manually drop the leftover column A new binding, or a backfill that takes the [drop-and-recreate path](/reference/backfilling-data/#schema-changes-during-backfill), creates the table without the `flow_document` column entirely. :::note -For an existing Estuary-managed table, the storage savings are only realized after a backfill clears the stored documents. +For an existing Estuary-managed table, the storage savings are only realized after a backfill clears the stored `flow_document` fields. ::: ### Migrating into a table another tool populated diff --git a/site/docs/guides/migrate-to-estuary.md b/site/docs/guides/migrate-to-estuary.md index 13c9c9cbcab..d137ee30000 100644 --- a/site/docs/guides/migrate-to-estuary.md +++ b/site/docs/guides/migrate-to-estuary.md @@ -71,7 +71,7 @@ materializations: - [`onIncompatibleSchemaChange: abort`](/concepts/advanced/evolutions/) replaces the default of `backfill`, which responds to an incompatible schema change by refreshing the table from the collection. Against history the collection does not have, that is data loss. :::warning -Do not use the **Backfill** button on this materialization. A backfill truncates the destination table, and paired with an incompatible schema change it drops and recreates it. Either way the pre-existing rows are gone, along with any partitioning. See [Schema changes during backfill](/reference/backfilling-data/#schema-changes-during-backfill). +Do not use the **Backfill** button on this materialization. A backfill truncates the destination table, which preserves your partitioning but removes the rows. If the publication also carries an incompatible schema change, or the collection key has changed, it drops and recreates the table instead, losing the partitioning too. See [Schema changes during backfill](/reference/backfilling-data/#schema-changes-during-backfill). ::: ### 5. Overlap, then decommission