diff --git a/site/docs/guides/advanced-usage/feature-flags.md b/site/docs/guides/advanced-usage/feature-flags.md index 54b32184f56..2f0af2f0ea5 100644 --- a/site/docs/guides/advanced-usage/feature-flags.md +++ b/site/docs/guides/advanced-usage/feature-flags.md @@ -115,13 +115,26 @@ Generates synthetic row identifiers for tables without primary keys. Allows materializations to write to tables that already exist in the destination, even for newly added bindings. - **Default:** Disabled. New bindings fail if the target table already exists. -- **Use case:** Migrating data from another system into Estuary-managed tables, or re-creating a materialization that was previously deleted. +- **Use case:** Migrating data from another system into Estuary-managed tables, re-creating a materialization that was previously deleted, or pre-creating a table so you control its DDL, for example [BigQuery partitioning](/reference/Connectors/materialization-connectors/BigQuery/#table-partitioning), which cannot be set on a table after it is created. - **Caveats:** - Enabling this flag makes the connector load existing keys from the destination before merging, instead of skipping that lookup for keys it expects to be new. It is slightly slower but ensures merges and updates work correctly against rows that already exist in the table. - The connector cannot verify that existing table schemas are compatible. - This flag alone does **not** prevent backfill of the source collection. To avoid backfilling data into the existing table, also configure [`notBefore`](/reference/time-travel) or use "Only Changes" mode on the binding. + - If the table already contains rows that Estuary did not write, this flag is not sufficient on its own. See [Migrate an existing pipeline to Estuary](/guides/migrate-to-estuary/). - **Applies to:** All SQL and warehouse materialization connectors (PostgreSQL, MySQL, Snowflake, BigQuery, Redshift, etc.) +#### What the connector does to the existing table + +On the first publication after attaching, the connector reconciles the table against the binding's field selection: + +- Columns in the field selection that the table does not have are added with `ALTER TABLE ... ADD COLUMN`, as nullable. This includes Estuary's own `flow_document`, `flow_published_at`, and `_meta/op` columns. +- Columns the table has that are **not** in the field selection are altered to drop `NOT NULL`, if they had it. Keep this in mind when you pre-create a table from DDL you care about: a `NOT NULL` constraint on a column Estuary does not materialize will be relaxed. +- The table itself is not dropped or recreated by this reconciliation, so partitioning, clustering, and other table-level DDL survive it. + +Because missing columns are added for you, you can pre-create a table with only the columns you need to control (the partition column and the collection key, for instance) and let the connector fill in the rest. + +Attaching is not the only thing that touches the table, though. A [backfill](/reference/backfilling-data/#schema-changes-during-backfill) truncates it by default, which preserves table-level DDL, but drops and recreates it when a selected field's type changes incompatibly, when the collection key changes, or when `always_drop_tables_on_backfill` is set. Since the default [`onIncompatibleSchemaChange`](/concepts/advanced/evolutions/) is `backfill`, an incompatible schema change can reach that path without anyone asking for a backfill. Set it to `abort` if the table's DDL or its pre-existing rows matter. + ### retain_existing_data_on_backfill Skips truncating destination tables when a backfill is triggered. diff --git a/site/docs/guides/customize-dataflows/customize-materialization-fields.md b/site/docs/guides/customize-dataflows/customize-materialization-fields.md index 9016c59d2f3..2787feb0b56 100644 --- a/site/docs/guides/customize-dataflows/customize-materialization-fields.md +++ b/site/docs/guides/customize-dataflows/customize-materialization-fields.md @@ -366,9 +366,15 @@ Since Estuary no longer writes to it, you can manually drop the leftover column A new binding, or a backfill that takes the [drop-and-recreate path](/reference/backfilling-data/#schema-changes-during-backfill), creates the table without the `flow_document` column entirely. :::note -The main motivation for this option is saving destination storage. For an existing table, those savings are only realized after a backfill clears the stored documents. +For an existing Estuary-managed table, the storage savings are only realized after a backfill clears the stored `flow_document` fields. ::: +### Migrating into a table another tool populated + +Saving storage is one reason to exclude `flow_document`. The other is migration: this option is **required** when a new binding attaches to a table that already holds rows Estuary did not write, because the connector cannot populate a stored document for rows it never wrote and reads a null instead. Reconstructing from the top-level columns is what makes those rows mergeable. + +That case has its own guide, including the rest of the configuration it needs: [Migrate an existing pipeline to Estuary](/guides/migrate-to-estuary/). + ## Pruned fields Estuary's schema inference tracks unique field locations. To prevent excessively large schemas, there's a complexity limit: diff --git a/site/docs/guides/migrate-to-estuary.md b/site/docs/guides/migrate-to-estuary.md new file mode 100644 index 00000000000..d137ee30000 --- /dev/null +++ b/site/docs/guides/migrate-to-estuary.md @@ -0,0 +1,108 @@ +--- +description: Move an existing pipeline to Estuary without dropping your destination tables, using allow_existing_tables_for_new_bindings, Exclude Flow Document, and a cutover filter. +slug: /guides/migrate-to-estuary/ +--- + +# Migrate an Existing Pipeline to Estuary + +When you replace another pipeline with Estuary, the destination tables it wrote are usually worth keeping. They hold history that is expensive to re-extract, they may be partitioned by hand, and downstream models already point at them. + +Estuary can attach to those tables and keep writing to them. By default that is blocked: + +``` +validating binding: table [warehouse schema orders] already exists for new binding +"acmeCo/production/orders". You must drop this table to continue. +``` + +The default is right whenever the destination holds nothing you cannot regenerate: drop the table and let Estuary rebuild it. This guide is for when it does hold something. + +## Which case are you in + +| Your table | What you need | +| --- | --- | +| Exists but is **empty**, pre-created to control its DDL | [`allow_existing_tables_for_new_bindings`](/guides/advanced-usage/feature-flags/#allow_existing_tables_for_new_bindings) only | +| Exists and **holds rows Estuary did not write** | That flag, plus [Exclude Flow Document](#why-exclude-flow-document-is-required) and a cutover filter | + +The trigger is rows existing, not the table existing. The first case is not really a migration: it is how you apply table-level DDL that Estuary does not manage, such as [BigQuery partitioning](/reference/Connectors/materialization-connectors/BigQuery/#partitioning-a-new-table), which cannot be set after a table is created. The rest of this page is the second case. + +## Cutover + +### 1. Run the capture alongside your existing pipeline + +Create the capture and let it run. Data lands in collections and nothing reaches your destination yet. Captures are read-only against the source, so both tools can read it concurrently. For CDC sources each tool needs its own replication slot or equivalent, so check your source's limits before adding a second. + +### 2. Decide how to avoid re-loading history + +[`notBefore`](/reference/time-travel) filters on when a document was **published into the collection**, not on when the row changed in your source. A timestamp copied from your outgoing tool's last sync therefore does not work: if the capture backfills, every document is published today, nothing is filtered, and the whole backfill lands in your table. + +Two approaches that do work: + +- Capture with **Only Changes** backfill mode, so the collection only ever holds post-cutover events. +- Let the capture backfill, then set `notBefore` to when that backfill completed. Rows that changed between the old tool's last sync and the capture's start are then picked up only when CDC next touches them. + +Cleanest of all is to stop the old pipeline's writes before you publish the binding, and use that moment as the cutover point. + +### 3. Check the table against the collection + +Do this before cutting over, not after. See [Pre-cutover checks](#pre-cutover-checks). + +### 4. Create the materialization + +```yaml +materializations: + acmeCo/production/materialize-bigquery: + endpoint: + connector: + image: ghcr.io/estuary/materialize-bigquery:dev + config: + # ...other endpoint configuration... + advanced: + feature_flags: "allow_existing_tables_for_new_bindings" + no_flow_document: true + bindings: + - source: acmeCo/production/orders + resource: { table: orders, dataset: warehouse } + notBefore: 2026-01-01T00:00:00Z + onIncompatibleSchemaChange: abort +``` + +- `allow_existing_tables_for_new_bindings` lets the binding attach instead of failing. For what the connector then does to the table, see [What the connector does to the existing table](/guides/advanced-usage/feature-flags/#what-the-connector-does-to-the-existing-table). +- **Exclude Flow Document** (`no_flow_document`) is required once the table holds rows. See [why](#why-exclude-flow-document-is-required). +- [`onIncompatibleSchemaChange: abort`](/concepts/advanced/evolutions/) replaces the default of `backfill`, which responds to an incompatible schema change by refreshing the table from the collection. Against history the collection does not have, that is data loss. + +:::warning +Do not use the **Backfill** button on this materialization. A backfill truncates the destination table, which preserves your partitioning but removes the rows. If the publication also carries an incompatible schema change, or the collection key has changed, it drops and recreates the table instead, losing the partitioning too. See [Schema changes during backfill](/reference/backfilling-data/#schema-changes-during-backfill). +::: + +### 5. Overlap, then decommission + +Let both pipelines run long enough to compare row counts and spot-check recently changed rows, then disable the old one. + +## Why Exclude Flow Document is required + +Under [standard updates](/concepts/materialization/#delta-updates) the connector keeps each complete document in a `flow_document` column and reads it back to merge updates into. + +When a binding attaches to an existing table, the connector adds that column but cannot populate it for the rows already there. The load query selects it anyway and reads a null, and the task fails. The exact error varies by connector, but it is always a task failure rather than silent corruption: the pipeline stops instead of writing wrong rows. + +[Exclude Flow Document](/guides/customize-materialization-fields/#excluding-flow_document-with-standard-updates) reconstructs the document from the table's own top-level columns instead, and those do hold values for the pre-existing rows. + +Set it when you create the binding. Enabling it on a binding that already exists also works, and takes effect on that binding's next transaction, but the task will have been failing in the meantime. + +## Pre-cutover checks + +The connector cannot verify that an existing table is compatible with the collection. In rough order of how much damage they do when missed: + +- **Row uniqueness must match the collection key.** The connector merges on the collection key. If the existing rows are not uniquely identified by that key, updates will not find them and will insert instead, leaving duplicates and no error. This is the most common and least visible migration failure. If the old tool generated a surrogate key of its own, use [`groupBy`](/guides/customize-materialization-fields/#group-by-keys) to merge on the real one. +- **Column names and types.** Other tools lowercase names or add their own metadata, and type mappings differ for the same source column, particularly numeric precision and timestamps. Estuary uses the collection's field names and the mapping documented on your destination connector's page. +- **The old tool's metadata columns go stale.** They stay in the table but stop being updated, since Estuary does not write them. Check that no downstream model or incremental filter depends on one. Estuary's equivalent is `flow_published_at`. +- **Deletion behavior.** Confirm whether the table used hard deletes or a soft-delete marker, and configure the materialization to match. + +If you can, run the materialization against a copy of the table in a scratch schema first and compare. It catches all of the above at once. + +## Related + +- [Feature flags](/guides/advanced-usage/feature-flags/) for `allow_existing_tables_for_new_bindings`, and for `retain_existing_data_on_backfill` +- [Customize materialized fields](/guides/customize-materialization-fields/) for field selection, Exclude Flow Document, and `groupBy` +- [Time travel](/reference/time-travel) for `notBefore` +- [Backfilling data](/reference/backfilling-data/) for what a backfill does to a destination table +- [Schema evolution](/guides/schema-evolution/) for `onIncompatibleSchemaChange` diff --git a/site/docs/reference/Connectors/materialization-connectors/BigQuery.md b/site/docs/reference/Connectors/materialization-connectors/BigQuery.md index da8eda0ba1f..ee97eb0160e 100644 --- a/site/docs/reference/Connectors/materialization-connectors/BigQuery.md +++ b/site/docs/reference/Connectors/materialization-connectors/BigQuery.md @@ -180,7 +180,34 @@ Tables are automatically created with [clustering](https://cloud.google.com/bigquery/docs/clustered-tables) based on the Estuary collection primary keys. Tables are not created with any other [partitioning](https://cloud.google.com/bigquery/docs/partitioned-tables), but pre-existing partitioned tables can be materialized to. -It isn't possible to alter the partitioning of an existing table, but you can convert an existing table to one with partitioning by creating a new table and copying the data from the existing table into it. This can be done to tables that the connector is materializing to, as long as the materializing task is temporarily disabled while doing the conversion. +BigQuery does not allow the partitioning of a table to be altered after it is created, so there are two ways to get a partitioned destination table. Which one you need depends on whether the table exists yet. + +### Partitioning a new table + +If the binding does not exist yet, create the table yourself with the partitioning you want, then attach a binding to it. This avoids the copy-and-rename conversion below entirely, which matters if you are adding many tables. + +1. Create the table in BigQuery with your partitioning. You only need the columns you want to control, such as the partition column and the collection key: +```sql +create table . ( + timestamp, + string +) +partition by ; +``` +Partitioning on `flow_published_at` is a common choice, since Estuary writes that column on every row and incremental transformations often filter on it. + +2. Enable the [`allow_existing_tables_for_new_bindings`](/guides/advanced-usage/feature-flags/#allow_existing_tables_for_new_bindings) feature flag on the materialization, under **Endpoint Config → Advanced Options**. Without it, a new binding whose table already exists fails validation. + +3. Add the binding. On the first publication the connector adds the remaining columns from the binding's field selection with `ALTER TABLE ... ADD COLUMN`, including its own `flow_document`, `flow_published_at`, and `_meta/op` columns. The table is not recreated, so your partitioning is preserved. + +Two things to note: + +* A `NOT NULL` constraint on a column that Estuary does not materialize will be dropped when the connector reconciles the table. +* This procedure assumes the table you create is **empty**. If it already holds rows that Estuary did not write, more configuration is needed: see [Migrate an existing pipeline to Estuary](/guides/migrate-to-estuary/). + +### Repartitioning an existing table + +If the table is already being materialized to, you can convert it by creating a new table and copying the data from the existing table into it. This can be done to tables that the connector is materializing to, as long as the materializing task is temporarily disabled while doing the conversion. To convert an existing materialized table to one with different partitioning: 1. Pause your materialization by disabling it from the [UI](../../../concepts/web-app.md) or editing the task specification with the [CLI](../../../guides/flowctl/edit-specification-locally.md).