[fas89]

fix(gcp): a BigQuery replace or rollback keeps the table's governance

Branch fix/bq-replace-keeps-governance, two commits on origin/main 63f3b40: 0954326 fix(iac), 0739491 fix(gcp).

Summary

Nothing forge-cli runs recreates a governed BigQuery table to replace its rows any more.

• fluid rollback restores a BigQuery table with one query job: SELECT * FROM <backup>
  into the live table, with WRITE_TRUNCATE_DATA and CREATE_NEVER.
• The GCP planner's replace mode does the same for a SQL build. It keeps the table
  definition planned, and its backup is a zero-copy CREATE SNAPSHOT TABLE … CLONE.
• When tofu replaces a BigQuery table, its row access policies and table IAM members
  are now recreated in the same apply.

dbt's table materialisation still recreates the table. That is a known gap, written
up in docs/governance-parity.md, and a follow-up. The dbt runner is unchanged.

Why

BigQuery documents that CREATE OR REPLACE TABLE drops every row access policy on the
table. The new table also takes its schema from the SELECT, so the column policy tags
went too, and with them column-level security and masking. Every principal that a row
filter or a column restriction limits then read every row and every column, until the
next fluid apply.

This class of bug showed up in three places:

1. fluid rollback (cli/rollback.py::_restore_bigquery). This one executes on
   main. It ran CREATE OR REPLACE TABLE <orig> AS SELECT * FROM <backup> for any
   BigQuery entry in .fluid/rollback-state.json.
2. The GCP planner in replace mode. It emitted that same CTAS for SQL builds and
   dropped bq.ensure_table for their targets. Its backup was a plain
   CREATE TABLE … AS SELECT copy with no row access policy and no policy tag, so any
   reader of the dataset could read all of it. These paths are latent on the OpenTofu
   engine but still wrong. GcpProvider.restore_ddl recorded the same CTAS.
3. The IaC. When tofu replaced a table (adding partitioning or a key to a live
   table, with --allow-data-loss), BigQuery deleted the table's row access policies
   and table IAM along with it. tofu planned them as unchanged, because their
   table_id is the same string before and after, so the new table stood ungoverned
   until the next apply.

What changes

fluid_build/cli/rollback.py

• _restore_bigquery builds SELECT * FROM <validated backup FQN>.
• The destination is a bigquery.TableReference built from the identifiers that
  _validated_bq_fqn checked. No destination SQL is interpolated.
• write_disposition comes from _bigquery_load.write_disposition("full_refresh"):
  WRITE_TRUNCATE_DATA, or WRITE_TRUNCATE under BIGQUERY_EMULATOR_HOST.
  create_disposition is CREATE_NEVER.
• The client comes from _bigquery_load.bigquery_client: ADC, or anonymous credentials
  on an emulator.
• The baked ddl[] is still never executed, and state-file input is still validated
  before any client exists.
• New guard (not run on an emulator): before writing, the restore compares the rows the
  restoring identity can read (COUNT(*)) with the backup's own numRows, which no
  policy filters. If a row access policy shows the identity only some of the rows, the
  restore stops with rollback_bigquery_backup_filtered instead of replacing the table
  with that subset. It also refuses when numRows is missing.
• The execution half is split into _run_bigquery_restore and _readable_backup_rows.

fluid_build/providers/gcp/provider.py

• GcpProvider.restore_ddl returns [], as the AWS provider does, with a docstring
  saying that fluid rollback restores BigQuery with a data-only query job that keeps
  the table and its governance.

fluid_build/providers/gcp/plan/planner.py

• In replace mode, a SQL build's action carries sql (the SELECT), a structured
  destination ({project, dataset, table}, validated) and
  write_disposition: WRITE_TRUNCATE_DATA / create_disposition: CREATE_NEVER.
• Amend mode still emits INSERT INTO.
• SQL that starts with INSERT/CREATE/MERGE/UPDATE/DELETE/COPY/TRUNCATE passes through
  untouched, as before.
• _bq_wrap_sql_for_target became _bq_build_for_target, with the target lookup
  pulled out into _bq_build_target.
• _plan_exposures no longer skips bq.ensure_table for SQL-build targets.
• _plan_replace_snapshots emits
  CREATE SNAPSHOT TABLE IF NOT EXISTS <backup> CLONE <table> OPTIONS(expiration_timestamp = TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)).
  allow_failure and the rollback_snapshot marker are kept, and the "BigQuery has no
  CLONE" comment is corrected.
• The docstrings of plan_actions, GcpPlanner and the provider are fixed.

fluid_build/providers/_planner_base.py

• The docstrings now describe destructive routing per provider: Snowflake uses CTAS,
  GCP a WRITE_TRUNCATE_DATA query job.

fluid_build/iac/providers/gcp.py

• Every google_bigquery_row_access_policy gets
  lifecycle.replace_triggered_by = ["google_bigquery_table.<t>.id"].
• So does every google_bigquery_table_iam_member on a table the module declares. That
  covers the shared-dataset table grants, and a planned iam.bind_bq_table on the
  module's own table, matched by _module_table.

Docs

• docs/apply.md: what --mode replace and replace-and-build do on the OpenTofu
  engine, and how rollback restores BigQuery.
• docs/governance-parity.md: replace and rollback keep governance; a replaced table
  comes back governed; the dbt gap is stated plainly; the "not proven yet" list is
  updated.
• AGENTS.md and fluid_build/cli/scaffold_ide.py: a note under the mode matrix.
• CHANGELOG.md: an [Unreleased] Fixed entry.

Tests

• tests/providers/test_gcp_replace_keeps_governance.py: the research tests, extended.
• tests/cli/test_rollback.py::TestRestoreBigQuery: rewritten. The old assertions
  pinned the CTAS.
• tests/iac/_fake_bigquery.py: now serves row access policies and table IAM, and
  deletes both along with their table, as BigQuery does.
• tests/iac/test_iac_gcp_governance_plan.py: three real-tofu tests.
• tests/iac/test_iac_gcp_governance.py: emit tests, plus two new tofu validate
  shapes.

Decisions taken (beyond the founder's binding ones)

1. The trigger references the table's id, not google_bigquery_table.<t>. I
   measured this with tofu 1.12 and terraform-provider-google 6.50 against the fake
   BigQuery (script and output in the scratch dir, rtb/):

   • Whole-resource reference (the brief's literal form): it fixes the replacement,
     but OpenTofu fires a resource reference on every in-place update too. A label
     change or a new retention period replaced both row access policies. That is
     remove: 2 in the plan, which the data-loss gate refuses without
     --allow-data-loss, because row access policies are not access-only resources.
     It would also briefly leave the table with no policies.
   • creation_time: correct, but tofu validate checks a replace_triggered_by
     attribute against the schema of the resource that carries it (an OpenTofu quirk I
     confirmed), and a table IAM member has no creation_time, so the module failed to
     validate.
   • id: valid on both resource types. It is unknown in the plan only when a new
     table is created, so it replaces with the table and is a no-op on an in-place
     update.

   A seeded test proves the difference.

2. The rollback refuses a filtered backup. A table snapshot can carry row-level
   security, and the restore reads the backup as the operator. A filtered operator
   would otherwise replace the table with the rows it can see and silently delete the
   rest. The guard fails closed, including when numRows is missing.

3. Snapshot expiry is 30 days. The old copy had no expiry. It was deleted only by
   the rollback-state pruning (the last 20 per env and product), which still deletes
   snapshots through cleanup_backups. There is no GCP setting for this, so 30 days
   matches the Snowflake provider's BackupConfig.backup_retention default.

4. The rollback result keeps the ddl key. For BigQuery it now holds the query
   text, so consumers that read ddl keep working. It also adds destination,
   write_disposition and create_disposition.

5. The renamed planner helper (_bq_wrap_sql_for_target became
   _bq_build_for_target) returns action fields instead of a SQL string. Its only
   caller is the planner.

OpenTofu-first audit

Audited on 6 October against git diff origin/main...HEAD, with tofu 1.12 and
terraform-provider-google 6.50. The audit changed no code.

1. Governance stays in the module. Row access policies, policy tags, data policies and
IAM are only ever emitted as OpenTofu resources (google_bigquery_row_access_policy,
google_bigquery_table_iam_member, the Data Catalog and data policy resources). The diff
adds three client calls, none of which touches governance: the restore's query job into the
existing table (client.query with WRITE_TRUNCATE_DATA / CREATE_NEVER), a get_table
read for numRows, and the existing _bigquery_load._count_table COUNT(*). Nothing in
the branch creates, recreates or patches a row policy, policy tag, data policy or IAM
binding through the BigQuery or Data Catalog SDK. The restore's job is the reason the table
keeps its governance: BigQuery documents that WRITE_TRUNCATE_DATA "keeps the constraints
and schema of the existing table" (REST discovery document, revision 20260922).

2. replace_triggered_by on .id is the documented attribute form. The OpenTofu
lifecycle page lists three triggers: any plan to update or replace a referenced resource
instance, and for a single attribute reference "any change to the attribute value". The
repo already uses the same meta-argument for the table itself
(replace_triggered_by = [terraform_data.<trigger>]). The real-tofu plan tests cover:

Case	Test (real tofu + provider 6.50 + _fake_bigquery)	Result
Table replaced, row policies	test_a_replaced_table_gets_its_row_access_policies_back	table and both policies delete+create, policies live after apply
Table replaced, shared-dataset grants	test_a_replaced_table_in_a_shared_dataset_gets_its_grants_back	member delete+create, grant live after apply
In-place update, row policies	test_an_in_place_table_update_leaves_its_row_access_policies (label, retention period)	table update, policies no-op, remove 0

The in-place case for table IAM members has no committed real-tofu test. It was run as a
probe outside the branch (retention period and label, shared dataset): the table is an
update, the member a no-op, remove 0; with the whole-resource reference the member is
delete+create on a label change. Same mechanism as the row policies, so no change.

3. Snapshot and restore stay data operations. They are not declared policy, and the
google provider cannot express them as desired state:

• google_bigquery_job copy takes only create_disposition, write_disposition,
  source_tables, destination_table and destination_encryption_configuration.
  tofu validate against provider 6.50 rejects operation_type ("No argument or block type
  is named operation_type"), though the REST API has operationType SNAPSHOT / RESTORE.
• Its query block rejects write_disposition = "WRITE_TRUNCATE_DATA" (expected ... to be one of ["WRITE_TRUNCATE" "WRITE_APPEND" "WRITE_EMPTY" ""]). The one overwrite it accepts,
  WRITE_TRUNCATE, removes every row access policy, which is the bug this branch fixes.
• The provider has no BigQuery snapshot or clone resource and google_bigquery_table has no
  snapshot or clone argument (1,096 resources searched).
• The resource's own documentation says a job cannot be changed or deleted once created,
  so a job per backup would accumulate in state and never be undone by teardown. A
  pre-replace backup must run before the table's destroy, and a resource that depends on the
  table is created after it; I found no OpenTofu feature that runs a step before a
  resource's destroy (Terraform's action triggers gained before_destroy only in 1.16).
  A rollback is an operator's request, not an apply. OpenTofu's provisioner documentation
  calls actions its plan cannot model a last resort.
• The planner snapshot is native-engine only. On the OpenTofu engine, which is the only
  engine for GCP, no snapshot is taken at all (docs/apply.md), so these actions do not
  run there.
• The restore uses the client, the disposition helper and the emulator handling of
  build_runners/_bigquery_load.py, the same way dlt's BigQuery destination loads.

Validation

All commands ran from the clone with the shared venvs and -p no:randomly.

Step	Command	Result
Failing tests now pass	pytest tests/providers/test_gcp_replace_keeps_governance.py tests/cli/test_rollback.py tests/iac/test_iac_gcp_governance.py	194 passed
Seeded defect, red	git stash push -- fluid_build/cli/rollback.py fluid_build/providers/gcp/provider.py fluid_build/providers/gcp/plan/planner.py fluid_build/iac/providers/gcp.py, then the 37 new or changed tests	22 failed, 15 passed. The 15 are invariants that hold on main too: amend still inserts, sink pass-through, a crafted destination refused, no snapshot when additive, missing-field errors, and the in-place no-op
Seeded defect, green	git stash pop, same 37 tests	37 passed
Seeded literal trigger	replace_triggered_by set to [google_bigquery_table.<t>], then the two real-tofu row-policy tests	2 failed (the in-place update tests: ['delete','create'] != ['no-op']), 1 passed. Fix restored afterwards
tofu-gated tests (tofu on PATH, PLUGIN_UNIX_SOCKET_DIR=/tmp/wfi1, TF_PLUGIN_CACHE_DIR=wf78/tfcache)	pytest tests/iac/test_iac_gcp_governance.py tests/iac/test_iac_gcp_governance_plan.py tests/iac/test_iac_tofu_validate.py tests/iac/test_iac_runner.py	148 passed in 446 s
Full suite, tofu not on PATH	env PATH=/usr/bin:/bin:/usr/sbin:/sbin:<venv>/bin pytest tests -q -x -p no:randomly	Run 1 stopped at test_dbt_model_contracts.py::TestLiveDbtBuildEnforcement::test_conformant_model_builds_then_drift_fails (No module named 'dbt.adapters.duckdb'), after 4845 passed. Run 2 deselected the six dbt-environment tests: 20067 passed, 660 skipped, 15 deselected (the six plus the nine the config deselects), 17 xfailed, 16 xpassed, rc=0 in 775 s
The six deselected tests on origin/main	a worktree of origin/main 63f3b40, the same six tests	6 failed there too (dbt-duckdb is not in the shared venv): pre-existing, not this change
Lint	lintvenv/bin/ruff check fluid_build/ tests/	All checks passed
Format	lintvenv/bin/black --check fluid_build/ tests/ (24.10.0)	1810 files would be left unchanged
Guardrail	python3 tools/product_guardrail/check.py --self-test && python3 tools/product_guardrail/check.py	self-test passed; 0 failing findings (13 pre-existing warnings, 1 grace)
Secret scan	dsvenv/bin/detect-secrets-hook --baseline .secrets.baseline <16 changed files>	First run exited 1 (it refreshed the baseline: one AGENTS.md line number moved 597 to 599). The (filename, type, hashed_secret) set is identical, 250 before and 250 after. The rerun with the baseline staged exited 0
Quality delta	ripwire <clone> --quality-delta	Run against origin/main (via a soft reset, files untouched): exit 2, 13 gating. Twelve are in test code: six duplication rows pairing the research test's _snapshot dict helper or a pytest.raises pattern with unrelated test helpers, two new-clone-of-reused-helper rows on the same helper, and verbosity of TestRestoreBigQuery (107 to 152 lines) and FakeBigQuery (179 to 251), which grew with the new coverage. The thirteenth is short-horizon-churn on _emit_row_policies, added in #700 the day before. In source, only minor or new-symbol rows remain: _plan_replace_snapshots 73 to 79 lines, and _bq_build_for_target at 6 params by ripwire's count (the same keyword arguments the old _bq_wrap_sql_for_target took). Before review, _restore_bigquery (131 to 172 lines) and _bq_build_for_target (complexity 17) also gated; both were refactored (_run_bigquery_restore, _readable_backup_rows, _bq_build_target) and no longer appear. I did not reshape test code just to lower the count
Live probe dry run	live_probe.py --dry-run under a wrapper that refuses every socket connection	71 calls printed, 0 connection attempts, exit 0. The live run is left to the main session

Live probe

r1-scratch/live_probe.py has not been run. It runs against project
fluid-testing-a5d2b1 in europe-west3.

Identity. It asserts that the ADC identity is tordering@gmail.com, using tokeninfo
with the token in a POST body, before it creates anything.

What it creates. Everything is prefixed fluid_probe_<UTC timestamp>:

• a dataset with a 1-day default table expiration;
• a taxonomy with one policy tag;
• a governed customer_profile table: a tagged email column, DAY ingestion-time
  partitions with an expiration, 3 rows of which 2 are consented, and the row access
  policy probe_consented;
• a governed copy for the control, an ungoverned source table, and an ungoverned
  mapping table.

Steps.

1. Before: check the table's governance as set up.
2. Snapshot governance: take snapshot S0 through the fixed backup path, and report
   what it keeps.
3. Elevated snapshot: add a temporary TRUE policy for the restorer, take snapshot
   S1, then drop the policy from the live table.
4. Planner replace: run the replace-mode build action the way the action describes
   it, then check governance again (after-planner).
5. Rollback as the plain owner: run fluid rollback from S0 (expected: refused by
   the new guard), then from S1 without Fine-Grained Reader (expected: refused on the
   tag).
6. Rollback with the access a restorer needs: grant Fine-Grained Reader, poll until
   it takes effect, run the rollback from S1, revoke it, and poll until the column is
   refused again. Then check governance (after-rollback).
7. Column mapping: a WRITE_TRUNCATE_DATA query job with the SELECT's columns
   reordered: two STRING columns swapped, then mismatched types. A third job with a
   NULLABLE column missing.
8. Control: run the old CREATE OR REPLACE TABLE … AS SELECT on the copy. It is
   expected to drop the policy and the tag.

Teardown runs in a finally. It deletes the dataset with its contents and the
taxonomy, then lists datasets and taxonomies to show that nothing prefixed
fluid_probe_ remains. The probe prints a JSON summary with per-step results and
verdicts.

Risks

• Unmeasured live. That a WRITE_TRUNCATE_DATA query job keeps the policy tags and
  row access policies is documented, and was measured for a load job on 5 October, but
  not for a query job. The probe settles it.
• Column mapping. Whether such a job maps the SELECT's columns by name or by
  position, and what it does with a missing NULLABLE column, is unknown until the probe
  runs. A planner action whose SELECT column order differs from the table could
  misplace values if the mapping is positional.
• Guard false refusals. If BigQuery reports no numRows for a snapshot, every
  BigQuery rollback is refused, because the guard fails closed. The probe records
  snapshot_num_rows.
• What a restorer needs. A restore must run as an identity that a TRUE row access
  policy covers and that is a Fine-Grained Reader of the backup's policy tags. Before,
  the old CTAS backup needed the same access when it was taken; now it is needed at
  restore time.
• Snapshot retention. A snapshot outlives a crashed probe or an un-pruned state
  for up to 30 days. It is zero-copy until the base table changes.
• The trigger relies on how id plans. Depending on id being unknown in the plan
  when a resource is replaced is standard Terraform and OpenTofu behaviour, and the
  tests pin it against provider 6.50.

Follow-ups

• dbt into a governed BigQuery table. dbt-bigquery's create or replace table
  under the table materialisation, and under --full-refresh in replace-and-build,
  drops row access policies, and drops policy tags unless persist_docs re-applies
  them. Founder decision: separate follow-up.
• The Snowflake analogue. providers/snowflake/plan/planner.py::_wrap_sql_for_target
  emits CREATE OR REPLACE TABLE … AS when destructive. The CREATE OR REPLACE TABLE
  example in docs/apply.md under embedded SQL runs verbatim and would drop directly
  attached masking and row access policies.
• The keyword check. _bq_wrap_sql_for_target (now _bq_build_for_target) checks
  only the first 32 characters for a sink keyword, so SQL that begins with a comment is
  wrapped into invalid SQL. The Snowflake wrapper has the same check.
• forge/core/apply_modes.py. Its docstring mode matrix still says
  "auto-snapshot → CREATE OR REPLACE".
• GcpProvider.cleanup_backups builds bigquery.Client(project=...) directly
  rather than _bigquery_load.bigquery_client, so it is not emulator-aware.

Live validation on real BigQuery (6 Oct 2026)

A probe ran this branch's code against real BigQuery: one temporary dataset in europe-west3 and one Data Catalog taxonomy, both deleted afterwards and verified absent by listing.

Setup: a DAY-partitioned table (7-day partition expiration) with email under a policy tag, and a row access policy probe_consented (consent = true) for the probing identity, which is no fine-grained reader of the tag. The identity got a TRUE row policy and Fine-Grained Reader only while it wrote, the access a pipeline identity holds; both were removed before the final checks.

Step	Row access policies	Policy tag on email	Partitioning / expiration	Restricted reader sees
Before	probe_consented	kept	DAY, 7 days	2 of 3 rows, email refused
Backup: CREATE SNAPSHOT TABLE IF NOT EXISTS … CLONE … OPTIONS(expiration_timestamp = …)	both copied to the snapshot	kept on the snapshot	DAY (partition expiration not copied, as documented)	–
Replace: query job, WRITE_TRUNCATE_DATA, CREATE_NEVER	kept	kept	kept	rows replaced
fluid rollback restore from the snapshot	kept	kept	kept	3 rows restored
After the write access is removed	probe_consented	kept	kept	2 of 3, email refused
fluid rollback without that access	refused (rollback_bigquery_execute_failed, 403 on the tagged column); nothing changed	kept	kept	2 of 3
Control: the old CREATE OR REPLACE TABLE … AS SELECT (unpartitioned copy)	none	dropped	–	3 of 3, email readable

Also measured:

• a WRITE_TRUNCATE_DATA query job maps the SELECT's columns by name, not by position, and a missing NULLABLE column lands as NULL;
• BigQuery refuses the replace job and the snapshot from an identity without full row access or Fine-Grained Reader (403), so a restricted identity fails closed;
• on a partitioned table, the old CTAS fails outright ("Cannot replace a table with a different partitioning spec"), so the silent drop happens on unpartitioned tables.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.