feat(governance): column masking, row filters and governance labels on AWS and GCP by fas89 · Pull Request #700 · Agenticstiger/forge-cli

fas89 · 2026-10-05T17:59:31Z

What this adds

A contract can say that a principal reads a column masked rather than not at all, and which rows a principal reads, in fields that name no cloud (fluid-schema 0.7.6). Each emitter writes them in its platform's own terms. A binding that cannot enforce one is refused, never left unenforced.

Contract	GCP	AWS
`columnRestrictions[] {access: mask, mask: last_four \| first_four \| nullify}`	Its own policy tag + `google_bigquery_datapolicy_data_policy` (LAST_FOUR_CHARACTERS / FIRST_FOUR_CHARACTERS / ALWAYS_NULL) + `maskedReader` for the masked principals	The table grant excludes the columns; a protected multi-dialect Glue view (`VIRTUAL_VIEW`, definer, Athena validation connection) returns them masked by the same rule. Needs `governance.lakeFormation.maskedViews`, else `column-mask-unenforceable`
`policy.authz.rowFilters[] {principal, where, name?}`	A `google_bigquery_row_access_policy` per filter + `fluid_all_rows` (TRUE) for every other granted principal	An `aws_lakeformation_data_cells_filter` per filter; the principal's SELECT is on the filter, not the table
labels, classification, jurisdiction, regulation, residency	GCP labels on the dataset, table and key	AWS tags on the bucket and key, Glue table parameters

Row filter predicates are parsed with sqlglot first. A statement separator, a comment, a subquery or an undeclared column is refused.

A bug this would have shipped, fixed here

A full-refresh BigQuery load was WRITE_TRUNCATE. BigQuery documents that it "remove[s] all existing row access policies", so every build would have dropped the filters the apply had just made. Every filtered principal would then have read every row until the next apply. The load is now WRITE_TRUNCATE_DATA, the disposition dlt's BigQuery destination uses for the same reason. On an emulator it stays WRITE_TRUNCATE, because goccy appends for any disposition it does not know (server/handler.go).

Measured on real BigQuery before the change:

After WRITE_TRUNCATE: no policy left, and every row visible.
After WRITE_TRUNCATE_DATA: the policy kept.
On a DAY-partitioned table with a partition expiration: every partition replaced, and the partitioning and expiration kept.

Three more found before merging, each fixed in its own commit

A data policy id over BigQuery's limit (eaa4485). BigQuery refuses an id of 200 characters or more; I measured the limit live (200 accepted, 300 refused). A longer id now keeps its first characters and ends in a hash. A short id is unchanged.
The masked view's definer lacked CREATE_TABLE (d3a825c). Lake Formation's view prerequisites require it on the view's database. The live probe had granted it by hand, which is how the gap went unseen.
Emulator mode on provider 6 (effa30b). Provider 6 builds a Glue id from the catalog and refuses :<name> ([Bug]: Read of aws_glue_catalog_database after creation fails due to account ID missing from ARN hashicorp/terraform-provider-aws#31626). On an emulator the account id stays skipped, because 6.23+ reads bucket tags through S3 Control at <account>.<endpoint>, which neither moto nor LocalStack routes ([Bug] After bumping to v6.23.0, localstack cannot create an S3 bucket due to S3 Control API hashicorp/terraform-provider-aws#45292). So each Glue database, table and masked view takes its catalog_id from data.aws_caller_identity there. Real AWS modules are unchanged.

Security review, and what it changed (`f82f90f`)

A security review of this change found three ways the emitted policies gave a principal more than the contract allows. Each was checked independently and reproduced by a test that fails without its fix.

AWS, a masked view reading past its reader's grant. The protected view reads the table as its definer, so whatever it selects, its reader reads. A masked principal whose grant listed its columns (columns:) read every other column raw through the view, the contract's denied ones included. The view now selects exactly what the principal's own grant reads raw, plus its masked columns.
GCP, a write-only filtered principal reading every row. A filter's grantees were its principal's readers only. A principal that only writes (roles/bigquery.dataEditor, which reads rows) was left out of its own filter and landed in fluid_all_rows. A filter now names every identity of its principal, and no filtered identity is ever in fluid_all_rows.
GCP, a row filter dropped on a view or a non-BigQuery target. It is refused now (row-filter-view, gcp-governance-unsupported-target), as a column restriction already was.

Rejected after tracing: label or tag text reaching OpenTofu interpolation (_label keeps [a-z0-9_-]), identifier quoting in the view SQL (validate_ident allow-list), two logical principals mapped to one identity (handled per identity), and an AWS masked or filtered principal keeping a raw table grant (refused).

The moto lane on provider 6.23+ (`c52a4fe`)

Provider 6.23+ tags a bucket inside its CreateBucket request and skips PutBucketTagging. moto 5.2.3 drops those tags, so the state-drift lane read a clean apply as tag drift. Real S3 keeps them. A test-only shim (tests/iac/_moto_s3.py) makes moto keep them, the fix fakecloud made in its own S3 (faiscadev/fakecloud#2557). Locally, the drift lane goes from 2 of 5 to 5 of 5, and the two lanes after it pass 140 of 140.

Not in this change

BigQuery SQL-transform builds and their rollback write with CREATE OR REPLACE TABLE ... AS SELECT. BigQuery documents that this drops the table's row access policies, and it rebuilds the schema without the policy tags. The DuckDB runner's load, which the lab uses, now keeps them.
fluid verify does not yet check a masked view's columns, the GCP data policy and row access policies, or the AWS filter expression (stated in the parity doc).
On AWS, a row filter or mask on a binding that is not a Glue table is dropped along with its Lake Formation grants. Nothing is granted, but nothing is refused either.
A binding's lakeFormation.tags without contract-level tagDefinitions emits depends_on on undeclared resources.
The row-filter predicate check refuses node types (a denylist) in sqlglot's default dialect. An allow-list of expression nodes would be stricter.

terraform-provider-aws `~> 6.0`

The view needs aws_glue_catalog_table.view_definition, which 5.x does not have. With only the pin moved, tofu plan with 6.67 against the live state of eleven applied AWS products planned no change for any of them. Provider 6.23+ reads bucket tags through S3 Control (s3:ListTagsForResource). The lab's deploy role is allowed it, checked by IAM policy simulation.

Evidence

New: tests/iac/test_iac_mask_rowfilters_labels.py (25), plus disposition tests in tests/build_runners/test_duckdb_bigquery_load.py. With the old disposition put back, 4 tests fail.
Local: tests/iac 1547 passed; tests/build_runners and tests/cli 2809 passed.
Measured by hand on a real lab, with the resources this emits made by hand:
- BigQuery masking: the analyst read XXXXX plus the last four digits; the steward read the clear value.
- AWS protected view: every row masked for the analyst, who was still refused the base column.
- Lake Formation row filter: 752 of 10,172 rows.
Refactoring left the generated modules byte-identical (AWS and GCP smoke contract).

Not checked by `fluid verify` yet (stated in the parity doc)

The data policy and its masked readers.
The row access policies on GCP.
The filter expression on AWS.
The labels.

An end-to-end apply of exactly what this emits is not proven yet either.

…n AWS and GCP A contract can now say that a principal reads a column masked rather than not at all, and which rows a principal reads, in fields that name no cloud. Each emitter writes them in its platform's own terms, and a binding that cannot enforce one is refused, never left unenforced. Column masking, `columnRestrictions[] {access: mask, mask: last_four | first_four | nullify}` (fluid-schema 0.7.6): - GCP: the masked columns get their own policy tag with a BigQuery data policy on it (DATA_MASKING_POLICY, LAST_FOUR_CHARACTERS, FIRST_FOUR_CHARACTERS or ALWAYS_NULL) and roles/bigquerydatapolicy.maskedReader for the masked principals. The tag's fine-grained readers read the clear value. - AWS: Lake Formation cannot mask. The masked principal's grant excludes the columns from the table, and a protected multi-dialect Glue Data Catalog view (VIRTUAL_VIEW, run as its definer, validated through an Athena connection) returns them masked by the same rule, with the principal's row filter as its WHERE. It needs governance.lakeFormation.maskedViews {definer, validationConnection}; without it a mask is refused (column-mask-unenforceable). Row filters, `exposes[].policy.authz.rowFilters[] {principal, where, name?}`: - The predicate is parsed with sqlglot before it is written into a policy. A statement separator, a comment, a subquery or an undeclared column is refused. - GCP: a row access policy per filter, plus fluid_all_rows (TRUE) for every other principal the expose grants anything to, because BigQuery shows a principal no policy names no row at all. - AWS: a data cells filter per filter, and the principal's SELECT granted on the filter instead of the table. fluid verify's column check reads grants on data cells filters too, so a filtered principal's excluded columns are still checked. Governance labels: the contract's and the expose's labels, and its classification, jurisdiction, regulatory frameworks and residency, as GCP labels on the dataset, table and key, and as AWS tags and Glue table parameters. A full-refresh BigQuery load is now WRITE_TRUNCATE_DATA. BigQuery documents that "operations that use the WRITE_TRUNCATE write disposition remove all existing row access policies", so every build would have dropped the row filters the apply had just made, and each filtered principal would have read every row until the next apply. Measured on BigQuery before the change: after a WRITE_TRUNCATE load the table had no row access policy and showed every row; after WRITE_TRUNCATE_DATA it kept the policy, and on a DAY-partitioned table with a partition expiration it replaced every partition's rows and kept the partitioning. dlt's BigQuery destination loads with the same disposition for the same reason. On an emulator the load stays WRITE_TRUNCATE: goccy/bigquery-emulator appends for a disposition it does not know. terraform-provider-aws is pinned ~> 6.0: the view needs aws_glue_catalog_table view_definition, which 5.x does not have. A tofu plan with 6.67 against the live state of eleven applied AWS products planned no change for any of them. docs/governance-parity.md gains the three rows, what fluid verify does not check yet (the data policy, the row access policies, the filter expression and the labels), and the prerequisites.

github-actions · 2026-10-05T17:59:45Z

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

Link a docs PR — check the "Docs PR linked" box and paste the URL
Mark as no docs needed — check "No docs needed" with a justification
Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

BigQuery refuses a data policy id of 200 characters or more ("should only contain letters, numbers and underscores while under 200 characters"; measured on 5 October 2026: 200 accepted, 300 refused). The id is the policy tag's resource stem, built from the product, dataset, table and columns, so a product with long names would have failed at apply. A longer id now keeps its first characters and ends in a hash of the whole, and a short one is unchanged.

…tabase Lake Formation's prerequisites for a Data Catalog view: the role that defines it holds SELECT with the grant option on every table it reads and CREATE_TABLE on the database the view is created in. The definer was granted DESCRIBE only, so on an account where nothing else had granted it CREATE_TABLE the view would not have been created. The probe the view's shape was measured with had granted the definer CREATE_TABLE by hand, which is how the gap went unseen.

terraform-provider-aws 6 makes a Glue database's id <catalog id>:<name> (a table's <catalog id>:<database>:<name>) from the resource's catalog_id or the provider's account id, and refuses to read back an id whose catalog is empty ("unexpected format for ID (:sales), expected catalog-id:database-name", hashicorp/terraform-provider-aws#31626). On an emulator the provider block skips requesting the account id, so every Glue database and table applied against moto failed right after its create. Requesting it is not the way out: given one, provider 6.23 and later read a bucket's tags through S3 Control at <account>.<endpoint>, which neither moto nor LocalStack routes (hashicorp/terraform-provider-aws#45292). So on an emulator the Glue database, table and masked view take their catalog id from data.aws_caller_identity, which the emulator answers. On real AWS the module is unchanged. The moto retention test now emits its module as an emulator user would, with AWS_ENDPOINT_URL set, and keeps its own routing as the one provider block. The new test file gets its license header.

…does terraform-provider-aws 6.23 and later tag a bucket in its CreateBucket request (CreateBucketConfiguration Tags, added to S3 in 2025) and skip PutBucketTagging when the create succeeds. moto 5.2.3 reads only LocationConstraint from that body and drops the tags, so every bucket the moto lanes applied was left untagged, and the state-drift lane read a clean apply as tag drift (tags.fluid_contract, tags.managed_by). On real S3 the tags are kept: eleven applied AWS products planned no change on provider 6.67. tests/iac/_moto_s3.py wraps moto's bucket PUT so a create also stores the body's tags, the fix fakecloud made in its own S3 (faiscadev/fakecloud#2557), and tests/iac/conftest.py applies it once per process. The drift lane passes locally with it (5 of 5), and failed without it (2 of 5).

…never give more than the contract Three findings of a security review of this change, each reproduced by a test that fails without its fix. - A protected Glue view reads the table as its definer, so whatever it selects, its reader reads. It selected every column but the reader's derived exclusions, and a grant written with a columns allow-list has none, so a masked principal whose grant listed its columns read every other column raw through the view, the contract's denied ones included. The view now selects exactly what the principal's own grant reads raw (its columns list, or every column less its exclusions), plus its masked columns, masked. - On GCP a filter's grantees were its principal's readers only. A principal that only writes (roles/bigquery.dataEditor, which reads rows) was left out of its own filter and landed in fluid_all_rows, so a partner limited to its own rows read every row. A filter now names every identity of its principal, whatever it holds (a row access policy never grants the table), and no filtered identity is ever in fluid_all_rows, since BigQuery shows a principal the union of the policies naming it. - A row filter on a BigQuery view, or on a GCP binding that is not a BigQuery table, was dropped. It is refused now (row-filter-view, and gcp-governance-unsupported-target), as a column restriction already was.

…ial to its new line The moto shim's import and call moved conftest.py's dummy moto credential ("secret_key": "test", already in the baseline) from line 835 to 843. detect-secrets-hook fails a pull request whose baseline line numbers are stale. Nothing new is allowlisted: the same hash, at its new line.

…ator terraform-provider-aws 6 refuses to read back a Glue database whose id has no catalog (":engine_silver"). The apply engine compiled this lane's module as for real AWS (no catalog id) while the sidecar provider skipped requesting the account id, so two of its four tests failed on provider 6. The moto fixture now names the server in AWS_ENDPOINT_URL, as an emulator user does, so the module carries each Glue resource's catalog id, and the sidecar is provider_override.tf.json, merged into the provider block the plugin emits for an emulator instead of duplicating it. 4 of 4 pass locally. The secrets baseline follows the file's known dummy credential to its new line.

CHANGELOG.md: the [0.19.0] section, with #700 (column masking, row filters and governance labels on AWS and GCP; terraform-provider-aws 6; a full-refresh BigQuery load that keeps row access policies) and #699 (a DuckDB landing outside the contract's directory lands on every run), and the compare links for 0.18.1, 0.18.2 and 0.19.0.

… emulator, so provider 6 reads a Glue database back (#708) * test(iac): the LocalStack and moto e2e rigs compile the module for an emulator, so provider 6 reads a Glue database back The heavy emulated lane went red on 2026-10-06, and stayed red: six round-trips in test_iac_aws_localstack_e2e.py failed at tofu apply with "unexpected format for ID (:mesh_silver), expected catalog-id:database-name". All six apply a Glue database. Every LocalStack round-trip that applies none of them passed in the same run. The three Glue tests in test_iac_moto_e2e.py fail the same way against moto, and that file runs in no CI lane. #700 moved hashicorp/aws from ~> 5.0 to ~> 6.0. Provider 6 gives a Glue database the id <catalog id>:<name>, the catalog id being the resource's catalog_id or else the provider's account id, and refuses to read back an id whose catalog is empty. Both rigs skip requesting the account id in a sidecar provider block, as an emulator needs, so the id was ":<name>". #700 handled this in the plugin: when AWS_ENDPOINT_URL names an emulator, each Glue database and table names its catalog from data.aws_caller_identity (aws._emulator_catalog_id). It moved the apply-engine moto rig onto that path. The LocalStack rig and the moto e2e rig aim tofu at the emulator only through the sidecar's endpoints map and never set AWS_ENDPOINT_URL, so the plugin compiled their module for real AWS. Both rigs now compile the module with AWS_ENDPOINT_URL naming the emulator, as an emulator user sets it. The module then carries its own provider "aws" block, so each sidecar is written as provider_override.tf.json, which merges into it. A second plain block would be a duplicate provider configuration. LocalStackProject scopes the variable to the emit, so it reaches neither the tofu child nor the rest of the test. The moto e2e fixture also resets moto between tests, as the apply-engine rig does. moto's backends are process-wide, so the database a failed test leaves behind made the next test's CreateDatabase fail with AlreadyExistsException, a second failure with the same cause. No product code changes. The plugin already emits the right module for an emulator user, and nothing changes on real AWS. * chore: the secrets baseline follows the two rig files' moved lines detect-secrets 1.5.0 (the CI pin) moves the line numbers of the two dummy emulator credentials already in the baseline, in tests/iac/conftest.py and tests/iac/test_iac_moto_e2e.py. No secret is added.

github-actions Bot added cli Changes to the CLI surface or implementation docs Documentation changes needs-docs Pull request needs a linked docs update or justification tests Test coverage or test infrastructure changes labels Oct 5, 2026

fas89 added 6 commits October 5, 2026 20:13

github-actions Bot added the security Security-related changes or reports label Oct 5, 2026

fas89 merged commit ff26cb1 into main Oct 5, 2026
39 checks passed

fas89 mentioned this pull request Oct 6, 2026

chore(release): prepare 0.19.0 #701

Merged

Navigation Menu

Repository navigation

feat(governance): column masking, row filters and governance labels on AWS and GCP - #700

feat(governance): column masking, row filters and governance labels on AWS and GCP#700
fas89 merged 8 commits into
mainfrom
feat/mask-rowfilters-labels

fas89 commented Oct 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fas89 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

A bug this would have shipped, fixed here

Three more found before merging, each fixed in its own commit

Security review, and what it changed (f82f90f)

The moto lane on provider 6.23+ (c52a4fe)

Not in this change

terraform-provider-aws ~> 6.0

Evidence

Not checked by fluid verify yet (stated in the parity doc)

Uh oh!

github-actions Bot commented Oct 5, 2026

📄 Documentation Reminder

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fas89 commented Oct 5, 2026 •

edited

Loading

Security review, and what it changed (`f82f90f`)

The moto lane on provider 6.23+ (`c52a4fe`)

terraform-provider-aws `~> 6.0`

Not checked by `fluid verify` yet (stated in the parity doc)