fix(aws): refuse a row filter on a binding that names no Glue-catalog table

[fas89]

fix(aws): refuse a row filter on a binding that names no Glue-catalog table

Summary

A policy.authz.rowFilters entry on an AWS binding that is not a Glue-catalog table
(redshift_serverless, redshift_table, redshift_external_schema, kinesis_stream,
glue_table, ..., or a Glue format with no location.database) was silently dropped:
fluid validate passed, and fluid generate iac and fluid apply wrote the Redshift
namespace and workgroup with no filter and exited 0. It is now refused with the existing
kind row-filter-unenforceable.

fluid validate now runs the same Lake Formation derivations as the emitter
(lf_governance), so it reports at stage 2 every refusal they raise, with the emitter's
own message.

Why

On AWS a row filter is a Lake Formation data cells filter on a Glue table.
_emit_lakeformation returns early for any format outside _GLUE_CATALOG_FORMATS and
for a binding with no location.database. The refusal that should guard that early
return, lf_row_filters, checked only for grants and a location.table, never the
format or the database.

Measured on origin/main (63f3b40), on a redshift_serverless contract with Lake
Formation grants and a row filter:

• fluid validate: Valid FLUID contract (schema v0.7.6), exit 0.
• fluid generate iac: exit 0, module holds only aws_redshiftserverless_namespace and
  aws_redshiftserverless_workgroup.
• A parquet binding with table but no database: the module held only the S3 bucket,
  with no grant and no filter.

There was a second gap with the same symptom. validate_governance ran only
lf_column_exclusions for an aws binding. Its docstring says it runs the emitter's
derivations so each refusal is reported at stage 2, but lf_row_filters and
lf_masked_views never ran. So a row filter with no read grant or with an unmapped
principal, a mask with no maskedViews, and a text mask on a non-string column all
passed fluid validate and failed first at generate iac or apply.

Masks and denies on a non-Glue binding were already refused. Every mask is a column
restriction, and lf_column_exclusions runs first and refuses non-Glue formats
(column-restriction-unenforceable). That kind is kept; the mask case had no test and
now has one.

What changes

• fluid_build/iac/providers/aws.py

  • lf_row_filters: before the grants check, it refuses when the format (default
    parquet, compared case-blind) is not in _GLUE_CATALOG_FORMATS, or when
    location.database or location.table is missing. This is the same predicate
    lf_column_exclusions uses. The message names the format and lists the Glue-catalog
    formats, and the remediation is one sentence, in the style of
    gcp_governance.refuse_unsupported_target:

    exposes[profile].policy.authz.rowFilters filters rows, but its redshift_serverless
    binding names no Glue-catalog table; the AWS emitter enforces row filters only as
    Lake Formation data cells filters on one, a binding of format avro, csv, delta,
    iceberg, json, orc or parquet with a location.database and location.table. It
    would not be enforced. Move the filter to such an expose, or remove it from this one.

    The missing-table test moved from the grants check into this guard. The grants
    check keeps its message.

  • New lf_governance(exposure, binding, index): runs lf_column_exclusions, then
    lf_row_filters, then lf_masked_views, and returns
    (exclusions, row_filters, masked_views). AwsIacPlugin.emit calls it in place of
    the three calls.

• fluid_build/iac/governance_validation.py
  • The aws branch goes through a new _aws_refusal(exposure, binding, index). It runs
    lf_governance and returns the refusal text, or "" when there is none. Before,
    the branch called only lf_column_exclusions.
  • lf_masked_views quotes names through validate_ident, which raises a plain
    ValueError. Uncaught, that would escape validate_governance, and
    cli/validate.py treats any exception there as "Governance binding check skipped"
    (shown only with --verbose). Every governance finding of the contract would then
    be lost. I measured this with the first version of this change: a contract with a
    Redshift row filter plus a masked expose on a silver-zone Glue database printed
    "Valid FLUID contract", exit 0. _aws_refusal reports the ValueError as that
    expose's error (exposes[masked]: Invalid SQL identifier: 'silver-zone'), and the
    check goes on. The helper also keeps validate_governance's complexity at 27 (it
    was 26); the inline version measured 33.
  • The module docstring now names the real refusal surfaces.
• docs/governance-parity.md: refusals happen at fluid validate, fluid generate iac,
  fluid apply and the state check of fluid diff, not at fluid plan (cli/plan.py
  never runs the IaC emitter). The list now includes the non-Glue row-filter refusal,
  and the Row filters row of the table says where AWS can enforce one.
• CHANGELOG.md: [Unreleased] gets Fixed and Documentation entries.
• tests/iac/test_iac_aws_nonglue_governance.py (new, 55 tests): the research's 22
  tests, kept, with the mask and combined-policy assertions tightened to the exact kind,
  plus these:
  • the message lists every Glue-catalog format;
  • a missing database or table is refused;
  • every Glue-catalog format, upper-case PARQUET and an omitted format still emit the
    filter;
  • lf_governance equals the three derivations in order, and the filter reaches the
    masked view's WHERE;
  • emit_module (the fluid apply and fluid diff path) raises unsupported_binding;
  • a 15-case parity matrix: for each refusal lf_governance can raise,
    validate_governance reports exactly the emitter's message and remediation;
  • a filter, a mask and a deny on Glue emit and validate clean;
  • the ValueError case above;
  • the real fluid validate CLI exits 1 on the Redshift repro and 0 on the Glue
    control.

Prior art (borrow before build)

• In repo: lf_column_exclusions (same guard, same exception) and
  gcp_governance.refuse_unsupported_target (message shape: where, what was asked,
  "It would not be enforced.", one-sentence remediation). The existing stable kind
  row-filter-unenforceable is reused; no kind is minted.
• Terraform plugin framework ValidateConfig
  (https://developer.hashicorp.com/terraform/plugin/framework/providers/validate-configuration):
  configuration errors are reported at validate, plan and apply, before any change.
  That is why validate now runs the emitter's own derivation.
• Kubernetes KEP-2885 (https://www.kubernetes.dev/resources/keps/2885/): silently
  dropped fields were treated as a bug class and moved to strict validation.
• AWS: Lake Formation row and cell filters reach Redshift only through Spectrum over
  Glue-catalog tables
  (https://aws.amazon.com/blogs/big-data/use-amazon-redshift-spectrum-with-row-level-and-cell-level-security-policies-defined-in-aws-lake-formation/).
  That confirms the refusal is right for native Redshift bindings.
• Considered and rejected: dbt's warn-and-drop for unsupported constraints
  (https://docs.getdbt.com/reference/resource-properties/constraints). It is the
  behaviour being fixed, and wrong for an access-control policy.

Decisions

• Missing location.database is refused too, not only a non-Glue format.
  _emit_lakeformation drops a filter in that case through the same early return. I
  measured it on main (parquet, table, no database: bucket only). The guard is exactly
  lf_column_exclusions's predicate, so the two refusals agree on what "a Glue table"
  means.
• The message lists the formats from _GLUE_CATALOG_FORMATS, sorted, so it follows
  the set if the set changes. It includes avro, orc and delta, which are in the set
  but not in the 0.7.6 binding-format enum. I listed the emitter's truth rather than add
  a second list.
• The remediation is one sentence: "Move the filter to such an expose, or remove it
  from this one." It does not say "use a Glue table", which would mislead a glue_table
  author.
• Kinds are unchanged. Masks and denies on non-Glue keep
  column-restriction-unenforceable. column-mask-unenforceable is not minted for
  non-Glue, because it cannot fire there.
• New stage-2 errors are errors, not warnings, matching the emitter. A row filter
  with no Lake Formation grants used to get, at most, the generic "accessPolicy.grants
  are not enforced" warning at validate. It is now the emitter's error.
• The ValueError catch sits only in the aws branch (_aws_refusal). The GCP
  branch is untouched, to keep this self-contained. A GCP-side ValueError would still
  skip the check; that is listed as a follow-up.
• The guard is inline, mirroring lf_column_exclusions, rather than a shared
  predicate helper. That keeps the diff small for the sibling Lake Formation tags change
  to rebase over. lf_row_filters complexity goes from 16 to 18 (ripwire minor).
• Not changed, per the brief: labels are not refused, _GLUE_CATALOG_FORMATS is
  not widened, and a governance.lakeFormation block on non-Glue is not refused.

Validation

All commands ran in the clone at fix/aws-row-filter-nonglue-refused (commit
b6575e1, on origin/main 63f3b40), with PYTHONPATH set to the clone (import fluid_build resolves into it) and -p no:randomly. Unless a row says otherwise, the
results are for the final code (the md5 of the three code and test files was
recorded before the final runs and checked again before the commit).

Step	Command	Result
Red on main	pytest tests/iac/test_iac_aws_nonglue_governance.py (research version, unfixed source)	11 failed, 11 passed
Final tests on main source	the same file, both source files stashed	collection error: lf_governance does not exist on main
Green	pytest tests/iac/test_iac_aws_nonglue_governance.py	55 passed
Seed 1: guard disabled (if False:)	same	20 failed, 35 passed (non-Glue filter, missing database/table, generate iac, emit_module, validate, parity, identifier, CLI)
Seed 2: _aws_refusal runs only lf_column_exclusions	same	18 failed, 37 passed (every validate and parity row-filter/mask case, identifier, CLI)
Seed 3: except ValueError clause removed	same	1 failed (the identifier test), 54 passed
Restored	md5 of both source files equal to the saved fix; same tests	55 passed
Targeted, tofu hidden	pytest tests/iac tests/perf/test_startup_budget.py tests/test_validate_cmd.py tests/test_validate_ext.py	1661 passed, 243 skipped
Full suite, tofu hidden, -x	env PATH=/usr/bin:/bin:/usr/sbin:/sbin:<venv>/bin pytest tests -q -x -p no:randomly	Stopped at tests/engines/test_dbt_model_contracts.py::TestLiveDbtBuildEnforcement ("Could not find adapter type duckdb!"). The shared venv has dbt-core but not dbt-duckdb, which CI installs.
Full suite, no -x (code before the _aws_refusal refactor)	same, without -x	20097 passed, 654 skipped, 6 failed. All 6 are dbt tests needing dbt-duckdb (or a dbt executable): test_dbt_model_contracts x2, test_dbt_semantic_models::TestLiveDbtParse, test_build_runners::...test_uses_configured_dbt_command_prefix_verbatim, test_golden_path_e2e::test_deterministic_lane_golden_path, test_speed_transformation_dbt_e2e::test_forge_generate_dbt_parse_dimensional
Those 6 on unfixed main	the 6 node ids, both source files stashed (fluid_build == origin/main)	6 failed, the same way
Full suite, final code, -x, the 6 deselected	pytest tests -q -x -p no:randomly --deselect <the 6>	20097 passed, 654 skipped, 15 deselected, 18 xfailed, 15 xpassed, exit 0
Tofu-gated, tofu on PATH (code before the ValueError handling)	the 12 tofu-gated tests/iac files plus the new file, with PLUGIN_UNIX_SOCKET_DIR=/tmp/wfi3, TF_PLUGIN_CACHE_DIR set and cloud credential sources blanked; against moto and the in-process BigQuery fake only	255 passed, 5 skipped
Tofu-gated, final code	test_iac_gcp_governance.py, test_iac_aws_column_restrictions.py, the new file, tofu on PATH	175 passed, 0 skipped
Real CLI	fluid validate / fluid generate iac on the Redshift repro	exit 1, row-filter-unenforceable, no main.tf.json written
Real CLI control	same on the parquet Glue control	exit 0; module has aws_lakeformation_data_cells_filter with filter_expression: consent = true
Real CLI, bad identifier	fluid validate on Redshift filter + masked silver-zone expose	before the handling: "Valid FLUID contract", exit 0; final: 2 errors, exit 1
Example corpus	real fluid validate on all 53 example/template contracts, main vs final	identical: 45 exit 0, 8 exit 1 (pre-existing, unrelated), same governance lines
Repo-wide YAML	validate_governance over 76 contract documents, origin/main source vs final	0 changed, 0 errors either side
Lint	ruff check fluid_build/ tests/ (ruff 0.16.10)	All checks passed
Format	black --check fluid_build/ tests/ (24.10.0)	1810 files unchanged
Guardrail	python3 tools/product_guardrail/check.py --self-test && python3 tools/product_guardrail/check.py	self-test passed; 0 failing, 13 pre-existing warnings, 1 graced
Secrets	detect-secrets-hook --baseline .secrets.baseline on the 5 changed files	exit 0; no changed file holds a baselined entry; baseline byte-identical, not refreshed
Quality delta	ripwire <clone> --quality-delta (vs HEAD = origin/main)	exit 2, gating=10, details below

Quality delta, weighed. The 10 gating rows:

• 3 short-horizon-churn rows (AwsIacPlugin, emit, lf_row_filters, churn=self).
  These symbols landed in feat(governance): column masking, row filters and governance labels on AWS and GCP #700 days ago, and the fix has to touch them.
• 3 duplication rows pair TestTheValidateCommand::test_the_same_filter_on_a_glue_table_passes_fluid_validate
  (31 to 33 tokens: run the CLI, assert exit 0) with the same idiom in
  test_iac_gcp_governance.py and test_iac_binding_target_resolution.py.
• 4 duplication rows pair the test helper _without (a dict-filter comprehension,
  21 to 26 tokens) with unrelated helpers: _kwargs, canonical_model_aliases,
  without_yaml, _only_sensitive_keys.

I judged these clone rows false positives. Sharing a helper across unrelated modules
would be a wrong abstraction. One real row I fixed: my first version nested a
try in validate_governance and took its complexity from 26 to 33 (gating). The
_aws_refusal helper brought it to 27 (minor). lf_row_filters went from 16 to 18
(minor). The remaining rows are new-symbol and do not gate: dead-code on pytest test
methods (collected by pytest), two parallel tests the research wrote, which I kept as
the brief asked, _contract's keyword parameters, and the length of two test classes.

No AWS or GCP call was made. The refusal fires during emit, before any module is
written or any tofu run, so no live test applies.

Risks

• A contract that today applies with a silently dropped row filter on a non-Glue
  binding, or on a Glue format with no database, now fails fluid validate,
  generate iac and apply. That is the intent. No contract in examples/, the
  templates or the test fixtures is affected.
• fluid validate now runs lf_row_filters and lf_masked_views for every aws
  binding. A contract with a row filter or a mask may see a stage-2 error it used to
  get first at apply, for example principal-unmapped or column-mask-type.
• The format value is echoed into the message, as lf_column_exclusions already does.
  It is the contract author's own input, printed to their own console through Rich,
  which reads bracketed text as markup (see follow-ups).
• In the full suite, one non-strict xfail test flipped from xpass to xfail between two
  runs (17/16 to 18/15). No xfail test touches iac governance; the only xfail markers
  under tests/iac are on emulator e2e tests, which are skipped here.

Follow-ups (not in this change)

• A governance.lakeFormation block on a non-Glue binding (grants, tags,
  registerLocation, maskedViews, a hand-written rowFilter) is still skipped
  silently by _emit_lakeformation, and so are _check_lf_grant_columns's checks.
  Refusing it breaks existing contracts, so it needs a decision.
• _GLUE_CATALOG_FORMATS omits the schema-valid glue_table, athena_table,
  delta_table and s3_file, and holds avro, orc and delta, which the 0.7.6
  binding enum does not accept. Widening it changes what apply creates and what
  fluid diff inspects.
• fluid validate prints errors through Rich markup, so exposes[profile] renders as
  exposes and the expose id is lost. This predates the change: the existing
  column-restriction message loses it the same way.
• validate_governance does not run the emitter's other refusals:
  _check_lf_grant_columns, lakeformation-grant-columns in _emit_lakeformation, and
  the pool-prefix check. In its GCP branch, a plain ValueError would still end the
  whole check silently. A broader fix belongs in cli/validate.py, which shows "check
  skipped" only under --verbose.
• The two tests of TestLiveDbtBuildEnforcement need dbt-duckdb, which CI installs
  and the local shared venv lacks.
• Native Redshift enforcement (RLS and dynamic data masking through redshift-data, or
  Lake Formation on a Redshift datashare) is future feature work.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.