fix(aws): a Lake Formation tag the contract does not define is associated, and the module validates

[fas89]

fix(aws): a Lake Formation tag the contract does not define is associated, and the module validates

Summary

A binding's governance.lakeFormation.tags can name a tag that the contract's
governance.lakeFormation.tagDefinitions does not define. That is the normal case
when a platform owns the account's LF-tag ontology, which is the model AWS recommends.
Before this change the emitted association named such a key in its depends_on, as an
aws_lakeformation_lf_tag the module never declares. tofu validate, fluid plan and
fluid apply then rejected the whole module with "Reference to undeclared resource".

Now a key the contract does not define is a tag the platform owns. It is associated
by its literal key, with no reference, no warning and no refusal. A key the contract
defines names its tag the way the provider documents it,
key = aws_lakeformation_lf_tag.<name>.key, and the association carries no
depends_on at all. OpenTofu orders the association after the tag from that
reference. The reference evaluates to the same key, so an association applied from
0.19.0 plans no change. This was applied and planned with real OpenTofu against moto.
Four more ways to get a module that fails at apply, or that drops tags without a word,
are refused at emit and at fluid validate.

Why

• Reproduced on main (63f3b40) with OpenTofu 1.12.0 and hashicorp/aws 6.67.0:
  fluid generate iac wrote
  aws_lakeformation_resource_lf_tags.gold_sales_orders_lf_tags_orders.depends_on = [aws_lakeformation_lf_tag.gold_sales_orders_lf_tag_classification] with no such
  resource. tofu validate failed: "There is no managed resource
  aws_lakeformation_lf_tag gold_sales_orders_lf_tag_classification definition in
  the root module."
• The same unguarded edge also came from three other inputs: a definition with an empty
  values list (skipped by the emitter, but still a dependency); an association key with
  an empty value (dropped from lf_tag, but still a dependency); and keys that collide
  under safe_ident (a-b and a_b), where the edge could point at the wrong tag.
• The only test that covered LF-tag associations was a real-AWS e2e, and it always
  defines its tags. No offline test caught this.
• The depends_on edges for the tags the contract defines were themselves the
  pattern OpenTofu calls a last resort. The provider's own example orders the
  association with a reference to the tag's key.

What changes

fluid_build/iac/providers/aws.py

• lf_tag_definitions(contract): the aws_lakeformation_lf_tag bodies, by resource
  name. _emit_lf_account_settings now emits from it. It refuses a key with no values,
  which was skipped before, and two keys that are one resource name, where the later
  used to replace the earlier. Both are lakeformation-tag-definition.
• lf_tag_associations(contract, exposure, binding, index): the lf_tag blocks, one
  per key that carries a value. It refuses a value outside a contract-defined key's
  values (lakeformation-tag-value). The comparison is case-folded, because Lake
  Formation stores keys and values lower-cased. It also refuses tags on a Glue-catalog
  binding that names no Glue table (lakeformation-tag-association); those were
  dropped before.
• _emit_lf_tag_association: section 3 of _emit_lakeformation, moved into a helper
  the way _emit_lf_filtered_grant and _emit_lf_masked_views are. A key this module
  owns (resources["aws_lakeformation_lf_tag"] holds the matching safe_ident name,
  and that resource's key equals the association key) becomes
  "key": "${aws_lakeformation_lf_tag.<name>.key}", built with tofu_ref, as every
  other cross-reference in the emitter is. Any other key stays the literal the contract
  wrote, which the renderer escapes. No association carries depends_on.
• AwsIacPlugin.emit derives lf_tags beside lf_column_exclusions and
  lf_row_filters and passes it down, the same pattern those use.

fluid_build/iac/governance_validation.py

• validate_governance calls the same two functions. lf_tag_definitions runs once
  per contract, and only when some expose is bound to AWS. lf_tag_associations runs
  per AWS binding. So fluid validate reports each refusal at stage 2 with the emit's
  own message. It adds no warning, because the generated pipeline's stage 2 runs
  fluid validate --strict.

fluid_build/schemas/fluid-schema-0.7.3.json to 0.7.6.json

• The tags and tagDefinitions descriptions are edited in place, as plain text. A
  key the contract does not define must already exist in the account, and the principal
  that runs the apply needs the Lake Formation ASSOCIATE permission on it. A key the
  contract defines must not also be created elsewhere. The descriptions also state the
  value rule and the refusals. They say the association "waits for" a defined tag, which
  the reference still does, so the second commit leaves them alone.
• A structural diff of each schema against HEAD shows that only those two
  description strings changed.

tests/iac/test_iac_lakeformation_tags.py (new, 32 tests)

• Platform-owned keys: no dangling address anywhere in the module (a depends_on or a
  tag-key reference), a literal key, no depends_on, and
  validate_governance == ([], []).
• A key the contract defines alongside one it does not: only the defined key is a
  reference. No association carries depends_on, for a defined, a mixed, a
  platform-owned and a colliding contract.
• A contract that defines every tag it associates: the exact association body is
  pinned, with both keys as references.
• An empty-valued key adds no reference. A key that shares a defined tag's resource
  name (a-b beside a_b) stays a literal. A platform-owned key is still escaped as
  contract text (${ becomes $${).
• Each refusal is checked at emit (its kind and message) and at validate_governance
  (the same message and remediation). Case-folded values are accepted. Tags with no
  value on a binding with no table are not refused. Tags on a non-Glue format are
  ignored as before, and a contract bound only to GCP is not checked.
• The schema wording is pinned in every schema that carries the fields, the way
  test_iac_lakeformation_admins.py pinned the admins correction.
• tofu init -backend=false plus tofu validate on a platform-owned module, alone and
  beside a defined tag. The test skips without tofu, as test_iac_aws_validate.py does.
  The file is marked unit, so iac-tests Stage 1, which has tofu on PATH, runs it.

tests/iac/test_iac_lakeformation_tags_moto.py (new, 3 tests; real tofu apply,
plan and destroy against an in-process moto server)

• A contract that defines its tags: the module as 0.19.0 emitted it (literal keys and
  depends_on; the test pins that body) is applied, and the module this branch emits
  then plans No changes on the same state (tofu plan -detailed-exitcode exits 0).
  After an apply, the state lists both tags among the association's dependencies, and
  the destroy leaves no tag behind.
• A platform-owned tag: created in moto with boto3 first, then associated beside a
  tag the contract creates. GetResourceLFTags reads both back, the module plans
  clean, and after the destroy the platform's tag is still in the account while the
  contract's tag is gone.
• A platform-owned key that does not exist: tofu validate passes, and the apply fails
  with EntityNotFoundException, with no association in the state.
• Same harness as test_iac_aws_retention_encryption_moto.py and
  test_iac_aws_apply_engine_moto.py: emitted with AWS_ENDPOINT_URL set (emulator
  mode, so Glue resources name their catalog for provider 6), a
  provider_override.tf.json sidecar routing each service to moto, every AWS_*
  variable dropped from tofu's environment, moto reset per test, and the conftest's
  CreateBucket-tags shim. Skipped without tofu or moto's server extra.

.github/workflows/iac-tests.yml

• Stage 1 runs the moto file in its own step and fails if it only skipped
  (scripts/ci/assert_lane_coverage.py), as it does for every other moto file. The
  step adds 11 lines, which moves an existing .secrets.baseline entry for that
  workflow from line 247 to 258; the baseline carries the new line number.

Docs and changelog

• docs/governance-parity.md: a Lake Formation tags row that names the reference, the
  three new refusal kinds, a short "Lake Formation tags the platform owns" section, a
  prerequisite line, and entries in "What is proven, and what is not", including what
  moto does not check.
• HONESTLY_TESTED.md: a "Stage 1 LF tags" row, with what is not proven.
• CHANGELOG.md: a ### Fixed entry under [Unreleased].

Prior art (borrow-before-build)

• terraform-provider-aws, aws_lakeformation_resource_lf_tags docs
  (website/docs/r/lakeformation_resource_lf_tags.html.markdown): the resource
  "Manages an attachment between one or more existing LF-tags", the lf_tag.key
  argument is "Key name for an existing LF-tag", and the Database example writes
  key = aws_lakeformation_lf_tag.example.key. That reference is what this change
  emits for a defined tag. The provider has no LF-tag data source (checked with
  tofu providers schema -json on 6.67.0), so an external tag cannot be checked at
  plan time.
• terraform-provider-aws source, internal/service/lakeformation/resource_lf_tags.go
  at v6.67.0: create calls AddLFTagsToResource and turns each returned Failures
  entry into an error, and every argument is ForceNew. So a missing platform tag
  fails the apply, and any change to a key's value would replace the association,
  which is why the no-change plan matters.
• OpenTofu depends_on docs: "we recommend using expression references to imply
  dependencies when possible", and depends_on should be "a last resort because it
  can cause OpenTofu to create more conservative plans". OpenTofu JSON syntax docs: a
  JSON string that is a single interpolation evaluates to the referenced value directly.
• cloudposse/terraform-aws-lakeformation (main.tf): its association writes
  key = aws_lakeformation_lf_tag.default[lf_tag.key].key, with no depends_on on
  the tags. It supports only tags the module defines, so the platform-owned literal is
  where this change goes further.
• AWS Lake Formation API reference, AddLFTagsToResource: it returns Failures, and
  EntityNotFoundException means "A specified entity does not exist." AWS CloudFormation
  AWS::LakeFormation::TagAssociation names tag keys literally. AWS's TBAC guidance and
  the Big Data Blog post "Decentralize LF-tag management with AWS Lake Formation": a
  central steward creates the tags, and domains hold ASSOCIATE on them.
• moto's Lake Formation docs and source (5.2.3): create_lf_tag,
  add_lf_tags_to_resource, get_resource_lf_tags, delete_lf_tag and
  list_lf_tags are implemented. add_lf_tags_to_resource reports a key that does not
  exist as a failure ("Tag or tag value does not exist.") but does not check the value.
• This repo: tofu_ref/TofuExpr is how the emitter writes every cross-reference;
  _wire_aws_deps and the Snowflake _container_deps add an edge only to a resource
  in the same module; _glue_db_ref inlines the literal name of a container the module
  does not own; lf_column_exclusions is the shared emit and validate refusal shape;
  the moto harness is copied from test_iac_aws_retention_encryption_moto.py and
  test_iac_aws_apply_engine_moto.py.

Decisions taken while implementing (beyond the founder's option B)

• Kinds: lakeformation-tag-value and lakeformation-tag-definition come from the
  brief. The safe_ident collision also uses lakeformation-tag-definition. Tags on a
  binding with no table use the new lakeformation-tag-association.
• "No table" means no Glue table: location.database or location.table is missing.
  This is the same condition lf_column_exclusions uses. The refusal applies only to
  Glue-catalog formats. A non-Glue binding keeps ignoring its whole
  governance.lakeFormation block, as documented, so its tags are still ignored.
• Key matching is exact, which is the founder's ownership rule. Only the value
  comparison is case-folded. Keys that differ only in case are listed under follow-ups.
• A defined tag is named by reference, not by depends_on (second commit, on the
  founder's OpenTofu-first rule). The first commit had kept depends_on so that the
  module stayed byte for byte the same. The emitted module now changes for every
  contract that associates a tag it defines, but only in the association's lf_tag keys
  and the removed depends_on. The moto test shows such an association, applied from
  0.19.0, plans no change.
• The .key attribute is referenced, not .id. The id is <catalog id>:<key>;
  pointing at it plans a replacement of every association (the seeded defect below).
• The moto file gets its own iac-tests Stage 1 step with the lane-coverage assert.
  Without one it would run in no CI lane: test_iac_moto_e2e.py is in that state today
  and is red on main (reported separately, not changed here).
• The moto file keeps its own tofu_env, _tofu, _init and _provider_override, as
  each moto file in tests/iac does. The conftest's tofu_env keeps the inherited
  AWS_* variables, and these tests must not.
• An association key with an empty value is still skipped, not refused, even on a
  binding with no table.
• fluid validate checks tagDefinitions only when some expose is bound to AWS,
  because the AWS emitter is the only one that creates the tags.
• Section 3 was moved into _emit_lf_tag_association. Ripwire quality-delta had
  flagged _emit_lakeformation growing from 66 to 72 in complexity and from 218 to 229
  lines.
• The CHANGELOG entry has no PR number yet.

Validation

All commands were run in the clone on this branch. PY is the shared test venv's
python, run with PYTHONPATH=<clone> and -p no:randomly. OpenTofu 1.12.0,
hashicorp/aws 6.67.0, moto 5.2.3.

The second commit (reference wiring)

1. Real OpenTofu against moto, by hand. A localhost moto server (moto 5.2.3 in a
   scratch venv, with the repo's CreateBucket-tags shim), AWS_ENDPOINT_URL pointing at
   it, dummy keys, AWS_PROFILE unset, shared config files set to /dev/null.
   • (i) Defined tags. A git worktree of origin/main (63f3b40, the same fluid_build
     as v0.19.0) emitted the module. Its only difference from this branch's module is
     the association (depends_on and two literal keys against two references). The
     main module was applied (6 added), and planned clean on its own. Then this
     branch's module was swapped in, and tofu plan -detailed-exitcode exited 0 with
     "No changes. Your infrastructure matches the configuration." After a no-op apply,
     the state lists both aws_lakeformation_lf_tag resources among the association's
     dependencies. The destroy removed the association before the tags.
   • (ii) Platform-owned. create_lf_tag(classification, [confidential, public]) with
     boto3, then this branch's module (classification as a literal key, domain defined
     and referenced) was applied (5 added). GetResourceLFTags on sales.orders
     returned domain=[sales] and classification=[confidential], and the plan after
     was clean. After the destroy, list_lf_tags still held classification.
   • (iii) A platform-owned key that does not exist. tofu validate passed. The apply
     failed: "creating AWS Lake Formation Resource LF Tags (catalog id:123456789012, tag
     key:nonexistent, values:[x]): EntityNotFoundException: Tag or tag value does not
     exist." The bucket, database and table were created and are in the state; the
     association is not. When the key was then created with a different value only
     (y), moto accepted the association of x, so moto does not check values.
2. The tests. test_iac_lakeformation_tags.py and
   test_iac_lakeformation_tags_moto.py, with tofu on PATH: 35 passed.
3. Seeded defects, each on the final code, then restored:
   • The previous commit's depends_on edges restored: 9 failed. They are the new
     reference tests and the moto no-change test.
   • The reference pointed at .id instead of .key (and the moto test's pattern
     loosened to let it through): the moto no-change test fails at the plan,
     "aws_lakeformation_resource_lf_tags.gold_sales_orders_lf_tags_orders must be
     replaced ... Plan: 1 to add, 0 to change, 1 to destroy."
   • The ownership guard replaced by if True, so platform keys become references:
     12 failed, including both tofu validate tests and both platform-owned moto
     tests.
   • Restored (code byte-identical to the fix, cmp): 35 passed.
4. The CI step, locally. The moto file with --junitxml, then
   scripts/ci/assert_lane_coverage.py ... --require "LF tags moto=...": "OK LF tags
   moto: 3 passing", exit 0. actionlint on iac-tests.yml: clean.
   tests/test_ci_dbt_ceiling.py and tests/test_ci_lane_coverage_guard.py: 57 passed.
5. Targeted suites.
   • tests/iac, tofu hidden: 1581 passed, 248 skipped.
   • With tofu on PATH: the two tag files, test_iac_aws_validate.py,
     test_iac_tofu_validate.py, the Lake Formation bucket-policy and column-grant
     tofu plan vs moto files, test_iac_aws_column_restrictions.py,
     test_iac_aws_retention_encryption.py, test_iac_aws_retention_encryption_moto.py,
     test_iac_aws.py, test_iac_state_drift_moto.py and
     test_iac_aws_apply_engine_moto.py: 239 passed (20 min 45 s).
6. CI lint, format, guardrail and secret scan, as CI runs them.
   • ruff check fluid_build/ tests/: passed. The dead-code ruff gate: passed.
   • black --check fluid_build/ tests/ (24.10.0): 1811 files unchanged.
   • python3 tools/product_guardrail/check.py --self-test and the check: 0 failing
     findings. Its 13 warnings are all in files this change does not touch.
   • detect-secrets-hook --baseline .secrets.baseline on every file changed against
     origin/main: it first moved the iac-tests.yml entry's line number (247 to 258),
     as above; with that committed, it exits 0.
7. ripwire --quality-delta against the first commit (exit 2, 24 gating rows), weighed
   and left alone:
   • 18 rows are clone groups: the new moto file's tofu_env, _tofu, _init,
     _write, _plans_no_change and _lakeformation match the helpers every other
     moto file in tests/iac keeps for itself (retention, column grants, column
     restrictions, apply engine, cross-account LocalStack). That is the harness this
     file was asked to copy; a shared module is a follow-up.
   • 4 rows are short-horizon churn on _emit_lf_tag_association, _emit_lakeformation
     and AwsIacPlugin.emit: this branch revises what its own first commit wrote, and
     only comments changed in the last two.
   • 2 rows are the iac-tests.yml Stage 1 job growing by the 11-line step, the same
     shape as the five moto steps beside it.
   • The 10 new-symbol rows are the new tests (dead-code is how ripwire counts a test
     function) and two similar one-key tests.

The first commit

1. The failing tests now passed. 28 passed; on main the research agent's four
   tests gave 3 failed, 1 passed, and tofu validate reported "Reference to undeclared
   resource".
2. Seeded defects. The code half reverted: 13 failed. Only the ownership guard
   removed: 6 failed. The fluid validate mirror removed: 6 failed. No case folding:
   2 failed. The schema descriptions reverted: 4 failed. Restored: 28 passed.
3. Full suite, tofu hidden, without -x: 6 failed, 20068 passed, 656 skipped. All
   six are dbt tests that need the dbt-duckdb adapter or a dbt executable, which the
   shared venv lacks, and they fail the same way on an untouched origin/main copy.
4. End to end, with no AWS credentials and no cloud calls: fluid validate --strict
   passes the platform-owned contract and refuses the bad ones; fluid generate iac --validate passes; fluid generate iac on a bad value exits 1 with
   kind: lakeformation-tag-value.

Risks

• A typo in a platform-owned key or value is caught only at apply, by Lake Formation.
  The provider has no data source to check it earlier, and option B adds no warning.
  The apply fails loudly (proven against moto for a key), after the bucket, database
  and table were created; re-running once the tag exists completes it.
• Every contract that associates a tag it defines emits a changed module: the keys
  become references and depends_on goes. Proven to plan no change against moto for
  an association applied from 0.19.0, not yet against real Lake Formation. A defined
  key with upper-case letters is the open case: the reference carries the tag
  resource's key as the provider reads it back (see follow-ups).
• Four contract shapes that used to emit are now refused: a value outside a defined
  key's values, an empty definition, two definitions that collide, and tags with no
  Glue table. The first three already failed at apply or produced a wrong module. The
  last one silently dropped the tags.
• The schema descriptions of four released versions changed in place, as the founder
  accepted. Only description text changed.
• Security: LF-TBAC grants access by tag, so a contract can now associate any tag that
  exists in the account. The control is on the AWS side. The applying principal must
  hold ASSOCIATE on the tag, the same authority as before. The literal key is not a
  TofuExpr, so the renderer still escapes it, and a test pins that. The reference is
  built only from safe_ident output.

Follow-ups

• The association resource key safe_ident(f"{cid}_lf_tags_{table}") ignores the
  database. Two exposures with the same table name in different databases share one
  association, and the second silently replaces the first. The grant keys have the same
  shape. Fixing it changes resource addresses, so it needs a migration design (moved
  blocks). Not changed here.
• Keys that differ only in case. A definition Domain with an association domain is
  treated as platform-owned (no reference, no value check). Two definitions Domain and
  domain are both emitted. AWS stores keys lower-cased, so each pair is one tag, and a
  defined Domain may read back as domain. Decide whether to refuse case variants
  after checking live behaviour. The same applies to values that collapse under
  lower-casing.
• fluid validate and fluid generate iac print refusal messages through Rich markup,
  which strips [orders] from exposes[orders]. This is pre-existing and affects
  every refusal that names an expose in brackets.
• A gated real-AWS e2e for the platform-owned path, and for the no-change plan of an
  association applied from 0.19.0: this needs the sandbox account and approval.
• Each moto file in tests/iac carries its own tofu_env, _tofu, _init and
  provider override; ripwire counts them as clone groups. A shared moto harness module,
  like _moto_s3.py, would remove them across all of those files.
• Optional typo protection through an explicit opt-in list of external keys (option D
  in the research), if it is wanted.

Merged with main after #703

#703 combined the AWS emitter's column, row-filter and masked-view checks into one lf_governance call, which fluid validate shares through _aws_refusal. This branch's tag checks were rebuilt on that:

• Emitter: lf_governance, then lf_tag_associations.
• Validator: _aws_refusal takes the contract and runs the same two checks in the same order, so an expose gets one refusal, never a refusal plus an "unenforced grants" warning.
• Changelog: corrected the claim that the old bug broke fluid plan, which never runs the emitter. It broke tofu validate and fluid apply.

Checked on the merged code:

• tests/iac with tofu hidden: 1,636 passed. The validate and examples-unchanged tests plus both branches' new files: 332 passed.
• The tofu-against-moto tag tests with tofu on PATH: 90 passed, none skipped.
• A seeded defect: removing the tag check from _aws_refusal turns 3 tests red.
• ruff, black 24.10.0, the product guardrail and detect-secrets are clean.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.