[fas89]

What and why

Overlays deep-merge lists of maps by position (loader._deep_merge). In a contract with more than one build or expose, inserting or reordering an entry in the base silently retargets every overlay. Take the requester's case: the base has builds: [ingest, transform], and overlays/prod.yaml raises builds[1], meaning transform. Add a backfill build first, and prod now patches ingest. It also copies id: transform onto it, and fluid plan --env prod fails with planner_failed: 'schedule_transform', with no word of why (reproduced on main).

This PR does three things.

From fluidVersion 0.7.6 (the schema preview), overlays merge by identity.

• The keys are builds[].id, exposes[].exposeId, and consumes[] by productId + exposeId. They match an overlay entry to its base entry when the base's entries carry unique ones.
• An entry that matches nothing is appended and logged as overlay_entry_added. The warning names the base's ids, so a typo shows.
• An overlay entry the merge cannot match (no id, or an id named twice) fails the load with its index. Falling back to position would retarget it, which is the failure this fixes. fluid validate --env and load_with_overlay report it with the overlay's path, fluid bundle --env does too, and the contract API raises contract_load_failed. Id-less patches still apply to a base with a single entry.
• A base whose own ids are missing or repeated keeps the positional merge and logs overlay_merged_by_position.
• Only the three top-level lists are keyed. Lists inside an entry, such as an expose's schema, still merge by position.

Older contracts merge exactly as before, and now warn. overlay_patch_retargeted fires when a positional patch lands on an entry that names another id, or another part of a consumes key. It names both ids, the fix, and the opt-in.

One chokepoint. The rule sits in _deep_merge, so every overlay path uses it:

• the loader: validate, plan, apply, diff, verify, publish;
• fluid bundle --env and generate artifacts;
• the contract API.

Each warning names its contract and is logged once per process (apply loads its contract twice), as note_missing_overlay already does.

The requester's part 3, the safety net without keyed merge, is the 0.7.5 warning. It warns and does not fail under --strict; --strict does not see loader warnings today.

Borrowed, not built

• Kubernetes server-side apply list-map keys (x-kubernetes-list-map-keys, compound keys such as containerPort + protocol). An entry matches on every key field, and an unmatched entry is added. Like SSA, entries the merge cannot key are refused, not merged by position.
• jsonmerge 1.9.2 arrayMergeById (MIT): an id read from the entry, with compound ids.
• Strategy: adapt the pattern. The keyed merge is a few dozen lines, and jsonmerge would bring jsonschema onto the light CLI's cold path, so we borrow the semantics, not the dependency.

Validation

• Tests: tests/test_overlay_keyed_merge.py (28 tests, several parametrized). The 9 original behaviour tests fail on main.
• Mutation, per the test audit: all killed:
  • always or never keyed;
  • no append;
  • the len(base) > 1 guard;
  • first key field only;
  • no retarget warning;
  • no merged-by-position warning;
  • version read from the base only;
  • overlay ids not unique.
• Full suite: 26,600 passed; the 6 failures are the environmental baseline that reproduces on main. Perf startup budget: 38 passed. Lint and format: whole-tree ruff, black 24.10.0, the dead-code gate, license headers and lint-imports.
• Live through the fluid CLI, on the requester's scenario:
  • 0.7.6: plan --env prod and bundle --env prod patch only transform; backfill and ingest are untouched in plan.json and in contract.resolved.yaml.
  • 0.7.5: the merge is unchanged, and overlay_patch_retargeted is printed once per command.
  • An id-less overlay on 0.7.6: validate reports contract_load_failed, naming builds[0].
  • A typo'd id: appended with overlay_entry_added naming the base ids.
• ripwire --quality-delta: no gating rows.

Review verdicts

• security-review (three rounds): no findings. The keyed merge narrows where a patch lands (the ids must be equal). Merged ids stay unique. Every caller propagates the new error, so nothing fails open. The logs carry ids only.
• world-class-inspection: documentation, architecture, codebase and test suite reported CONCERNS in round 1. Every change-caused finding is fixed and re-verified:
  • the keyed fallback that retargeted;
  • the wrong remedy text;
  • the silent append;
  • the attribution;
  • the tests for later versions;
  • the API error type;
  • the per-contract warnings;
  • the bundle file name.
• defect-scan (hunter plus two re-verify rounds): all change-caused findings fixed. Pre-existing ones are follow-ups below.

Documentation

• Docs PR linked: docs: engine: flink, and overlays that merge by identity from 0.7.6 (forge-cli #731, #732) forge_docs#155 (Flink jobs section, identity-merge overlay rules; merge after this PR)
• No docs needed
• Docs TODO

Follow-ups (not in this PR)

• An overlay holding a $ref gives different results per command (plan drops it; bundle merges the raw $ref).
• datamesh-manager publish --overlay <file> passes the path as an env name.
• fluid plan and fluid apply report an overlay load failure as planner_failed / "Unexpected error", while validate says contract_load_failed.
• validate does not reject duplicate build ids.
• Declare merge keys per list in the schema (like x-kubernetes-list-map-keys) so a later version that adds a keyed list carries it.

[github-actions]

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

• Link a docs PR — check the "Docs PR linked" box and paste the URL
• Mark as no docs needed — check "No docs needed" with a justification
• Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.