docs: engine: flink, and overlays that merge by identity from 0.7.6 (forge-cli #731, #732)

docs/advanced/contract-loading-api.md

@@ -180,7 +180,10 @@ In the engine's order:
 3. **Overlay**: for `env`, the first of `overlays/<env>.yaml|yml|json`,
    `<env>.yaml|yml|json`, `<contract-stem>.<env>.yaml|yml|json` next to the
    contract, deep-merged over the base (objects key by key, lists of objects
-   by position, anything else replaced). A bundle is never re-overlaid.
+   by position, anything else replaced). *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* From
+   `fluidVersion` 0.7.6, `builds`, `exposes` and `consumes` merge by identity,
+   and an overlay entry the merge cannot match raises `ContractLoadError`
+   (`contract_load_failed`). A bundle is never re-overlaid.
 4. **Alias values**: human-friendly values rewritten to the schema's enum
    value, for example `source.kind: pg` → `postgres`,
    `source.mode: incremental` → `incremental_append`,

docs/concepts/builds-exposes-bindings.md

@@ -56,6 +56,7 @@ fluid apply contract.fluid.yaml --yes --mode amend-and-build
 
 Under `--mode amend-and-build`, each build goes to the runner that matches it:
 
+- *(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* a Flink build (`engine: flink`, or `properties.language: flink_sql`) is not run: it is reported as run outside forge-cli (see [Flink jobs](#flink-jobs-engine-flink));
 - an `acquisition` build runs on its ingestion engine;
 - a build with `engine: sql`, or `pattern: embedded-logic` with `properties.sql`, runs as embedded SQL;
 - a dbt build runs the dbt project at `repository`;
@@ -106,6 +107,38 @@ only the first expose (by_region); by_customer is not written by this build
 
 After that run `out/by_region.parquet` held the second build's rows (`customer_id`, `revenue`). Until this changes, give each embedded-SQL build its own contract. An extra expose bound to a cloud store or a warehouse is refused instead of skipped (from the v0.18.1 source).
 
+### Flink jobs: `engine: flink`
+
+*(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* A long-running Flink job, typically Flink SQL from Kafka to Kafka on the [Flink Kubernetes Operator](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/) or on AWS Managed Service for Apache Flink, declares `engine: flink` (schema 0.7.6 preview). forge-cli validates and plans the build but ships no runner for it: the platform that runs the job executes it.
+
+```yaml
+fluidVersion: "0.7.6"
+builds:
+  - id: reconcile
+    engine: flink
+    pattern: embedded-logic
+    outputs: [service_instances_full]
+    properties:
+      language: flink_sql
+      sql: |
+        INSERT INTO service_instances_full
+        SELECT i.id, i.status, v.location
+        FROM service_instances i
+        JOIN inventory v ON i.id = v.instance_id
+      parameters:
+        parallelism: 2
+        upgradeMode: last-state
+    execution:
+      runtime:
+        platform: kubernetes
+```
+
+- **Deployment settings go under `properties.parameters`.** `embeddedLogicPattern` is closed, so `parallelism` beside `sql` fails validation with `Additional properties are not allowed ('parallelism' was unexpected)`. Name them as the operator does: `parallelism` and `upgradeMode` (`savepoint`, `last-state`, `stateless`) from its job spec, and `flinkConfiguration`.
+- **`fluid apply --mode amend-and-build`** prints `Build 'reconcile' - engine 'flink' runs on kubernetes, outside forge-cli; there is nothing for forge-cli to run.` and counts the build on its own summary line, `Run outside forge-cli: 1`. A contract whose only builds run elsewhere exits 0; it is not "every build was skipped". The apply run report records the build as `skipped` with `reason: runs_outside_forge`.
+- **No planner runs its SQL.** The local, AWS, GCP and Snowflake planners never turn it into a DuckDB, Athena, BigQuery or Snowflake statement, and the local planner lands nothing for an expose the job writes. In a contract that mixes a Flink build with a local one, name each build's exposes in `outputs`: when it cannot tell which build writes an expose, the local planner lands neither and warns.
+- **Schedules and generated code.** `fluid generate schedule` gives the build no task or DAG, and a contract with no other build has nothing to schedule (exit 0). `fluid generate transformation` says there is no code to generate for it. Leave `execution.trigger` out for a continuous job.
+- **On 0.7.5**, `engine: flink` fails validation. A 0.7.5 contract declares the same job with `properties.language: flink_sql` (with `engine: sql`, say), and forge-cli treats it the same way.
+
 ## `exposes[]` - the consumer-facing API
 
 ```yaml
@@ -510,6 +543,8 @@ Schema `0.7.6` is a preview. It validates only when the file names it, and the s
 - **`consumes[].upstreamWorkspace` and `upstreamDigest`** pin an upstream that lives in another mesh ([Federated upstreams](./federation.md)). `upstreamWorkspace` is the federated workspace's id; `upstreamDigest` is the `sha256:<64 hex>` digest of the upstream contract, which `fluid contract digest <file>` prints. `fluid apply` compares it with the upstream's live digest.
 - **`binding.principals`, `binding.encryption`, `binding.packaging`** (the first two, with `exposes[].lifecycle.expire`, are covered in [Governance parity](./governance-parity.md)) map the contract's logical principals to identities on a platform, set encryption at rest for S3 and BigQuery, and override packaging per expose.
 - **`exposes[].lifecycle.expire`** deletes stored data once it is older than `retention`; it is `false` unless you set it.
+- *(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* **`builds[].engine: flink`** declares a job that runs outside forge-cli; see [Flink jobs](#flink-jobs-engine-flink).
+- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* **Overlays merge `builds`, `exposes` and `consumes` by identity**; see [How an overlay merges](./environments-and-overlays.md#how-an-overlay-merges).
 
 ## Where to look next
 

docs/concepts/environments-and-overlays.md

@@ -71,6 +71,39 @@ not name the file.
 - A list of maps is patched by position: `exposes: [ {...} ]` in an overlay
   amends `exposes[0]` of the base and leaves the rest of that expose alone.
   Overlay entries past the end of the base list are appended.
+- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* In a contract whose `fluidVersion` is 0.7.6 (the schema
+  preview) or later, `builds`, `exposes` and `consumes` at the top of the
+  contract are patched by identity instead: `builds[].id`, `exposes[].exposeId`,
+  and `consumes[]` by `productId` + `exposeId`. Inserting or reordering an entry
+  in the base no longer retargets the overlay.
+
+  ```yaml
+  # contract.fluid.yaml (base): backfill was just added first
+  builds:
+    - {id: backfill, engine: spark}
+    - {id: ingest, engine: kafka-connect}
+    - {id: transform, engine: spark, properties: {executors: 2}}
+  ```
+
+  ```yaml
+  # overlays/prod.yaml: patches transform wherever it sits
+  builds:
+    - id: transform
+      properties: {executors: 8}
+  ```
+
+  An overlay entry whose id the base lacks is appended, and logged as
+  `overlay_entry_added` with the base's ids, so a typo shows. An overlay entry
+  that names no id, or an id named twice, fails the load
+  (`contract_load_failed`, naming the overlay file and the index): give each
+  entry the id of the entry it patches. Id-less patches still apply to a base
+  list with at most one entry. When the base's own ids are missing or repeated,
+  the list merges by position and logs `overlay_merged_by_position`. Lists
+  inside an entry, such as an expose's `contract.schema`, still merge by
+  position.
+- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* Before 0.7.6 the merge is by position, and a patch that
+  lands on an entry naming another id logs `overlay_patch_retargeted`: Put the
+  overlay's entries in the base's order, or set `fluidVersion: "0.7.6"`.
 - Base keys an overlay does not set survive inside a patched map. A base
   `binding.location.path` is still present in the resolved binding of an overlay
   that sets `project`, `dataset` and `table`: `fluid bundle --env gcp` shows

docs/recipes/per-environment-overlays.md

@@ -116,7 +116,7 @@ fluid apply contract.fluid.yaml --env prod --yes
 
 ## What the overlay loader does
 
-- **Deep-merges** the overlay over the base — maps merge key-by-key; a list of maps is patched **positionally**, so `exposes: [ { ... } ]` in an overlay amends `exposes[0]` of the base rather than replacing the whole list, and overlay entries past the end of the base list are appended. Scalar lists, empty lists and mismatched types are replaced wholesale.
+- **Deep-merges** the overlay over the base — maps merge key-by-key; a list of maps is patched **positionally**, so `exposes: [ { ... } ]` in an overlay amends `exposes[0]` of the base rather than replacing the whole list, and overlay entries past the end of the base list are appended. Scalar lists, empty lists and mismatched types are replaced wholesale. *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* With `fluidVersion: "0.7.6"`, `builds`, `exposes` and `consumes` are patched by their ids instead, so each overlay entry names the `id` / `exposeId` (or `productId` + `exposeId`) of the entry it patches; see [How an overlay merges](../concepts/environments-and-overlays.md#how-an-overlay-merges).
 - **Resolves `$ref` first.** Fragments the base pulls in with `$ref` are resolved before the overlay is merged, so an overlay can patch a field that lives in a fragment. Do not put a `$ref` in an overlay itself: as of 0.18.1 such an overlay is silently ignored by `fluid validate` and `fluid plan`.
 - **Announces the merge** — an `overlay_applied` line is printed whenever an overlay is merged. The line does **not** name which overlay file was used.
 - **Warns when no overlay matches** `--env <name>` and runs the base contract unchanged. The warning names the overlays that do exist:

docs/reference/preview-fields.md

@@ -85,6 +85,10 @@ The [`consumes`](./contract-0.7.6-preview.md#consumes) and [`exposes`](./contrac
 
 *([forge-cli #712](https://github.com/Agenticstiger/forge-cli/pull/712), unreleased)* `builds[].properties.sink.catalogAuth` is a 0.7.6 field too. No release includes it yet, so it has no row above and none in the preview delta, which is generated from a released schema. It names the environment variable that authenticates a streaming sink to an Iceberg REST catalog; see [Authenticate a sink to a REST catalog](../advanced/source-aligned-acquisition.md#authenticate-a-sink-to-a-rest-catalog).
 
+*(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* `builds[].engine: flink` is a 0.7.6 value too, for a job that runs on the Flink Kubernetes Operator, AWS Managed Service for Apache Flink, or another Flink platform. forge-cli validates and plans it but ships no runner; see [Flink jobs](../concepts/builds-exposes-bindings.md#flink-jobs-engine-flink).
+
+*(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* In a contract that names 0.7.6, an overlay patches `builds`, `exposes` and `consumes` by identity instead of by position; see [How an overlay merges](../concepts/environments-and-overlays.md#how-an-overlay-merges).
+
 ### Federated upstreams
 
 `upstreamWorkspace` and `upstreamDigest` travel together. This pair of fields on a `consumes[]` entry validates under 0.7.6: