From b78f78224026c4909775f86bb3b64eb8b60a3ff1 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Mon, 19 Oct 2026 16:56:59 +0200 Subject: [PATCH] docs: engine: flink, and overlays that merge by identity from 0.7.6 (forge-cli #731, #732) builds-exposes-bindings.md gets a "Flink jobs: engine: flink" section: the example contract, deployment settings under properties.parameters named as the Flink Kubernetes Operator names them, and what apply, the planners and the schedule and transformation generators do with a build forge-cli validates and plans but never runs. environments-and-overlays.md, the per-environment-overlays recipe and the contract-loading API page describe the 0.7.6 identity merge for builds, exposes and consumes, the load error for an overlay entry it cannot match, and the warnings; preview-fields.md lists both as unreleased 0.7.6 additions. --- docs/advanced/contract-loading-api.md | 5 +++- docs/concepts/builds-exposes-bindings.md | 35 ++++++++++++++++++++++ docs/concepts/environments-and-overlays.md | 33 ++++++++++++++++++++ docs/recipes/per-environment-overlays.md | 2 +- docs/reference/preview-fields.md | 4 +++ 5 files changed, 77 insertions(+), 2 deletions(-) diff --git a/docs/advanced/contract-loading-api.md b/docs/advanced/contract-loading-api.md index 36653e5..9a0dc1e 100644 --- a/docs/advanced/contract-loading-api.md +++ b/docs/advanced/contract-loading-api.md @@ -180,7 +180,10 @@ In the engine's order: 3. **Overlay**: for `env`, the first of `overlays/.yaml|yml|json`, `.yaml|yml|json`, `..yaml|yml|json` next to the contract, deep-merged over the base (objects key by key, lists of objects - by position, anything else replaced). A bundle is never re-overlaid. + by position, anything else replaced). *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* From + `fluidVersion` 0.7.6, `builds`, `exposes` and `consumes` merge by identity, + and an overlay entry the merge cannot match raises `ContractLoadError` + (`contract_load_failed`). A bundle is never re-overlaid. 4. **Alias values**: human-friendly values rewritten to the schema's enum value, for example `source.kind: pg` → `postgres`, `source.mode: incremental` → `incremental_append`, diff --git a/docs/concepts/builds-exposes-bindings.md b/docs/concepts/builds-exposes-bindings.md index beaaf9b..ac56997 100644 --- a/docs/concepts/builds-exposes-bindings.md +++ b/docs/concepts/builds-exposes-bindings.md @@ -56,6 +56,7 @@ fluid apply contract.fluid.yaml --yes --mode amend-and-build Under `--mode amend-and-build`, each build goes to the runner that matches it: +- *(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* a Flink build (`engine: flink`, or `properties.language: flink_sql`) is not run: it is reported as run outside forge-cli (see [Flink jobs](#flink-jobs-engine-flink)); - an `acquisition` build runs on its ingestion engine; - a build with `engine: sql`, or `pattern: embedded-logic` with `properties.sql`, runs as embedded SQL; - a dbt build runs the dbt project at `repository`; @@ -106,6 +107,38 @@ only the first expose (by_region); by_customer is not written by this build After that run `out/by_region.parquet` held the second build's rows (`customer_id`, `revenue`). Until this changes, give each embedded-SQL build its own contract. An extra expose bound to a cloud store or a warehouse is refused instead of skipped (from the v0.18.1 source). +### Flink jobs: `engine: flink` + +*(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* A long-running Flink job, typically Flink SQL from Kafka to Kafka on the [Flink Kubernetes Operator](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/) or on AWS Managed Service for Apache Flink, declares `engine: flink` (schema 0.7.6 preview). forge-cli validates and plans the build but ships no runner for it: the platform that runs the job executes it. + +```yaml +fluidVersion: "0.7.6" +builds: + - id: reconcile + engine: flink + pattern: embedded-logic + outputs: [service_instances_full] + properties: + language: flink_sql + sql: | + INSERT INTO service_instances_full + SELECT i.id, i.status, v.location + FROM service_instances i + JOIN inventory v ON i.id = v.instance_id + parameters: + parallelism: 2 + upgradeMode: last-state + execution: + runtime: + platform: kubernetes +``` + +- **Deployment settings go under `properties.parameters`.** `embeddedLogicPattern` is closed, so `parallelism` beside `sql` fails validation with `Additional properties are not allowed ('parallelism' was unexpected)`. Name them as the operator does: `parallelism` and `upgradeMode` (`savepoint`, `last-state`, `stateless`) from its job spec, and `flinkConfiguration`. +- **`fluid apply --mode amend-and-build`** prints `Build 'reconcile' - engine 'flink' runs on kubernetes, outside forge-cli; there is nothing for forge-cli to run.` and counts the build on its own summary line, `Run outside forge-cli: 1`. A contract whose only builds run elsewhere exits 0; it is not "every build was skipped". The apply run report records the build as `skipped` with `reason: runs_outside_forge`. +- **No planner runs its SQL.** The local, AWS, GCP and Snowflake planners never turn it into a DuckDB, Athena, BigQuery or Snowflake statement, and the local planner lands nothing for an expose the job writes. In a contract that mixes a Flink build with a local one, name each build's exposes in `outputs`: when it cannot tell which build writes an expose, the local planner lands neither and warns. +- **Schedules and generated code.** `fluid generate schedule` gives the build no task or DAG, and a contract with no other build has nothing to schedule (exit 0). `fluid generate transformation` says there is no code to generate for it. Leave `execution.trigger` out for a continuous job. +- **On 0.7.5**, `engine: flink` fails validation. A 0.7.5 contract declares the same job with `properties.language: flink_sql` (with `engine: sql`, say), and forge-cli treats it the same way. + ## `exposes[]` - the consumer-facing API ```yaml @@ -510,6 +543,8 @@ Schema `0.7.6` is a preview. It validates only when the file names it, and the s - **`consumes[].upstreamWorkspace` and `upstreamDigest`** pin an upstream that lives in another mesh ([Federated upstreams](./federation.md)). `upstreamWorkspace` is the federated workspace's id; `upstreamDigest` is the `sha256:<64 hex>` digest of the upstream contract, which `fluid contract digest ` prints. `fluid apply` compares it with the upstream's live digest. - **`binding.principals`, `binding.encryption`, `binding.packaging`** (the first two, with `exposes[].lifecycle.expire`, are covered in [Governance parity](./governance-parity.md)) map the contract's logical principals to identities on a platform, set encryption at rest for S3 and BigQuery, and override packaging per expose. - **`exposes[].lifecycle.expire`** deletes stored data once it is older than `retention`; it is `false` unless you set it. +- *(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* **`builds[].engine: flink`** declares a job that runs outside forge-cli; see [Flink jobs](#flink-jobs-engine-flink). +- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* **Overlays merge `builds`, `exposes` and `consumes` by identity**; see [How an overlay merges](./environments-and-overlays.md#how-an-overlay-merges). ## Where to look next diff --git a/docs/concepts/environments-and-overlays.md b/docs/concepts/environments-and-overlays.md index a0f7eb8..61d7b5b 100644 --- a/docs/concepts/environments-and-overlays.md +++ b/docs/concepts/environments-and-overlays.md @@ -71,6 +71,39 @@ not name the file. - A list of maps is patched by position: `exposes: [ {...} ]` in an overlay amends `exposes[0]` of the base and leaves the rest of that expose alone. Overlay entries past the end of the base list are appended. +- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* In a contract whose `fluidVersion` is 0.7.6 (the schema + preview) or later, `builds`, `exposes` and `consumes` at the top of the + contract are patched by identity instead: `builds[].id`, `exposes[].exposeId`, + and `consumes[]` by `productId` + `exposeId`. Inserting or reordering an entry + in the base no longer retargets the overlay. + + ```yaml + # contract.fluid.yaml (base): backfill was just added first + builds: + - {id: backfill, engine: spark} + - {id: ingest, engine: kafka-connect} + - {id: transform, engine: spark, properties: {executors: 2}} + ``` + + ```yaml + # overlays/prod.yaml: patches transform wherever it sits + builds: + - id: transform + properties: {executors: 8} + ``` + + An overlay entry whose id the base lacks is appended, and logged as + `overlay_entry_added` with the base's ids, so a typo shows. An overlay entry + that names no id, or an id named twice, fails the load + (`contract_load_failed`, naming the overlay file and the index): give each + entry the id of the entry it patches. Id-less patches still apply to a base + list with at most one entry. When the base's own ids are missing or repeated, + the list merges by position and logs `overlay_merged_by_position`. Lists + inside an entry, such as an expose's `contract.schema`, still merge by + position. +- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* Before 0.7.6 the merge is by position, and a patch that + lands on an entry naming another id logs `overlay_patch_retargeted`: Put the + overlay's entries in the base's order, or set `fluidVersion: "0.7.6"`. - Base keys an overlay does not set survive inside a patched map. A base `binding.location.path` is still present in the resolved binding of an overlay that sets `project`, `dataset` and `table`: `fluid bundle --env gcp` shows diff --git a/docs/recipes/per-environment-overlays.md b/docs/recipes/per-environment-overlays.md index 5c7f628..e1f0c8c 100644 --- a/docs/recipes/per-environment-overlays.md +++ b/docs/recipes/per-environment-overlays.md @@ -116,7 +116,7 @@ fluid apply contract.fluid.yaml --env prod --yes ## What the overlay loader does -- **Deep-merges** the overlay over the base — maps merge key-by-key; a list of maps is patched **positionally**, so `exposes: [ { ... } ]` in an overlay amends `exposes[0]` of the base rather than replacing the whole list, and overlay entries past the end of the base list are appended. Scalar lists, empty lists and mismatched types are replaced wholesale. +- **Deep-merges** the overlay over the base — maps merge key-by-key; a list of maps is patched **positionally**, so `exposes: [ { ... } ]` in an overlay amends `exposes[0]` of the base rather than replacing the whole list, and overlay entries past the end of the base list are appended. Scalar lists, empty lists and mismatched types are replaced wholesale. *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* With `fluidVersion: "0.7.6"`, `builds`, `exposes` and `consumes` are patched by their ids instead, so each overlay entry names the `id` / `exposeId` (or `productId` + `exposeId`) of the entry it patches; see [How an overlay merges](../concepts/environments-and-overlays.md#how-an-overlay-merges). - **Resolves `$ref` first.** Fragments the base pulls in with `$ref` are resolved before the overlay is merged, so an overlay can patch a field that lives in a fragment. Do not put a `$ref` in an overlay itself: as of 0.18.1 such an overlay is silently ignored by `fluid validate` and `fluid plan`. - **Announces the merge** — an `overlay_applied` line is printed whenever an overlay is merged. The line does **not** name which overlay file was used. - **Warns when no overlay matches** `--env ` and runs the base contract unchanged. The warning names the overlays that do exist: diff --git a/docs/reference/preview-fields.md b/docs/reference/preview-fields.md index 7687277..9917c7c 100644 --- a/docs/reference/preview-fields.md +++ b/docs/reference/preview-fields.md @@ -85,6 +85,10 @@ The [`consumes`](./contract-0.7.6-preview.md#consumes) and [`exposes`](./contrac *([forge-cli #712](https://github.com/Agenticstiger/forge-cli/pull/712), unreleased)* `builds[].properties.sink.catalogAuth` is a 0.7.6 field too. No release includes it yet, so it has no row above and none in the preview delta, which is generated from a released schema. It names the environment variable that authenticates a streaming sink to an Iceberg REST catalog; see [Authenticate a sink to a REST catalog](../advanced/source-aligned-acquisition.md#authenticate-a-sink-to-a-rest-catalog). +*(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* `builds[].engine: flink` is a 0.7.6 value too, for a job that runs on the Flink Kubernetes Operator, AWS Managed Service for Apache Flink, or another Flink platform. forge-cli validates and plans it but ships no runner; see [Flink jobs](../concepts/builds-exposes-bindings.md#flink-jobs-engine-flink). + +*(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* In a contract that names 0.7.6, an overlay patches `builds`, `exposes` and `consumes` by identity instead of by position; see [How an overlay merges](../concepts/environments-and-overlays.md#how-an-overlay-merges). + ### Federated upstreams `upstreamWorkspace` and `upstreamDigest` travel together. This pair of fields on a `consumes[]` entry validates under 0.7.6: