Re-vendor 0.7.3-0.7.6 from forge-cli v0.21.0: engine flink, row filters, masked columns, catalogAuth

docs/README.md

@@ -66,6 +66,6 @@ exposes:
 - **[Concepts](/fluid/concepts/)** — the agentic-native layer and how FLUID compares to ODCS / ODPS.
 - **[Schema Reference](/fluid/schema/anatomy)** — every top-level block, a cheatsheet, and the full specification.
 - **[What's New in 0.7.5](/fluid/releases/0.7.5)** — the latest stable version: streaming Kafka → Iceberg sink, Confluent Cloud Tableflow, and a pgvector output port.
-- **[0.7.6 (preview)](/fluid/releases/0.7.6)** — packaging modes, declared consumers, cross-mesh pins; opt-in, and still subject to change.
+- **[0.7.6 (preview)](/fluid/releases/0.7.6)** — packaging modes, declared consumers, cross-mesh pins, row filters and masked columns, Flink builds; opt-in, and still subject to change.
 - **[See the deck](/fluid/deck/)** — the FLUID story in slides.
 - **[Contributing & governance](/fluid/contributing/)** — how the standard is maintained, and how to take part.

docs/releases/0.7.6.md

@@ -4,7 +4,7 @@
 0.7.6 is a **preview**. A contract uses it only by declaring `fluidVersion: "0.7.6"`, and its fields and their meaning can still change before it is promoted to stable. The latest stable version is [0.7.5](/fluid/releases/0.7.5). See [Stable and preview versions](/fluid/schema/versions#stable-and-preview-versions).
 :::
 
-0.7.6 adds declarative ownership of infrastructure containers (`packaging`), a declared list of downstream consumers, pins for upstreams in another mesh, and binding-level fields that let one cloud-neutral contract carry encryption, identity mapping and data expiry to each platform it is deployed on.
+0.7.6 adds declarative ownership of infrastructure containers (`packaging`), a declared list of downstream consumers, pins for upstreams in another mesh, binding-level fields that let one cloud-neutral contract carry encryption, identity mapping and data expiry to each platform it is deployed on, row filters and masked columns in an expose's access policy, Apache Flink as a build engine, and credentials by environment variable for a REST catalog an Iceberg sink writes to.
 
 - **Schema:** [`fluid-schema-0.7.6.json`](/fluid/schema/fluid-schema-0.7.6.json) · **Reference:** [`specs/0.7.6/fluid-spec.html`](/fluid/specs/0.7.6/fluid-spec.html) · **Diff:** [`diff-0.7.5-to-0.7.6.md`](https://github.com/open-data-protocol/fluid/blob/main/schema-diffs/diff-0.7.5-to-0.7.6.md)
 - **Compatibility:** additive over 0.7.5. `scripts/check-compat.py --from 0.7.5 --to 0.7.6` reports no narrowing change, so a valid 0.7.5 contract stays valid when it declares `"0.7.6"`.
@@ -27,8 +27,12 @@ Every addition below is optional. The list is derived from the schemas: these ar
 | `exposes[].binding.governance.lakeFormation.bucketPolicy` | Which grantees may read a Lake Formation location directly from object storage. |
 | `exposes[].lifecycle.expire` | Whether data older than the expose's `retention` is deleted. |
 | `exposes[].policy.privacy.masking[].params.{keepFirst, keepLast, saltEnv, keyEnv}` | Typed parameters for masking rules. |
+| `exposes[].policy.authz.rowFilters[]` | Which rows each principal may read. |
+| `exposes[].policy.authz.columnRestrictions[].mask` | The masking rule an `access: mask` restriction applies. |
+| `exposes[].binding.governance.lakeFormation.maskedViews` | The role and Glue connection behind the views that serve masked columns on Lake Formation. |
+| `build(s).properties.sink.catalogAuth` | How an acquisition sink authenticates to an Iceberg REST catalog, by naming environment variables. |
 
-`fluidVersion` accepts `"0.7.6"` in addition to `"0.7.3"`, `"0.7.4"` and `"0.7.5"`.
+`fluidVersion` accepts `"0.7.6"` in addition to `"0.7.3"`, `"0.7.4"` and `"0.7.5"`. Three enums gain values: `build(s).engine` accepts `flink`, `build(s).properties.sink.catalog` accepts `lakekeeper` and `polaris`, and `exposes[].policy.authz.columnRestrictions[].access` accepts `mask`.
 
 0.7.6 also writes down the meaning of a field that already existed: `exposes[].policy.authz.columnRestrictions` ([below](#column-restrictions-semantics-written-down)).
 
@@ -165,20 +169,125 @@ policy:
 
 `params` stays open (`additionalProperties: true`). The typed keys name secrets by environment variable rather than carrying them; the schema's description adds that the reference implementation's runner refuses any other key, including a literal salt or key. The 0.7.6 description of `strategy` also gives each strategy's output form — for example, `hash` is the lowercase hex SHA-256 of salt followed by value — and notes that `k_anonymity` is accepted by the schema but is a property of a whole table, not a per-value transform.
 
+## Row filters and masked columns
+
+<!-- fluid-check: merge-into=minimal@0.7.6 at=exposes/0 -->
+```yaml
+policy:
+  authz:
+    readers: ["group:support", "group:analysts-eu"]
+    columnRestrictions:
+      - principal: "group:support"
+        columns: [phone]
+        access: mask
+        mask: last_four            # last_four | first_four | nullify
+    rowFilters:
+      - principal: "group:analysts-eu"
+        name: eu_only              # optional; a lowercase identifier
+        where: "region = 'EU'"
+```
+
+**`rowFilters[]`** lets a principal read only the rows its `where` predicate selects. A reader that no filter names reads every row, and the schema's description allows one filter per principal per expose. `where` is one SQL boolean expression over the expose's columns, with no statement separator or comment. `principal` and `where` are required, and `name`, the filter's name on the platform, must match `^[a-z][a-z0-9_]{0,62}$`. Principals resolve through `binding.principals`, as for column restrictions.
+
+**`access: mask`** lets the principal read the restricted columns masked, never their raw values, by the rule in **`mask`**: `last_four` (`XXXXX` and the value's last four characters), `first_four` (its first four characters and `XXXXX`) or `nullify` (`NULL`). The schema requires `mask` with `access: mask` and refuses it with any other `access`. A deny beats a mask, and a mask beats an allow for the principal it names.
+
+<!-- fluid-check: merge-into=minimal@0.7.6 at=exposes/0 -->
+```yaml
+binding:
+  platform: aws
+  format: parquet
+  location: { bucket: acme-lake, path: support/contacts/, region: eu-west-1, database: support, table: contacts }
+  governance:
+    lakeFormation:
+      maskedViews:
+        definer: "arn:aws:iam::123456789012:role/fluid-view-definer"
+        validationConnection: athena-view-validation
+```
+
+Lake Formation cannot mask a column. **`governance.lakeFormation.maskedViews`** says where the views that serve masked columns come from: `definer`, the IAM role the views are defined as, and `validationConnection`, the Glue connection Glue validates each view's SQL through. Both are required. In the reference implementation, a masked principal's grant on the table excludes the columns, and the principal reads a protected `SECURITY DEFINER` Glue Data Catalog view, `<table>_masked`, that returns them masked by the same rule and applies the principal's row filter. On BigQuery it attaches a data policy to the columns' policy tag. Like the other descriptions of what `fluid apply` emits, this is non-normative.
+
+## `engine: flink`
+
+<!-- fluid-check: merge-into=minimal@0.7.6 -->
+```yaml
+build:
+  pattern: embedded-logic
+  engine: flink
+  properties:
+    language: flink_sql
+    sql: |
+      INSERT INTO payments_reconciled
+      SELECT p.payment_id, p.amount, s.settled_at
+      FROM payments AS p JOIN settlements AS s ON p.payment_id = s.payment_id
+    parameters:
+      parallelism: 4
+      upgradeMode: savepoint       # the operator's savepoint | last-state | stateless
+      flinkConfiguration:
+        execution.checkpointing.interval: 60s
+```
+
+`flink` names Apache Flink as the build's engine, usually for a long-running Flink SQL job (`properties.language: flink_sql`). The embedded-logic properties are a closed set, so deployment settings go under `properties.parameters`; the schema's description recommends the names the Flink Kubernetes Operator's job spec uses (`parallelism`, `upgradeMode`) and `flinkConfiguration`. A 0.7.5 contract cannot declare `engine: flink`.
+
+The reference implementation validates and plans a `flink` build but ships no runner for it: the platform that runs the job, such as the Flink Kubernetes Operator or AWS Managed Service for Apache Flink, executes it. It never runs such a build's SQL on DuckDB or turns it into a warehouse statement, and it treats a build with `properties.language: flink_sql` the same way on any `fluidVersion`.
+
+## REST catalog credentials: `sink.catalogAuth`
+
+<!-- fluid-check: merge-into=minimal@0.7.6 -->
+```yaml
+exposes:
+  - exposeId: events
+    kind: table
+    contract:
+      schema:
+        - { name: event_id, type: STRING, required: true }
+    binding:
+      platform: local
+      format: iceberg
+      location: { catalog: lakekeeper, uri: "http://lakekeeper:8181/catalog", warehouse: analytics, database: default, table: events }
+builds:
+  - id: ingest
+    pattern: acquisition
+    engine: kafka-connect
+    outputs: [events]
+    properties:
+      source: { kind: postgres, mode: incremental_append }
+      sink:
+        format: iceberg
+        catalog: lakekeeper
+        catalogAuth:
+          type: oauth2
+          credentialEnv: LAKEKEEPER_CLIENT   # a variable holding client_id:client_secret
+          oauth2ServerUri: https://keycloak.example.com/realms/lake/protocol/openid-connect/token
+          scope: lakekeeper
+```
+
+`catalogAuth` says how the sink authenticates to an Iceberg REST catalog. It names environment variables and never holds a secret.
+
+| `type` | Requires | Refuses |
+|---|---|---|
+| `oauth2` (client credentials) | `credentialEnv`, `oauth2ServerUri`, `scope`; `audience` and `resource` are optional | `tokenEnv` |
+| `bearer` (a static token) | `tokenEnv` | `credentialEnv`, `oauth2ServerUri`, `scope`, `audience`, `resource` |
+
+- `credentialEnv` and `tokenEnv` must be environment variable names. The schema refuses one shaped like a token: a run of 20 or more letters and digits, with a digit, between underscores.
+- `oauth2ServerUri` must be an `http://` or `https://` URL with no `user:password@` part. It, `scope`, `audience` and `resource` may not hold a `${...}` expression or a control character, or start with `env:`.
+
+The reference implementation writes each named variable into the sink config it derives as `env:<NAME>`, which Apache Iceberg's REST client resolves from the environment of the process that runs the sink. It refuses `catalogAuth` on a catalog it does not reach over REST, and next to a hand-written override of the same settings. It also refuses an Iceberg sink unless an expose the build writes is bound with `format: iceberg`, as `events` is above; the catalog's address is on that binding.
+
 ## Column restrictions: semantics written down
 
 `exposes[].policy.authz.columnRestrictions` exists in 0.7.5 with the one-line description "Column-level access control". The 0.7.6 description defines it:
 
 - A column named in any restriction is restricted.
 - `access: deny` — the principal may not read the columns. `access: allow` — the columns are readable only by the principals an `allow` names.
 - A deny beats an allow, and a restriction never grants access: the readers are still the expose's readers.
+- `access: mask` (added in 0.7.6) — the principal reads the columns masked by the rule in `mask` ([above](#row-filters-and-masked-columns)). A deny beats a mask, and a mask beats an allow for the principal it names.
 - Principals are logical, resolved per platform through `binding.principals`.
 
 ---
 
 ## Two complete contracts
 
-Both validate against the 0.7.6 JSON Schema (Draft 2020-12) and with the reference implementation, `data-product-forge` 0.18.1 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.6)`).
+Both validate against the 0.7.6 JSON Schema (Draft 2020-12) and with the reference implementation, `data-product-forge` 0.21.0 (`fluid validate` prints `✅ Valid FLUID contract (schema v0.7.6)`).
 
 **BigQuery:** column restriction, mapped principals, a product-owned key, expiring partitions, a percentile measure, packaging and a consumer.
 

docs/releases/README.md

@@ -6,7 +6,7 @@ Release notes for the FLUID specification, newest first. Each entry links to its
 
 | Version | Status | Codename | Compatibility | Summary |
 |---|---|---|---|---|
-| **[0.7.6](/fluid/releases/0.7.6)** | **Preview** | Declarative Packaging Modes | ✅ Additive over 0.7.5 | Adds top-level `packaging` and `consumers`, cross-mesh pins on `consumes[]` (`upstreamWorkspace` + `upstreamDigest`), `aggParams` on semantic measures, and on bindings `encryption.kms`, `principals` and Lake Formation `bucketPolicy`; an expose's `lifecycle` gains `expire`, and masking rules gain typed `params`. |
+| **[0.7.6](/fluid/releases/0.7.6)** | **Preview** | Declarative Packaging Modes | ✅ Additive over 0.7.5 | Adds top-level `packaging` and `consumers`, cross-mesh pins on `consumes[]` (`upstreamWorkspace` + `upstreamDigest`), `aggParams` on semantic measures, and on bindings `encryption.kms`, `principals` and Lake Formation `bucketPolicy`; an expose's `lifecycle` gains `expire`, masking rules gain typed `params`, and access policies gain `rowFilters` and `access: mask`. Builds can declare `engine: flink`, and an Iceberg sink can name a REST catalog's credentials by environment variable (`sink.catalogAuth`). |
 | **[0.7.5](/fluid/releases/0.7.5)** | **Stable (latest)** | Streaming Kafka → Iceberg Sink & Confluent Tableflow | ✅ Additive over 0.7.4 | Adds an opt-in Iceberg streaming sink on the `kafka-connect` acquisition engine, the `confluent` (Tableflow) and `pgvector` binding platforms, the `pgvector_table` format and `binding.vectorConfig`, and location fields for Tableflow, Iceberg catalogs, Redshift Serverless and Kinesis. |
 | **[0.7.4](/fluid/releases/0.7.4)** | Stable | Runtime agentPolicy Enforcement at the MCP Gateway | ✅ Additive over 0.7.3 | Adds the optional `exposes[].mcp` block, the `postgres` binding platform, and the `postgres_table` / `athena_table` / `glue_table` formats. |
 | **[0.7.3](/fluid/releases/0.7.3)** | Stable | Source-Aligned Acquisition | ✅ Additive over 0.7.2 | The `acquisition` build pattern and six ingestion engines, top-level `retention`, `governance` (AWS Lake Formation) and `extensions`, `binding.governance`, `metadata.productType`, and `exposes[].contract.schemaPolicy`. |

docs/schema/changelog.md

@@ -21,6 +21,11 @@ No narrowing change. **Preview:** the 0.7.6 schema can still change before it is
 - `exposes[].binding.governance.lakeFormation.bucketPolicy` (`cross-account` | `none` | `all-grantees`; `cross-account` is the default). Its description says that an emitted `aws_s3_bucket_policy` is authoritative and replaces every other statement on the bucket; see the [0.7.6 notes](/fluid/releases/0.7.6#binding-fields-encryption-principals-lake-formation-bucket-policy).
 - `exposes[].lifecycle` now refers to a new `exposeLifecycle` definition: the root `lifecycle` fields plus `expire` (default `false`).
 - Typed `exposes[].policy.privacy.masking[].params`: `keepFirst`, `keepLast`, `saltEnv`, `keyEnv`.
+- `exposes[].policy.authz.rowFilters[]` (`principal` and `where` required; `name` a lowercase identifier; `description`).
+- `exposes[].policy.authz.columnRestrictions[].access` value `mask`, with `columnRestrictions[].mask` (`last_four` | `first_four` | `nullify`), required with `access: mask` and refused without it.
+- `exposes[].binding.governance.lakeFormation.maskedViews` (`definer` and `validationConnection`, both required).
+- `build(s).engine` value `flink`; see the [0.7.6 notes](/fluid/releases/0.7.6#engine-flink).
+- `build(s).properties.sink.catalog` values `lakekeeper` and `polaris`, and `build(s).properties.sink.catalogAuth` (`type`: `oauth2` | `bearer`), which names the environment variables holding a REST catalog's credentials.
 - `fluidVersion` accepts `"0.7.6"`.
 
 **Described:** `exposes[].policy.authz.columnRestrictions` gains a description of its semantics (deny beats allow; a restriction never grants access; principals are logical).
@@ -147,3 +152,4 @@ No narrowing change. Note that 0.1.0 and 0.1.1 declare the same `$id` (see [Vers
 
 - **Lake Formation `admins` (0.7.3, 0.7.4, 0.7.5).** These schemas were first published with a description of `governance.lakeFormation.admins` saying that principals not listed are not removed. That is the opposite of what applying it does: the list is **authoritative**, and an admin not listed — including the identity running the apply — loses Lake Formation admin. The description was corrected in the reference implementation and the published files carry the corrected text. Only the description changed; the same documents validate.
 - **0.7.5** was re-published with the stable content (above).
+- **Lake Formation tags (0.7.3, 0.7.4, 0.7.5, 0.7.6).** These schemas were first published with a description of `exposes[].binding.governance.lakeFormation.tags` saying that its keys must reference tag definitions in the contract's top-level `governance.lakeFormation.tagDefinitions`. The schema never required that, and a key the contract does not define names a tag owned outside it, such as a platform's central LF-tag ontology, which must already exist in the account. The descriptions of `tags` and `tagDefinitions` were corrected in the reference implementation, and the published files carry the corrected text. Only the descriptions changed; the same documents validate.

schema-diffs/diff-0.7.2-to-0.7.3.md

@@ -579,7 +579,7 @@
         }
         tags: {
           type: "object"
-          description: "LF-tag (TBAC) associations applied to this table. Keys must reference tag definitions declared in th..."
+          description: "LF-tag (TBAC) associations applied to this table, as tag key to value. Emitted as aws_lakeformation_..."
           additionalProperties: {
             type: "string"
           }