From 4e130d86c538e8c44472c6494c366f6c51ff193d Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Tue, 27 Oct 2026 03:15:38 +0100 Subject: [PATCH 1/2] docs: the seven new 0.7.6 preview fields (forge-cli #753) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit forge-cli #753 adds seven optional fields to the 0.7.6 preview schema, for things a contract could not previously say. All seven need `fluidVersion: "0.7.6"`, all are additive, and a contract that sets none of them validates and exports byte for byte as before. `reference/preview-fields.md` is the page that maps the preview, so the seven go there as their own section with a table and the detail each one needs. The generated 0.7.6 delta cannot carry them yet — it is rendered from the schema bundled in the pinned CLI — and the section says so rather than leaving a reader to wonder why the rows are missing. Written from `fluid_build/schemas/fluid-schema-0.7.6.json` on `origin/fix/odps-export-import-defects`, whose field descriptions carry the reasoning, and checked against the exporters that read each field. Three have a caveat that belongs in the open rather than in a footnote: - `metadata.team.members[].email` is exported as a `fluidEmail` custom property, because there is no Bitol field to export it as: all three vendored schemas (ODCS v3.1.0, ODPS v1.0.0, ODPS v1.1.0) define `TeamMember` with `additionalProperties: false` and none has `email`. `username` is the only required member field. - `exposes[].contract.id` has no pattern, only `minLength: 1`, so a catalog can supply a URN or a UUID. Two exposes resolving to one id — both pinned, or one pinning what the other derives — now fail `fluid validate` and every export. - `binding.serialization` is the only thing that moves a Kafka server's ODCS `format` off `kafka_topic`; the two `-confluent` spellings collapse onto the wire format they carry, and the block rides along in `fluidExtras` so the registry is not forgotten. A contract that does not declare it exports exactly as before. A column's `quality` gets the fuller treatment, because the two things that surprise are not in its one-line summary: it REPLACES the derived rules for that column rather than adding to them, and rules parked in the importer's pass-through bucket are dropped by a FLUID → ODPS → FLUID cycle, so they belong on the column. The key table is the schema's own list, with which `type` each key applies to. Co-Authored-By: Claude Opus 5 --- docs/reference/preview-fields.md | 109 +++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/docs/reference/preview-fields.md b/docs/reference/preview-fields.md index f30d977..639d9fe 100644 --- a/docs/reference/preview-fields.md +++ b/docs/reference/preview-fields.md @@ -146,6 +146,115 @@ Replace `` with the domain of your own groups; left as written, val The same refusal applies to a placeholder in a reserved top-level domain (`.example`, `.test`, `.invalid`, `.localhost`), per the `exposes[].binding.principals` row. A value that is not valid for the binding's platform is refused too: an AWS role ARN mapped on a `gcp` binding fails with "is not a GCP IAM member". +## Seven more fields, in the next release after `0.22.0` + +::: tip Not in a released CLI yet +[forge-cli #753](https://github.com/Agenticstiger/forge-cli/pull/753) adds seven optional +fields to the 0.7.6 preview schema, for things a contract could not previously say. All seven +require `fluidVersion: "0.7.6"`, all are additive, and a contract that sets none of them +validates and exports byte for byte as before. The names are taken from Bitol ODCS v3.1.0 and +the released ODPS v1.1.0 rather than invented, so import and export are a pass-through instead +of a translation. + +The release is a minor bump from `0.22.0` and its number is not settled yet, so this page names +the release rather than the number. The generated [0.7.6 preview delta](./contract-0.7.6-preview.md) +does not list these rows: it is rendered from the schema bundled in the CLI release this site is +pinned to, and will carry them once that pin moves. +::: + +| Field | What it declares | What it changes on export | +| --- | --- | --- | +| `metadata.team` | The team as a roster — `{name, members[]}`, each member `{username, name, role, email}` — where `metadata.owner` holds one team name and one mailbox and no role at all | Fills the ODCS and ODPS team block, which came out empty before. See [the `email` caveat](#the-email-caveat) | +| `description` (root) | Either a string, as today, or `{purpose, usage, limitations}` — the shape both Bitol standards define | An export could only ever carry `description.purpose`; `usage` and `limitations` had no home a contract could declare, so every export left them empty | +| `consumes[].contractId` | The upstream contract's own id, for an upstream that has one — a product not built with FLUID, registered in a catalog under an id FLUID did not derive | The canonical-port builder already preferred an explicit id; with `consumes[]` closed there was no way to supply one, so the derived id was exported as if the upstream answered to it | +| `exposes[].contract.id` | An explicit, stable contract id for one output port | Used verbatim as the ODCS `id` and the ODPS output port `contractId`. See [pinning a contract id](#pinning-a-contract-id) | +| `binding.location.host` | Host or endpoint; for a Kafka binding, the bootstrap servers | The ODCS server mapper already read `host` and fell back to `account`, while the schema accepted only `account` — so the only valid way to set bootstrap servers was a field named for something else. With neither set, the exported server is the placeholder `${KAFKA_BOOTSTRAP_SERVERS}` | +| `exposes[].binding.serialization` | `{value, key}`, each one of `json`, `avro`, `protobuf`, `xml`, `avro-confluent`, `json-confluent` | The only thing that moves a Kafka server's ODCS `format` off `kafka_topic`. See [declaring the wire format](#declaring-the-wire-format) | +| `quality` on a column | Column-level data quality rules, in ODCS's own shape | Written straight into `schema[].properties[].quality[]` of the exported ODCS contract. See [column quality rules](#column-quality-rules) | + +`binding.location.host` is usually set per environment in an overlay rather than in the base +contract, since the brokers differ by environment and the rest of the binding does not. + +### The `email` caveat + +`metadata.team.members[].email` is **not** exported as a Bitol `TeamMember` field, because +there is no such field to export it as. All three vendored schemas — ODCS v3.1.0, ODPS v1.0.0 +and ODPS v1.1.0 — define `TeamMember` with `additionalProperties: false` and none of them has +an `email` property. Writing one would produce a document that fails the standard's own schema. + +So the roster's `email` is carried in that member's `fluidEmail` custom property instead. It is +not dropped, and it is not smuggled into a field that means something else. + +`username` is the only member field Bitol requires, and the only one this schema requires. The +other three, `name`, `role` and `email`, are optional. `role` is free-form, because Bitol does +not constrain it — `owner`, `steward`, `data engineer`. + +### Pinning a contract id + +Without `exposes[].contract.id` a port's contract id is derived as `.`, +so renaming the product or the expose silently changes the contract id and breaks every +consumer reference to it. A catalog that requires an id format of its own — a URN, a UUID — +cannot be served at all. + +The field has **no pattern**, only a minimum length of 1. That is deliberate: a catalog's id +format is the catalog's business, so the schema does not have an opinion about what a URN or a +UUID should look like. + +Each expose needs its own. Two exposes that resolve to one contract id — whether both pin it, +or one pins what the other derives — fail `fluid validate` and every export path, and the +message names both exposes and the shared id. That export used to succeed and write one +per-port ODCS document for two output ports, so one port's schema was lost; see +[Read this first if you consume the exported artifacts](../upgrading.md#read-this-first-if-you-consume-the-exported-artifacts). + +### Declaring the wire format + +A Kafka binding declares `format: kafka_topic`, which tells a consumer nothing about whether +the values on the topic are Avro or JSON — the one thing it has to know to read the topic at +all. `binding.serialization.value` says so, and `key` says the same for a topic whose keys are +not opaque bytes. + +The first four values are the message formats ODCS v3.1.0 documents for its own Kafka server +`format`. The two `-confluent` values name those same wire formats carried through a Confluent +schema registry, spelled as Apache Flink spells them; on export they collapse onto the plain +wire format, because that is what the ODCS field means. The registry is not forgotten: the +whole `binding.serialization` block rides along verbatim in the `fluidExtras` custom property. + +A contract that does not declare it exports exactly as it does today. + +### Column quality rules + +The ODCS exporter has always replayed a column's `quality` key verbatim and always read it back +on import — but `$defs.column` is `additionalProperties: false` and did not define it. So a +column written the way the exporter supports was **refused by `fluid validate`**: the only way +to get a hand-written column rule into a published contract was to write an invalid contract. + +Two things are worth knowing before you set it. + +- **It replaces the derived rules for that column** rather than adding to them. That is what + makes an imported ODCS column round-trip byte for byte. A column without it exports exactly + as it does today, with the rules forge derives from `required`, the primary-key tag and + `validationRules`. +- **Put the rules on the column, not in a pass-through bucket.** The ODPS importer merges a + resolved port's `contract` block into its output-port stub and discards the sub-import's root + `extensions`, so rules parked in that bucket were dropped outright by a + FLUID → ODPS → FLUID cycle. + +The authority for an item's contents is ODCS v3.1.0 `$defs.DataQuality`, which the schema +deliberately does not re-declare — a copy would drift from the vendored schema, and +`fluid odcs validate` already checks a document against it. The keys the +schema does name: + +| Key | Applies to | Notes | +| --- | --- | --- | +| `id` | every type | Stable identifier, unique within the column's rules; the charset is ODCS `$defs.StableId`. Rules forge derives itself are named `-`. Optional in ODCS, but a DQ engine that labels results by rule id rejects a contract whose rules have none, so give every rule one | +| `type` | — | `text` (prose), `library` (a predefined ODCS metric), `sql` (a query whose result is compared) or `custom` (a named engine's own check) | +| `metric` | `library` | One of `nullValues`, `missingValues`, `invalidValues`, `duplicateValues`, `rowCount` — the five ODCS v3.1.0 defines. A name outside the list is not a library rule | +| `arguments` | `library` | Extra arguments for the metric: `pattern` or `validValues` for `invalidValues`, as DataHub's ODCS ingestion reads them | +| `query` | `sql` | The query whose result the comparison operator is applied to | +| `engine`, `implementation` | `custom` | The engine that runs the check (`soda`, `great-expectations`, `dbt`) and the check as that engine expresses it | +| `dimension` | every type | One of `accuracy`, `completeness`, `conformity`, `consistency`, `coverage`, `timeliness`, `uniqueness` | +| `description`, `severity`, `unit` | every type | ODCS types `severity` as a free string and documents `info` / `warning` / `error` by example. `unit` is usually `rows` (the default) or `percent` | + ## Behaviour that does not wait for the preview `bucketPolicy` is a 0.7.6 key, but the behaviour it controls is not. When a binding with Lake Formation grants omits it, the AWS emitter in 0.18.1 applies `cross-account`: only grantees in another AWS account get a bucket-policy statement, and same-account grantees get none. Setting the key under 0.7.6 changes that choice; leaving it out does not return the earlier output. The `exposes[].binding.governance.lakeFormation.bucketPolicy` row has the security reasoning. From 9dd19536146a87369c8edb2db4cd0550d353de14 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Tue, 27 Oct 2026 03:15:38 +0100 Subject: [PATCH 2/2] docs: exported ODCS and ODPS artifacts change shape (forge-cli #753) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six changes to exported documents, each a fix for output that was wrong, and each with old values that may be baked into whatever reads them. So they go in the upgrade guide, as the DuckDB 2.x notes did — but above the per-release checklists rather than below, because this is the section a reader has to see before upgrading rather than one they reach by working through their version. Leading with the one that is urgent: a Kafka column's `physicalType` was silent data corruption. Kafka had no physical-type table, so the exporter fell back to the ODCS logicalType, and ODCS defaults an unqualified `integer` to `i32` — so a tool deriving an Avro schema read 32 bits for a column a producer such as Flink `BIGINT` writes as 64. The page says plainly that anything building Avro schemas from these exports should treat every 64-bit column as having been wrong and re-register. The rest, with what to do and how to check each: - A per-port ODCS contract is named after its port (title, else `exposeId`), with the product name in a `fluidProductName` custom property. The ODCS `id` is unchanged and a whole-contract export is untouched, but a single-expose product's name changes too whenever the port's title differs from the product's — deliberately, since a special case for one port would change the name the day a second is added. - ODPS `inputPorts[].contractId` is `.`, the id something actually publishes, and a `consumes[].versionConstraint` that used to be dropped rides in `fluidVersionConstraint`. - A Kafka server's ODCS `format` becomes a message format only once the contract declares `binding.serialization`; the page states that the `json` default behind it is pre-existing, so the no-op case is genuinely byte-identical. - Derived quality rules carry a stable, derived `id`, and declared `dq.rules` export as ODCS quality for the four types ODCS can express. The other four are logged by name and still ride in `fluidExtras`; the table says which is which, because "some rules export" is not actionable. - `fluid odps import` validates before writing and writes nothing on failure, and its pass-through bucket moved to root `extensions.odps` because `metadata` is closed and the old address could never validate. `--lenient` is noted as a widened flag, not a new one. - The breaking case: two ports resolving to one contract id now fail everywhere, where the export used to succeed and lose one port's schema under a MANIFEST merkle root. Co-Authored-By: Claude Opus 5 --- docs/upgrading.md | 149 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/docs/upgrading.md b/docs/upgrading.md index 3852082..b38cdda 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -95,6 +95,155 @@ prints). Some fields added in these releases exist only in `0.7.6`: 5. Work through the section for each version you cross, below. +## Read this first if you consume the exported artifacts + +::: warning Shipping in the next release after `0.22.0` +Exported ODCS and ODPS documents change shape. Every change below corrects output that was +wrong, but the old values may be baked into whatever reads them, so this is the section to read +before you upgrade rather than after. The work is +[forge-cli #753](https://github.com/Agenticstiger/forge-cli/pull/753); the release is a minor +bump from `0.22.0` and its number is not settled yet, so this page names the release rather +than the number. + +Nothing here changes a FLUID contract. If you do not consume `fluid generate standard`, +`fluid odcs export`, `fluid odps export` or `fluid generate artifacts` output, skip it. +::: + +| Change | What to do | How to check | +|---|---|---| +| A Kafka column's ODCS `physicalType` is an Avro type name. **`bigint` now exports as `long`, where it exported `integer`** | Re-derive and re-register any Avro schema built from a forge ODCS export of a Kafka topic with a 64-bit column | Export the contract and read `physicalType` on that column | +| A **per-port** ODCS contract's `name` is the port's, not the product's; the product name moves to a `fluidProductName` custom property | Re-key anything that keys files, registry subjects or display names on the contract `name`; read the product from `fluidProductName` | Export a product with more than one expose: the two documents no longer share a `name` | +| ODPS `inputPorts[].contractId` is `.`, not the bare upstream product id | Point any resolver that looks an input port up by that id at the port-scoped form | Export a contract with a `consumes[]` entry and read `inputPorts[].contractId` | +| A Kafka server's ODCS `format` becomes a message format **once the contract declares `binding.serialization`** | Nothing, until you adopt that field | A contract without it exports the same bytes as before | +| Derived quality rules carry an `id`, and rules declared in `dq.rules` are exported as ODCS quality | Nothing. A DQ engine that labels results by rule id stops refusing the document | `quality[].id` is present on every rule the exporter mints | +| `fluid odps import` refuses to write a contract that does not validate, and its pass-through bucket moved from `metadata.odps_passthrough` to the root `extensions.odps` | Read the bucket at its new address. Pass `--lenient` if you need the old write-anyway behaviour | An import that used to print a success tick and write an invalid file now fails and writes nothing | +| **Breaking.** Two exposes resolving to one contract id fail `fluid validate` and every export | Give each expose its own `exposes[].contract.id`, or rename the expose whose id is derived | `fluid validate` names both exposes and the shared id | + +### The Kafka `physicalType` change is the urgent one + +This was silent data corruption, not a cosmetic fix. Kafka had no physical-type table, so +`fluid_to_physical` fell back to the ODCS *logicalType* — and ODCS's own schema defaults an +unqualified `integer` to `i32`. A tool deriving an Avro schema from the exported contract +therefore read 32 bits for a column a producer such as Flink `BIGINT` writes as 64. The +registered schema and the data disagreed, and nothing signalled it. + +The physical type is now the Avro type name, from a table measured against the Avro 1.12 +specification and against datacontract-cli's own ODCS→Avro exporter. The integer family is +split by width, because Avro has no 8- or 16-bit integer: `int`, `integer`, `int2`, `int4` and +`serial` are `int`, while `bigint`, `long`, `int8`, `int64`, `longint` and `bigserial` are +`long`. `decimal` carries its precision and scale in `customProperties`. + +If you build Avro schemas from these exports, treat every 64-bit column as having been wrong +and re-register. + +### The per-port ODCS name + +A per-port export — `fluid generate standard --format odcs`, and the per-expose documents +`fluid generate artifacts` writes — used to name every document after the **product**, so a +product with two exposes published two contracts both called "Orders Product". The `id` was +already per-port; the `name` was not, so a tool that keys files, registry subjects or display +names on the name could not tell them apart. + +The name is now the port's `title`, falling back to its `exposeId`, and the product's name is +in a `fluidProductName` custom property. + +There is deliberately **no** special case for a single-port product: making the exported name +depend on how many siblings a port has would change that name the day a second port is added. +So a single-expose product's exported name changes too, whenever its port's title (or +`exposeId`) differs from the product's name. The ODCS `id` is unchanged in every case, and a +whole-contract export is untouched. + +### Input ports, and the version constraint that used to be dropped + +`inputPorts[].contractId` carried the bare upstream *product* id, which is not the id anything +publishes: an output port is published under `.`. It is now that +port-scoped form — or the upstream's own id when the contract names one in the new +`consumes[].contractId`. + +A `consumes[].versionConstraint` was dropped outright, because ODCS's input-port object has no +field for it. It now rides in a `fluidVersionConstraint` custom property. + +### The Kafka server `format` + +A Kafka binding declares `format: kafka_topic`, which says nothing about whether the values on +the topic are Avro or JSON — the one thing a consumer has to know to read it at all. With the +new `binding.serialization.value`, the exported ODCS server `format` is the message format ODCS +means by that field: `avro`, `json`, `protobuf` or `xml`, with `avro-confluent` and +`json-confluent` collapsing onto the wire format they carry. Whether a schema registry is +involved is not lost — `binding.serialization` rides along verbatim in the `fluidExtras` +custom property, unless `ODCS_FLUID_EXTRAS=false` opts out of that blob. + +**A contract that has not adopted the field exports exactly as before.** The exporter will not +guess a wire format from a contract that never stated one: `binding.format` passes through as +it always has, and where a contract sets neither, the pre-existing `json` default still +applies. So this changes a Kafka export only once the contract says what its messages are. + +### Quality rules + +Rules the exporter derives from `required`, the primary-key tag and `validationRules` now carry +an `id`. The ids are derived rather than counted, so two renders of one contract produce the +same ids and an export is not a spurious diff. That matters twice: a DQ engine that labels its +results by rule id refuses a whole contract whose rules have none, and DataHub's ODCS ingestion +seeds its assertion URNs from `quality.id`, so a stable id is what keeps an assertion's +identity across a rename or a reorder. + +Rules an author declared in `dq.rules` used to be parked in the `fluidExtras` custom property +and nowhere else, so a consumer running the published contract's quality saw only the checks +forge had synthesised. They are now exported as ODCS quality for the four types ODCS can +express: + +| `dq.rules` type | Exported as | +|---|---| +| `completeness` | library metric `nullValues`, dimension `completeness` | +| `uniqueness` | library metric `duplicateValues`, dimension `uniqueness` | +| `valid_values` | library metric `invalidValues`, dimension `conformity` | +| `freshness` | a `text` rule — ODCS has no library metric for it | + +The other four — `accuracy`, `schema`, `anomaly_detection` and `drift_detection` — have no +executable ODCS form, so each is **logged by name with the reason** and still carried in +`fluidExtras`. Silence was the bug; a rule with no ODCS form is now reported rather than +dropped without a word. + +### `fluid odps import` + +The command's own help said it "always emits one validated FLUID contract" while never running +the validator, so importing its own export printed a success tick and wrote a file +`fluid validate` then rejected with ten errors. It now validates before writing: + +```bash +fluid odps import product.odps.yaml -o contract.fluid.yaml +``` + +A document that does not produce a valid contract fails, names the problem and **writes +nothing**. `--lenient` writes it anyway — note that this widens a flag that already existed: +it used to mean only "downgrade output-port resolution failures to warnings", and it now also +means "write the contract even when it does not validate". + +The pass-through bucket the importer writes for everything ODPS carries that FLUID has no field +for moved from `metadata.odps_passthrough` to the root `extensions.odps`. `metadata` is +`additionalProperties: false`, so the old address could never validate; root `extensions` is +the one object the FLUID schema declares open. Root `extensions` arrived in FLUID 0.7.3, so an +import whose source document records an older `fluidVersion` warns and declares the default +instead, because the older schema cannot hold the bucket. + +### The breaking one: two ports under one contract id + +A contract whose `exposes[].contract.id` collides with a sibling's **resolved** id — pinned or +derived — now fails `fluid validate` and every export door. The check compares the ids the +exporters will actually mint, so a pin equal to a sibling's derived `.` +is caught too: + +```bash +fluid validate contract.fluid.yaml +``` + +The message names both exposes and the shared id, and the advice fits which side pinned it: +change the `contract.id` where one is pinned, rename the expose where the id is derived. + +That export used to succeed. It wrote one per-port ODCS document for two output ports, so the +losing port's schema was absent from the bundle and both ports resolved to the survivor — and +`fluid generate artifacts` sealed it under a MANIFEST merkle root. The failure is the fix. + ## From 0.15.x to 0.16 Release notes: [`0.16.0`](./RELEASE_NOTES_0.16.0.md) (covers `0.15.1` to `0.16.2`).