docs: the seven new 0.7.6 preview fields, and the exported ODCS/ODPS artifact changes (forge-cli #753)

docs/reference/preview-fields.md

@@ -146,6 +146,115 @@ Replace `<your-domain>` with the domain of your own groups; left as written, val
 
 The same refusal applies to a placeholder in a reserved top-level domain (`.example`, `.test`, `.invalid`, `.localhost`), per the `exposes[].binding.principals` row. A value that is not valid for the binding's platform is refused too: an AWS role ARN mapped on a `gcp` binding fails with "is not a GCP IAM member".
 
+## Seven more fields, in the next release after `0.22.0`
+
+::: tip Not in a released CLI yet
+[forge-cli #753](https://github.com/Agenticstiger/forge-cli/pull/753) adds seven optional
+fields to the 0.7.6 preview schema, for things a contract could not previously say. All seven
+require `fluidVersion: "0.7.6"`, all are additive, and a contract that sets none of them
+validates and exports byte for byte as before. The names are taken from Bitol ODCS v3.1.0 and
+the released ODPS v1.1.0 rather than invented, so import and export are a pass-through instead
+of a translation.
+
+The release is a minor bump from `0.22.0` and its number is not settled yet, so this page names
+the release rather than the number. The generated [0.7.6 preview delta](./contract-0.7.6-preview.md)
+does not list these rows: it is rendered from the schema bundled in the CLI release this site is
+pinned to, and will carry them once that pin moves.
+:::
+
+| Field | What it declares | What it changes on export |
+| --- | --- | --- |
+| `metadata.team` | The team as a roster — `{name, members[]}`, each member `{username, name, role, email}` — where `metadata.owner` holds one team name and one mailbox and no role at all | Fills the ODCS and ODPS team block, which came out empty before. See [the `email` caveat](#the-email-caveat) |
+| `description` (root) | Either a string, as today, or `{purpose, usage, limitations}` — the shape both Bitol standards define | An export could only ever carry `description.purpose`; `usage` and `limitations` had no home a contract could declare, so every export left them empty |
+| `consumes[].contractId` | The upstream contract's own id, for an upstream that has one — a product not built with FLUID, registered in a catalog under an id FLUID did not derive | The canonical-port builder already preferred an explicit id; with `consumes[]` closed there was no way to supply one, so the derived id was exported as if the upstream answered to it |
+| `exposes[].contract.id` | An explicit, stable contract id for one output port | Used verbatim as the ODCS `id` and the ODPS output port `contractId`. See [pinning a contract id](#pinning-a-contract-id) |
+| `binding.location.host` | Host or endpoint; for a Kafka binding, the bootstrap servers | The ODCS server mapper already read `host` and fell back to `account`, while the schema accepted only `account` — so the only valid way to set bootstrap servers was a field named for something else. With neither set, the exported server is the placeholder `${KAFKA_BOOTSTRAP_SERVERS}` |
+| `exposes[].binding.serialization` | `{value, key}`, each one of `json`, `avro`, `protobuf`, `xml`, `avro-confluent`, `json-confluent` | The only thing that moves a Kafka server's ODCS `format` off `kafka_topic`. See [declaring the wire format](#declaring-the-wire-format) |
+| `quality` on a column | Column-level data quality rules, in ODCS's own shape | Written straight into `schema[].properties[].quality[]` of the exported ODCS contract. See [column quality rules](#column-quality-rules) |
+
+`binding.location.host` is usually set per environment in an overlay rather than in the base
+contract, since the brokers differ by environment and the rest of the binding does not.
+
+### The `email` caveat
+
+`metadata.team.members[].email` is **not** exported as a Bitol `TeamMember` field, because
+there is no such field to export it as. All three vendored schemas — ODCS v3.1.0, ODPS v1.0.0
+and ODPS v1.1.0 — define `TeamMember` with `additionalProperties: false` and none of them has
+an `email` property. Writing one would produce a document that fails the standard's own schema.
+
+So the roster's `email` is carried in that member's `fluidEmail` custom property instead. It is
+not dropped, and it is not smuggled into a field that means something else.
+
+`username` is the only member field Bitol requires, and the only one this schema requires. The
+other three, `name`, `role` and `email`, are optional. `role` is free-form, because Bitol does
+not constrain it — `owner`, `steward`, `data engineer`.
+
+### Pinning a contract id
+
+Without `exposes[].contract.id` a port's contract id is derived as `<product id>.<exposeId>`,
+so renaming the product or the expose silently changes the contract id and breaks every
+consumer reference to it. A catalog that requires an id format of its own — a URN, a UUID —
+cannot be served at all.
+
+The field has **no pattern**, only a minimum length of 1. That is deliberate: a catalog's id
+format is the catalog's business, so the schema does not have an opinion about what a URN or a
+UUID should look like.
+
+Each expose needs its own. Two exposes that resolve to one contract id — whether both pin it,
+or one pins what the other derives — fail `fluid validate` and every export path, and the
+message names both exposes and the shared id. That export used to succeed and write one
+per-port ODCS document for two output ports, so one port's schema was lost; see
+[Read this first if you consume the exported artifacts](../upgrading.md#read-this-first-if-you-consume-the-exported-artifacts).
+
+### Declaring the wire format
+
+A Kafka binding declares `format: kafka_topic`, which tells a consumer nothing about whether
+the values on the topic are Avro or JSON — the one thing it has to know to read the topic at
+all. `binding.serialization.value` says so, and `key` says the same for a topic whose keys are
+not opaque bytes.
+
+The first four values are the message formats ODCS v3.1.0 documents for its own Kafka server
+`format`. The two `-confluent` values name those same wire formats carried through a Confluent
+schema registry, spelled as Apache Flink spells them; on export they collapse onto the plain
+wire format, because that is what the ODCS field means. The registry is not forgotten: the
+whole `binding.serialization` block rides along verbatim in the `fluidExtras` custom property.
+
+A contract that does not declare it exports exactly as it does today.
+
+### Column quality rules
+
+The ODCS exporter has always replayed a column's `quality` key verbatim and always read it back
+on import — but `$defs.column` is `additionalProperties: false` and did not define it. So a
+column written the way the exporter supports was **refused by `fluid validate`**: the only way
+to get a hand-written column rule into a published contract was to write an invalid contract.
+
+Two things are worth knowing before you set it.
+
+- **It replaces the derived rules for that column** rather than adding to them. That is what
+  makes an imported ODCS column round-trip byte for byte. A column without it exports exactly
+  as it does today, with the rules forge derives from `required`, the primary-key tag and
+  `validationRules`.
+- **Put the rules on the column, not in a pass-through bucket.** The ODPS importer merges a
+  resolved port's `contract` block into its output-port stub and discards the sub-import's root
+  `extensions`, so rules parked in that bucket were dropped outright by a
+  FLUID → ODPS → FLUID cycle.
+
+The authority for an item's contents is ODCS v3.1.0 `$defs.DataQuality`, which the schema
+deliberately does not re-declare — a copy would drift from the vendored schema, and
+`fluid odcs validate` already checks a document against it. The keys the
+schema does name:
+
+| Key | Applies to | Notes |
+| --- | --- | --- |
+| `id` | every type | Stable identifier, unique within the column's rules; the charset is ODCS `$defs.StableId`. Rules forge derives itself are named `<column>-<metric>`. Optional in ODCS, but a DQ engine that labels results by rule id rejects a contract whose rules have none, so give every rule one |
+| `type` | — | `text` (prose), `library` (a predefined ODCS metric), `sql` (a query whose result is compared) or `custom` (a named engine's own check) |
+| `metric` | `library` | One of `nullValues`, `missingValues`, `invalidValues`, `duplicateValues`, `rowCount` — the five ODCS v3.1.0 defines. A name outside the list is not a library rule |
+| `arguments` | `library` | Extra arguments for the metric: `pattern` or `validValues` for `invalidValues`, as DataHub's ODCS ingestion reads them |
+| `query` | `sql` | The query whose result the comparison operator is applied to |
+| `engine`, `implementation` | `custom` | The engine that runs the check (`soda`, `great-expectations`, `dbt`) and the check as that engine expresses it |
+| `dimension` | every type | One of `accuracy`, `completeness`, `conformity`, `consistency`, `coverage`, `timeliness`, `uniqueness` |
+| `description`, `severity`, `unit` | every type | ODCS types `severity` as a free string and documents `info` / `warning` / `error` by example. `unit` is usually `rows` (the default) or `percent` |
+
 ## Behaviour that does not wait for the preview
 
 `bucketPolicy` is a 0.7.6 key, but the behaviour it controls is not. When a binding with Lake Formation grants omits it, the AWS emitter in 0.18.1 applies `cross-account`: only grantees in another AWS account get a bucket-policy statement, and same-account grantees get none. Setting the key under 0.7.6 changes that choice; leaving it out does not return the earlier output. The `exposes[].binding.governance.lakeFormation.bucketPolicy` row has the security reasoning.

docs/upgrading.md

@@ -95,6 +95,155 @@ prints). Some fields added in these releases exist only in `0.7.6`:
 
 5. Work through the section for each version you cross, below.
 
+## Read this first if you consume the exported artifacts
+
+::: warning Shipping in the next release after `0.22.0`
+Exported ODCS and ODPS documents change shape. Every change below corrects output that was
+wrong, but the old values may be baked into whatever reads them, so this is the section to read
+before you upgrade rather than after. The work is
+[forge-cli #753](https://github.com/Agenticstiger/forge-cli/pull/753); the release is a minor
+bump from `0.22.0` and its number is not settled yet, so this page names the release rather
+than the number.
+
+Nothing here changes a FLUID contract. If you do not consume `fluid generate standard`,
+`fluid odcs export`, `fluid odps export` or `fluid generate artifacts` output, skip it.
+:::
+
+| Change | What to do | How to check |
+|---|---|---|
+| A Kafka column's ODCS `physicalType` is an Avro type name. **`bigint` now exports as `long`, where it exported `integer`** | Re-derive and re-register any Avro schema built from a forge ODCS export of a Kafka topic with a 64-bit column | Export the contract and read `physicalType` on that column |
+| A **per-port** ODCS contract's `name` is the port's, not the product's; the product name moves to a `fluidProductName` custom property | Re-key anything that keys files, registry subjects or display names on the contract `name`; read the product from `fluidProductName` | Export a product with more than one expose: the two documents no longer share a `name` |
+| ODPS `inputPorts[].contractId` is `<productId>.<exposeId>`, not the bare upstream product id | Point any resolver that looks an input port up by that id at the port-scoped form | Export a contract with a `consumes[]` entry and read `inputPorts[].contractId` |
+| A Kafka server's ODCS `format` becomes a message format **once the contract declares `binding.serialization`** | Nothing, until you adopt that field | A contract without it exports the same bytes as before |
+| Derived quality rules carry an `id`, and rules declared in `dq.rules` are exported as ODCS quality | Nothing. A DQ engine that labels results by rule id stops refusing the document | `quality[].id` is present on every rule the exporter mints |
+| `fluid odps import` refuses to write a contract that does not validate, and its pass-through bucket moved from `metadata.odps_passthrough` to the root `extensions.odps` | Read the bucket at its new address. Pass `--lenient` if you need the old write-anyway behaviour | An import that used to print a success tick and write an invalid file now fails and writes nothing |
+| **Breaking.** Two exposes resolving to one contract id fail `fluid validate` and every export | Give each expose its own `exposes[].contract.id`, or rename the expose whose id is derived | `fluid validate` names both exposes and the shared id |
+
+### The Kafka `physicalType` change is the urgent one
+
+This was silent data corruption, not a cosmetic fix. Kafka had no physical-type table, so
+`fluid_to_physical` fell back to the ODCS *logicalType* — and ODCS's own schema defaults an
+unqualified `integer` to `i32`. A tool deriving an Avro schema from the exported contract
+therefore read 32 bits for a column a producer such as Flink `BIGINT` writes as 64. The
+registered schema and the data disagreed, and nothing signalled it.
+
+The physical type is now the Avro type name, from a table measured against the Avro 1.12
+specification and against datacontract-cli's own ODCS→Avro exporter. The integer family is
+split by width, because Avro has no 8- or 16-bit integer: `int`, `integer`, `int2`, `int4` and
+`serial` are `int`, while `bigint`, `long`, `int8`, `int64`, `longint` and `bigserial` are
+`long`. `decimal` carries its precision and scale in `customProperties`.
+
+If you build Avro schemas from these exports, treat every 64-bit column as having been wrong
+and re-register.
+
+### The per-port ODCS name
+
+A per-port export — `fluid generate standard --format odcs`, and the per-expose documents
+`fluid generate artifacts` writes — used to name every document after the **product**, so a
+product with two exposes published two contracts both called "Orders Product". The `id` was
+already per-port; the `name` was not, so a tool that keys files, registry subjects or display
+names on the name could not tell them apart.
+
+The name is now the port's `title`, falling back to its `exposeId`, and the product's name is
+in a `fluidProductName` custom property.
+
+There is deliberately **no** special case for a single-port product: making the exported name
+depend on how many siblings a port has would change that name the day a second port is added.
+So a single-expose product's exported name changes too, whenever its port's title (or
+`exposeId`) differs from the product's name. The ODCS `id` is unchanged in every case, and a
+whole-contract export is untouched.
+
+### Input ports, and the version constraint that used to be dropped
+
+`inputPorts[].contractId` carried the bare upstream *product* id, which is not the id anything
+publishes: an output port is published under `<productId>.<exposeId>`. It is now that
+port-scoped form — or the upstream's own id when the contract names one in the new
+`consumes[].contractId`.
+
+A `consumes[].versionConstraint` was dropped outright, because ODCS's input-port object has no
+field for it. It now rides in a `fluidVersionConstraint` custom property.
+
+### The Kafka server `format`
+
+A Kafka binding declares `format: kafka_topic`, which says nothing about whether the values on
+the topic are Avro or JSON — the one thing a consumer has to know to read it at all. With the
+new `binding.serialization.value`, the exported ODCS server `format` is the message format ODCS
+means by that field: `avro`, `json`, `protobuf` or `xml`, with `avro-confluent` and
+`json-confluent` collapsing onto the wire format they carry. Whether a schema registry is
+involved is not lost — `binding.serialization` rides along verbatim in the `fluidExtras`
+custom property, unless `ODCS_FLUID_EXTRAS=false` opts out of that blob.
+
+**A contract that has not adopted the field exports exactly as before.** The exporter will not
+guess a wire format from a contract that never stated one: `binding.format` passes through as
+it always has, and where a contract sets neither, the pre-existing `json` default still
+applies. So this changes a Kafka export only once the contract says what its messages are.
+
+### Quality rules
+
+Rules the exporter derives from `required`, the primary-key tag and `validationRules` now carry
+an `id`. The ids are derived rather than counted, so two renders of one contract produce the
+same ids and an export is not a spurious diff. That matters twice: a DQ engine that labels its
+results by rule id refuses a whole contract whose rules have none, and DataHub's ODCS ingestion
+seeds its assertion URNs from `quality.id`, so a stable id is what keeps an assertion's
+identity across a rename or a reorder.
+
+Rules an author declared in `dq.rules` used to be parked in the `fluidExtras` custom property
+and nowhere else, so a consumer running the published contract's quality saw only the checks
+forge had synthesised. They are now exported as ODCS quality for the four types ODCS can
+express:
+
+| `dq.rules` type | Exported as |
+|---|---|
+| `completeness` | library metric `nullValues`, dimension `completeness` |
+| `uniqueness` | library metric `duplicateValues`, dimension `uniqueness` |
+| `valid_values` | library metric `invalidValues`, dimension `conformity` |
+| `freshness` | a `text` rule — ODCS has no library metric for it |
+
+The other four — `accuracy`, `schema`, `anomaly_detection` and `drift_detection` — have no
+executable ODCS form, so each is **logged by name with the reason** and still carried in
+`fluidExtras`. Silence was the bug; a rule with no ODCS form is now reported rather than
+dropped without a word.
+
+### `fluid odps import`
+
+The command's own help said it "always emits one validated FLUID contract" while never running
+the validator, so importing its own export printed a success tick and wrote a file
+`fluid validate` then rejected with ten errors. It now validates before writing:
+
+```bash
+fluid odps import product.odps.yaml -o contract.fluid.yaml
+```
+
+A document that does not produce a valid contract fails, names the problem and **writes
+nothing**. `--lenient` writes it anyway — note that this widens a flag that already existed:
+it used to mean only "downgrade output-port resolution failures to warnings", and it now also
+means "write the contract even when it does not validate".
+
+The pass-through bucket the importer writes for everything ODPS carries that FLUID has no field
+for moved from `metadata.odps_passthrough` to the root `extensions.odps`. `metadata` is
+`additionalProperties: false`, so the old address could never validate; root `extensions` is
+the one object the FLUID schema declares open. Root `extensions` arrived in FLUID 0.7.3, so an
+import whose source document records an older `fluidVersion` warns and declares the default
+instead, because the older schema cannot hold the bucket.
+
+### The breaking one: two ports under one contract id
+
+A contract whose `exposes[].contract.id` collides with a sibling's **resolved** id — pinned or
+derived — now fails `fluid validate` and every export door. The check compares the ids the
+exporters will actually mint, so a pin equal to a sibling's derived `<productId>.<exposeId>`
+is caught too:
+
+```bash
+fluid validate contract.fluid.yaml
+```
+
+The message names both exposes and the shared id, and the advice fits which side pinned it:
+change the `contract.id` where one is pinned, rename the expose where the id is derived.
+
+That export used to succeed. It wrote one per-port ODCS document for two output ports, so the
+losing port's schema was absent from the bundle and both ports resolved to the survivor — and
+`fluid generate artifacts` sealed it under a MANIFEST merkle root. The failure is the fix.
+
 ## From 0.15.x to 0.16
 
 Release notes: [`0.16.0`](./RELEASE_NOTES_0.16.0.md) (covers `0.15.1` to `0.16.2`).