Skip to content

docs: the seven new 0.7.6 preview fields, and the exported ODCS/ODPS artifact changes (forge-cli #753) - #167

Merged
fas89 merged 2 commits into
mainfrom
docs/odps-odcs-export-fields
Oct 27, 2026
Merged

fas89 merged 2 commits into
mainfrom
docs/odps-odcs-export-fields

Conversation

@fas89

@fas89 fas89 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Companion docs for Agenticstiger/forge-cli#753 — the seven new 0.7.6 preview fields, and the exported-artifact changes a consumer needs before upgrading. Branched off agenticstiger/main, one commit per group.

Everything here was read out of the code and the schema on origin/fix/odps-export-import-defects, not out of the PR description or the changelog. That turned up four places where those two say something other than what the code does; they are in Premise corrections below, and three of them changed what I wrote.

1. The seven preview fields

They go in reference/preview-fields.md — the page whose whole job is mapping the 0.7.6 preview — as a new section with a table and the detail each field needs. The generated 0.7.6 delta cannot carry the rows yet, since it is rendered from the schema bundled in the pinned CLI, and the section says so instead of leaving a reader to wonder why they are missing.

I verified the count and the shape by diffing the 0.7.6 schema on the branch against origin/main's, rather than trusting either summary. Seven distinct additions, 28 new property paths, 2 changed (description and metadata, each gaining a member), 0 removed:

Field New paths
metadata.team team, team.name, team.members[] + $defs.teamMember.{username,name,role,email}
root description $defs.descriptionObject.{purpose,usage,limitations}, and description becomes a oneOf
consumes[].contractId $defs.consumeRef.contractId
exposes[].contract.id $defs.exposeContract.id
binding.location.host $defs.bindingLocation.host
binding.serialization $defs.binding.serialization{,.value,.key}
column quality $defs.column.quality[] + 11 item keys

Your three caveats are all on the page, and all three check out:

  • members[].email cannot be a Bitol TeamMember field. Confirmed by reading all three vendored schemas: ODCS v3.1.0, ODPS v1.0.0 and ODPS v1.1.0 each define TeamMember with additionalProperties: false, and none has an email property. It exports as a fluidEmail custom property (odcs/mappers/team.py:203). The page also notes that username is the only required member field and that role is free-form, since Bitol does not constrain it.
  • exposes[].contract.id has no pattern, only minLength: 1, so a catalog's own id format is the catalog's business. The page says that outright rather than leaving a reader to infer it from the absence of a constraint.
  • binding.serialization is the only thing that moves a Kafka server's ODCS format off kafka_topic, and a contract that has not adopted it exports exactly as before. See the fourth premise note for the one nuance I added.

A column's quality gets more than a table row, because the two things that surprise are not in its summary: setting it replaces the derived rules for that column rather than adding to them, and rules parked in the importer's pass-through bucket are dropped outright by a FLUID → ODPS → FLUID cycle (the ODPS importer merges a resolved port's contract block into its output-port stub and discards the sub-import's root extensions), so they belong inline on the column. The key table is the schema's own list, with which type each key applies to.

2. The exported-artifact changes

A new Read this first if you consume the exported artifacts section in upgrading.md, with the Change / What to do / How to check table the rest of that page uses — same treatment as the DuckDB 2.x notes in #166, with one deliberate difference: it sits above the per-release checklists rather than below them. These are not changes a reader reaches by working through their version; they are changes they have to see before they upgrade at all, which is how #753's own description frames them.

Leading with the urgent one. The Kafka physicalType change was silent data corruption, and the page says so in those words. Kafka had no physical-type table, so fluid_to_physical fell back to the ODCS logicalType, and ODCS's own schema defaults an unqualified integer to i32 — so a tool deriving an Avro schema read 32 bits for a column a producer such as Flink BIGINT writes as 64. The registered schema and the data disagreed with nothing to signal it. The page tells a reader to treat every 64-bit column in such an export as having been wrong and to re-register, and gives the integer family split by width from the new table (int/integer/int2/int4/serial → int; bigint/long/int8/int64/longint/bigserial → long).

Then the other five, each with what to do and how to check:

  • The per-port ODCS name is the port's title, falling back to exposeId, with the product name in fluidProductName. The id is unchanged and a whole-contract export is untouched — but a single-expose product's name changes too whenever the port's title differs from the product's, and the page explains why there is deliberately no single-port special case (it would change the name the day a second port is added).
  • ODPS inputPorts[].contractId is <productId>.<exposeId>, the id something actually publishes, and a consumes[].versionConstraint that used to be dropped rides in fluidVersionConstraint.
  • The Kafka server format, with the no-op case stated precisely.
  • Quality rules: derived rules carry a stable, derived id (so two renders are not a spurious diff), and declared dq.rules export as ODCS quality for the four types ODCS can express. The page gives the mapping as a table — completeness → nullValues, uniqueness → duplicateValues, valid_values → invalidValues, freshness → a text rule — and names the other four, which are logged by name and still ride in fluidExtras. "Some rules export now" is not something a reader can act on.
  • fluid odps import validates before writing and writes nothing on failure, and the pass-through bucket moved to root extensions.odps. The page adds a constraint the summaries do not mention: root extensions arrived in FLUID 0.7.3, so an import whose source records an older fluidVersion warns and declares the default instead, because the older schema cannot hold the bucket (odps_standard/provider.py:1057).
  • The breaking case, two ports resolving to one contract id, with what the old behaviour actually cost.

Premise corrections

1. The changelog says six fields and lists seven. The ### Added heading on the branch reads "Six optional fields in the 0.7.6 preview schema" and then describes seven. Your message and the PR description both say seven, and the schema diff above confirms seven. Nothing in the docs is affected — I counted from the schema — but the changelog bullet is wrong in the PR as it stands.

2. --lenient is not a new flag. It exists in 0.18.1 and on origin/main, where its help reads "Downgrade output-port resolution failures to warnings (input ports are always lenient)". The branch widens it: the help becomes that plus "and write the FLUID contract even when it does not validate against the FLUID schema". So "--lenient restores the old behaviour" is right about the effect and wrong about the flag's provenance — a reader who grep'd their scripts for a new flag would find it already there, doing something else. The page says it is a widened flag carrying two meanings, not a new one.

3. The schema's own text cites a command that does not exist. The column-quality description in fluid-schema-0.7.6.json says "fluid generate odcs --validate already checks the emitted document against it". There is no fluid generate odcs: generate takes transformation, speed-transformation, dbt, schedule, ci, standard, artifacts, dbt-tests, iac, vector. The real command is fluid odcs validate, which exists on both origin/main and the branch. This one matters beyond this PR: scripts/gen_contract_reference.py renders schema descriptions verbatim into docs/reference/contract-0.7.6-preview.md, so once the pin moves that sentence publishes as written. My page says fluid odcs validate; the schema text wants the same fix in #753.

4. The Kafka json default is pre-existing, and the changelog can be read as saying otherwise. The changelog argues against falling back to ODCS's json default — "would swap a meaningless token for a confident claim that is wrong for every Avro topic" — which reads as though forge never defaults format. It does: details.setdefault("format", "json") is in odcs/mappers/servers.py on origin/main too, unchanged by the branch, and it applies when a contract sets neither binding.format nor binding.serialization. The no-op claim still holds exactly, because the default is not new. The page states it, so a reader comparing exports is not surprised by a json they thought this release introduced.

One place where your phrasing was the accurate one and the changelog's was not. You wrote "two exposes resolving to one contract id"; the changelog's ### Added bullet says "two exposes that pin the same contract id". The code is yours: contract_id_collisions compares the ids the exporters will mint, so pinned-vs-pinned, pinned-vs-derived and (defensively) derived-vs-derived are all caught, and _collision_message has a different remedy for each. The ### Fixed entry and the PR description describe the broader behaviour, so only that one bullet undersells it. The docs describe the resolved-id rule.

Merge safety

All three open docs PRs are independent, checked rather than assumed:

pairwise, both orders:   #165 ↔ #166, #165 ↔ this, #166 ↔ this   →  0 conflicts
all three in one tree:   git merge-tree --write-tree             →  clean tree, 0 markers

The combined upgrading.md holds all three sections in a sensible order — "Read this first" above the checklists, "Changes in 0.20.0" and "DuckDB 1.x to 2.x" below them. They can land in any order.

Verified locally

Reproduced from the workflow files, on python3.12 with data-product-forge==0.18.1 in a venv:

Command From Result
check_cli_docs.py --version-only cli-consistency.yml OK (0.18.1)
check_cli_docs.py cli-consistency.yml OK — 69 subcommands, 1335 invocations, every version claim
check_providers.py cli-consistency.yml OK
gen_contract_reference.py --check cli-consistency.yml OK (untouched)
npm run docs:build docs-pr-check.yml exit 0, 242 pages
node scripts/check-dist-links.mjs docs-pr-check.yml Clean — 137,872 absolute references
base-prefix-check link-check.yml OK, self-test passed
lychee-source link-check.yml 0 errors over 909 unique links; canaries 0 / 2

Every fluid … invocation I put in a fenced block was checked against the pinned 0.18.1 CLI before writing it, which is what the flag oracle enforces — so nothing new from #753 is in a fence. fluid odcs export --env and fluid odps-bitol export --env, which #753 adds, are mentioned only in prose for that reason.

Heading anchors are the one link class no gate here covers, so I read the five cross-references out of the built HTML: read-this-first-if-you-consume-the-exported-artifacts, the-email-caveat, pinning-a-contract-id, declaring-the-wire-format, column-quality-rules all resolve.

docs/.vuepress/cli-version.json is untouched, and docs/reference/contract-0.7.6-preview.md is not regenerated — the generator refuses unless the installed package equals the pin, so that is the post-release follow-up.

Follow-ups

  • After the release: bump the pin, regenerate contract-0.7.6-preview.md, and swap the two "the next release after 0.22.0" admonitions (one in upgrading.md, one in preview-fields.md) for the number. The regeneration is also what will finally give that page engine: flink and trigger.type: continuous.
  • In #753: the six-versus-seven count, the fluid generate odcs --validate command name in the schema description, and the ### Added collision bullet.
  • Not written here: the pre-existing follow-ups in #753's own list are forge-cli work, not docs. The one with a docs surface is the --env-on-export addition, which is a docs/cli/ reference edit once the release exists.

fas89 and others added 2 commits October 27, 2026 03:15
forge-cli #753 adds seven optional fields to the 0.7.6 preview schema,
for things a contract could not previously say. All seven need
`fluidVersion: "0.7.6"`, all are additive, and a contract that sets none
of them validates and exports byte for byte as before.

`reference/preview-fields.md` is the page that maps the preview, so the
seven go there as their own section with a table and the detail each one
needs. The generated 0.7.6 delta cannot carry them yet — it is rendered
from the schema bundled in the pinned CLI — and the section says so
rather than leaving a reader to wonder why the rows are missing.

Written from `fluid_build/schemas/fluid-schema-0.7.6.json` on
`origin/fix/odps-export-import-defects`, whose field descriptions carry
the reasoning, and checked against the exporters that read each field.

Three have a caveat that belongs in the open rather than in a footnote:

- `metadata.team.members[].email` is exported as a `fluidEmail` custom
  property, because there is no Bitol field to export it as: all three
  vendored schemas (ODCS v3.1.0, ODPS v1.0.0, ODPS v1.1.0) define
  `TeamMember` with `additionalProperties: false` and none has `email`.
  `username` is the only required member field.
- `exposes[].contract.id` has no pattern, only `minLength: 1`, so a
  catalog can supply a URN or a UUID. Two exposes resolving to one id —
  both pinned, or one pinning what the other derives — now fail
  `fluid validate` and every export.
- `binding.serialization` is the only thing that moves a Kafka server's
  ODCS `format` off `kafka_topic`; the two `-confluent` spellings
  collapse onto the wire format they carry, and the block rides along in
  `fluidExtras` so the registry is not forgotten. A contract that does
  not declare it exports exactly as before.

A column's `quality` gets the fuller treatment, because the two things
that surprise are not in its one-line summary: it REPLACES the derived
rules for that column rather than adding to them, and rules parked in
the importer's pass-through bucket are dropped by a FLUID → ODPS → FLUID
cycle, so they belong on the column. The key table is the schema's own
list, with which `type` each key applies to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six changes to exported documents, each a fix for output that was wrong,
and each with old values that may be baked into whatever reads them. So
they go in the upgrade guide, as the DuckDB 2.x notes did — but above
the per-release checklists rather than below, because this is the
section a reader has to see before upgrading rather than one they reach
by working through their version.

Leading with the one that is urgent: a Kafka column's `physicalType` was
silent data corruption. Kafka had no physical-type table, so the
exporter fell back to the ODCS logicalType, and ODCS defaults an
unqualified `integer` to `i32` — so a tool deriving an Avro schema read
32 bits for a column a producer such as Flink `BIGINT` writes as 64. The
page says plainly that anything building Avro schemas from these exports
should treat every 64-bit column as having been wrong and re-register.

The rest, with what to do and how to check each:

- A per-port ODCS contract is named after its port (title, else
  `exposeId`), with the product name in a `fluidProductName` custom
  property. The ODCS `id` is unchanged and a whole-contract export is
  untouched, but a single-expose product's name changes too whenever the
  port's title differs from the product's — deliberately, since a
  special case for one port would change the name the day a second is
  added.
- ODPS `inputPorts[].contractId` is `<productId>.<exposeId>`, the id
  something actually publishes, and a `consumes[].versionConstraint`
  that used to be dropped rides in `fluidVersionConstraint`.
- A Kafka server's ODCS `format` becomes a message format only once the
  contract declares `binding.serialization`; the page states that the
  `json` default behind it is pre-existing, so the no-op case is
  genuinely byte-identical.
- Derived quality rules carry a stable, derived `id`, and declared
  `dq.rules` export as ODCS quality for the four types ODCS can express.
  The other four are logged by name and still ride in `fluidExtras`; the
  table says which is which, because "some rules export" is not
  actionable.
- `fluid odps import` validates before writing and writes nothing on
  failure, and its pass-through bucket moved to root `extensions.odps`
  because `metadata` is closed and the old address could never validate.
  `--lenient` is noted as a widened flag, not a new one.
- The breaking case: two ports resolving to one contract id now fail
  everywhere, where the export used to succeed and lose one port's
  schema under a MANIFEST merkle root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant