Repository navigation
docs: the seven new 0.7.6 preview fields, and the exported ODCS/ODPS artifact changes (forge-cli #753) - #167
Merged
Merged
Conversation
forge-cli #753 adds seven optional fields to the 0.7.6 preview schema, for things a contract could not previously say. All seven need `fluidVersion: "0.7.6"`, all are additive, and a contract that sets none of them validates and exports byte for byte as before. `reference/preview-fields.md` is the page that maps the preview, so the seven go there as their own section with a table and the detail each one needs. The generated 0.7.6 delta cannot carry them yet — it is rendered from the schema bundled in the pinned CLI — and the section says so rather than leaving a reader to wonder why the rows are missing. Written from `fluid_build/schemas/fluid-schema-0.7.6.json` on `origin/fix/odps-export-import-defects`, whose field descriptions carry the reasoning, and checked against the exporters that read each field. Three have a caveat that belongs in the open rather than in a footnote: - `metadata.team.members[].email` is exported as a `fluidEmail` custom property, because there is no Bitol field to export it as: all three vendored schemas (ODCS v3.1.0, ODPS v1.0.0, ODPS v1.1.0) define `TeamMember` with `additionalProperties: false` and none has `email`. `username` is the only required member field. - `exposes[].contract.id` has no pattern, only `minLength: 1`, so a catalog can supply a URN or a UUID. Two exposes resolving to one id — both pinned, or one pinning what the other derives — now fail `fluid validate` and every export. - `binding.serialization` is the only thing that moves a Kafka server's ODCS `format` off `kafka_topic`; the two `-confluent` spellings collapse onto the wire format they carry, and the block rides along in `fluidExtras` so the registry is not forgotten. A contract that does not declare it exports exactly as before. A column's `quality` gets the fuller treatment, because the two things that surprise are not in its one-line summary: it REPLACES the derived rules for that column rather than adding to them, and rules parked in the importer's pass-through bucket are dropped by a FLUID → ODPS → FLUID cycle, so they belong on the column. The key table is the schema's own list, with which `type` each key applies to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six changes to exported documents, each a fix for output that was wrong, and each with old values that may be baked into whatever reads them. So they go in the upgrade guide, as the DuckDB 2.x notes did — but above the per-release checklists rather than below, because this is the section a reader has to see before upgrading rather than one they reach by working through their version. Leading with the one that is urgent: a Kafka column's `physicalType` was silent data corruption. Kafka had no physical-type table, so the exporter fell back to the ODCS logicalType, and ODCS defaults an unqualified `integer` to `i32` — so a tool deriving an Avro schema read 32 bits for a column a producer such as Flink `BIGINT` writes as 64. The page says plainly that anything building Avro schemas from these exports should treat every 64-bit column as having been wrong and re-register. The rest, with what to do and how to check each: - A per-port ODCS contract is named after its port (title, else `exposeId`), with the product name in a `fluidProductName` custom property. The ODCS `id` is unchanged and a whole-contract export is untouched, but a single-expose product's name changes too whenever the port's title differs from the product's — deliberately, since a special case for one port would change the name the day a second is added. - ODPS `inputPorts[].contractId` is `<productId>.<exposeId>`, the id something actually publishes, and a `consumes[].versionConstraint` that used to be dropped rides in `fluidVersionConstraint`. - A Kafka server's ODCS `format` becomes a message format only once the contract declares `binding.serialization`; the page states that the `json` default behind it is pre-existing, so the no-op case is genuinely byte-identical. - Derived quality rules carry a stable, derived `id`, and declared `dq.rules` export as ODCS quality for the four types ODCS can express. The other four are logged by name and still ride in `fluidExtras`; the table says which is which, because "some rules export" is not actionable. - `fluid odps import` validates before writing and writes nothing on failure, and its pass-through bucket moved to root `extensions.odps` because `metadata` is closed and the old address could never validate. `--lenient` is noted as a widened flag, not a new one. - The breaking case: two ports resolving to one contract id now fail everywhere, where the export used to succeed and lose one port's schema under a MANIFEST merkle root. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion docs for Agenticstiger/forge-cli#753 — the seven new 0.7.6 preview fields, and the exported-artifact changes a consumer needs before upgrading. Branched off
agenticstiger/main, one commit per group.Everything here was read out of the code and the schema on
origin/fix/odps-export-import-defects, not out of the PR description or the changelog. That turned up four places where those two say something other than what the code does; they are in Premise corrections below, and three of them changed what I wrote.1. The seven preview fields
They go in
reference/preview-fields.md— the page whose whole job is mapping the 0.7.6 preview — as a new section with a table and the detail each field needs. The generated 0.7.6 delta cannot carry the rows yet, since it is rendered from the schema bundled in the pinned CLI, and the section says so instead of leaving a reader to wonder why they are missing.I verified the count and the shape by diffing the 0.7.6 schema on the branch against
origin/main's, rather than trusting either summary. Seven distinct additions, 28 new property paths, 2 changed (descriptionandmetadata, each gaining a member), 0 removed:metadata.teamteam,team.name,team.members[]+$defs.teamMember.{username,name,role,email}description$defs.descriptionObject.{purpose,usage,limitations}, anddescriptionbecomes aoneOfconsumes[].contractId$defs.consumeRef.contractIdexposes[].contract.id$defs.exposeContract.idbinding.location.host$defs.bindingLocation.hostbinding.serialization$defs.binding.serialization{,.value,.key}quality$defs.column.quality[]+ 11 item keysYour three caveats are all on the page, and all three check out:
members[].emailcannot be a BitolTeamMemberfield. Confirmed by reading all three vendored schemas: ODCS v3.1.0, ODPS v1.0.0 and ODPS v1.1.0 each defineTeamMemberwithadditionalProperties: false, and none has anemailproperty. It exports as afluidEmailcustom property (odcs/mappers/team.py:203). The page also notes thatusernameis the only required member field and thatroleis free-form, since Bitol does not constrain it.exposes[].contract.idhas no pattern, onlyminLength: 1, so a catalog's own id format is the catalog's business. The page says that outright rather than leaving a reader to infer it from the absence of a constraint.binding.serializationis the only thing that moves a Kafka server's ODCSformatoffkafka_topic, and a contract that has not adopted it exports exactly as before. See the fourth premise note for the one nuance I added.A column's
qualitygets more than a table row, because the two things that surprise are not in its summary: setting it replaces the derived rules for that column rather than adding to them, and rules parked in the importer's pass-through bucket are dropped outright by a FLUID → ODPS → FLUID cycle (the ODPS importer merges a resolved port'scontractblock into its output-port stub and discards the sub-import's rootextensions), so they belong inline on the column. The key table is the schema's own list, with whichtypeeach key applies to.2. The exported-artifact changes
A new Read this first if you consume the exported artifacts section in
upgrading.md, with the Change / What to do / How to check table the rest of that page uses — same treatment as the DuckDB 2.x notes in #166, with one deliberate difference: it sits above the per-release checklists rather than below them. These are not changes a reader reaches by working through their version; they are changes they have to see before they upgrade at all, which is how #753's own description frames them.Leading with the urgent one. The Kafka
physicalTypechange was silent data corruption, and the page says so in those words. Kafka had no physical-type table, sofluid_to_physicalfell back to the ODCS logicalType, and ODCS's own schema defaults an unqualifiedintegertoi32— so a tool deriving an Avro schema read 32 bits for a column a producer such as FlinkBIGINTwrites as 64. The registered schema and the data disagreed with nothing to signal it. The page tells a reader to treat every 64-bit column in such an export as having been wrong and to re-register, and gives the integer family split by width from the new table (int/integer/int2/int4/serial→int;bigint/long/int8/int64/longint/bigserial→long).Then the other five, each with what to do and how to check:
nameis the port'stitle, falling back toexposeId, with the product name influidProductName. Theidis unchanged and a whole-contract export is untouched — but a single-expose product's name changes too whenever the port's title differs from the product's, and the page explains why there is deliberately no single-port special case (it would change the name the day a second port is added).inputPorts[].contractIdis<productId>.<exposeId>, the id something actually publishes, and aconsumes[].versionConstraintthat used to be dropped rides influidVersionConstraint.format, with the no-op case stated precisely.id(so two renders are not a spurious diff), and declareddq.rulesexport as ODCS quality for the four types ODCS can express. The page gives the mapping as a table —completeness→nullValues,uniqueness→duplicateValues,valid_values→invalidValues,freshness→ atextrule — and names the other four, which are logged by name and still ride influidExtras. "Some rules export now" is not something a reader can act on.fluid odps importvalidates before writing and writes nothing on failure, and the pass-through bucket moved to rootextensions.odps. The page adds a constraint the summaries do not mention: rootextensionsarrived in FLUID 0.7.3, so an import whose source records an olderfluidVersionwarns and declares the default instead, because the older schema cannot hold the bucket (odps_standard/provider.py:1057).Premise corrections
1. The changelog says six fields and lists seven. The
### Addedheading on the branch reads "Six optional fields in the 0.7.6 preview schema" and then describes seven. Your message and the PR description both say seven, and the schema diff above confirms seven. Nothing in the docs is affected — I counted from the schema — but the changelog bullet is wrong in the PR as it stands.2.
--lenientis not a new flag. It exists in 0.18.1 and onorigin/main, where its help reads "Downgrade output-port resolution failures to warnings (input ports are always lenient)". The branch widens it: the help becomes that plus "and write the FLUID contract even when it does not validate against the FLUID schema". So "--lenientrestores the old behaviour" is right about the effect and wrong about the flag's provenance — a reader who grep'd their scripts for a new flag would find it already there, doing something else. The page says it is a widened flag carrying two meanings, not a new one.3. The schema's own text cites a command that does not exist. The column-
qualitydescription influid-schema-0.7.6.jsonsays "fluid generate odcs --validatealready checks the emitted document against it". There is nofluid generate odcs:generatetakestransformation,speed-transformation,dbt,schedule,ci,standard,artifacts,dbt-tests,iac,vector. The real command isfluid odcs validate, which exists on bothorigin/mainand the branch. This one matters beyond this PR:scripts/gen_contract_reference.pyrenders schema descriptions verbatim intodocs/reference/contract-0.7.6-preview.md, so once the pin moves that sentence publishes as written. My page saysfluid odcs validate; the schema text wants the same fix in #753.4. The Kafka
jsondefault is pre-existing, and the changelog can be read as saying otherwise. The changelog argues against falling back to ODCS'sjsondefault — "would swap a meaningless token for a confident claim that is wrong for every Avro topic" — which reads as though forge never defaultsformat. It does:details.setdefault("format", "json")is inodcs/mappers/servers.pyonorigin/maintoo, unchanged by the branch, and it applies when a contract sets neitherbinding.formatnorbinding.serialization. The no-op claim still holds exactly, because the default is not new. The page states it, so a reader comparing exports is not surprised by ajsonthey thought this release introduced.One place where your phrasing was the accurate one and the changelog's was not. You wrote "two exposes resolving to one contract id"; the changelog's
### Addedbullet says "two exposes that pin the same contract id". The code is yours:contract_id_collisionscompares the ids the exporters will mint, so pinned-vs-pinned, pinned-vs-derived and (defensively) derived-vs-derived are all caught, and_collision_messagehas a different remedy for each. The### Fixedentry and the PR description describe the broader behaviour, so only that one bullet undersells it. The docs describe the resolved-id rule.Merge safety
All three open docs PRs are independent, checked rather than assumed:
The combined
upgrading.mdholds all three sections in a sensible order — "Read this first" above the checklists, "Changes in 0.20.0" and "DuckDB 1.x to 2.x" below them. They can land in any order.Verified locally
Reproduced from the workflow files, on python3.12 with
data-product-forge==0.18.1in a venv:check_cli_docs.py --version-onlycheck_cli_docs.pycheck_providers.pygen_contract_reference.py --checknpm run docs:buildnode scripts/check-dist-links.mjsEvery
fluid …invocation I put in a fenced block was checked against the pinned 0.18.1 CLI before writing it, which is what the flag oracle enforces — so nothing new from #753 is in a fence.fluid odcs export --envandfluid odps-bitol export --env, which #753 adds, are mentioned only in prose for that reason.Heading anchors are the one link class no gate here covers, so I read the five cross-references out of the built HTML:
read-this-first-if-you-consume-the-exported-artifacts,the-email-caveat,pinning-a-contract-id,declaring-the-wire-format,column-quality-rulesall resolve.docs/.vuepress/cli-version.jsonis untouched, anddocs/reference/contract-0.7.6-preview.mdis not regenerated — the generator refuses unless the installed package equals the pin, so that is the post-release follow-up.Follow-ups
contract-0.7.6-preview.md, and swap the two "the next release after0.22.0" admonitions (one inupgrading.md, one inpreview-fields.md) for the number. The regeneration is also what will finally give that pageengine: flinkandtrigger.type: continuous.fluid generate odcs --validatecommand name in the schema description, and the### Addedcollision bullet.--env-on-export addition, which is adocs/cli/reference edit once the release exists.