fix: exported lineage named no upstreams, claimed an unset version, and shipped unexecutable quality SQL by fas89 · Pull Request #756 · Agenticstiger/forge-cli

fas89 · 2026-10-27T08:55:09Z

Two defects found by feeding a v0.23.0 export to a lineage resolver and a data-quality engine, plus
the consequences of fixing the first one.

What was wrong

An ODPS inputPorts[].version was always 1.0.0. It came from the consume's own version or
the provider default, and a consume that states a range (versionConstraint: ^2.1) has neither — so
every such export asserted a pin the contract never made, and for a product whose 1.x line is
retired, a version that no longer exists. The range's lowest accepted version is now resolved
(^2.1 → 2.1.0, ~1.4 → 1.4.0, >=3.0.0 → 3.0.0). A range the resolver cannot read falls
back to the old constant rather than guessing: ODPS v1.0.0 has version in InputPort's required
so the field cannot be omitted, and a wrong exact version in exported lineage reads as a fact
somebody established. The refusals include compound ranges, wildcards, >1.0.0 (whose true lower
bound semver gives no way to name), non-ASCII digits, and leading zeros — ^01.2 is schema-valid
since $defs.semverRange carries no pattern, but 01.2.0 matches neither $defs.semver nor any
resolver.

An exported output port named none of its upstreams. outputPorts[].inputContracts was
populated only from an imported document's pass-through, so a FLUID-authored contract exported
output ports with no lineage at all and a catalog reading dependencies there saw a product with
none. The entries are now derived from the document's own resolved input ports, carrying exactly the
id and version Bitol's $defs.InputContract allows, read from those ports rather than
recomputed so the two halves of one document cannot disagree.

A quality expression exported as SQL no engine could execute. ODCS defines the sql rule's
query as an individual query returning a value that can be compared, and
exposes[].contract.quality[].expression is a row predicate — it was emitted verbatim, so an engine
got preferred_customer_flag IN ('Y', 'N') with mustBe: true and nothing to run. It is now a
counting query compared to zero, against the ${table} placeholder from ODCS's own example. The
negation is (…) IS NOT TRUE, not NOT (…): under three-valued logic a NULL row makes the naive
form NULL and counts no failure, though that row's predicate did not hold.

The consequences of the second fix

Deriving lineage put a product-to-product dependency in a payload it had been deliberately removed
from. Both Data Mesh Manager publish paths strip product-consume inputPorts, because Entropy draws
that lineage from Access agreements and a named upstream would have to be mirrored as a SourceSystem,
rendering duplicate graph nodes. Neither stripped the new key.

The strip now lives once, as remove_output_port_input_contracts beside the derivation that creates
it. The two paths strip by different rules — a missing sourceSystemId in the registrar, a canonical
product-consume match in the provider — so each passes the ports it dropped. Only consume-derived
lineage goes: an expects[] port is dropped by the same sourceSystemId rule but its expose replays
the upstream document's own inputContracts, and nothing else in the payload records that.

Fixed in passing, because this change would have doubled it: when two consumes[] share an
exposeId the exporter renames the second, and the match was by name — missing a productId tail
containing __ of its own (silver.crm__core → crm__core__customers) and the numeric-suffix
fallback (customers_2, no __ to find). Both published a port Entropy cannot resolve. The match
now also tests the contract id the exporter stamps on the port, via the exporter's own helper rather
than a second copy of the convention.

Added

consumes[].contractVersion, in the 0.7.6 preview schema only — 0.7.5's $defs.consumeRef is
additionalProperties: false, so adding it there would change validation for shipped contracts. It
is the exact upstream version this product was built against, within versionConstraint, and wins
over the resolved lower bound.

Reuse

Checked locally first: schema_manager.VersionConstraint rejects the caret outright and is the
FLUID schema version type, so pointing it at an upstream product's version would overload it and
put --min-version at risk. semver, semantic_version and packaging resolve in a dev venv but
none is a declared dependency of this project, and the core is kept lean — so the range reader is
deliberately not a library, and reads only the forms $defs.semverRange documents.

Borrowed rather than invented: the ${table} placeholder and the compare-a-returned-value shape come
from ODCS's own $defs.DataQualitySql.query example; the partial-version rule is npm's range
grammar, which is where $defs.semverRange's examples come from. The provider/render test helpers
are reused from tests/providers/test_odps_v11_type.py rather than copied, the way
test_duckdb_sandbox_workspace_root.py reuses a sibling module's fixtures.

Validation

Driven through the CLI, not just the test suite: fluid validate → fluid odps export →
fluid odps validate on a 0.7.6 contract. The emitted inputPorts[0].version is 2.1.0, the output
port carries inputContracts: [{id: silver.crm.core.customers, version: 2.1.0}], and the document is
valid against Bitol ODPS v1.0.0. The ODCS query was read back out of the emitted YAML and executed
on DuckDB: 0 failing rows, then 1 after a bad row, then 2 after a NULL row — while the naive
NOT (…) form returns 1 and misses the NULL.

Every fix is pinned by a test that fails when the fix is reverted; each was checked by neutralising
the change and confirming which tests fail. One test that looked like coverage was vacuous — its
three consumes shared a contractId, so the exporter's dedup collapsed them and no rename happened —
and was replaced with a case that genuinely produces customers_2.

Four full-suite runs, the last at this commit: 28,300 passed, 9 failed, the same 9 every time.
Eight fail identically on clean origin/main (stale dlt, local Snowflake config, a live claude
CLI test, $TMPDIR shape); the ninth makes a real boto3 call to a loopback listener and passes in
isolation and per-file on both worktrees. None is in a file this branch touches. ruff,
black --check (24.10.0, the CI pin) and detect-secrets 1.5.0 are clean whole-tree, and
tests/perf/test_startup_budget.py still passes.

ripwire --quality-delta exits 2 on clean origin/main as well (14 pre-existing gating findings in
untouched files). This branch adds one: a 30-token duplication between the version acceptance test
and an unrelated model-name test — two tests that each render once and assert one field.

Review

Four adversarial rounds, each finding reproduced by execution before being fixed.

Security, two rounds: NO HIGH OR MEDIUM FINDINGS. The range reader was fuzzed (60k, then
400k inputs) and returns only ^\d+\.\d+\.\d+$ or None. The quality field already carried
arbitrary SQL into an exported artifact verbatim before this branch, so that is pre-existing; the
new first-token match is a strict narrowing — 0 inputs reach the verbatim branch that did not
before.
Defects, three rounds: 7 real findings, all fixed — the startswith matching column names,
the trailing ;/-- breaking the wrapper, leading zeros, the wrong DMM path, an over-reach that
deleted imported lineage, and the renamed-port leak.
World-class bar: no FAIL. Seven concerns, all addressed, including two docstring claims that
did not hold and a ### Changed section that omitted the ODCS reshaping.

Known follow-ups

A consume with no productId — schema-invalid, fluid validate rejects it — has its derived lineage
stripped with no Access agreement to replace it. Several cases the audit enumerated are still
uncovered: the SDP-plus-consume interaction, the explicit-contractId path, and the four
malformed-input guards in remove_output_port_input_contracts. Both are filed on the board.

Documentation

Docs PR linked: docs: 0.24.0 — contractVersion, and the lineage a DMM publish strips forge_docs#170

Adds consumes[].contractVersion to the 0.7.6 preview tables and records, on the Data Mesh Manager
page, that a stripped product-consume input port's derived lineage entry goes with it. Also bumps
cli-version.json to 0.24.0 — which is why it is a draft: the docs drift gate runs fluid at the
pinned version, so it cannot go green until data-product-forge==0.24.0 is on PyPI. Merge order is
this PR, then the tag, then the docs PR.

ODCS defines the `sql` rule's `query` as an individual query returning a value that can be compared, and FLUID's `exposes[].contract.quality[].expression` is a row predicate. It was emitted verbatim, so a DQ engine reading an exported contract was handed `preferred_customer_flag IN ('Y', 'N')` with `mustBe: true` and nothing it could execute. A predicate now becomes a counting query compared to zero, against the `${table}` placeholder ODCS's own example uses. The negation is `(…) IS NOT TRUE`, not `NOT (…)`: under three-valued logic a NULL row makes the naive form NULL and counts no failure, although that row's predicate did not hold — verified against DuckDB, where the naive form misses the NULL row and this one does not. The whole-number comparison follows OpenMetadata's convention for the same shape of check. An expression already written as a `SELECT` or `WITH` query passes through unchanged.

…ersion Two defects in the same export, both found by feeding a 0.23.0 export to a lineage resolver. `inputPorts[].version` came from the consume's own `version` or the provider's `1.0.0` default, and a consume that states a range (`versionConstraint: ^2.1`) has neither — so every such export asserted `version: 1.0.0`, a pin the contract never made and, for a product whose 1.x line is retired, a version that no longer exists. The lowest version the range accepts is now resolved (`^2.1` → `2.1.0`, `~1.4` → `1.4.0`, `>=3.0.0` → `3.0.0`), reading only the forms `$defs.semverRange` documents. A range the resolver cannot read — a compound, a wildcard, or `>1.0.0`, whose true lower bound semver gives no way to name — falls back to the old constant rather than guessing: ODPS v1.0.0 has `version` in `InputPort`'s `required` so the field cannot be omitted, and a wrong exact version in exported lineage reads as a fact somebody established. Deliberately not a semver library: `semver`, `semantic_version` and `packaging` are all in the resolved environment but none is a declared dependency and the core is kept lean, while `schema_manager.VersionConstraint` rejects the caret outright and is the FLUID *schema* version type. `outputPorts[].inputContracts` was populated only from an imported document's pass-through, so a contract written in FLUID exported output ports with no lineage at all and a catalog reading dependencies there saw a product with none. The entries are now derived from the document's own resolved input ports, carrying exactly the `id` and `version` Bitol's `$defs.InputContract` allows, read from those ports rather than recomputed so the two halves of one document cannot disagree. An imported document's own `inputContracts` still win, and `expects[]` — the carrier an imported document's input ports arrive on — is excluded, so import then export stays a fixed point and never adds a dependency the author did not write. Adds `consumes[].contractVersion` to the 0.7.6 preview schema: the exact upstream version this product was built and tested against, within `versionConstraint`. It wins over the resolved lower bound. `_consume_input_port` is extracted so `_build_input_ports` does not grow.

…arry `str.isdigit()` is true for Arabic-Indic digits and for superscripts, so `versionConstraint: ^١.٢` was read as a range and its "lower bound" emitted verbatim into an exported `inputPorts[].version`. The check now requires `isascii()` as well, so an unreadable range falls back to the constant the way every other unreadable form does.

…paths All three were reproduced by execution before the fix and are pinned by tests that fail when it is reverted. `is_query` tested `expression.upper().startswith(("SELECT", "WITH"))`, which has no word boundary, so a predicate on any column whose name merely begins with those letters — `selected_at IS NOT NULL`, `with_tax > 0`, `withholding_amount >= 0` — was classified as an already-written query and exported verbatim with `mustBe: true`. For a whole class of columns that silently reinstated the unexecutable fragment this change exists to remove. The first token is now matched with `\b`. The wrapper was also built on one line, so a predicate ending in `;` produced `WHERE (amount > 0;) IS NOT TRUE` and one ending in a `-- comment` swallowed the closing paren and the comparison after it. Both are parse errors in DuckDB. A trailing `;` is now dropped and the predicate sits on its own line. The emitted query survives the YAML round-trip exactly and executes. `_range_lower_bound` accepted numeric components with a leading zero, so `versionConstraint: ^01.2` — schema-valid, since `$defs.semverRange` carries no pattern — resolved to `01.2.0`, which matches neither `$defs.semver` nor any resolver. Pre-change these exported `1.0.0`: wrong, but resolvable. They now fall back like every other unreadable range. `outputPorts[].inputContracts` is no longer derived from `builds[].properties.source`. That path synthesizes its `contractId` as `<kind>://<database>/<stream>` precisely because the source has no contract at all, to populate the input-port field ODPS v1.0.0 requires. `inputContracts` is optional, so naming an id no catalog can resolve is worse than naming nothing — the failure `_input_port_contract_id` already warns about. An SDP exported two such entries per product and nothing covered the branch; it is covered now. Also corrects two docstring claims that did not hold: `semver` and `semantic_version` resolve in some environments and not others, so the sentence now rests only on what is true of the project (neither is a declared dependency), and semver.org defines no ranges, so the partial-version rule is credited to npm's range grammar, which is where `$defs.semverRange`'s examples come from. `_consume_input_port` is renamed `_consume_to_input_port`, the shape this file spells for a one-to-one mapper without exception.

…th it `remove_odps_product_consume_input_ports` strips product-to-product consumes from the ODPS payload before publish, because Entropy draws that lineage from Access resources — keeping the consumes as input ports would require mirroring each upstream as a SourceSystem and render duplicate graph nodes. Deriving `outputPorts[].inputContracts` states the same dependency a second time in the same document, and nothing stripped it, so `fluid publish --target datamesh-manager` began POSTing product-to-product lineage on the one payload it had been deliberately removed from. A removed port's lineage entry now goes with it, and the key is dropped when nothing is left. An explicit source-system consume keeps its entry, because its input port is kept too.

…export The 0.7.6 root description's "NEW in v0.7.6" list named its six sibling opt-in fields and not `contractVersion`, so an author reading the preview summary would not learn the field exists. It also now says what `fluid validate` does not check: that the value falls inside `versionConstraint`. `consumes_to_canonical_ports`' docstring claims to describe every field it returns and no longer did — `contract_version` is in the dict. Listing it also disambiguates it from `get_contract_version` in the same module, which returns the FLUID *schema* version. The changelog's `### Changed` listed only the ODPS keys and omitted the ODCS `sql` rule reshaping — `query`, `mustBe: true → 0` and `description` all change for every expression rule, which is the value most likely already baked into a consumer; our own test had to be rewritten for it. The section now opens with the upgrade warning 0.23.0 uses and tells the reader what to do.

… path The previous commit fixed `DataMeshManagerProvider`, which backs `fluid datamesh-manager publish`. That is the secondary path. The canonical one — `fluid publish --target datamesh-manager` and `catalog.register: [datamesh_manager]` — goes through `DataMeshManagerRegistrar`, whose own module notes DMM has "exactly one publish path". It filters `inputPorts` to those carrying a `sourceSystemId`, dropping product-to-product ports, and never touched `inputContracts` — so the derived lineage still named those upstreams on the payload PUT to `/api/dataproducts/{id}`, and the changelog's claim about that command was false as written. The strip now lives once, as `remove_output_port_input_contracts` in the ODPS mapper that owns the derivation, so the two cannot drift. The two publish paths strip by different rules — a missing `sourceSystemId` in the registrar, a canonical product-consume match in the provider — so each passes the ports it dropped and the helper reads the `contractId`s off them. The registrar imports it inside the function, because that module is imported for its `register_catalog_backend` side effect and must stay off the light CLI's cold path; `tests/perf/test_startup_budget.py` still passes. A test drives the registrar end to end through `register_payload` with the HTTP layer mocked and asserts the dataproduct body names no upstream product. It fails when either the registrar call or the shared helper is reverted. Also corrects two docstrings: the canonical-shape block had `# semverRange` displaced onto the new `contract_version` line, which documented an exact `$defs.semver` field as a range, and `_build_input_ports` said `_input_port_contract_id` "warns about" a failure it only describes — it holds no logger.

… was for The previous commit drove the strip from every input port the registrar's filter dropped. That filter drops anything without a `sourceSystemId`, which catches an `expects[]` port as well as a product consume — and an `expects[]` port's expose replays the upstream document's OWN `inputContracts` verbatim through the ODPS pass-through. So publishing an imported product deleted the only record of its upstream: Access agreements are built from `consumes[]`, and that document has none, so nothing replaced it. Reproduced on the real wire path through `register_payload` with the HTTP layer mocked: with the strip as written the `vendor.customers_raw` entry was gone, and with it disabled the entry survived. `consume_input_port_contract_ids` now names the ids a `consumes[]` export actually produces, and only ports matching those are passed to the strip. Three cases are pinned: a consume's derived lineage is dropped, an `expects[]` port's own lineage is kept, and a contract with both keeps the pass-through, since that wins in the mapper. The provider path never had this reach — its `_is_product_consume` matches canonical product consumes only. Also corrects the deferred import's comment. The claim that this module is on the light CLI's cold path is false in both directions: `fluid --help` already loads `providers.odps_standard`, and `catalog_registrars` is not imported on that path at all. The import stays deferred for the reason that does hold — building the backend registry should not pull the ODPS provider in with it.

…lish Found while covering the lineage strip, and fixed here rather than carried because deriving `outputPorts[].inputContracts` would have doubled it: the leaked port and its mirrored lineage entry. When two `consumes[]` share an `exposeId` the exporter renames the second, and `_is_product_consume` matched the renamed port by name. That misses two of the three rename shapes: a `productId` tail containing `__` of its own (`silver.crm__core` → `crm__core__customers`, where splitting at the first `__` reads the boundary wrong) and the numeric-suffix fallback (`customers_2`, which carries no `__` at all). Either one published an input port Entropy cannot resolve without mirroring the upstream product as a SourceSystem — the duplicate graph node the strip exists to prevent. Reproduced through `apply(..., dry_run=True)` on contracts `fluid validate` accepts. The match now also tests the contract id the exporter stamps on the port, via `input_port_contract_id` — the exporter's own helper, made public rather than copied, since the id convention already carries a comment about not keeping two copies in step. That id is `{productId}.{exposeId}` or one the author pinned, so it cannot catch a source-system port that merely shares an upstream, and it is independent of the name the rename disturbs. Measured before settling on it: the predicate alone closes all three rename shapes, so the name checks are left exactly as they were instead of also being rewritten. An earlier draft changed the `__` split to a suffix search as well; that turned out to be redundant and was dropped, and the test that appeared to cover it was vacuous — its three consumes shared a `contractId`, so the exporter's dedup collapsed them to one port and no rename happened. The case now uses three consumes sharing an `exposeId` and a `productId` tail, which genuinely produces `customers_2`, and both leak shapes fail when the predicate is reverted. Four more cases the audit found uncovered are pinned too: a stated pass-through lineage surviving when the product also consumes, a stated entry naming the consumed contract still being removed, and a source-system consume keeping both its port and its lineage.

github-actions · 2026-10-27T08:55:24Z

📄 Documentation Reminder

This PR appears to be missing a documentation reference. Our docs live in a separate repo.

Please update the PR description with one of:

Link a docs PR — check the "Docs PR linked" box and paste the URL
Mark as no docs needed — check "No docs needed" with a justification
Acknowledge docs TODO — check "Docs TODO" and create the docs PR before merge

See the Contributing Guide for details.

fas89 added 10 commits October 27, 2026 06:57

docs(changelog): 0.24.0 — exported lineage and executable quality SQL

06a5b70

github-actions Bot added the needs-docs Pull request needs a linked docs update or justification label Oct 27, 2026

github-actions Bot added provider Changes or requests related to providers tests Test coverage or test infrastructure changes labels Oct 27, 2026

fas89 mentioned this pull request Oct 27, 2026

docs: 0.24.0 — contractVersion, and the lineage a DMM publish strips Agenticstiger/forge_docs#170

Merged

github-actions Bot removed the needs-docs Pull request needs a linked docs update or justification label Oct 27, 2026

fas89 merged commit 0d55f51 into main Oct 27, 2026
34 checks passed

fas89 deleted the fix/odps-lineage-and-quality-sql branch October 27, 2026 09:26

Navigation Menu

Repository navigation

fix: exported lineage named no upstreams, claimed an unset version, and shipped unexecutable quality SQL - #756

fix: exported lineage named no upstreams, claimed an unset version, and shipped unexecutable quality SQL#756
fas89 merged 10 commits into
mainfrom
fix/odps-lineage-and-quality-sql

fas89 commented Oct 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fas89 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

The consequences of the second fix

Added

Reuse

Validation

Review

Known follow-ups

Documentation

Uh oh!

github-actions Bot commented Oct 27, 2026

📄 Documentation Reminder

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fas89 commented Oct 27, 2026 •

edited

Loading