Skip to content

Re-vendor 0.7.3-0.7.6 from forge-cli v0.21.0: engine flink, row filters, masked columns, catalogAuth - #52

Merged
fas89 merged 2 commits into
open-data-protocol:mainfrom
fas89:chore/revendor-v0.21.0
Oct 19, 2026
Merged

fas89 merged 2 commits into
open-data-protocol:mainfrom
fas89:chore/revendor-v0.21.0

Conversation

@fas89

@fas89 fas89 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What

Re-vendors the schemas from forge-cli v0.21.0 (commit 91e071e2c2273e2ba06fd0deabfe42d78de82f2e), the release that adds engine: flink to the 0.7.6 preview. scripts/schema-versions.json moves from v0.18.1 to v0.21.0.

Vendored byte for byte from that commit: fluid-schema-0.7.3.json, 0.7.4, 0.7.5 and 0.7.6. specs/ and schema-diffs/ are regenerated with the locked renderer.

What changes in the schemas

0.7.6 (preview). Everything is additive (check-compat.py: 0.7.5→0.7.6 compatible, 0 breaking):

  • build(s).engine accepts flink (forge-cli#731).
  • exposes[].policy.authz.rowFilters[], columnRestrictions[].access: mask with columnRestrictions[].mask, and binding.governance.lakeFormation.maskedViews (forge-cli#700).
  • build(s).properties.sink.catalog accepts lakekeeper and polaris (forge-cli#711), and sink.catalogAuth names a REST catalog's credentials by environment variable (forge-cli#712).

0.7.3, 0.7.4, 0.7.5 and 0.7.6. Only the descriptions of the Lake Formation tags and tagDefinitions change (forge-cli#706). The same documents validate.

The overlay identity merge in forge-cli#732 is loader behaviour, not schema, so nothing here pins it.

Corpus

Without new cases, 0.7.6 mutation coverage falls to 39.9%, under the CI floor of 40. This PR adds 37 cases, each invalid one failing with exactly one error (keyword + pointer):

  • tests/0.7.6/build-consumes.json: a valid engine: flink build with its deployment settings under parameters.
  • tests/0.7.5/build-consumes.json: engine: flink is rejected (anyOf at /build/engine).
  • tests/0.7.6/acquisition.json: catalogAuth (both types, required members per type, the members each type refuses, variable names vs. tokens and ${...}, userinfo in the token URL, env: and empty values, closed set) and the two new catalogs. The existing "catalog outside the recognized set" case used polaris, which 0.7.6 now accepts; it uses hadoop instead.
  • tests/0.7.6/expose-policy.json: access: mask without mask, mask without access or with deny, the mask enum, and rowFilters (required members, minLength, name pattern, closed set).
  • tests/0.7.6/binding-governance.json: maskedViews (required members, closed set).

Mutation coverage: 0.7.5 41.0% (unchanged), 0.7.6 43.1% (518 of 1201, 130 redundant). tests/README.md carries the new figure.

Docs

  • docs/releases/0.7.6.md: table rows and new sections for row filters and masked columns, engine: flink and sink.catalogAuth, plus the access: mask bullet under column restrictions. Every example in the notes validates against the 0.7.6 schema, merged into the minimal contract where marked, and with fluid validate from data-product-forge 0.21.0. The catalogAuth example carries its own Iceberg expose, since 0.21.0 refuses an Iceberg sink with no expose bound format: iceberg.
  • docs/schema/changelog.md: the new 0.7.6 additions, and a correction entry for the Lake Formation tag descriptions.
  • One-line 0.7.6 summaries in docs/README.md and docs/releases/README.md.

Gates

All run locally on this branch:

  • conformance/run.py: 1098 cases, 1098 passed.
  • conformance/mutation_coverage.py --min-coverage 40: 41.0% (0.7.5), 43.1% (0.7.6).
  • tests/meta_test.py: 50/50.
  • scripts/check-compat.py: no breaking change.
  • scripts/check-schema-drift.py: every synced schema byte-identical to v0.21.0.
  • generate-docs.py --check and generate-schema-diffs.py --check: current.
  • conformance/check_reference.py against data-product-forge 0.21.0: 1094 agree, 0 disagree, 4 optional format behaviours not implemented.

fas89 added 2 commits October 19, 2026 21:59
…additions refuse

Vendored byte for byte from forge-cli v0.21.0 (91e071e2c2273e2ba06fd0deabfe42d78de82f2e).
0.7.6 gains engine: flink, authz.rowFilters, access: mask with its mask rule,
lakeFormation.maskedViews, the lakekeeper and polaris sink catalogs, and
sink.catalogAuth. 0.7.3-0.7.6 change only the Lake Formation tag descriptions.

Without new cases 0.7.6 mutation coverage fell to 39.9%, under the 40% floor.
36 cases bring it to 43.1%, each invalid one failing with exactly one error.
The 'sink catalog outside the recognized set' case used polaris, which 0.7.6
now accepts; it uses hadoop. 0.7.5 rejects engine: flink.

check_reference against data-product-forge 0.21.0: 1098 cases, 0 disagree.
Release notes for row filters and masked columns, engine: flink and
sink.catalogAuth; the schema changelog's 0.7.6 list; a correction entry for
the Lake Formation tag descriptions. Every example in the 0.7.6 notes
validates against the schema and with fluid validate from 0.21.0.
@fas89
fas89 merged commit 0fa98e6 into open-data-protocol:main Oct 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant