Repository navigation
Re-vendor 0.7.3-0.7.6 from forge-cli v0.21.0: engine flink, row filters, masked columns, catalogAuth - #52
Merged
Conversation
…additions refuse Vendored byte for byte from forge-cli v0.21.0 (91e071e2c2273e2ba06fd0deabfe42d78de82f2e). 0.7.6 gains engine: flink, authz.rowFilters, access: mask with its mask rule, lakeFormation.maskedViews, the lakekeeper and polaris sink catalogs, and sink.catalogAuth. 0.7.3-0.7.6 change only the Lake Formation tag descriptions. Without new cases 0.7.6 mutation coverage fell to 39.9%, under the 40% floor. 36 cases bring it to 43.1%, each invalid one failing with exactly one error. The 'sink catalog outside the recognized set' case used polaris, which 0.7.6 now accepts; it uses hadoop. 0.7.5 rejects engine: flink. check_reference against data-product-forge 0.21.0: 1098 cases, 0 disagree.
Release notes for row filters and masked columns, engine: flink and sink.catalogAuth; the schema changelog's 0.7.6 list; a correction entry for the Lake Formation tag descriptions. Every example in the 0.7.6 notes validates against the schema and with fluid validate from 0.21.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Re-vendors the schemas from forge-cli v0.21.0 (commit
91e071e2c2273e2ba06fd0deabfe42d78de82f2e), the release that addsengine: flinkto the 0.7.6 preview.scripts/schema-versions.jsonmoves from v0.18.1 to v0.21.0.Vendored byte for byte from that commit:
fluid-schema-0.7.3.json,0.7.4,0.7.5and0.7.6.specs/andschema-diffs/are regenerated with the locked renderer.What changes in the schemas
0.7.6 (preview). Everything is additive (
check-compat.py: 0.7.5→0.7.6 compatible, 0 breaking):build(s).engineacceptsflink(forge-cli#731).exposes[].policy.authz.rowFilters[],columnRestrictions[].access: maskwithcolumnRestrictions[].mask, andbinding.governance.lakeFormation.maskedViews(forge-cli#700).build(s).properties.sink.catalogacceptslakekeeperandpolaris(forge-cli#711), andsink.catalogAuthnames a REST catalog's credentials by environment variable (forge-cli#712).0.7.3, 0.7.4, 0.7.5 and 0.7.6. Only the descriptions of the Lake Formation
tagsandtagDefinitionschange (forge-cli#706). The same documents validate.The overlay identity merge in forge-cli#732 is loader behaviour, not schema, so nothing here pins it.
Corpus
Without new cases, 0.7.6 mutation coverage falls to 39.9%, under the CI floor of 40. This PR adds 37 cases, each invalid one failing with exactly one error (keyword + pointer):
tests/0.7.6/build-consumes.json: a validengine: flinkbuild with its deployment settings underparameters.tests/0.7.5/build-consumes.json:engine: flinkis rejected (anyOfat/build/engine).tests/0.7.6/acquisition.json:catalogAuth(both types, required members per type, the members each type refuses, variable names vs. tokens and${...}, userinfo in the token URL,env:and empty values, closed set) and the two new catalogs. The existing "catalog outside the recognized set" case usedpolaris, which 0.7.6 now accepts; it useshadoopinstead.tests/0.7.6/expose-policy.json:access: maskwithoutmask,maskwithoutaccessor withdeny, themaskenum, androwFilters(required members,minLength, name pattern, closed set).tests/0.7.6/binding-governance.json:maskedViews(required members, closed set).Mutation coverage: 0.7.5 41.0% (unchanged), 0.7.6 43.1% (518 of 1201, 130 redundant).
tests/README.mdcarries the new figure.Docs
docs/releases/0.7.6.md: table rows and new sections for row filters and masked columns,engine: flinkandsink.catalogAuth, plus theaccess: maskbullet under column restrictions. Every example in the notes validates against the 0.7.6 schema, merged into the minimal contract where marked, and withfluid validatefrom data-product-forge 0.21.0. ThecatalogAuthexample carries its own Iceberg expose, since 0.21.0 refuses an Iceberg sink with no expose boundformat: iceberg.docs/schema/changelog.md: the new 0.7.6 additions, and a correction entry for the Lake Formation tag descriptions.docs/README.mdanddocs/releases/README.md.Gates
All run locally on this branch:
conformance/run.py: 1098 cases, 1098 passed.conformance/mutation_coverage.py --min-coverage 40: 41.0% (0.7.5), 43.1% (0.7.6).tests/meta_test.py: 50/50.scripts/check-compat.py: no breaking change.scripts/check-schema-drift.py: every synced schema byte-identical to v0.21.0.generate-docs.py --checkandgenerate-schema-diffs.py --check: current.conformance/check_reference.pyagainst data-product-forge 0.21.0: 1094 agree, 0 disagree, 4 optionalformatbehaviours not implemented.