Repository navigation
feat: contract fragments, drafts 21-29 (text refs, siblings, pointers, one composition) and 0.25.0 - #760
Merged
Conversation
…fragment root
A $ref with sibling keys was not a ref at all: _is_ref_node required $ref to
be the only key, so {$ref: shared.yaml#/order_id, description: ...} stayed a
literal key and its target was never read. A .sql or .md target failed with
"YAML root must be an object/dict", or, for SQL holding ': ', with a raw
PyYAML error, or loaded silently as a mapping. A YAML fragment whose root is
a list was refused while the same list as .json loaded.
fluid_build/contract_refs.py now holds the resolver: one step for every ref,
read the target, apply its pointer, merge the siblings.
- Siblings are a JSON Merge Patch (RFC 7386) over a copy of the target, as
json-schema-ref-parser's extended $ref does: objects merge, lists and
scalars replace, null removes a key. A target that is not an object takes
no siblings, and says so.
- A .sql, .md, .markdown, .txt, .py, .scala or .r target is its UTF-8 text,
never parsed. A pointer into one is refused.
- A fragment may have any root; the root contract and overlays stay objects.
- Confinement, the system-path deny list, cycle detection and the depth limit
apply to every ref as before, refs inside siblings included. A new cap stops
refs that double the contract every step (the billion laughs of $ref).
- Pointers decode RFC 6901 escapes (~0, ~1).
loader.py keeps its names as wrappers over the module. The contract API
reads LoadedContract.files from the resolver's own trace instead of a second
walker that could not read a text target or see a sibling ref.
Drafts 21, 23 (siblings) and 24.
…efs, pointers resolve, nothing is dropped An overlay was parsed raw and merged with its $ref unresolved. Then two loaders disagreed. load_contract_with_overlay, behind validate, plan, apply, generate standard and the library, saw a $ref left in the merged contract and silently re-read the base, so `--env prod` planned the dev topic with exit 0. materialize_contract, behind bundle and generate artifacts, kept the literal $ref. Any $ref the resolver left behind (a same-document pointer) did the same, so one leftover pointer dropped every overlay on the CLI path. And generate artifacts without --env read a split contract's raw root. loader.compose_contract is now the one composition every entry point reads: 1. the contract's file refs (contract_refs); 2. the overlay's own refs, relative to the overlay file and confined to the same root (FLUID_REF_ROOT is consulted when only the overlay names a file), then the merge, so a keyed 0.7.6 merge sees the ids the overlay's fragments name; 3. same-document #/ pointers, resolved against that result. A pointer names a place in the composed contract whichever file it is written in, so splitting a contract never changes it, and an overlay that changes a shared value reaches every pointer to it (late binding, as Jsonnet's self). Pointers may cross other refs; cycles and pointers that name nothing fail, naming the file and pointer they were written at. No $ref survives: one that cannot be resolved fails every command. The auto-bundle fallback is gone, and with it the overlay drop. bundle and generate artifacts call the same composition; generate artifacts composes a contract holding a $ref even without --env. The contract API composes the same way and no longer mirrors the drop: LoadedContract.overlay and .files come from the composition (the overlay's fragments included), unresolved_refs is empty for every file load, and the in-memory forms take overlay_dir= for the overlay's own refs (api 1.2). docs/contract-refs.md and docs/CONTRACT_LOADING_API.md described the old rules; they now describe these. Drafts 22 and 23 (same-document pointers). They land together: 22's "a surviving $ref fails every command" is 23's rule.
…nd line
Validation reported the path in the composed document only, so an error in
fragments/exposes/orders.yaml read "exposes[0].contract.schema[1].type: 42
is not valid ..." and the author mapped exposes[0] to a file by hand. Now:
fragments/exposes/orders.yaml:14 (exposes[0].contract.schema[1].type): 42 is not valid ...
The composition records, in a side table keyed by path in the composed
contract, which file and which pointer inside it each part came from: each
fragment where it landed, the keys written beside a $ref, the overlay
(keyed-list entries at the index they landed on, matched by id), and
#/ pointers as aliases of their target. The contract stays plain dicts and
lists. A line is read only when an error needs one, from PyYAML compose
marks (no new dependency; the same size, alias and depth caps as the
loader's parse, via util.safe_yaml.compose_yaml_safe).
A value is located at the latest file that holds it. An unexpected key is
located at the key itself; a missing one at the object's definition, not at
an overlay or siblings that only merged other keys into it. Errors in the
root contract read as before. Extension validator errors name the fragment
of the block they are about.
The library gets the same answer: LoadedContract.origin_of(path) returns the
file and line a value was written in (api 1.2).
Draft 25.
…efs read regular files only
generate artifacts without --env read the raw root of a contract, so a split
contract failed ("FLUID expose has no usable name") or exported a literal
$ref, and reference-only detection never saw a build kept in a fragment.
generate ci's reference-only and engine detection read the raw root too.
Both now read the composed contract. A contract with no $ref is still read
exactly as before, so its artifacts are unchanged; one split into fragments
now gets byte-identical artifacts to the same contract written flat, with or
without --env. (An earlier commit on this branch routed the no-env case
through a temporary bundle; that path writes a differently ordered
contract, so it is reverted here in favour of composing in the reader.)
Hardening of the resolver, found by probing the scenario catalog:
- A target must be a regular file: a directory, FIFO or device is refused
before it is read (a FIFO or /dev/zero blocked forever).
- /proc, /sys and /dev stay blocked whatever the root: their files report
size 0, so the text cap could not stop /proc/self/environ.
- A file with no known suffix keeps the old rule: JSON, or a YAML object.
Otherwise a dotfile such as .env would compose as a YAML string.
- A fragment that contains itself through a YAML alias is refused instead of
overflowing the stack; nesting past Python's recursion limit is a ref
error, not a crash.
- The expansion cap counts key characters and every copy an alias makes.
- A list index must be ASCII digits; the depth error names its ref; an
overlay that composes to a list is refused.
fluid docs parsed each contract with a plain YAML load: a contract written by fluid split listed its port as expose-0, columns were read from exposes[].schema (no 0.7.x contract has them there, so every port showed "No schema columns defined."), a 0.7.6 description object aborted the whole run with "'dict' object has no attribute 'replace'" after truncating the page to 0 bytes, and Nullable/PII read keys the schema rejects. - Contracts load through the loader fluid validate uses, refs resolved and YAML guarded. One that fails to load gets an index row naming the error and no page; the rest of the catalog is written. - Ports are named by exposeId; columns come from contract.schema (a legacy top-level schema still reads); Nullable follows `required`, PII follows `sensitivity` (pii, phi), with the legacy keys still honoured. - A description object renders purpose, usage and limitations; the owner object as team (email); domain and tags are read from the root first; consumes show productId/exposeId. - A page is rendered before its file is opened, and two contracts with one id get a page each (-2) instead of overwriting one. index.json keeps its keys and value types. Draft 27.
…ontract
fluid split always moved the same four sections, rewrote the root with
yaml.dump (every comment lost, quotes and flow style rewritten), wrote YAML
into a .json root, overwrote one fragment with another when two ids made the
same file name, and on a second run wrapped its own $refs into new fragments
until the contract no longer loaded.
- --parts chooses what moves: sovereignty, accessPolicy, builds, exposes
(the default, as before), sql (a build's properties.sql to sql/<id>.sql,
.py/.scala/.r by its language), quality and schema (an expose's lists),
extensions or extensions.<name>, description (texts to docs/<part>.md).
A part inside another is referenced from the file it ends up in
(fragments/builds/x.yaml holds `sql: {$ref: ../../sql/x.sql}`).
- A YAML root is spliced, not re-dumped: every byte outside a moved part is
kept, comments included, and each part takes its own text, comments and
block scalars included, to its fragment. PyYAML compose marks give the
spans; a block node's end mark runs over trailing comments, so a span ends
at the node's own last character. No new dependency (ruamel is optional
here, and loses between-item and end-of-file comments). A root with YAML
aliases, or a JSON root, is split as data instead; a JSON root stays JSON.
- A part that is already a $ref is left alone, so a second run changes
nothing; a file that exists is never written over, and two parts bound for
one file get -2, -3. Text a file cannot give back exactly (a \r, a leading
BOM) stays inline. A CRLF root keeps its line ends.
- Before it ends, split composes the result and compares it with the
original; if they differ, every file it wrote is taken back.
The default layout is unchanged: the same files, refs and headers, and the
dict-level splitter fluid forge writes through produces exactly what it did.
Every shipped example (24) composes to the same document after a split, and
the v0.24.0 loader reads the default split output to the same documents.
Draft 28.
…s own tree
A workspace that keeps fragments its products share (a team, glossary
terms) outside each product's folder needed FLUID_REF_ROOT in every process
that loads one of them: CI, the editor, every tool on the library. Now the
workspace declares it once:
# fluid.workspace.yaml
ref_root: . # refRoot is read too
found by walking up from the contract, relative to that file, and only read
for a contract that names another file.
A project file may not widen confinement past its own tree: that is why a
project .env cannot set FLUID_REF_ROOT (GHSA-58px-v64p-6ppm). So the root is
used only when it is a directory at or below the workspace file's own,
contains the contract, and passes the vetting the DuckDB sandbox already
gives a workspace grant: not the filesystem root, a home or a temp root, and
not above the repository the contract is in (a fluid.workspace.yaml in ~/src
would otherwise reach every clone). Anything else is ignored with a WARNING
once per file, and the contract keeps its own directory. An operator turns
the setting off with FLUID_IGNORE_WORKSPACE_REF_ROOT, which a project .env
cannot set.
Precedence: ref_root= > FLUID_REF_ROOT > the workspace > the contract's
directory. An ignored FLUID_REF_ROOT falls back to the root the contract gets
without it, the workspace's when it has one.
Draft 29.
…luid validate --fragment
An editor checking a contract written by fluid split against the 0.7.6
schema reported every fragment: "Additional properties are not allowed
('$ref' was unexpected)", and the four required keys of an expose. A
fragment could not be checked on its own: fluid validate read it as a
contract, fell back to schema 0.7.5, then 0.7.4.
- fluid_build/schemas/fluid-authoring-schema-0.7.6.json is the canonical
0.7.6 schema with a {$ref} (siblings included) accepted at every value
position, wrapped the way the OpenAPI 3.1 meta-schema admits a Reference
Object (if {type: object, required: [$ref]} then fragmentRef else the
canonical subschema), so every other value keeps the canonical checks and
messages. Generated by python -m fluid_build.schema_authoring; a test
fails when it drifts. Its name stays out of the fluid-schema-*.json glob
that version discovery, the tests and the forge_docs generator read.
- The canonical schemas are unchanged: commands validate the composed
contract, and Command Center generates its models from them.
- fluid validate --fragment KIND FILE validates one fragment against the
$defs entry it stands for (expose, build, column, sovereignty, ..., or
quality / schema for an expose's lists), its own refs resolved under the
root its contract would have, the version from --schema-version or the
nearest contract.fluid.yaml above it. Its errors read as the whole
contract's do: fragments/exposes/orders.yaml:14 (contract.schema[1].type).
Draft 26.
…kept in fragments
Read raw, a fragment-first contract reached the tools with {$ref: ...} where
its builds and exposes are, so validate_contract reported errors fluid
validate does not, and an empty file validated nothing. The path form now
loads the contract the way every command does: composed, fragments confined
to the contract's directory unless widened.
…g every command
compose_contract asks iter_external_refs whether the root names another
file before composing it, and that walk visited every path: on a root such
as `extensions: &e {self: *e}` it never ended, so fluid validate, plan,
the library and the MCP tools hung on one line of input. origin/main failed
on it at once ("maximum recursion depth exceeded").
iter_external_refs now visits each container once, as find_ref_node does,
and the composition reports the cycle: "the document contains itself
through a YAML alias".
- fragment_sites (cyclomatic complexity 72) is one generator per part: sections, entries (with a build's SQL and an expose's lists), extensions, description; same sites, same order. - load_yaml_safe and compose_yaml_safe share the checked compose step. - The two ref-root warn-once registries share one first-time check. - A schema error's location is built by one helper (_error_path), and a docs column row by another (_column_row_html). - Removed describe_fragment_layout and loader._pointer_parts, which nothing calls since split and the resolver stopped using them. Found by ripwire --quality-delta over the branch. No behaviour change.
ContractOrigins._locate (cyclomatic complexity 31) asks each record what it wrote through _from_entry, and _line walks the YAML node tree a step at a time through _node_child; fragment_sites' per-entry branch is _item_sites. Same answers: the provenance and split suites pass unchanged.
Trimmed this branch's edits to the sentences the change made false: a ref may name any value or a text file, keys beside $ref merge, same-document refs resolve, an overlay's refs resolve, nothing is left, and the workspace can widen the root. The authoring schema, --fragment and the workspace details go to forge_docs, where the user docs live.
… per area 1,871 tests over the 394 scenarios of the fragment work: targets, siblings and encodings; pointers, cycles and the expansion budget; confinement; overlays and the one loader; provenance; fluid split; docs, the authoring schema and validate --fragment; the workspace ref_root and the compatibility gate. A scenario the code does not meet yet is a strict xfail naming its finding, so the suite turns green as each is fixed. describe_fragment_layout is back (0.24.0 had it, and the split compatibility tests import it): split --dry-run lists through it.
…xplained What was wrong: - A ref_root the file system cannot stat (a name longer than it allows, or a path under a directory the process may not enter) raised a bare OSError out of the loader on Python 3.10 to 3.13: Path.is_dir() after resolve() was unguarded. It failed even a contract whose refs all stay in its own directory, and a valid FLUID_REF_ROOT did not help, because the workspace file was read before the variable was looked at. (L-21) - A fluid.workspace.yaml that cannot be parsed (a list root, invalid YAML, an alias bomb, over the 5 MiB cap) was swallowed without a word. (L-15) - When a workspace ref_root was refused, the escape error never said so. Its hint told the author to declare the ref_root they had just declared, and the one WARNING per process never reaches a service's caller. (L-03) What is true now: - An unstattable ref_root is ignored with the workspace_ref_root_ignored WARNING, like any other unusable value; the contract keeps its own directory. - The workspace file is read only when its ref_root would be used: with no ref_root= and no FLUID_REF_ROOT, or with a FLUID_REF_ROOT that is ignored. - A workspace file that cannot be parsed is ignored with a WARNING that names the file and why. A YAML syntax error is placed (line, column), never quoted. - Every escape error of a contract whose workspace ref_root was ignored says so, why, and which file to fix; when FLUID_REF_ROOT was ignored too, it says both. With FLUID_IGNORE_WORKSPACE_REF_ROOT set, the hint says the switch is on instead of asking for a ref_root. - A value from the workspace file is shown cut to 200 characters: the file may hold megabytes there, and every escape error now repeats the reason. The strict xfails of L-03, L-15 (the four malformed files) and L-21 are removed; their tests pass and pin the messages.
…ommand reads it validate's composition rule looks up each consumed product's productType in an index of the workspace's contracts, and it built that index from the raw root of every *.fluid.yaml. An upstream whose metadata lives in a $ref fragment had no productType there, so an ADP consuming a CDP passed validate and the rule never fired. The index now reads each contract through the loader (load_contract), so the fragments are in and the loader's input protections apply. A contract that does not compose contributes nothing and is logged at DEBUG, as an unparseable file always was: the contract being validated reports its own errors, and one broken neighbour does not stop the walk. A metadata block that is not a mapping no longer aborts the whole walk either; that product is indexed with no type. tests/test_composition_workspace_scan.py proved that each file is read once by counting calls to this module's own load_yaml_safe. The read happens in the loader now, so it counts calls to load_contract instead; what it pins is unchanged.
…consumes are in a fragment When an upstream cursor rewinds, mark_downstream_dirty flags every product whose consumes[] names it. find_downstream_products read each *.fluid.yaml raw, so a product whose consumes[] is a $ref to a fragment was never found and got no replay-pending marker, with nothing said. The walk reads each contract through the loader, fragments in. A contract that does not compose is skipped with a DEBUG line, as an unparseable one was.
A file holding untrusted YAML or JSON was read whole with Path.read_text before any cap applied: a sparse 8 GiB file cost about 16 GiB, and a FIFO or a device standing in for a file blocked or read without end. And the ref root check, Path.is_relative_to, builds every parent of the path it tests, so a composition paid for that on every ref. safe_yaml.read_bounded_file reads a regular file only, checked before it is opened and again on the file opened without blocking, and never reads past the cap, sizing its read by the file; read_bounded_text returns its text as Path.read_text would. ref_confinement.is_within is is_relative_to as a string comparison of resolved paths, and confine_ref uses it. Nothing calls the readers yet; the loader and the resolver move onto them next.
…fuse a pointer merged into an object What was wrong, by scenario finding, and what is true now: - B14-hybrid, B14-scalar, C20-hybrid, P5-06: a '#/' pointer is a dict until the last pass, so a merge took it for data. Beside a file $ref whose target held an object at that key, or as an overlay value over an object of the base, its $ref landed beside the object's keys, which then beat what it named; the overlay's site was lost too. Now it is refused, naming the file that wrote it, its pointer there, the place and the remedy (write the keys there, or name a file, which is read before the merge). Where no object stands it replaces, as before. Over a bare pointer it replaces, and the pointer pass refuses it if that one named an object, so an overlay can still swap one pointer to a string for another. - E11-time, DX-04: every ref re-read and re-parsed its target, so a fan-in bomb inside the depth limit took about 70 s. A composition now parses each file once (every later ref walks its own copy, still charged), and resolves at most MAX_REF_RESOLUTIONS = 10,000 refs, file refs and pointers alike; the 19-level bomb ends at that cap in about 1.5 s. - S2-F05 (SG-34), E10-msg: fragments nest deeper than a file may, and past about 330 levels bundle's YAML writer died while validate passed. The composed contract is held to MAX_COMPOSED_DEPTH = 200 levels at compose time, pointer copies included, so both commands agree. A RecursionError while a fragment is parsed is reported as nesting depth, not as a parse failure of a healthy file. - D-38, D-37, F-overlay-fifo-hang, DX-02: the root contract, overlays and JSON or YAML fragments were read whole before the 5 MiB cap, and an overlay that was a FIFO blocked every --env run. Every file the composition reads is now a regular file read with the bound (util.safe_yaml). - DX-01, F31 (SG-15), D-34 (SG-14): an overlay reached through a symlink was read from outside the ref root, and a symlinked contract took its overlay beside the link. The overlay is looked up beside the resolved contract and read only from inside the ref root, symlinks resolved, never from a system directory. - D-24 (SG-21): validate, plan and bundle read the file an --env path named. The loader holds a non-empty env to the API's contract_env_invalid rule where it looks the overlay up; an empty env is still no env. - S8-07 (SG-22): the in-memory API forms get the ref root a contract file in base_dir gets (FLUID_REF_ROOT, the workspace's ref_root). - DX-03, P5-04: the deny list refused a fragment deep in the root through the CLI's path-depth rule, and named neither file nor pointer. It uses the CLI's set of system directories only, and names where the ref was written. - N23: './x.yaml/' named the file x.yaml; a ref ending as a directory does is refused. A18: a JSON root that is not an object came back from raw loads; it is refused naming the file. M-16: compile_contract parsed a contract twice; it parses it once. Characterization tests whose pinned behaviour these decisions change are rewritten to pin the new behaviour; the C-22 pointer bomb now ends at the resolution cap.
…r compatibility (N24) The scenario suite pinned './a#b.yaml' reading key 'b.yaml' of file 'a' as a bug (C-12) in a strict xfail. The maintainer keeps it: contracts 0.24.0 loaded may use 'file.yaml#key', so a file ref's fragment need not start with '/'. Only a same-document '#key' is refused. The xfail is now a characterization test of the accepted behaviour.
ContractOrigins found each error's line by scanning the mapping's keys one by one, so n invalid values in one mapping cost n squared: 40,000 took 38 s, a 5 MiB contract of invalid labels about an hour, with or without any $ref. A mapping's keys are now indexed once per report (P5-09). The file is read again for its lines under the rules every file the composition reads keeps: a regular file, read no further than the cap, so one replaced by a FIFO since cannot block the report.
…ctly A text holding U+0085 (NEL) was composed exactly (the loader, the JSON bundle, the plan of the contract) but written into `fluid bundle` YAML and the tgz's `contract.resolved.yaml` as a bare NEL, which every YAML reader folds into a space. The two members of one tgz then disagreed, and `plan <bundle.tgz>` planned different SQL than `plan <contract>`. It hit a key holding NEL too, and text refs make such content easy to bring in. Both writers now use a dumper (`ExactDumper` for the `fluid bundle` file, which used `yaml.dump`; `ExactSafeDumper` for the canonical member, which used `yaml.safe_dump`) that writes a string holding NEL, LS (U+2028) or PS (U+2029) double-quoted with escapes (`\N`, `\L`, `\P`). Every other string, number and key is written as before, byte for byte.
Split checks its result against what the original composes to. When the original did not compose in the splitter's environment (a ref that needs FLUID_REF_ROOT, which was not set), the check was skipped: the command exited 0 and could leave a moved part whose relative ref no longer resolved (SP-03). Now such a contract is refused, dry run included, saying it does not compose here and why; nothing is written. With the root its refs need, it splits and is checked as any other. Rebasing a relative ref inside a moved part (SP-02) stays a follow-up: split refuses that case instead.
Contract fragments: drafts 21-29 of the $ref work (text refs, siblings, pointers, list roots, one composition with overlays, provenance, the authoring schema and validate --fragment, docs, split --parts, the workspace ref_root), the P0/P1 findings of the scenario suite, and the two security fixes already on main since 0.24.0 (#758, #759).
…is an unknown kind validate dispatched to the fragment check on any truthy args.fragment, so a caller handing run() an args object without the attribute set (a mock, a plugin's namespace) took the fragment path with a non-text kind; and --fragment '' was ignored, validating the file as a whole contract. The fragment check now runs for a kind given as text, and an empty kind is refused as an unknown kind (half of K14). schema_authoring reads the canonical schema with encoding='utf-8' spelled as the keyword the text-I/O guard looks for.
…s of their own run() had grown to cyclomatic complexity 16 with the compose check of SP-03; _composed and _print_dry_run take those branches. No behaviour change: the split suites pass unchanged.
The confinement tests plant a fake secret marker and a dummy OpenSSH key to prove a ref never composes or echoes them, and the authoring-schema test pins the sha256 of each canonical schema 0.24.0 shipped. None is a credential; each line carries detect-secrets' allowlist pragma so the CI secret scan on changed files passes.
…leaf, not cli The composition screened every ref target against the CLI's deny set by importing fluid_build.cli.security. Once apply's replay walk composed the contracts it reads, build_runners reached cli through loader -> contract_refs -> cli.security, and the import-linter contract 'build_runners must not depend on cli' broke (the CI import-hygiene job). The set (build_forbidden_paths, FORBIDDEN_PATHS) moves verbatim to the stdlib-only leaf fluid_build._path_policy; cli.security re-exports both names, so its checks and callers are unchanged, and the composition reads the same set from the leaf. loader now reaches no cli module at all. The leaf joins the tier-0 contract.
… not a name under it load_contract_from_text / load_contract_from_dict with base_dir composed the document under the root a contract file in base_dir would get, found by resolving base_dir/contract.fluid.yaml. That file is never read, but resolve() follows a symlink at that name, so whoever controls base_dir's contents chose the root: an upload holding 'contract.fluid.yaml -> .' (a link tarfile's 'data' filter keeps) widened it to the parent directory and composed another tenant's files; '-> /' widened it to the whole host bar the system directories. FLUID_IGNORE_WORKSPACE_REF_ROOT did not stop it. Found by the security-review gate; introduced on this branch. The root is now taken from base_dir itself, resolved; a name under it moves nothing.
…k or from above the repository validate --fragment chose the fragment's ref root from the nearest contract.fluid.* above it, accepting a symlink and walking up to /. A repository shipping 'contract.fluid.yaml -> ~/.profile' moved the root to the home directory, and a fragment's ref then composed ~/.docker/config.json and printed it in a schema error; the CLI already refuses a symlinked contract path for this reason. Without a link, an unrelated contract high up (~/contract.fluid.yaml) widened the root the same way. Found by the security-review gate; introduced on this branch. The lookup skips symlinks and stops at the fragment's repository or workspace (.git or fluid.workspace.yaml), and a fragment outside the root its contract gives is refused.
…used, naming the overlay The mirror of the round-2 fix: an overlay value written as a #/ pointer to a list of objects, over the base's builds or an expose's columns, replaced the base list where the same value inline (or as a file ref) merges by id or by position, so plan --env prod dropped a build and overlaid columns lost their keys with exit 0. Found by the defect-scan gate, round 3. It is refused, as a pointer merged into an object is. A refusal raised while merging an overlay now names the overlay file when its message does not.
…efore reading Round 2 checked every file a composition read against --readable-paths, but only after the composition succeeded: a composition that failed on a file outside the readable paths raised its error first, quoting that file's key names (a pointer miss) or a line of it (a YAML error), and mcp 1.x, which the pin allows, hands that text to the client. Found by the defect-scan and world-class-inspection gates, round 3. Under an active policy the contract is now composed under a ref root capped to the readable path that holds it, when the root it would get is wider: a ref outside fails as an escape before the file is opened. The post-check stays.
…s split as data Found by the defect-scan gate, round 3: the data split (a JSON root, a root with aliases) wrote text holding U+0085, U+2028 or U+2029 bare, which YAML reads back as a space, so split refused valid contracts; it now writes with the bundle's ExactDumper. A file with bare CR line ends gave the text splice line arithmetic it does not handle and an unparsable root; it is now split as data, with the notice that its comments were not kept.
…ad_contract_fields does
…ut parsing the contract _readable_ref_root parsed the contract itself only to learn whether it names another file, which the raw-reader guard (test_g20_no_unlisted_module_reads_a_contract_itself) rightly flags. It now asks for the root a contract that names one would get (reads_outside): one that names none reads nothing past its own directory anyway.
…f objects is refused The last bypass of the round-3 refusal: with the base list itself written as a bare #/ pointer, an overlay pointer replaced it, and the pointer pass checked only whether the replaced pointer named an object, so the overlay's list replaced the base's whole (plan --env prod dropped a build, exit 0) where, inline, it merges by id. Found by the defect-scan and world-class-inspection gates, round 4. A replacement an overlay made is now refused when the replaced pointer names a non-empty list of objects too; keys beside a file $ref stay a merge patch, where a list replaces. The refusal says 'list of objects' and 'the entries' for that case.
- mcp: a --readable-paths entry that names a file is not a ref root; the cap uses the deepest readable directory holding the contract, else its own directory (a regression of the round-3 cap: 'ref_root=... is not a directory'). - split: NEL, LS and PS end a line for YAML too, so a file using them as line breaks is split as data, like a bare CR. - validate --fragment: a nearest contract whose fluidVersion is a $ref gives no version instead of crashing (cli_unhandled_exception). - the parse cache keys a file by name when its file system reports no inode (st_ino 0). The subprocess hang guards of tests/fragments wait 60 s, not 6: a cold fluid validate under a loaded runner took longer (round-4 gate finding).
github-actions
Bot
added
cli
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
contract_refs' unused LOG, schema_manager's unused Sequence import, cli.security's unused platform import and _build_forbidden_paths re-export (the set lives in fluid_build._path_policy), two unused test imports and a needless lambda. The loader's resolver aliases (_MAX_REF_DEPTH, _is_ref_node, _parse_ref, _resolve_pointer) stay for the callers 0.24.0 had, and are listed in __all__ as the export they are; the contract API reads parse_ref from contract_refs itself.
…eview's next findings The fragment scenario suite (tests/fragments, ~1,900 tests) made the Coverage & Audit job run past its 30-minute timeout (it ran 19 on main) and brought the 3.10 test leg to 24 of its 25 minutes. The test legs get 35 and the coverage job 45: still minutes, not GitHub's 6-hour default. The code-quality review's next batch: unused test imports, two needless lambdas, and validate --fragment importing its own module where it can import SchemaVersion and ValidationResult directly.
The code-quality review flags a module imported with both `import` and `from ... import` in one file. Eleven fragment test modules imported the package only to read `fluid_build.__file__` for their "this checkout runs" check, beside `from fluid_build import loader`; they now read the path from the module they already import. Two MCP tests import the `mcp` package the way their file's other imports do.
This was referenced Oct 29, 2026
fas89
added a commit
that referenced
this pull request
Oct 30, 2026
…n parallel (#770) * ci: coverage measures on Python 3.14, where branch coverage runs on sys.monitoring The Coverage & Audit job went from ~15 to 37-41 minutes after the fragment suite landed (#760): with `branch = true`, coverage.py on 3.12 can only use the C tracer, and that tracer took tests/fragments/test_split (390 in-process tests) from 6.7 to 31.3 minutes in CI. coverage.py's sys.monitoring core ("sysmon") measures branches only on 3.14+, where it is the default (https://coverage.readthedocs.io/en/latest/config.html). Measured locally on that module: 3.12 226s plain, 546s with coverage; 3.14 217s plain, 220s with coverage. The job now runs on 3.14 with COVERAGE_CORE=sysmon set (a fallback then at least logs no-sysmon), and its timeout drops from 45 to 30, under the ~40 min a fallback would take. Gate margins on main are 5 to 49 points (core 82/70, V1 85/80, local 80/50, cloud 69/20). Nothing outside ci.yml uses the job or artifact name; neither is a required check. * ci(iac-tests): Stage 1's tofu lanes run as parallel jobs, and its unit step under xdist Stage 1 ran its six tofu lanes one after another in a single job, about 27 minutes on main for a month: the job took the sum of them. Each lane runs its own files against moto or an in-process stand-in and asserts its own junit, so each is now a matrix job, and Stage 1 takes the longest lane plus setup. A lane shares one TF_PLUGIN_CACHE_DIR across its modules (the tests honour an external one), one provider download per lane instead of per module; a lane is one pytest process, so nothing writes that cache concurrently. The unit step ran ~7.4k tests in one process (five minutes); it now runs with `-n auto --dist loadscope`, as ci.yml's test matrix runs the same tests on every PR, under the same MALLOC_ARENA_MAX cap. The emulator extra stays out of that job, so it still cannot change what the unit step runs. Check names: "Stage 1 — unit + tofu validate" stays; the lanes report as "Stage 1 — <lane>". None is a required check. * ci: the secrets baseline follows the test credential iac-tests.yml moved Splitting Stage 1 into lanes moved `AWS_SECRET_ACCESS_KEY: test` from line 258 to 318, so the changed-files detect-secrets step failed on the stale line number. Regenerated with detect-secrets 1.5.0, the CI pin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contract fragments: the
$refwork of drafts 21–29, and the 0.25.0 release notes.What changes
$refcan name a text file (.sql,.md, …) and becomes its text; a fragment may be any JSON or YAML value.$refare merged over its target (RFC 7386), and same-document#/pointers resolve against the composed contract.$refis left.fluid validate --fragment KIND FILEchecks one fragment, and an authoring schema lets editors check fragment-first contracts.fluid docsreads composed contracts and the 0.7.x shapes.fluid split --partschooses what moves, and keeps the root's comments.ref_rootinfluid.workspace.yamlwidens the ref root once for a workspace.Details and upgrade notes are in
CHANGELOG.mdunder[0.25.0]. Some inputs 0.24.0 accepted are now refused; the upgrade notes list them.Compatibility
$refloads, bundles and exports byte-identically to 0.24.0.fluid splitoutput stays readable by 0.24.0.Testing
tests/fragments(about 1,900 tests).main.Smaller remaining items are tracked separately.