Repository navigation
fix: fluid split left a moved $ref pointing off, wrote over files in --out, and half-undid a refused split - #765
Merged
Merged
Conversation
In a CRLF root, the span of a moved block that ended in a trailing comment ran to the line's LF, so it took the CR with it: the $ref line that replaced the block ended in a bare LF among CRLF lines, in the root and in a fragment a nested part was cut from (finding SP-01). The span now stops before the CR, which stays to end the ref's line. The CRLF fixtures and the generated roots keep the comments they left out to avoid the bug.
A block scalar's span ended at its last character of text, which is right for |, |-, > and >-, whose blank lines after the text are not their value. A keep-chomped one (|+, >+) keeps them: when it ended a moved part, the fragment lost them, the split contract no longer composed to the original one, and split refused a valid contract (finding SP-04). The span of a keep-chomped scalar now runs over its blank lines, up to the line break that ends the last of them, which stays in the root to end the ref's line. The value is the same in the fragment, with LF or CRLF.
A node's start mark is at its tag or anchor when it has one. Written on the
key's line ('sovereignty: !!map' or '&s', the mapping below), that start put
the splice on the key's line, which got 'key: $ref: ...', a line that does
not parse; split refused the contract (finding SP-06).
The ref now takes the key's line as a flow mapping, and the tag or anchor
starts the fragment, so the text is still spliced and every comment kept.
The fragment's lines are dedented by the indentation of the part's first
entry, not by the anchor's column, so a block scalar under it keeps its
indentation.
'sql: | # the main query' put the comment inside the span a moved text part replaces with a ref, and a text file cannot hold a YAML comment: the comment was in no file after the split, and nothing said so (finding SP-10). The comment on a block scalar's header line now stays in the file that holds the ref, on the ref's line: the root, or the fragment the part was cut from. The generated roots of the comment check put trailing comments on header lines too.
…there
A relative $ref resolves against the directory of the file that holds it.
A part that moved to fragments/builds/t.yaml took its refs along unchanged,
so './sql/t.sql' now named fragments/builds/sql/t.sql: the split did not
compose, and split refused any contract with a ref in a build or an expose,
including one split once with --parts sql, while --dry-run listed the files
and exited 0 (finding SP-02).
Split now rewrites each relative ref in a part that moves to name the same
file from the part's new one ('../../sql/t.sql', '../../cols.yaml#/a'), in
the text splice (in the ref's own quoting, every comment kept) and in the
data split. The ref's own path is kept as written: only the climb out of
the fragment's directory goes in front, so a '..' in it is never folded
past a symlinked directory. A URL, an absolute path and a same-document
'#/...' pointer are left alone, by the predicate the ref resolver uses
(ref_confinement.is_relative_file_ref). A second pass ('--parts sql', then
'--parts builds') ends where one split of both parts does.
What rebasing cannot fix is still refused and taken back: a file ref whose
pointer runs through a part that moves out of the root
('./contract.fluid.yaml#/builds/0/id'). The compose-and-compare check stays.
A refused split said 'nothing was changed', and the module promises as much, but it left behind every directory it had made: an empty fragments/ and fragments/builds/, or the whole --out directory. A leftover fragments/ changes what 'fluid bundle <contract>' does without --out (it writes contract.bundled.fluid.yaml instead of printing). A write that failed on a lone surrogate also left the empty file it had just created, which the next run numbered around (finding SP-05). Split now makes each directory itself and records the ones it creates, records a fragment the moment its file exists, and takes all of it back on a refusal, directories last and deepest first. Every text is encoded before anything is made, so one with no UTF-8 form is refused with nothing to take back, and the message names its file. 'nothing was changed' now ends the message only after the take-back.
Every failed write ended in exit 1, 'Split not written', except the last one: a read-only root failed to open, and the undo then wrote the old text back to that same root, which raised the same PermissionError inside the handler: exit 2, 'Unexpected error', and the directories made stayed (finding SP-09). The undo now puts the root back only when it was opened for writing, so a root that could not be opened is left as it was, and every step of it runs whatever an earlier one did and none raises. A read-only root is now a clean exit 1 with the tree as it was; if a step of the undo does fail, the message names what is left instead of saying nothing was changed.
The slug of an id was never shortened: an id of 300 characters named a 305-byte file, over the 255 bytes a file name may have. The dry run listed it and exited 0; the real run failed with ENAMETOOLONG (finding SP-07). A slug over 100 characters is now cut to 91 and ends in '-' and 8 hex digits of the SHA-256 of the whole id, so every name stays far under 255 bytes with a '-N' and the longest suffix, and two long ids that start alike still get a name each. Builds, exposes, their sql, schema and quality, and extension blocks are all named this way, and the dry run lists exactly the names the real run writes. The cap is pants' safe_filename default; the hash is kubespawner's strip_and_hash.
…does Finding SP-08: with <out>/fragments a symlink to a directory outside <out>, split wrote the fragments through the link before a later check refused the split, and left the directories it made out there. The real run has refused such a link before writing anything since the fragments PR; the dry run never checked, so it listed fragments under the link and exited 0 for a split the real run refuses. The checks (a symlink at the root's name, a symlinked directory on the way to any fragment, however deep) now run before the dry run lists anything, with the same message and exit 1, and again before the first write. The test pins the whole tree, the outside directory included, after both runs.
Fragments never overwrote a file, but the root did: 'fluid split c --out DIR' replaced a different DIR/contract.fluid.yaml without a word and exited 0, and the dry run did not mention it. A second split into the same --out met its own first-run files, wrote '-2' copies of every fragment and left the first run's fragments unreferenced (finding SP-11). The root may now be written only over the contract itself (an in-place split) or a byte-for-byte copy of it, where nothing is lost. Any other file there is refused before anything is written, the dry run included, with the file named and what to do: move it away or choose another --out. When the file there composes to the same document (the second-run case) the message says so and gives the command that splits that contract further. A split already there is not reused: the source may have changed since. A hard link to the contract in another directory is refused too, as writing it would change the contract in its own directory.
… to data The SP-02 rebase writes the new ref over the old one's scalar in the text splice, in the quoting it was written in; those branches (plain, single, double, tagged, a block mapping) and the fallback for a ref written as a block scalar, which the text splice refuses and the data split rebases, had no test of their own. Each one now checks the fragment's line, that every comment is kept, and the invariant.
…place
The data split (a root with YAML aliases or merge keys, or a JSON root)
replaced each part by its ref in the container that held it. An anchor or
a merge key ('<<: *defaults') puts one container at several places, so the
part was replaced at all of them, and the next part there moved that ref
instead of itself (finding N1). With the sql of two builds merged from one
defaults block, the second sql file got a dict: 'Unexpected error', exit 2
(main refused cleanly). With one aliased contract mapping at two exposes,
the second schema fragment got the splitter's own ref, which the SP-02
rebase then sent out of the ref root: exit 1, where main split it.
Each container on the way to a part is now replaced, in the one above it,
by a shallow copy of itself, so the part is replaced where it moves from
and nowhere else; what the part shares with the places that keep it stays
as it is. Six contracts that share containers through anchors, aliases and
merge keys pin it, both repros among them, each with the invariant. And a
part that is not text when it is written, whatever put it there, is now a
clean refusal with nothing made, never an unexpected error.
… is made
JSON can escape a lone surrogate ('"\ud800"'), and a key with one that
stays in a JSON root comes back in the root's text, which then has no
UTF-8 form. The fragments were encoded before anything was made, but the
root was encoded outside that guard: 'Unexpected error', exit 2 (finding
N2).
The root is now encoded in the same step, so it is refused as a part with
no UTF-8 form is: 'Split not written: <root> cannot be written as UTF-8',
exit 1, with nothing made.
…near time The comment on a block scalar's header line was found with '[ \t]+#.*$', searched from every position: on a run of blanks with no '#' after it, each blank started a match that took the rest of the run and then gave it back one blank at a time. 80,000 blanks after 'sql: |' took 17 s, against 2.3 s for the whole split on main. A match now starts only where a run of blanks starts (a lookbehind), which finds the same comment, the blanks before it included, in time linear in the line. A test times the search over 100,000 blanks, with and without a comment after them.
rebased_ref stripped the './' steps a relative ref starts with one at a
time, each step copying the rest of the ref: a ref of 200,000 './' steps
took seconds to rebase.
The steps ('./', './/', '././') are now matched once, by one anchored
pattern with one way to match, and the rest is split from there. A dot
that starts a name ('./.x/y.yaml') and a '../' after them stay as they
were. A test rebases four 200,000-step refs under a second each.
…fused The SP-11 guard lets the root be written over a byte-for-byte copy of the contract in --out, as nothing is lost. But the root is written in place, and a copy that is also another name of a third file (a hard link) took that file with it: the third file was rewritten as the split root. The copy branch now also needs the file to have one name (st_nlink == 1); a hard link with the same bytes is refused like any other file there, before anything is written. A test hard-links a third file into --out and checks that file, and the whole tree, are as they were.
The take-back ran for an Exception only. A KeyboardInterrupt (Ctrl-C) or a
SystemExit while the fragments and the root were written, or while the
result was checked, went past it and left the split unchecked: fragments
written, the root rewritten (C1).
Any BaseException now takes the split back first; one that is not an
Exception is then raised again, so the CLI reports the interrupt as it
reports any ('Operation cancelled by user', exit 1), and what could not be
taken back, if anything, is printed before it goes on. A test interrupts
the check of a written split with each of the two and finds the tree as it
was.
_write_checked named the text of each fragment 'content' when it encoded it, then the same name for the bytes it wrote, so mypy took the second for a str: 'Incompatible types in assignment' and an 'incompatible type str' for the write (C2). The bytes are now 'encoded'.
Four properties had no test that fails when they break: - SP-05, the file being written: a fragment's file is made, then its write fails (a full disk, the second 'xb' open). It is taken back with the rest; recording it only after its write would leave it behind. - SP-02 in the data split: one ref object an alias puts in the root's metadata and in a build is rebased in the build's fragment only; a rebase in place would change the root's too. - SP-02 in the text splice: a ref in the columns, cut out of an expose that moves as well, is rewritten once, in the columns' own file; walking the columns again from the expose would write over the cut. - The generated roots now hold refs too (parameters with and without a sibling, a column, a whole schema), with their files beside the root, in the root's line ends. 300 seeds past the 24 in the suite pass.
…s letters
What the docstrings and the help said had drifted from what split does:
- The module docstring said no file is written over but the contract split
in place; a byte-for-byte copy of it in --out, the only name of its file,
is written over too (SP-11). It says so now.
- A rebased ref was said to keep 'its own quoting', and a test named so,
but a plain './cols.yaml#/a' comes back double-quoted: its new path needs
quotes. It keeps its quoting where the new path allows it, which is what
the docstring and the test say now, with a case for each such path.
- The double quotes were written without allow_unicode, so a ref with a
letter that is not ASCII ('./sql/my file é.sql') came back escaped
('\xE9'). The letter is written as it is now; NEL, LS, PS and what is
not printable are escaped as before, so the value reads back exactly.
- The help did not say that with --out the files a relative $ref names
must be under the output directory too (C5). One sentence says it.
- The test module said findings are kept as strict xfails; none is left.
The './' steps a relative ref starts with were matched with '(?:\./+)+', linear in time but holding state for every step it took: a million steps peaked at 124 MB, 2.5 million at 330 MB of RSS. Python 3.10 is supported, so a possessive quantifier cannot drop that state. The steps are now scanned: the run of '.' and '/' the path starts with, cut before its first '..' (a climb, kept) or before a last lone '.' (a name that starts with one). The scan keeps nothing per step. A test pins it against the pattern it replaced on every path of up to eight '.', '/' and 'x', and bounds the memory: the scan under 1 MB for a million steps, the whole rebase under 32 MB (the rest is reading the path for the ref root's own checks, a few bytes per character).
…rupt Two gaps were left in taking back an interrupted split. The flag that sent the take-back to the root was set after open(root, 'wb') returned, but open() empties the root before it returns: an interrupt just then left the contract empty, and the take-back passed it by. The flag is now set before the open, and the root is written back whenever its bytes differ from what they were, so a read-only root that could not be opened is still left alone. The take-back removed the fragments before it put the root back, and caught only an Exception, so a second Ctrl-C during it stopped it there: a split root over fragments it had already removed. The root now goes back first, every step runs whatever an earlier one raised, an interrupt included, and the first interrupt is raised again at the end (also when the split was being taken back after a plain refusal), with what could not be taken back printed before it. Tests interrupt the root's open just after it empties the file, and the take-back's first fragment removal after an interrupt or a refusal at the check; they find the root put back before any fragment was touched.
…n escaped
The double quotes of a rebased ref were written with allow_unicode, which
PyYAML takes to mean writing a bidi control (U+202E RLO, the isolates,
LRM, ALM), a zero-width space or joiner, a soft hyphen, a private-use or
an unassigned code point as it is: a fragment line that shows other text
than it holds, the Trojan Source pattern.
A value is now written with its letters that are not ASCII ('é') only when
it is all printable, the rule the single-quoted branch already had; one
that holds anything else is escaped whole. NEL, LS and PS are escaped
either way. Tests check each such character comes out escaped and reads
back exactly, and that a ref the contract escapes ('\u202E') stays escaped
in the fragment, no file the split writes holding the character itself.
The test source spells every such character as an escape.
The copies a data split makes on each part's way (N1) share what else they
hold with the original, so one object a YAML alias put at two places can
end up at many. The dumper named shared objects, and wrote an alias for
each: a builds list aliased in the metadata, 60 builds with a 'tags' list
each and '--parts sql', gave 60 aliases, past the 50 the loader reads when
the split is checked. The split was refused ('more than 50 alias
references'), where the split before N1 split it.
The data split's root and fragments are now dumped with aliases off, every
object in full wherever it is. Nothing grows past what the contract's own
alias cap let in, and a document that contains itself is refused before (it
does not compose). A test splits that contract and finds no anchor or alias
in any file it writes.
- A fragment counted as written before its exclusive create: when another process makes a file at that name just before split does, the create fails and the take-back would remove that other file. A test makes one there at that moment and finds it kept, the refused split taken back. - The line an interrupt prints when its take-back leaves something: with the interrupt going on there is no message to carry it. A test fails a take-back step during an interrupt and finds the line, with the path and the reason, and the root put back all the same.
…a way out Two things the words got wrong after SP-11: - The help said 'a file that exists is never written over' without the one exception: a byte-for-byte copy of the contract in --out that is the only name of its file. It says so now. - When the file in --out composes to this contract but is a hard link (to the contract in its own directory, or to a third file), the refusal gave 'To split that one, run: fluid split <out>/contract.fluid.yaml', which splits it in place, through the link: the other name changes with it. The refusal now says it is a hard link and gives a way out that writes through nothing: move it away, or choose another --out. The command stays the way out for a file that is the only name of its own. The two hard-link tests check the way out given, and a test reads the help.
# Conflicts: # CHANGELOG.md
github-actions
Bot
added
cli
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…d of catching BaseException Ctrl-C and an exit are still taken back and raised again; nothing else that is not an Exception is caught (code-quality: except BaseException).
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups of the drafts 21-29 work (#760), Trello WffUBM9S (SP-01..06), plus what two review rounds and a final wave found in the same command. The card's strict xfails are gone and their tests pass.
What changed
A relative
$refinside a part split moves is rebased (SP-02).sql: {$ref: ./sql/transform.sql}was copied unchanged intofragments/builds/transform.yaml, where it named a file one or two directories off, so split refused the contract. A split in two passes (--parts sql, then--parts builds) could not work. The ref now names the same file from the fragment (../../sql/transform.sql, a#/pointerkept). A URL, an absolute path and a same-document pointer are left as written. A rebased ref keeps its quoting and its letters; a character that hides (a bidi control, a zero-width one) is written escaped.A refused split leaves the tree as it was, as its message says.
fragments/changed what other commands did), and after a failed write the empty file.A part in a container a YAML alias or
<<merge shares is split from its own place. Two builds merging the same defaults madefluid split --parts sqlfail with "write() argument must be str, not dict" and leave an emptysql/b.sql. Each build's sql is now a file of its own. The data split writes every object in full, with no alias, so its output stays under the loader's 50-alias cap.fluid split --out DIRnever writes over another file.DIR/contract.fluid.yamlthat is not the contract being split was replaced without a word. The one exception is a byte-for-byte copy of it that is no hard link to another file, and--helpnow says so.--outwrote-2copies and left the first run's fragments unreferenced.A long id gets a file name a file system takes (SP-05). An id of 300 characters gave a 305-byte name: the dry run listed it and the real run failed. A slug over 100 characters now ends in 8 hex digits of a hash of the whole id.
Split keeps more of the root as written (SP-01, SP-04).
|+,>+) keeps its trailing blank lines.$ref.Hardening of the new code, found by review:
./steps of a ref are linear in time and memory: 100k blanks or 1M steps in well under a second.--outthat is a hard link to a third file is never written through.Borrow before build
Bug fixes inside existing code (the design space was chosen in #760). Reused in-repo:
util/ref_confinement(a newis_relative_file_refbesideconfine_ref, checked against it on 13,253 edge strings);fluid docsuses.Validation
-n auto --dist loadscope --ignore=tests/perf): 30592 passed. The 7 failures and 2 collection errors fail identically on clean main: dlt credential-field drift, the DuckDB sandbox broad-root tests, the liveclaudeCLI test, and two CI-only modules. After the last refactor and merging main: the split test files, 395 passed.tests/perf(serial), ruff, black 24.10.0, dead-code gate, product guardrail, import-linter 6/6, mypy --strict hotspots, detect-secrets, license headers: green (tests/perf38 passed; the CHANGELOG gate 45 passed).rebased_ref: 190,909 random refs and containers, every rebased ref resolving to the same target. Fault injection at 46 points (mkdir, open, write, compose): always exit 1 and an exactly restored tree.ripwire --quality-delta(branch diff applied on main):_write_checkedstays under the complexity bar, with the encode step and the take-back report in_encodedand_taken_back. Left, with reasons:_rebased_refs(new) has complexity 19 and nesting 5, as one walk over every ref form a moved part can hold;run()length 67→73 (minor); dead-code rows on_put_back(called throughfunctools.partial) and_InFull.ignore_aliases(a PyYAML dumper hook);_root_in_the_waycatches every compose failure on purpose, since a file there that does not compose is another contract; test-only duplication (house style).--parts buildsover asqlref): rc 0 and the result bundles; main refuses with$ref target not found. Two builds merging one<<default with--parts sql: rc 0 witha.sqlandb.sql; main fails withwrite() argument must be str, not dictand leavessql/b.sql.--outholding another contract: refused, rc 1, file untouched; main overwrote it with rc 0. A 300-character id: rc 0 with a 105-character file name; main fails with[Errno 63] File name too long.Reviews
./scan.GATE: PASS.Follow-ups (pre-existing, same on main, re-checked on this branch)
--parts. The span is cut at.rstrip()(_split_text.py).$refpoints into a moved part splits with exit 0, thenfluid bundle --env devfails..json/.yaml/.yml, and for an untraversablefragments/.--out <a regular file> --dry-runexits 0; the real run refuses.Documentation
docs/cli/split.md: refs in moved parts are rebased, a refused split changes nothing,--outnever writes over another file (the copy exception), and the slug cap. Comes with the forge_docs pages PR of this wave.