Repository navigation
fix(release): close both edges the packer was missing (#1359) - #1364
Merged
Merged
Conversation
pack_python.py reaches a file two ways: by following imports out from coli, and by spotting scripts that are launched as subprocesses. Only the first kind had its own imports followed, so a module that only a subprocess-launched script imports was left out of the release archive while --check still reported the archive complete. On dev this happens twice: tools/iq3_pack.py, which tools/convert_fp8_to_int4.py imports inside quant_e8(), and tools/engine_evidence.py, which this branch's eval_glm.py imports. Both are now reached. The test suite grows a real-tree assertion as well as fixture cases, because every existing case builds a disposable tree and so could not have caught either omission. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Following the imports of a subprocess-launched script (previous commit) packages tools/iq3_pack.py, but --xbits e8 still dies from an archive: iq3_pack.py opens tools/iq3xxs_grid.json next to itself at line 50, and the packer copied only .py files. The import edge was closed; the data edge was not. needed() now collects, along the same walk, files named by a string literal that exist next to the module and carry a data suffix. Verified end to end from a packed archive: iq3_pack.encode() runs and loads its 256x4 grid; delete the JSON and it is FileNotFoundError again. The suffix allowlist is deliberate and not "anything that is not .py": coli names colibri.c, family_registry.py names five engine sources, and none of those belong in a release archive. tools/iq3_encode.c stays out for the same reason it is optional in iq3_pack.py's own docstring -- the numpy path works without it, and compiling it needs a toolchain on the user's machine. That is a decision, not an heuristic's side effect. The summary line now counts the two kinds separately. Saying "N Python files" while shipping data files is how a display starts agreeing with a bug instead of catching it. Tests: five new cases, and the real-tree assertion now names the two files this was about instead of an invented one. Both defects restored individually make the suite fail (3 and 2 failures).
JustVugg
added a commit
to monotophic/colibri
that referenced
this pull request
Sep 6, 2026
Resolves the conflict JustVugg#1364 created, so this branch does not need a rebase. JustVugg#1364 landed the packer commit from this PR (cherry-picked, authorship intact) plus a second edge it left open: a data file opened next to a reached module. tools/pack_python.py therefore takes dev's version wholesale, which is a strict superset of this branch's -- the only lines unique here were the ones that commit replaced. tests/test_pack_python.py takes dev's version and keeps this branch's engine_evidence assertion as an ADDITIONAL real-tree case rather than a replacement. On dev that assertion could not run, because engine_evidence.py is introduced here; on this branch it pins the edge this branch adds, next to the one that pins the historical bug. 12 tests, all green.
This was referenced Sep 6, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1359, both halves of it.
The first commit is @monotophic's, cherry-picked unchanged from #1355 so the authorship stays theirs. The second is the part their report identified and deliberately did not fix.
What was wrong
needed()reached a file two ways, and only one of them was a real graph edge. A subprocess-launched script was added to the result and never walked, so a module only that script imports was never found. Ondevexactly one file fell through:tools/iq3_pack.py, imported byconvert_fp8_to_int4.pyinsidequant_e8().The packer's own docstring promised "import, seguiti in chiusura" and then introduced the subprocess branch beside it. The closure held for one edge and not the other, in the same function, under a comment claiming both. I wrote that in #1296; the report is right.
The second edge, which packaging
iq3_pack.pydoes not fixiq3_pack.py:50opensiq3xxs_grid.jsonnext to itself, and the packer copied only.py. So the import fix alone moves the failure one step later rather than removing it. #1359 says this plainly and leaves it, which is why it is here instead.Measured, and with the negative control:
Why an allowlist and not "anything that is not .py"
Because the loose rule ships the repository.
colinamescolibri.c,family_registry.pynames five engine sources,doctor.pynames three more: all real sibling files, none of them archive contents. The rule is a string literal naming a file that exists next to the module and carries a data suffix, which over-packs at worst by a few KB and never under-packs. For a packer that is the correct direction: under-shipping breaks a release, over-shipping does not.tools/iq3_encode.cstays out on purpose.iq3_pack.py's own docstring calls it optional, the numpy path works without it at ~25x, and using it needs a compiler on the user's machine. Shipping it should be a decision, not a side effect.Tests
Six cases from #1355 (five fixtures plus a real-tree assertion), five new ones here. Both defects restored individually make the suite red:
data_filesdisabledThe real-tree test now asserts
iq3_pack.pyandiq3xxs_grid.json, the two files this was actually about, so it catches either edge regressing. #1355's version assertedengine_evidence.py, a file #1355 itself introduces, so it could not run ondev; that assertion belongs back in #1355 as an extra case.One correction to the issue text, since it affects what could land when: it says the fix "asserts against the real tree rather than a fixture, because every fixture case in that module would have passed on dev today." Run against
dev's packer, the fixture case fails as intended:The fixtures do catch it, which is what made it safe to split this out of a 3400-line feature PR instead of waiting for it.
Checked and left alone
tools/k3_tokenizer.pyis copied by hand inrelease.ymland is not in the computed set. That is correct, not a leftover: no Python file names it. A human runs it. Static analysis cannot reach a caller who is a person, so the explicit copy is the right mechanism.@monotophic: #1355 will conflict on these two files once this lands. Say the word and I will resolve it on your branch so you do not have to rebase.