Skip to content

fix(release): close both edges the packer was missing (#1359) - #1364

Merged
JustVugg merged 2 commits into
devfrom
fix/pack-closure-and-data
Sep 6, 2026
Merged

JustVugg merged 2 commits into
devfrom
fix/pack-closure-and-data

Conversation

@JustVugg

@JustVugg JustVugg commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Closes #1359, both halves of it.

The first commit is @monotophic's, cherry-picked unchanged from #1355 so the authorship stays theirs. The second is the part their report identified and deliberately did not fix.

What was wrong

needed() reached a file two ways, and only one of them was a real graph edge. A subprocess-launched script was added to the result and never walked, so a module only that script imports was never found. On dev exactly one file fell through: tools/iq3_pack.py, imported by convert_fp8_to_int4.py inside quant_e8().

The packer's own docstring promised "import, seguiti in chiusura" and then introduced the subprocess branch beside it. The closure held for one edge and not the other, in the same function, under a comment claiming both. I wrote that in #1296; the report is right.

The second edge, which packaging iq3_pack.py does not fix

iq3_pack.py:50 opens iq3xxs_grid.json next to itself, and the packer copied only .py. So the import fix alone moves the failure one step later rather than removing it. #1359 says this plainly and leaves it, which is why it is here instead.

Measured, and with the negative control:

$ python3 tools/pack_python.py . /tmp/dist
15 Python files and 1 data files reachable from coli (copied)
  added:   iq3_pack.py, iq3xxs_grid.json          removed: nothing

$ cd /tmp/dist && python3 -c "...; iq3_pack.encode(...)"
grid loaded from the archive: (256, 4)          <- the documented --xbits e8 path works

$ rm tools/iq3xxs_grid.json && <same command>
FileNotFoundError: .../tools/iq3xxs_grid.json   <- and the file is why

Why an allowlist and not "anything that is not .py"

Because the loose rule ships the repository. coli names colibri.c, family_registry.py names five engine sources, doctor.py names three more: all real sibling files, none of them archive contents. The rule is a string literal naming a file that exists next to the module and carries a data suffix, which over-packs at worst by a few KB and never under-packs. For a packer that is the correct direction: under-shipping breaks a release, over-shipping does not.

tools/iq3_encode.c stays out on purpose. iq3_pack.py's own docstring calls it optional, the numpy path works without it at ~25x, and using it needs a compiler on the user's machine. Shipping it should be a decision, not a side effect.

Tests

Six cases from #1355 (five fixtures plus a real-tree assertion), five new ones here. Both defects restored individually make the suite red:

restored defect failures
data_files disabled 3
subprocess script not queued 2

The real-tree test now asserts iq3_pack.py and iq3xxs_grid.json, the two files this was actually about, so it catches either edge regressing. #1355's version asserted engine_evidence.py, a file #1355 itself introduces, so it could not run on dev; that assertion belongs back in #1355 as an extra case.

One correction to the issue text, since it affects what could land when: it says the fix "asserts against the real tree rather than a fixture, because every fixture case in that module would have passed on dev today." Run against dev's packer, the fixture case fails as intended:

found: ['bar.py', 'foo.py']       assertIn('baz.py') -> FAILS

The fixtures do catch it, which is what made it safe to split this out of a 3400-line feature PR instead of waiting for it.

Checked and left alone

tools/k3_tokenizer.py is copied by hand in release.yml and is not in the computed set. That is correct, not a leftover: no Python file names it. A human runs it. Static analysis cannot reach a caller who is a person, so the explicit copy is the right mechanism.

@monotophic: #1355 will conflict on these two files once this lands. Say the word and I will resolve it on your branch so you do not have to rebase.

monotophic and others added 2 commits September 6, 2026 01:58
pack_python.py reaches a file two ways: by following imports out from
coli, and by spotting scripts that are launched as subprocesses. Only the
first kind had its own imports followed, so a module that only a
subprocess-launched script imports was left out of the release archive
while --check still reported the archive complete.

On dev this happens twice: tools/iq3_pack.py, which
tools/convert_fp8_to_int4.py imports inside quant_e8(), and
tools/engine_evidence.py, which this branch's eval_glm.py imports. Both
are now reached. The test suite grows a real-tree assertion as well as
fixture cases, because every existing case builds a disposable tree and
so could not have caught either omission.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Following the imports of a subprocess-launched script (previous commit)
packages tools/iq3_pack.py, but --xbits e8 still dies from an archive:
iq3_pack.py opens tools/iq3xxs_grid.json next to itself at line 50, and
the packer copied only .py files. The import edge was closed; the data
edge was not.

needed() now collects, along the same walk, files named by a string
literal that exist next to the module and carry a data suffix. Verified
end to end from a packed archive: iq3_pack.encode() runs and loads its
256x4 grid; delete the JSON and it is FileNotFoundError again.

The suffix allowlist is deliberate and not "anything that is not .py":
coli names colibri.c, family_registry.py names five engine sources, and
none of those belong in a release archive. tools/iq3_encode.c stays out
for the same reason it is optional in iq3_pack.py's own docstring -- the
numpy path works without it, and compiling it needs a toolchain on the
user's machine. That is a decision, not an heuristic's side effect.

The summary line now counts the two kinds separately. Saying "N Python
files" while shipping data files is how a display starts agreeing with
a bug instead of catching it.

Tests: five new cases, and the real-tree assertion now names the two
files this was about instead of an invented one. Both defects restored
individually make the suite fail (3 and 2 failures).
@JustVugg
JustVugg merged commit 33dd2e7 into dev Sep 6, 2026
25 checks passed
@JustVugg
JustVugg deleted the fix/pack-closure-and-data branch September 6, 2026 07:34
JustVugg added a commit to monotophic/colibri that referenced this pull request Sep 6, 2026
Resolves the conflict JustVugg#1364 created, so this branch does not need a rebase.

JustVugg#1364 landed the packer commit from this PR (cherry-picked, authorship
intact) plus a second edge it left open: a data file opened next to a
reached module. tools/pack_python.py therefore takes dev's version
wholesale, which is a strict superset of this branch's -- the only lines
unique here were the ones that commit replaced.

tests/test_pack_python.py takes dev's version and keeps this branch's
engine_evidence assertion as an ADDITIONAL real-tree case rather than a
replacement. On dev that assertion could not run, because
engine_evidence.py is introduced here; on this branch it pins the edge
this branch adds, next to the one that pins the historical bug.

12 tests, all green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants