You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
clean/runner/pipeline.py read_wgcm reads every non-float32 tensor as uint8 (int32, float16, uint16, uint32 fail) #241
read_wgcm in clean/runner/pipeline.py maps the tensor dtype code to a numpy type with dt = np.uint8 if dtype else "<f4": code 0 is read as float32 and every other code as uint8. The container format
(clean/converter/OUTPUT-FORMAT.md, tensor table) has more codes: 0 float32, 1 uint8, 2 int32, and with container
version 2 also 3 float16, 4 uint16, 5 uint32, 6 e4m3.
A preset-K model file has the group net with the int32 tensor window_offsets (shape 8 x 2, 64 bytes). Reading it
fails:
ValueError: cannot reshape array of size 64 into shape (8,2)
so read_wgcm cannot load any Model K model file (and read_model_tensors, which calls it when there is no raw/
folder, fails the same way). float16 / uint16 / uint32 tensors of a version-2 container would also be read with the
wrong element size; e4m3 (one byte) happens to work.
Minimal repro
importstruct, numpyasnpfromclean.runner.pipelineimportread_wgcm# a one-group container with one int32 tensor [2, 2] (layout per clean/converter/OUTPUT-FORMAT.md; header fields that# read_wgcm does not use are zero)name, tname=b"net", b"t"index=struct.pack("<H", len(name)) +name+bytes(32) +struct.pack("<I", 1)
index+=struct.pack("<H", len(tname)) +tname+bytes([2, 2]) +struct.pack("<II", 2, 2)
data_off=88+len(index) +16index+=struct.pack("<QQ", data_off, 16)
header=b"WGCMODEL"+struct.pack("<II", 1, 1) +bytes(64) +struct.pack("<Q", len(index))
open("t.wgcm", "wb").write(header+index+np.arange(4, dtype="<i4").tobytes())
print(read_wgcm("t.wgcm"))
Expected: {"net": {"t": array([[0, 1], [2, 3]], dtype=int32)}}. Actual: ValueError: cannot reshape array of size 16 into shape (2,2).
(Checked on main: the repro prints exactly that error.)
Expected vs actual
Expected: each tensor read with the element type of its dtype code; a code the reader does not know raises a clear
error naming the code.
Actual: every nonzero code is read as uint8, so multi-byte types fail to reshape (or, for a size that happens to
fit, would silently give wrong values).
Proposed fix
Map the codes explicitly and check the byte count:
and a unit test with one tensor of each dtype code (a container written by the test, no model data). The other
readers (clean/k/tests/test_k_runtime_tables.py parse_wgcm, clean/rr_f/runtime/tools/pack_model.py wgcm_logical,
the converter tests' read_container) may want the same check.
Activity