Skip to content

clean/runner/pipeline.py read_wgcm reads every non-float32 tensor as uint8 (int32, float16, uint16, uint32 fail) #241

Open

Description

@phantoslayer21

Symptom

read_wgcm in clean/runner/pipeline.py maps the tensor dtype code to a numpy type with
dt = np.uint8 if dtype else "<f4": code 0 is read as float32 and every other code as uint8. The container format
(clean/converter/OUTPUT-FORMAT.md, tensor table) has more codes: 0 float32, 1 uint8, 2 int32, and with container
version 2 also 3 float16, 4 uint16, 5 uint32, 6 e4m3.

A preset-K model file has the group net with the int32 tensor window_offsets (shape 8 x 2, 64 bytes). Reading it
fails:

ValueError: cannot reshape array of size 64 into shape (8,2)

so read_wgcm cannot load any Model K model file (and read_model_tensors, which calls it when there is no raw/
folder, fails the same way). float16 / uint16 / uint32 tensors of a version-2 container would also be read with the
wrong element size; e4m3 (one byte) happens to work.

Minimal repro

import struct, numpy as np
from clean.runner.pipeline import read_wgcm

# a one-group container with one int32 tensor [2, 2] (layout per clean/converter/OUTPUT-FORMAT.md; header fields that
# read_wgcm does not use are zero)
name, tname = b"net", b"t"
index = struct.pack("<H", len(name)) + name + bytes(32) + struct.pack("<I", 1)
index += struct.pack("<H", len(tname)) + tname + bytes([2, 2]) + struct.pack("<II", 2, 2)
data_off = 88 + len(index) + 16
index += struct.pack("<QQ", data_off, 16)
header = b"WGCMODEL" + struct.pack("<II", 1, 1) + bytes(64) + struct.pack("<Q", len(index))
open("t.wgcm", "wb").write(header + index + np.arange(4, dtype="<i4").tobytes())
print(read_wgcm("t.wgcm"))

Expected: {"net": {"t": array([[0, 1], [2, 3]], dtype=int32)}}. Actual: ValueError: cannot reshape array of size 16 into shape (2,2).

(Checked on main: the repro prints exactly that error.)

Expected vs actual

  • Expected: each tensor read with the element type of its dtype code; a code the reader does not know raises a clear
    error naming the code.
  • Actual: every nonzero code is read as uint8, so multi-byte types fail to reshape (or, for a size that happens to
    fit, would silently give wrong values).

Proposed fix

Map the codes explicitly and check the byte count:

DTYPES = {0: "<f4", 1: np.uint8, 2: "<i4", 3: "<f2", 4: "<u2", 5: "<u4", 6: np.uint8}
...
if dtype not in DTYPES:
    raise ValueError(f"{path}: tensor {stage}/{name}: unknown dtype code {dtype}")
dt = np.dtype(DTYPES[dtype])
count = int(np.prod(shape)) if shape else 1
if size != count * dt.itemsize:
    raise ValueError(f"{path}: tensor {stage}/{name}: {size} bytes for {count} values of {dt}")
a = np.frombuffer(data[off:off + size], dtype=dt).reshape(shape)
out[stage][name] = a.astype(np.float32) if dtype in (0, 3) else a

and a unit test with one tensor of each dtype code (a container written by the test, no model data). The other
readers (clean/k/tests/test_k_runtime_tables.py parse_wgcm, clean/rr_f/runtime/tools/pack_model.py wgcm_logical,
the converter tests' read_container) may want the same check.

Activity

  1. added this to the 0.6.5 milestone on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions