Skip to content

Converter: the delivered preset-FG map, storage order and offset axes - #177

Merged
phantoslayer21 merged 1 commit into
mainfrom
fg/real-map
Nov 14, 2026
Merged

phantoslayer21 merged 1 commit into
mainfrom
fg/real-map

Conversation

@phantoslayer21

Copy link
Copy Markdown
Owner

Adds the spec side's frame-generation converter map. It replaces the synthetic stand-in layout from #148 (lane h).

Map: data/fg.v1.map.json

  • A byte-for-byte copy of the delivered map (v1). It is map version 1, "preset": "FG", with 67 block sources (A01..A17 for net1, B01w/B01b..B25w/B25b for net2) and 42 stages (dl1.l01..dl2.l25), in upload order.
  • It holds no weights: only lengths, hashes, spans and stage digests.
  • Its SHA-256 is pinned in the converter tests (FG_MAP_SHA256) and in clean/fg/tests/test_fg_map.py, which also checks every source and stage against clean/fg/layout.py. They match exactly: the blob lengths, net1's weight spans, the skipped zero area and copy 0 of each bias group.

Converter: two map features this map uses

  • "channel_order": "storage": the weight tensors (W_storage) are the stored spans, 1-D, not yet in natural axes. It is accepted only in a version-1 map of preset FG; every other preset still needs "natural".
  • Axes given by explicit offsets, {"offsets": [...]}, with one entry per index in place of stride/perm, in map versions 1 and 4. The net1 biases use it.
    • The index range is checked when the map loads.
    • A list of the wrong length, a non-integer, or offsets together with stride or perm is "converter data damaged" (exit 20). An offset past the source is "does not fit" (exit 21).
    • A version-5 map with offsets is damaged.
  • Note for the spec side: the map notes say explicit per-axis offsets were already supported in the version-1 format. Our converter did not support them before this PR; this PR adds them.

Tests

  • test_fg_real_map: the digest pin, the header, the sources and stages against the layout, and the map validating. A file without the blobs gives FG's message (exit 12); a file smaller than the key's blob gives exit 11.
  • test_fg now uses the delivered map's exact layout, filled with our own random blobs (cases.fg_case). The previous checks are kept: shuffled, aligned, even and odd placements, a decoy, the NaN-guarded zero areas and bias copies, --preset FG, and missing blobs giving exit 12 or 13.
  • New checks: malformed offset axes; negative offsets that stay inside the blob (same output); storage in a preset-M map is damaged.

Refs #96

Tested

  • Converter ctest with a local MSVC + Ninja build: unit, decompression vectors, synthetic (3,323 checks in quick mode; the full run passes) and synthetic5 all pass.
  • The clean/fg unit tests (including the new test_fg_map) and the line-ending check pass. The public gates pass, with no hits in the changed files.

Not tested

  • No real library in this PR. A conversion of a real file, checked by digests only, follows in its own PR.
  • Natural-order weights wait for the map's v1.1 revision.

… order and offset axes

The spec side's map v1 replaces the synthetic stand-in layout: 67 block sources
(A01..A17, B01w/B01b..B25w/B25b), 42 stages, weights as stored. Two map
features it needs: channel_order "storage" (preset FG, map version 1 only)
and axes by explicit offsets (map versions 1 and 4; range-checked at load,
malformed forms damaged), used for net1's bias copy 0.

The map's SHA-256 is pinned (converter tests and clean/fg/tests/test_fg_map.py),
its sources and stages are checked against clean/fg/layout.py, and the
synthetic end-to-end case now converts our own blobs in exactly its layout.

Refs #96

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant