Skip to content

test(dpt): use reproducible native-scale hash references - #44

Merged
noelEOS merged 9 commits into
mainfrom
models/dpt-integer-reference
Nov 3, 2026
Merged

noelEOS merged 9 commits into
mainfrom
models/dpt-integer-reference

Conversation

@noelEOS

@noelEOS noelEOS commented Nov 3, 2026

Copy link
Copy Markdown
Owner

SMP DPT's full-width and reduced numerical references now use a counter-based SplitMix64 hash of each parameter's name and flat index, scaled to the standard deviation of its pinned native initializer. Fixed BatchNorm calibration uses separate integer-generated inputs. Head and logits meet 1e-4 against float32 PyTorch; every intermediate's mirror error against float64 PyTorch is no larger than float32 PyTorch's own. No strict expected failure remains.

The three reference designs, in order:

  1. Unscaled integer pattern, declared at 24e1b6f before its one-shot recorded measurement. The period-31 pattern produces rank-at-most-30 matrices; full-width intermediate values reached 3.8e8. The procedure was superseded, and dpt-integer-offline.json remains unchanged as history.
  2. Fan-in-scaled integer pattern, declared at 80e92f3 before its one-shot recorded measurement. It still failed 1e-4: full-width head error 0.0005536079406738281 and logit error 0.0005426406860351562 (about 5.4e-4). That run stopped before reduced width. dpt-fan-in-offline.json, committed at ce619d9, remains unchanged as history.
  3. Final counter-hash at native standard deviation, declared at 6197082 before its one-shot recorded measurement at both widths. It passes the final-output and intermediate rules at both widths. dpt-hash-offline.json is committed unchanged, with per-parameter initializer/std/fan-in metadata and per-output scale/error/decision measurements. Five known hash answers and bit-identical second tensor builds are asserted.

Each design was declared before it ran and its evidence measurement ran once. The final recipe was not adjusted after seeing results. The native-init records also remain unchanged. The periodic pattern's coherent low-rank matrices caused amplification even after fan-in scaling; the final hash removes that repeated structure while matching native initializer scales.

As approved by the maintainer, CONTRIBUTING now specifies name-and-position integer hashes scaled to each native initializer's standard deviation for model numerical references. AGENTS corrects the cross-reference to the bold “Model references follow the same rule” paragraph under “Add a block”. The graph fixtures' _fixed_weights implementation stays unchanged.

Base revision: 127f66c.

Changed files by layer:

  • Numerical references/tests: DPT reference.py, diagnostics.py, new initialization.py, and test_vit_b16_dpt224.py.
  • Checkout tooling: vit_b16_dpt224_numerics.py and the compact fixture's description in vit_dpt.py.
  • Evidence/documentation: the three new offline records; vit-b16-dpt224.md, converter.md, api.md, model.md; the approved CONTRIBUTING sentence and AGENTS cross-reference.
  • No production model, block, converter or graph-gate implementation changed relative to main.

Validation on clean merge head 08406f7, including main at 127f66c (two threads; heavy jobs serialized):

  • graph_identity.py --check, using l00-golden-graphs.json: 36 fixture graphs equal; 9 trained graphs not compared because checkpoints are unavailable.
  • graph_identity.py --check-ext, using x00-g1ext-golden-graphs.json: 19/19 equal. SegFormer asset-backed raw inputs differ from the golden environment; all compared graphs remain byte-identical.
  • block_catalogue.py --check: in sync.
  • live_fixtures.py --offline vit_dpt_reduced: passed, zero cloud calls. Largest absolute mirror/PyTorch errors: reassemble0 3.4924596548080444e-10; reassemble1 2.9103830456733704e-10; reassemble2 5.820766091346741e-10; reassemble3 5.238689482212067e-10; fusion0/1/2 9.313225746154785e-10 each; fusion3 1.862645149230957e-9; head/logits 9.313225746154785e-10 each.
  • live_fixtures.py --plan vit_dpt_reduced: 7 requests, 0 assets/tasks, largest payload 1,944,662 bytes and 31,360 values; within budget. Fixture weights remain unchanged, so no new live run was needed.
  • Focused pytest: test_vit_b16_dpt224.py, test_smp_dpt_converter.py, test_timm_vit_converter.py: 92 passed. Includes DPT converter boundaries at both widths, all seven pinned reduced live hashes, and all 68 Prithvi preservation graphs. The reduced inline converter gate compares native SMP literals on both sides, independently of the hash numerical references.
  • Same tests with timm and SMP blocked by temporary sitecustomize: 14 passed, 8 skipped, no failures. No packages changed.
  • The full offline suite remains for CI.

Final one-shot numerical result:

Reference Head error vs float32 Logit error vs float32 Largest mirror/float64 error Argmax flips, every output
Full width 2.059340476989746e-5 2.0056962966918945e-5 4.644907568440715e-5 0
Reduced 7.599592208862305e-6 7.286667823791504e-6 3.139735947632616e-5 0

All 20 intermediates per width meet the float64 ordering rule. Full-width fusion3 peaks at 51.930015563964844 (float32 step 3.814697265625e-6); mirror/float32 error is 0.00018787384033203125, mirror/float64 error 4.644907568440715e-5, and float32/float64 error 0.00018706553028202677. This is not a 1e-4 intermediate agreement claim against float32. Reduced fusion3 peaks at 28.775081634521484 (step 1.9073486328125e-6).

Graph identity, offline mirror agreement, reduced live agreement, and model accuracy remain separate. The new numerical references have offline evidence only. The compact reduced live fixture remains unchanged. Full-width live execution remains deferred; model accuracy is unmeasured. No weights, inputs, calibration or tolerances were adjusted after the final comparison.

@noelEOS
noelEOS marked this pull request as ready for review November 3, 2026 19:59
@noelEOS
noelEOS merged commit 7b1f5db into main Nov 3, 2026
3 checks passed
@noelEOS
noelEOS deleted the models/dpt-integer-reference branch November 3, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant