Repository navigation
Model L oracle: both width-32 blocks in the FP16 form, FP8 op model switch, decoder skip; reproducible negative control - #218
Merged
Conversation
… control against the current kernels The decoder block of Model L's oracle used Model M's block arithmetic (float64 sums, exp softmax with the soft cap, the kernels' GELU, the float norm), the same form as the C32 kernels, so the kernels passing it did not test the block form. The decoder block now runs in its FP16 form (oracle._block_tokens_q7, table consts form "q7"): - every FP8 matrix op is one 32-term sum started from an FP16 value and rounded once to FP16; chunk chains ascending; - residuals enter as accumulator starts: the Wp chain starts from RN16(bp + x), the W2 chain from RN16(b2 + z); - the RMS norm sums FP16 squares in a fixed even / odd lane tree, adds 2^-13, and uses the gain RN16(gamma x sqrt 32); - the attention weights come from an FP16 polynomial and a bit shift, with an FP16 row-sum tree and an approximate reciprocal; the attention product P V is one FP16 op; - the MLP runs in four 32-wide slices with the FP16 GELU polynomial. The encoder block keeps Model M's form until its form is confirmed (table PROVISIONAL "E7"). Negative control (clean/l/records/l_q7_decoder_oracle_negative_control.json, real-weight Model L, 11 frames, CPU on a recorded HIP run): the current kernels pass the Model M-form decoder oracle on 11 of 11 frames and fail the FP16-form oracle on 11 of 11 (A, B, latent codes and C values far over the 0.5 % rule). The C32 kernels are re-implemented in the FP16 form in a follow-up; until then l_frame_check's decoder stage fails on Model L by design. Tests: test_l_reference Q7FormTest (known answers of the attention weights, GELU and norm, the norm's sum order, zero weights pass the input, residual starts, decoder "q7" / encoder "m"). Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
phantoslayer21
marked this pull request as ready for review
November 15, 2026 00:11
…coder skip; reproducible negative control; known answers Review of #218: - FP8 matrix-op model switch (table "mm8", oracle.MM8_MODELS): "exact" (the exact sum of the products and the FP16 start, rounded once; the default) or "trunc13" (two blocks of 16 products, the start in block 1, terms truncated toward zero to 13 bits below the largest, block 1's FP16 result block 2's start). It applies to every FP8 op of Model L's own stages (input linears, the 2x2 projection, the decoder projection, the blocks); records name it. - The negative control is a checked-in program (clean/l/runner/l_q7_negative_control.py): one recorded l_frame_check run checked against the Model M-form oracle and the FP16-form oracle in one process, with the command line, the runtime commit, the recorded run's file and frame SHA-256, the model's and map's SHA-256, the runtime library's SHA-256 and the per-frame alignment checks in its record. The old record is removed; the regenerated one follows. - Known answers with nonzero weights, worked out independently (an exact rational reference with its own FP16 / FP8 rounding, and a whole block worked out by hand), each with negative controls that mutate the oracle (test_l_q7_known: chunk order, a missing chunk, a rounding moved across a chunk boundary, the wrong W2 slice, MLP slice order, per-product accumulation, the other op model as default, ties away from zero, subnormals flushed, overflow saturated, the attention-weight, GELU and norm constants and trees, the trunc13 model's quantum, rounding instead of truncation and block size). Also: - The FP16 block form for any width 32 / 64 / 96 / 128 (the norm's tree per width, gain RN16(float32(gamma) x sqrtf(C))), K / 32 chains for every linear. - The encoder block now runs in the FP16 form too (its input is the FP16 result of the second input linear, as before). - The decoder block reads the encoder block's FP8 output as a skip: each window member's input is RN16(projection + skip), the projection taken with the coarse border and the skip with the fine mirror about the grid size. encode() returns the skip, network() and l_frame_check hand it to decode(). - The ten shared stages still run Model M's oracle (table PROVISIONAL "S8"). Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…f4da7 (l_q7_negative_control.py, FP8 op model exact) Model M-form oracle 11 of 11 frames (dec0 and enc0), FP16-form oracle 0 of 11, frames aligned 11 of 11; command line, runtime commit, recorded-run and frame SHA-256, model and map SHA-256 and the runtime library's SHA-256 in the record. Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…fore a check; the FP8 op model in every comparison record l_frame_check --run writes into the recording the checkout's git commit (git rev-parse HEAD, run by the runner), whether tracked files differ from it, and the SHA-256 of the runtime library, params.pkl and source.json. l_q7_negative_control.py re-hashes the files it loads and refuses when the recording has no identity or a hash differs; its record carries the verified identity (no caller-supplied commit). Every Model L comparison record names the effective FP8 op model (also when defaulted), per frame and per record, and l_frame_check refuses to write one without it (check_l_record). Tests: test_l_records. Review of #218 (REVIEW-218b). Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…identity (4ef40d2, verified) Model M-form oracle 11 of 11 frames (dec0 and enc0), FP16-form oracle 0 of 11, aligned 11 of 11, FP8 op model exact; the identity the runner captured (git commit, runtime library, params.pkl, source.json SHA-256) re-hashed and verified before the comparison. Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…hanges flag is required; every oracle comparison writer names the FP8 op model capture_identity raises when git rev-parse or git status fails or cannot run, so a failure never reads as a clean tree; verify_identity requires git_tracked_changes to be present and boolean. clean/l/runner/l_record.py write_l_record adds the effective FP8 op model (also when the table leaves it out) and checks it; l_frame_check (both comparisons), l_enc_check, l_ops_check, l_c32_check, l_default_post_check, l_d3d12_own_post_check and l_q7_negative_control write through it. test_l_records: both capture failure paths, a missing or non-boolean flag, the writer with a defaulted table, and that every runner comparing with the oracle writes through the writer. The negative-control record is rewritten through it from the same recorded run (results unchanged). Review of #218 (REVIEW-218c). Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #89
Model L's oracle: both width-32 blocks now use the FP16 form, and the decoder block reads the encoder's skip. The
current C32 kernels fail it, and that's the point of this PR.
Until now Model L's blocks in the oracle used Model M's block arithmetic, the same form the C32 kernels compute. The
kernels passing that oracle therefore said nothing about the block form.
What changes
clean/l/reference/oracle.py,table.py:The FP16 block form (
_block_tokens_q7; tableconstsform: "q7", now the default for both of Model L's blocks;any width 32 / 64 / 96 / 128):
ascending order (q / k / v from 0, W1 slices from b1, merges and projections from their bias).
chain from RN16(b2 + z).
approximate reciprocal. P V is one FP16 op.
FP8 op model switch (table
mm8,oracle.MM8_MODELS): what happens inside one FP8 op is not settled.exact(default): the exact sum, rounded once to FP16.trunc13: two blocks of 16 products, the start value in block 1, terms truncated toward zero to 13 bits below thelargest, block 1's FP16 result is block 2's start.
blocks). Every record names the model it used.
Decoder skip (table
skip_map, swin opskip):fine mirror about the grid size.
encode()returns the skip;network()andl_frame_check.compare_modelhand it todecode().Shared stages: the ten shared stages still run Model M's oracle (
PROVISIONAL["S8"], a follow-up).clean/l/runner/l_pack.pyandclean/l/README.mdfollow.Negative control (reproducible):
clean/l/runner/l_q7_negative_control.pyOne recorded
l_frame_checkrun (HIP, every kernel separate) is checked in one process, frame by frame on the run'sown stage inputs (
compare_model), against both:Record
clean/l/records/l_q7_oracle_negative_control.jsonholds:Record (commit 4ef40d2, captured by the runner at run time with the SHA-256 of the runtime library, params.pkl and source.json, and verified by the check before it compares; real-weight Model L (model SHA-256 d3c04019...ad806, map 58e6d0e7...e857), l_frame_check's 11 frames, HDR, HIP run with every kernel separate, FP8 op model exact; recorded-run and frame SHA-256 in the record):
With the FP16 form, per frame 10078-18089 of 18432 A codes and 10367-18408 of 18432 C values differ; with the Model M form at most 3 and 6.
Every Model L comparison record names the FP8 op model (also when defaulted): per record and per frame in
l_frame_check --compare. Every check that compares with the oracle writes its record throughclean/l/runner/l_record.pywrite_l_record:l_frame_check(both comparisons),l_enc_check,l_ops_check,l_c32_check,l_default_post_check,l_d3d12_own_post_checkandl_q7_negative_control. The writer adds the effective model, also when the table leaves it out, and refuses a record without it or with a contradicting frame.test_l_recordstests the writer with a defaulted table, and checks that every runner comparing with the oracle writes through it. A failed or unrunnablegit statusorgit rev-parsefails the capture, and the verification requires the tracked-changes flag to be present and boolean. Both failure paths are tested.Known answers with nonzero weights:
clean/l/tests/test_l_q7_known.pyExpected values are worked out independently of the oracle: an exact rational reference with its own FP16 / FP8
round-to-nearest-even, and a whole block (nonzero Wv, Wp, b1, W2, biases) worked out by hand. Every check has
negative controls that mutate the oracle and must fail it:
The earlier tests are updated for the skip and for the encoder's FP16 form.
Consequence:
l_frame_check's Model L encoder and decoder stages fail on the current kernels by design. The C32kernels (HIP and D3D12) are re-implemented in the FP16 form in a follow-up PR, checked against this oracle.
Tested / Not tested
test_l_reference,test_l_q7_known,test_l_recon,test_l_pack,test_l_map: 75 OK (incl. test_l_records);test_model_tables9 OK;test_l_runtime_pack4 OK;exactis the default, and the kernels will follow it).