Skip to content

qwen27: pin the pair against the single Mac in the test suite - #1435

Open
akol1 wants to merge 3 commits into
ashhart:mainfrom
akol1:qwen27/pair-tests
Open

akol1 wants to merge 3 commits into
ashhart:mainfrom
akol1:qwen27/pair-tests

Conversation

@akol1

@akol1 akol1 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Step 4 of 6 of the two-Mac Qwen3.8-27B series, rebased onto main (94b90c9db), stacked on step 3 (qwen27/pair-lifecycle, eb8f417cf) and step 2's qwen27/rank-slice (1af209fce); step 1 (#1432) is merged and its content arrived in 3f0a715c. Step 5 stacks on this commit.

The test suite gains the pair-vs-single gate. zig build tf-qwen27-tp-check builds it (zig/build/qwen27.zig:111), and the suite launches it; it needs --model, so a host without the checkpoint prints SKIP qwen27-tp-check: no --model DIR, so this host has no checkpoint to check and the suite passes rather than fails. Stage 1 loads the checkpoint once, then rank 1 through loadShard, and requires the same greedy ids, the same logits bytes and the same prompt-state digest; stage 2 starts two rank-local shards joined by pair.Loopback (a host-memory link with the pair's seam ABI) and requires both ranks to emit stage 1's ids, reporting pair_logits_byte_equal, pair_first_logits_difference and bad_pick_at, and the merged top two when a mismatch names one. --stage 1 runs the single-Mac parity only, --stage 2 the loopback pair only, and --stage 3 (the default) both. --taps 0,1,2,3,4 hashes five layers' post-block residuals on every side so a divergence names its layer, --prompt-file/--forced-file replay ids and picks, and --dump prints each side's top two and state digests as [dump]/[state] lines (a side whose top two is unavailable prints top2=error in place of the line's ids).

--chunk N feeds a prompt in N-row rounds through prefill, and --decode-prompt feeds the same rounds through decodeChunk — an earlier reply's arithmetic rather than the prompt kernels, which is the path a served chat's follow-up takes. N may reach 512, model.promptRows(); a Mac whose prompt side fuses takes all 512 rows a round, elsewhere a round is 128 rows and a wider chunk is fed in those rounds. The pair's stages keep the 128 rows the seam window allows, and a wider --chunk with --stage 2 is itself refused (error.PairChunkWiderThanTheSeamWindow) rather than silently narrowed. The review point was that --chunk had to reach decodeChunk and that the round size had to reach 512: an earlier head of this PR fed --decode-prompt one row a round whatever --chunk said, so the earlier-reply path had no round-size control. Both are wired now, and the --decode-prompt receipts below show --chunk 512 and --chunk 1 reaching the same tokens, with prompt_arithmetic: "decode" in both.

zig/tests/qwen27/tp_report.zig holds the digests and the dump lines the receipt quotes: stateDigest/committedDigest over both KV caches, the GDN states and the slot offsets, tapDigests over five post-block planes, and mergeTop2, which shifts a rank's local top two to the global vocabulary for the mismatch report. The receipt itself is JSON on stdout with rank0_tokens, rank1_tokens, reference_tokens, ranks_token_equal, reference_token_equal, first_difference, single_top2, pair_merged_top2, pair_logits_compared, pair_logits_byte_equal, pair_first_logits_difference, bad_pick_at, the two seam_gave_up counters, both prompt-state digests, chunk, prompt_arithmetic, and the memory triple (rank0_alone_resident_bytes, both_ranks_resident_bytes, second_rank_increment_bytes) against device_max_working_set.

A rank's verify never runs a tail. forward.zig's finish runs the head seam first and hands the reopened encoder back to the tail (zig/src/families/qwen27/forward.zig:230, :241); when the seam is head == .none it is skipped and the tail still runs, which is the one-Mac path. Because the head writes only this rank's vocabulary columns while the tail's matcher reads the whole row, decode_round.zig:221 refuses a round that has both with error.TailOnRank before the cache checks or any GPU work; zig/qwen27_bank_boundary_host.zig:53 pins that refusal with model-free stubs, and from step 3 on a pair refuses --drafter at the front door, so the two never meet.

Receipt

  • Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: this commit (2d224ac7f46bf5aaf2e6539d2dff06835b47f5de) on main (94b90c9db65dcb71bbc39715901d6fb1f00407d0) plus steps 2-3; no MLX or PyTorch; macOS 27.0.1 on an Apple M4 Pro (Mac16,11), Metal only (applegpu_g16, no tensor units), Apple metal version 32023.921; checkpoint TensorFold/Qwen3.8-27B-MLX-4bit, revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c, 4-bit affine, group 64, loaded from one directory on both ranks. The receipts below were re-run on this commit on 2026-10-30; the earlier table was recorded on the pre-rebase tree, and every row of the table reproduced these values when run here against the checkpoint above.

  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: the loopback pair caught a single-Mac vs pair token difference at the first token with error.PairTokensDifferFromSingleMac when the head merge was deliberately made to stride one Mac's 24 heads instead of the rank's 12; with the merge fixed, the stage-2 runs below report pair_logits_byte_equal: true and pair_first_logits_difference: null, and no bad_pick_at. The two 512-row (--ids 512) --decode-prompt receipts below differ in exactly the chunk key (512 and 1): the chunk reaches decodeChunk, the tokens and the prompt-state digest are the same, and prompt_arithmetic is decode in both. A 512-row run through the prompt kernels reaches the same [1011,220] with prompt_arithmetic: "prompt", so the option changes which arithmetic runs, not the picks.

  • Decode speed before and after, from tools/bench_openai.py: n/a, no served pair in this commit; the pair's decode rate is step 5's receipt.

  • Prompt speed before and after: n/a, this commit adds a test gate; the chunk cap it pins is a correctness bound (the seam window), not a throughput change.

  • Tests run, with pass, fail and skip counts: zig build test: 584/584 steps succeeded; 8771/8824 tests passed (53 skipped) (run here with a cold local cache, so these are the counts of the whole suite; every step succeeded), exit 0; that run launches the gate, and its log prints SKIP qwen27-tp-check: no --model DIR, so this host has no checkpoint to check (exit 0, so a host without the checkpoint passes instead of failing). zig build test-kernel-contract: 20/20 steps succeeded; 141/141 tests passed, exit 0. zig build dev-tools: 220/220 steps succeeded. zig build test-qwen27-served-host: 7/7 steps succeeded; 48/49 tests passed (1 skipped), exit 0. zig build tf-qwen27-tp-check: 42/42 steps succeeded (build only; the suite is what launches it with --model). zig fmt --check build.zig zig: clean. Filtered contract runs: SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 34 ms, limits 121 ms and SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 30072 ms, limits 119 ms. Lean rule: lean: 424 problems, 424 on the base, 0 new (tools/zig/lean_new.py, exit 0).

    The gate's own receipts, all with --model on the checkpoint above:

    Every run below exits 0 on --model "$CHECKPOINT" (the 4-bit MLX checkpoint named in the Environment bullet) except the last, which is the refusal; each line is a receipt this host printed, not a summary of one.

    Run (on this commit) What it pins
    tf-qwen27-tp-check --model "$CHECKPOINT" --ids 4 --steps 8 stage 1: shard1_tokens_equal: true, shard1_logits_byte_equal: true, shard1_first_logits_difference: null, shard1_prompt_state_equal: true; stage 2 (the loopback pair): ranks_token_equal: true, reference_token_equal: true, first_difference: null, bad_pick_at: null, seam_gave_up_rank0: 0, seam_gave_up_rank1: 0, tokens [828,318,951,2445,8,310,279,1001] on both ranks and on the single Mac
    … --ids 4 --steps 8 --dump stage 2 pair_logits_compared: true, pair_logits_byte_equal: true, pair_first_logits_difference: null, and [logits] steps_compared=8 first_differing_step=null — the full 248,320-wide row was hashed on every decoded step, not only the two picks
    … --ids 512 --steps 2 --chunk 512 --stage 1 the prompt path at the served round: tokens [1011,220], shard1_tokens_equal: true, shard1_logits_byte_equal: true, prompt_arithmetic: "prompt"
    … --ids 512 --steps 2 --chunk 512 --decode-prompt --stage 1 the same 512 rows through decodeChunk's 128-row rounds: tokens [1011,220], shard1_tokens_equal: true, prompt_arithmetic: "decode", and a different single_mac_prompt_state than the prompt-kernel run — the digest shows the other arithmetic actually ran
    … --ids 512 --steps 2 --chunk 1 --decode-prompt --stage 1 compared with the row above, the two receipts differ in exactly the chunk key: the 128-row rounds and the 1-row rounds reach the same tokens and the same prompt-state digest
    … --ids 128 --steps 2 --chunk 128 --decode-prompt --dump the pair's widest allowed round with decode arithmetic: stage 2 pair_logits_compared: true, pair_logits_byte_equal: true, pair_first_logits_difference: null, bad_pick_at: null, tokens [68,220], [audit] x_rank0=514 x_rank1=514 mismatched_exchanges=0 first_bad=null

    A wider chunk under a pair stage is refused by name instead of panicking: --ids 512 --steps 2 --chunk 512 --stage 3 exits 1 with error: PairChunkWiderThanTheSeamWindow.

  • Not run, and why: a two-Mac pair_accept.py run (step 5's scripts); this gate's loopback link is the same seam ABI over host memory, and the two-Mac receipt is step 5's, recorded before the series was rebased onto this base (the second Mac is gone now). CUDA, HIP/RDNA, M1-M3, M5 and Windows are not in this room.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. — the gate decodes one token a step through the lane rounds; both sides are greedy, and the pair is the loopback of the same lanes, not a serial comparison path.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — the gate compares greedy ids only and both sides run with no drafter; --decode-prompt is the earlier-reply arithmetic a resumed chat takes, and its receipts above match the prompt-kernel rounds token for token.
  • No precision traded for speed. If the bits change, the description says which sums change and why. — no arithmetic changes in this commit; pair_logits_byte_equal is the assertion that the pair's bits are one Mac's bits, not merely its tokens.
  • A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. — no GPU kernel changes in this commit; the filtered contract runs in the receipt are the regression check for the pair's four records core: add the row-split pair seam and split simd_qmm over K #1432 put on main.
  • Prompt processing is no slower than the last release. — n/a, this commit adds a test gate; the chunk differences above are a correctness comparison at different round sizes, not a speed comparison.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. — Metal on an M4 Pro (applegpu_g16); no CUDA, HIP, M1-M3, M5 or Windows.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. — the gate is new in this commit: the parent has no tf-qwen27-tp-check step (no such step, the same shape core: add the row-split pair seam and split simd_qmm over K #1432's body reported) and no gate source; the gate fails a pair whose merged logits or ids drift from the single Mac ([logits] first_differing_step, bad_pick_at, error.PairTokensDifferFromSingleMac), and the planted wrong stride in the exactness bullet above is the one this comparison catches. Without --model the built gate prints its SKIP line and exits 0, which the suite does on hosts without the checkpoint.
  • Comments and docstrings are one line. No measurements or history in the source. — lean: 424 problems, 424 on the base, 0 new.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines. — the gate takes the checkpoint directory as --model DIR and names no host or user path.

Stacked on: qwen27/rank-slice, qwen27/pair-lifecycle. This commit is step 4 of 6.

@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Keep this gate. It's the right one: tp_check.zig returns PairTokensDifferFromSingleMac on any difference. Two problems with the receipt:

  • It shows first_difference at 236 and 7, so the gate fails on both fixtures, yet the PR reports "0 failures".
  • --stage 2 on its own always fails with StageTwoNeedsTheSingleMacStream, because the reference is set only in stage 1. The receipt says stage 2 ran on its own.

When the seam is exact, both fixtures should pass with no exemption.

@akol1
akol1 force-pushed the qwen27/pair-tests branch 3 times, most recently from 42debf0 to 0c2df0e Compare October 27, 2026 22:38
@akol1

akol1 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Gate kept as you asked, and both receipt problems are gone on 0c2df0ec:

  • Every receipt is --stage 3, so stage 1 runs first and stage 2 uses its reference; stage 2 never claims to run on its own. Run as --stage 2 alone it still returns error.StageTwoNeedsTheSingleMacStream by design.
  • first_difference is null on all four fixtures (31-id chat at 96 and 256 steps, 4-id short at 8 steps with --chunk 1, in full and mlp), as are pair_first_logits_difference, bad_pick_at and both seam_gave_up_* counts. ranks_token_equal, reference_token_equal, pair_logits_compared and pair_logits_byte_equal are all true, with no exemption in the code or in the body.

@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks @akol1 for keeping the gate strict. Any token difference, any logits difference under --dump, disagreeing ranks or a seam that gave up all fail, and stage 2 still refuses to run without stage 1's reference. Two asks:

  1. The gate never runs a multi-row round outside the prompt kernel. Session.prefill always sets kernels.prompt (zig/src/families/qwen27/session.zig:12), and decode steps are one row. But a served chat feeds every earlier reply through decodeChunk in multi-row rounds, and that path runs q27_attn_merge, the stride bug in qwen27: rank-slice the 27B family and gather the whole embedding table #1433's first point. Please add an option that feeds the prompt through decodeChunk at --chunk N, in both stages. It should be red on the current heads and green once the merge takes the rank's heads. If qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 keeps 512-row chunks on the M5, --chunk must reach them too.
  2. It inherits core: add the row-split pair seam and split simd_qmm over K #1432's conflicts; please rebase.

Next we will run stage 3 on an M5 Ultra. There the projections take the lane kernels rather than simd_qmm, so none of these cells has run yet: full and mlp at chunk 1 and 128, a 512-id prompt, and the decoded rounds, all with --dump.

@akol1
akol1 force-pushed the qwen27/pair-tests branch 2 times, most recently from 053b403 to 64afee0 Compare October 28, 2026 04:09
@akol1

akol1 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed and force-pushed: qwen27/pair-tests is now 64afee0d, rebased onto current main (db3fd5ab), with the body rewritten around this head and the receipt table built on the six-step tree.

  1. --chunk now reaches decodeChunk, in both stages, and it reaches 512. --chunk N feeds a prompt in N-row rounds through prefill; --decode-prompt feeds the same rounds through decodeChunk — an earlier reply's arithmetic. N may reach 512 (model.promptRows()), and a Mac whose prompt side fuses takes all 512 rows a round. The pair's stages keep the 128 rows the seam window allows, and a wider --chunk with --stage 2 is refused by name (error: PairChunkWiderThanTheSeamWindow) rather than silently narrowed.

    Red before: an earlier head of this PR fed --decode-prompt one row a round whatever --chunk said, and the loopback gate caught the difference at the first token with error: PairTokensDifferFromSingleMac when the merge was deliberately made to stride one Mac's 24 heads — [headin] single/shard1 rank=0 in_hash 2478266508041925645 sums0 -1.759460 against [headin] pair rank=0|1 in_hash 14694376810239997583 sums0 -1.871216.

    Green now, from tf-qwen27-tp-check on the refreshed six-step tree: --ids 512 --steps 2 --chunk 512 --decode-prompt --stage 1 and --chunk 1 differ in exactly the chunk key — both reach [1011,220], the same single_mac_prompt_state, and prompt_arithmetic: "decode"; the 128-row pair round with --decode-prompt --dump reports pair_logits_compared: true, pair_logits_byte_equal: true, pair_first_logits_difference: null, bad_pick_at: null, [audit] x_rank0=514 x_rank1=514 mismatched_exchanges=0 first_bad=null, [logits] steps_compared=2 first_differing_step=null. Those rows, and the 512-row stage-3 refusal, are in the body's table.

  2. Rebased onto db3fd5ab; the body's environment bullet cites this head.

Gates on 64afee0d: zig build test 565/565 steps succeeded; 319/322 tests passed (3 skipped); test-kernel-contract 43/43; dev-tools 216/216; zig build tf-qwen27-tp-check 42/42 steps succeeded; zig fmt --check build.zig zig clean; pair SUMMARY 15 kernels: 15 pass gates 1-4, 0 fail, 0 known findings; 35 controls: 35 red as planted, 0 missed; 0 limit problems; gates 295 ms, limits 140 ms; simd SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 31418 ms, limits 106 ms; lean 428 problems, 428 on the base, 0 new. What I cannot run here is stage 3 on an M5 — the lane-kernel cells you named (full and mlp at chunk 1 and 128, the 512-id prompt, the decoded rounds, all with --dump) are green on this Mac through simd_qmm and unexercised through the lane kernels, which is the run you said you would make.

@ashhart

ashhart commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Thanks @akol1. --decode-prompt feeds the prompt through decodeChunk at --chunk N in both stages, --chunk reaches 512 for the single Mac, and a wider chunk under a pair stage is refused by name rather than narrowed. The planted 24-head stride going red at the first token is the evidence we wanted. We'll run the cells we listed on an M5 Ultra, as set out on #1432.

@akol1
akol1 force-pushed the qwen27/pair-tests branch 3 times, most recently from 1d8db14 to c7114a1 Compare October 28, 2026 17:00
@akol1

akol1 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — and thank you for the cell list; it is what makes the lane path checkable from here. Both asks are in on the current head c7114a1a, rebased onto dd0a9cc5, and the rebase did not touch them:

  1. --decode-prompt takes the prompt through decodeChunk in --chunk N rounds in both stages: feedPrompt (zig/tests/qwen27/tp_check.zig:49-57) branches to s.decodeChunk(ids[first..end], end == ids.len) and is called by both the stage 1 and stage 2 paths (:84, :170), so the gate runs the multi-row merge a served chat runs, not only the prompt kernel. --chunk accepts up to 512 (:310); on a pair stage a wider round is refused by name, error.PairChunkWiderThanTheSeamWindow (:311-312), rather than narrowed, so the flag cannot quietly mean a different arithmetic than the one it names. The planted 24-head stride is what turns that path red at the first token: error: PairTokensDifferFromSingleMac after the [headin] rounds, the run I posted earlier, and the code it exercises is the merge this branch fixes.
  2. Rebase: c7114a1a on dd0a9cc5.

Fresh receipts on c7114a1a: zig build test 576/576 steps succeeded; 1195/1215 tests passed (20 skipped), exit 0; zig build dev-tools 216/216; zig build test-kernel-contract 43/43; zig build tf-qwen27-tp-check 42/42; the pair filter SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 35 ms, limits 133 ms, the simd filter SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 34017 ms, limits 125 ms; lean 425 problems, 425 on the base, 0 new.

The lane kernels are still the one thing no M4 can show, so your stage-3 M5 Ultra run — full and mlp at chunk 1 and 128, the 512-id prompt, and the decoded rounds, all with --dump — remains the receipt for the projections path itself.

@akol1
akol1 force-pushed the qwen27/pair-tests branch from c7114a1 to c1b3040 Compare October 29, 2026 04:21
@akol1

akol1 commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the current main (7c068895) and force-pushed; the head is now c1b30409d115e6ff0a0a09506feee473cd2e6fd3. The series order is unchanged, and these are re-run numbers from the rebased head, not carried over.

zig build test: 576/576 steps succeeded; 328/331 tests passed (3 skipped); zig build test-kernel-contract: 43/43; zig build dev-tools: 216/216; zig build tf-qwen27-tp-check: 42/42; zig fmt --check build.zig zig clean; filtered pair run gates 32 ms, limits 104 ms, filtered simd run gates 29761 ms, limits 103 ms; lean rule 425 problems, 425 on the base, 0 new.

The gate's own receipts in the body were re-read on the rebased final tree (7718b1c2): --ids 4 --steps 8 prints stage 1 shard1_tokens_equal: true, shard1_logits_byte_equal: true, shard1_prompt_state_equal: true, and stage 2 ranks_token_equal: true, pair_logits_byte_equal: true, bad_pick_at: null with 0 gave-ups; --dump prints [logits] steps_compared=8 first_differing_step=null; the 512-id prompt prints [1011,220] at --chunk 512 and at --chunk 1, and both decode runs share the prompt-state digest 69abf87a… (the prompt-kernel run's is b511cad4…); the pair's 128-row round prints [68,220] and [audit] x_rank0=514 x_rank1=514 mismatched_exchanges=0 first_bad=null; the 512-row pair chunk still exits 1 with PairChunkWiderThanTheSeamWindow.

This room is one Mac now: the second Mac the pair receipts were recorded on is gone (the cluster was broken), so a fresh two-Mac pair_accept.py run is not possible here. The bodies keep that run labelled as a dated receipt from before the rebase and put the single-Mac loopback gate (tf-qwen27-tp-check) and the host tests next to it. No other question from the earlier comments is still open.

@akol1
akol1 force-pushed the qwen27/pair-tests branch from c1b3040 to bd05c1a Compare October 30, 2026 03:05
@akol1

akol1 commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (581cbe65, the head when the series was rebased; main has since taken 7a9fc417, which only changes tools/glm_index_emulate/emulate.cpp, and GitHub reports these branches conflict-free against it). qwen27/pair-tests is bd05c1a0697fb4fdfa5d263db1ea70e2a5677eba, one commit on step 3 (ebb337d4).

The gate's own receipts were re-run on this head against the checkpoint TensorFold/Qwen3.8-27B-MLX-4bit at 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c, not carried over:

  • --ids 4 --steps 8: stage 1 tokens [828,318,951,2445,8,310,279,1001], shard1_tokens_equal: true, shard1_logits_byte_equal: true, state 3f9bca28d03c38995491017e57f2f4656f87586102bf2882ada3a9a2b489f2f1; stage 2 ranks_token_equal: true, reference_token_equal: true, first_difference: null, bad_pick_at: null, both seam_gave_up counters 0.
  • --ids 4 --steps 8 --dump: [logits] steps_compared=8 first_differing_step=null; stage 2 pair_logits_compared: true, pair_logits_byte_equal: true, pair_first_logits_difference: null, [audit] x_rank0=2824 x_rank1=2824 mismatched_exchanges=0 first_bad=null.
  • --ids 512 --steps 2 --chunk 512 (prompt arithmetic): tokens [1011,220], shard1_tokens_equal: true, shard1_logits_byte_equal: true, state b511cad44fa322437e6e9317823351b4967b058ec9055362ba0d7b5b85397bb5.
  • --ids 512 --steps 2 --chunk 512 --decode-prompt: prompt_arithmetic "decode", tokens [1011,220], equal, state 69abf87a5c17544f16e9420400b0e8afab3447a99005333b53e778233f5e980a.
  • --chunk 1 --decode-prompt: the same tokens [1011,220] and the same state 69abf87a… as --chunk 512, so the round size reaching decodeChunk changes the rounds, not the arithmetic.
  • --ids 128 --steps 2 --chunk 128 --decode-prompt --dump: [68,220] on every side, pair_logits_compared: true, pair_logits_byte_equal: true, pair_first_logits_difference: null, [audit] x_rank0=514 x_rank1=514 mismatched_exchanges=0 first_bad=null.
  • --ids 512 --steps 2 --chunk 512 --stage 3 exits 1 with error: PairChunkWiderThanTheSeamWindow, the refusal by name rather than a silent narrowing.

Numbers from this head, run here, not carried over: zig build test 580/580 steps succeeded; 335/340 tests passed (5 skipped); zig build test-qwen27-served-host 7/7 steps succeeded; 37/38 tests passed (1 skipped); zig build tf-qwen27-tp-check 42/42 steps succeeded; zig build test-kernel-contract 20/20 steps succeeded; 114/114 tests passed; zig build dev-tools 216/216 steps succeeded; zig fmt --check build.zig zig clean; filtered pair run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 35 ms, limits 127 ms; filtered simd run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 32565 ms, limits 128 ms; lean rule 425 problems, 425 on the base, 0 new.

The suite's own tf-qwen27-tp-check step in this room prints SKIP qwen27-tp-check: no --model DIR, so this host has no checkpoint to check, so the gate rows above are the explicit --model runs. Stage 2 is the loopback pair, not two Macs: the second Mac the pair receipts were recorded on is gone, so the two-Mac pair_accept.py receipt stays dated.

@ashhart

ashhart commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Thanks @akol1. Every point from our earlier reviews is answered on this head, and the gate's own commit is unchanged since we last looked; it rebases cleanly onto current main. Nothing more is needed from you here. This lands right after #1433 and #1434, which it sits on (#1434 has one fix to make first), and we'll run the M5 Ultra stage-3 cells we listed before it goes in.

@akol1
akol1 force-pushed the qwen27/pair-tests branch from bd05c1a to 2e13027 Compare October 30, 2026 16:11
@akol1

akol1 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: qwen27/pair-tests is 20179805, one commit on step 3 (e550d50a), onto main (40cb15b4ebbbae4aa5ef028b264363bfd5a18524), which carries the cost-rule retarget itself, so the tree under this gate is green. The gate commit itself is unchanged: three of its four files (qwen27_core.zig, tp_check.zig, tp_report.zig) are byte-identical to the head you reviewed, and the fourth (families/qwen27/qwen27.zig) differs only by main's own advance inside it. Under this branch the non-main deltas are the step-3 qwen27_link.Workers move, the retired qwen27_warm.zig (the timing is main's qwen27_costs.zig).

The loopback gate was run here at this head against the 4-bit checkpoint, and it reproduces the body's receipt table: the 8-step fixture with [audit] x_rank0=2824 x_rank1=2824 mismatched_exchanges=0 first_bad=null and [logits] steps_compared=8 first_differing_step=null; the 128-row decode dump [68,220] with x_rank0=514 x_rank1=514; the 512-row prompt and decodeChunk runs agreeing on [1011,220] with different single_mac_prompt_state digests; and --ids 512 --steps 2 --chunk 512 --stage 3 refused with error: PairChunkWiderThanTheSeamWindow.

Numbers from this head, run here, not carried over: zig build test 580/580 steps succeeded; 8565/8617 tests passed (52 skipped); zig build test-qwen27-served-host 7/7 steps succeeded; 43/44 tests passed (1 skipped); zig build tf-qwen27-tp-check 42/42 steps succeeded (the step builds and skips without --model); zig build dev-tools 216/216 steps succeeded; zig build test-kernel-contract 20/20 steps succeeded; 138/138 tests passed; zig fmt --check build.zig zig clean; filtered pair run gates 36 ms, limits 133 ms; filtered simd run gates 32300 ms, limits 125 ms; lean rule 425 problems, 425 on the base, 0 new.

The M5 Ultra and M3 Ultra stage-3 cells are yours, as listed; this room has an M4 Pro only. Nothing else changes on a single Mac.

@akol1
akol1 force-pushed the qwen27/pair-tests branch 3 times, most recently from 2017980 to 46ee8b3 Compare October 30, 2026 19:23
@akol1

akol1 commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Rebased: qwen27/pair-tests is 46ee8b32, one commit on step 3 (161b323f), onto main (40cb15b4ebbbae4aa5ef028b264363bfd5a18524), which carries the cost-rule retarget itself, so the tree under this gate is green. Against the head you reviewed, the only change is the slotResume fix step 3 now carries: four files under zig/src/native, +15/-8, inherited here unchanged; there is no base advance, since both heads sit on main's 40cb15b4. This commit's own patch is byte-identical to the one you reviewed.

The loopback gate was run here at this head against the 4-bit checkpoint, and it reproduces the body's receipt table: the 8-step fixture with [audit] x_rank0=2824 x_rank1=2824 mismatched_exchanges=0 first_bad=null and [logits] steps_compared=8 first_differing_step=null; the 128-row decode dump [68,220] with x_rank0=514 x_rank1=514; the 512-row prompt and decodeChunk runs agreeing on [1011,220] with different single_mac_prompt_state digests; and --ids 512 --steps 2 --chunk 512 --stage 3 refused with error: PairChunkWiderThanTheSeamWindow.

Numbers from this head, run here, not carried over: zig build test 580/580 steps succeeded; 8565/8617 tests passed (52 skipped); zig build test-qwen27-served-host 7/7 steps succeeded; 43/44 tests passed (1 skipped); zig build tf-qwen27-tp-check 42/42 steps succeeded (the step builds and skips without --model); zig build dev-tools 216/216 steps succeeded; zig build test-kernel-contract 20/20 steps succeeded; 138/138 tests passed; zig fmt --check build.zig zig clean; filtered pair run gates 34 ms, limits 127 ms; filtered simd run gates 32341 ms, limits 125 ms; lean rule 425 problems, 425 on the base, 0 new.

The M5 Ultra and M3 Ultra stage-3 cells are yours, as listed; this room has an M4 Pro only. Nothing else changes on a single Mac.

@akol1
akol1 force-pushed the qwen27/pair-tests branch from 46ee8b3 to e392c10 Compare October 31, 2026 01:45
akol1 added 3 commits October 30, 2026 18:54
Step 2 of 6, stacked on step 1 (`qwen27/pair-seam`) and main
(`40cb15b4`); step 3 stacks on this commit.

A rank's layer program now carries one `Reduce` interface instead of
assuming the whole matrix. Attention's o_proj and the MLP's down_proj
seam this rank's output columns, gather the whole frame, and let the
producer kernel reduce over full K on its own rank; the head seams the
last row's logits and takes the pair's argmax over the whole vocab, so
the picked token is the single Mac's token, not rank 0's half-vocab
pick. On one Mac `reduce` is null and every seam is compiled out of the
program: the single-Mac program, its digests and its tests are the ones
that ran before.

The rank-local K-split is fixed while the slice is. A projection's split
is keyed on the checkpoint tensor's rows, not the view's rows: `Triple`
carries `whole` (the matrix's row count) and `prepare` derives
`tensorRows` from it, while the M5 and prompt tables keep the checkpoint
rows they always keyed. `in_proj_qkv` loads as three blocks of one tensor
(10,240 rows), so its gate/up-style stack keeps main's 4 K slices on
6,144 rows instead of reading the q block's 2,048 rows and splitting into
8; every width main served on one Mac keeps main's split, and host tests
"a rank's row slice keeps the checkpoint matrix's K split" and "every
one-Mac stack keeps main's K split" cover the eight stacks (qkv 10,240
rows: 4, q 6,144: 4, against the 2,048-row view's 8). `Part` drops its
unused `k0`, `Triple` its `k`/`k0`, and `Shard`/the family's private
`cols`/`attentionCols`/`gdnCols`/`downCols` and the column-window code
`prepare`/`packTiles` used them for are gone, with `config_test.zig`'s
column-span assertions deleted and the row fields left as they were.

`simd_qmm.call` takes the rank's `whole` width and `chunks(n, whole)` picks
the chunk fence from it, so a rank's K tree is its own full matrix's while
a one-Mac stack passes 0 and keeps exactly main's fence (the core test
pins `chunks` on 6,144/12,288 and 5,120/10,240 at 8, and on 6,144 and
5,120 alone at 16, which is the pair's own-width split the `gn` forwarding
is there to avoid). `quant_gpu.strided` forwards the `gn` of the linear it
serves.

`q27_attn_merge` takes the rank's head count (buffer 11) and strides a row
by that many heads instead of 24 (`G * 4`), so the non-prompt multi-row
rounds a served chat feeds every earlier reply through merge the rank's
own rows; `attention.zig` binds `g.heads`. The buffered form is only what
a pair runs. One Mac has its 24 heads at compile time, and main's kernel
used them there (`G * 4`, and `PG * 4` in the prompt merge), so each site
now has a second instance: `q27_attn_merge_single` and
`q27_attn_prompt_merge_single` carry main's body verbatim — same
reduction, same float arithmetic, main's indexing, no `heads` or
`kv_heads` buffer — and the encoder takes a trailing `split` and picks
`io[4]`/`prompt[1]` for a pair and `io[10]`/`prompt[2]` for one Mac
(`decode_round.zig` passes `r.reduce != null`). The two instances compute
the same values, so only the emitted code differs. A function constant
would need `MTLFunctionConstantValues`, which the runtime library never
creates (`PipelineBuild` would fail the contract's first gate), so the
second compiled instance is the route that needs no new Metal machinery. The config gains its shard
parse (`config_test.zig` covers it), the embedding table stays whole on
every rank — only the head is sliced — with `embedBound` reading the
table's bound rather than the family's `vocab_size`, and the session keeps
the pair's pick as the round's token and surfaces `error.PairFailed` from
the head seam's sentinel rather than returning `maxInt(u32)`. The seam's
`Reduce` is threaded through `forward.encodeTp`/`session`, and the prompt
attention kernel's signature follows the seam's reopened encoder.
…ith the peer

Step 3 of 6, stacked on steps 1-2 (`qwen27/pair-seam`, `qwen27/rank-slice`)
and main (`40cb15b4`); step 4 stacks on this commit.

The serving path learns the pair: a rank reads its rank-local shard, the
serve loop routes every seam through the link, and the head's pick is
the reply. A pair refuses what it cannot keep: vision, a rank count the
shard does not match, and a drafter whose state only rank 0 holds, and
it fails closed when its watchdog cannot be started. The slot accounting
understands that both ranks hold the frame.

A pair also keeps three serving rules. Its prefill chunk is capped at the
seam window's rows (`pair.chunk`), because each chunk stages a seam and
the window holds `batch_rows` of them; the host test pins that the
served chunk times the widest seam fits `pairBytes()`. A sampled or
grammar-constrained request is refused at the head of `prefillFn`
(`error.PairGreedyOnly`) instead of being answered greedily and failing
on the second token, with a name in `words()` and a host test. And a
pair never resumes a slot in place: rank 1 resets the slot at each
round's `begin` (`link.follow`), so `attach` passes the switch
`Metal.init` read through `lb.slotResume(init, pair)`, which keeps
`TF_QWEN27_SLOT_RESUME=0` off on one Mac and turns resume off for a pair
(`lb.resumesInPlace`), or the ranks would part at the resumed offset; the
host test pins that composition on both paths.

A start-up step that fails after the watchdog is up is undone: the two
threads live in `qwen27_link.Workers`, so `attach` sets their `quit`,
stops following and joins the watchdog, and a host already started is
stopped; `close` joins both before the pair it watched is freed, so
neither thread reads a pair that `open` has already freed.

`qwen27_link.zig` is the pair's link over the existing relay, and holds
the watchdog and rank 1's follower in one `Workers`; `qwen27_pair.zig`
holds the rank-local shard and the seam transport for the serve loop;
main's `qwen27_costs.zig` times the window both ranks share; and
`qwen27_served.zig`/`qwen27_slots.zig`/`qwen27_lanes.zig` the accounting
a shard adds. GLM-5.3's transport is not touched: `pair.zig`/`pair.metal`
are as on main, and no `glm_dsa` file changes in this series.
Step 4 of 6, stacked on steps 1-3 (`qwen27/pair-seam`, `qwen27/rank-slice`,
`qwen27/pair-lifecycle`) and main (`40cb15b4`); step 5 stacks on this commit.

`tf-qwen27-tp-check` is built by `zig build test` and launched by the
suite. It needs `--model`, so CI's run prints `SKIP
qwen27-tp-check: no --model DIR` and carries on, and a host with the
checkpoint gets both stages. Stage 1 loads the model, then rank 1
through `loadShard`, and requires the same greedy ids, logits bytes and
prompt-state digest; stage 2 starts two rank-local shards joined by
`pair.Loopback`, a host-memory link, and requires both ranks to emit
the same ids as the single Mac, with `pair_logits_byte_equal` and
`pair_first_logits_difference: null` reported, and `bad_pick_at` and the
seam give-up counters naming any loss. `--stage 1|2` narrows it and
`--dump` prints each side's top two.

`--decode-prompt` feeds the prompt through `Session.decodeChunk` in
`--chunk N`-row rounds, an earlier reply's arithmetic rather than the
prompt kernels: the multi-row round a served chat gives every earlier
reply, which runs the tree merge (`q27_attn_merge_single` on one Mac) and not the prompt kernels' merge. Both
stages use the same feed, and the receipt reports `prompt_arithmetic:
"decode"` next to the chunk. `--chunk` may then reach 512, the served
chunk (`model.promptRows()`); the pair's stages keep 128, the rows the
seam window holds (`pair.stageBytes`), so `--chunk 512 --stage 2` is
refused by name rather than panicking at the seam. Stage 1 at chunk 128 is
where main's merge stride is exercised: it takes `q27_attn_merge` at
`chunk` rows a round, so the single-Mac logits bytes are the reference the
pair has to match.

`zig/qwen27_core.zig` exports the pair and `zig/build/qwen27.zig`
registers the program; `--forced-file` replays one side's picks on the
other side for divergence localisation, and `--taps` hashes five layers'
post-prompt residuals.
@akol1
akol1 force-pushed the qwen27/pair-tests branch from e392c10 to 2d224ac Compare October 31, 2026 02:25
@akol1

akol1 commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main at 94b90c9db; the head is 2d224ac7f46bf5aaf2e6539d2dff06835b47f5de, the single commit of step 4 on step 3 (qwen27/pair-lifecycle eb8f417cf0c01a7021784a3295f4595f693d2cfd). The base advance carried 30 commits; the two that matter here are #1726's lane loop (q27-m5-lane-loop, merged at 24358c1e2; this gate's loopback lanes exercise it) and 2713b9ee0 (the engine choice behind qwen27_engine.owns). git range-diff against the chain you reviewed reports no hunk change in this commit: the gate file, the two build/qwen27.zig lines and qwen27_core.zig are the patch you read.

tf-qwen27-tp-check builds from zig/tests/qwen27/tp_check.zig through zig/build/qwen27.zig:111 and is a dependency of test (:112); it needs --model, so a host without the checkpoint prints its SKIP line and the suite stays green. Stage 1 runs one Mac and the same model as one shard, stage 2 runs two ranks over loopback and compares both ranks against the single-Mac tokens and logits, stage 3 (the default) runs both. On this head: stage 1 with 512 prompt ids at chunk 512 gives "single_mac_tokens":[1011,220], "shard1_tokens_equal":true, "shard1_logits_byte_equal":true, "shard1_first_logits_difference":null; stage 2 with 4 ids and 8 steps gives "ranks_token_equal":true, "reference_token_equal":true, "first_difference":null, "seam_gave_up_rank0":0, "seam_gave_up_rank1":0, "bad_pick_at":null, "global_vocab":248320, "local_vocab":124160; the 128-id decode case gives [68,220] on both ranks with "pair_logits_compared":true and "pair_logits_byte_equal":true; and --ids 512 --steps 2 --chunk 512 --stage 3 exits 1 with error.PairChunkWiderThanTheSeamWindow, which is the refusal the body's receipt already lists. The same seven rows are byte-identical on the final v27 head 6e8cd9973636de1440ed4690be8c21b6dcd1c45c.

Run here, on this head, not carried over: zig build test 584/584 steps succeeded; 8771/8824 tests passed (53 skipped) (cold local cache, so the whole suite ran; every step succeeded, exit 0); zig build test-kernel-contract 20/20 steps succeeded; 141/141 tests passed; zig build dev-tools 220/220 steps succeeded; zig build test-qwen27-served-host 7/7 steps succeeded; 48/49 tests passed (1 skipped); zig build tf-qwen27-tp-check build-only 42/42 steps succeeded; zig build test-qwen27-bank-boundary-host 7/7 steps succeeded; 4/4 tests passed; zig fmt --check build.zig zig clean; filtered pair run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 34 ms, limits 121 ms; filtered simd run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 30072 ms, limits 119 ms; lean: 424 problems, 424 on the base, 0 new (tools/zig/lean_new.py, exit 0).

This room has M4 Pro hardware only (applegpu_g16, macOS 27.0.1, metal 32023.921, no tensor units), so the M5 and M3 Ultra stage-3 cells you planned remain yours; the loopback rows above are the single-Mac stand-in, and no fresh two-Mac acceptance ran because the second Mac is gone.

The PR body is refreshed with these same receipts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants