qwen27: document two-Mac 27B serving and the row split's exactness by akol1 · Pull Request #1437 · ashhart/TensorFold

akol1 · 2026-10-27T02:56:34Z

What this changes

Step 6 of 6 of the two-Mac Qwen3.8-27B series, rebased onto main (94b90c9db), stacked on step 5 (qwen27/pair-tools, 3b6e996d5), on top of steps 2-4; step 1 (#1432) is merged. Docs only, three commits: RUNBOOK.md, docs/known-issues.md and docs/recipes/qwen3.8-27b.md.

The recipe gains a "Two Macs over Thunderbolt" section: both Macs hold the same TensorFold/Qwen3.8-27B-MLX-4bit checkpoint, each serves it from its own directory with its own settings file, rank 1 starts before rank 0, --parallel 1 is required, and requests go to rank 0 only. It says what the split covers (12 of 24 attention heads, 2 of 4 key/value heads, half of the linear attention's heads and value channels, 8,704 of 17,408 MLP columns, 124,160 of 248,320 lm_head rows), that each rank computes its own output rows at the full-K reduction a single Mac uses so no cross-rank partial sum exists, and that embeddings and norms stay whole on both Macs. At a decided token each rank takes the argmax over its own half of the 248,320 logits and pair_pick chooses between the two half-vocab winners; the pair never merges the halves into one whole-vocabulary argmax. A non-finite logit on either rank is flagged by pair_argmax and pair_pick answers its sentinel: the request fails with error.NonfiniteLogits, as on one Mac, and a dropped link fails it with error.PairFailed.

The docs list the refused set by name: --drafter (rank 1 has no drafter checkpoint; the served pair reads the flag the user passes, before the shard or the vision tower loads, so the name is the refusal), --prompt-cache-gib and --learn (each Mac holds only its own halves of a slot's state), --vision (rank 1 replays rank 0's token ids, so an image prompt would run different rows on the two ranks; it is listed with the other refused flags in docs/known-issues.md and RUNBOOK.md), --parallel above 1, and a request that samples or carries a grammar, refused at the head of prefill with speed-up mode: a pair answers greedy only; send the request with sampling and grammar off rather than answered greedily for one token. The formats section says which checkpoints a pair takes: the MLX affine stacks a single Mac reads on Metal, reading the same checkpoint directory on each Mac. The 4-bit TensorFold/Qwen3.8-27B-MLX-4bit stack is the one qualified on a pair — the two-Mac acceptance run and tf-qwen27-tp-check replay one Mac's recorded 4-bit replies against it — while the 2-, 3-, 5-, 6- and 8-bit affine stacks of groups 64 load through the same readers and split the same way, each rank keeping its row slice with the K split staying the checkpoint matrix's, but no pair run has produced a receipt on them yet: a pair serve there is unqualified. The branch's second commit is that qualification, with the receipt behind it. The third makes the non-finite-logit sentence say error.NonfiniteLogits for a healthy link and error.PairFailed only for a broken one, as the maintainer asked, and joins the short line break in the docs/known-issues.md bullet. Nothing CUDA-only is read (no NVFP4, no EXL3); a tied embedding head is refused at load, as it is on one Mac.

RUNBOOK.md gets the two-Mac commands in its speed-up section and the lifecycle in words: rank 1 first and rank 0 within five minutes, the ready-word timeout, the three-second link watchdog that ends a rank when its peer stops signalling so no rank serves alone, and "restart both after a link failure". docs/known-issues.md gains the same limits as an issue-shaped entry: one greedy stream, no drafts, no prompt cache, no learning, the refused flags, and the output-row split with pair_gather joining bf16 halves without arithmetic so pair logits are byte-identical to one Mac's on the recorded fixtures.

A rank's verify never runs a tail. forward.zig's finish runs the head seam first and hands the reopened encoder back to the tail (zig/src/families/qwen27/forward.zig:230, :241); when the seam is head == .none it is skipped and the tail still runs, which is the one-Mac path. Because the head writes only this rank's vocabulary columns while the tail's matcher reads the whole row, decode_round.zig:221 refuses a round that has both with error.TailOnRank before the cache checks or any GPU work; zig/qwen27_bank_boundary_host.zig:53 pins that refusal with model-free stubs, and from step 3 on a pair refuses --drafter at the front door, so the two never meet.

Receipt

Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: this commit (6e8cd9973636de1440ed4690be8c21b6dcd1c45c) on main (94b90c9db65dcb71bbc39715901d6fb1f00407d0) plus steps 2-5; no MLX or PyTorch; macOS 27.0.1 on an Apple M4 Pro (Mac16,11), Metal only (applegpu_g16, no tensor units), Apple metal version 32023.921. No checkpoint is loaded by this commit's gates: it edits three Markdown files, and the tree's tests are step 5's.
Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a, this commit changes no code and makes no exactness claim of its own; the exactness text it documents is measured by step 4's tf-qwen27-tp-check (byte-identical logits, error.PairTokensDifferFromSingleMac on any token difference), which this Mac still runs, and by step 5's pair_accept.py oracle, whose receipt predates the rebase because the second Mac is no longer available.
Decode speed before and after, from tools/bench_openai.py: n/a, docs-only.
Prompt speed before and after: n/a, docs-only.
Tests run, with pass, fail and skip counts: zig build test: 584/584 steps succeeded; 8772/8825 tests passed (53 skipped) (run here with a cold local cache, so these are the counts of the whole suite; every step succeeded), exit 0. zig build test-kernel-contract: 20/20 steps succeeded; 141/141 tests passed, exit 0. zig build dev-tools: 220/220 steps succeeded. zig build test-qwen27-served-host: 7/7 steps succeeded; 49/50 tests passed (1 skipped), exit 0. zig build tf-qwen27-tp-check: 42/42 steps succeeded (it needs --model, so on a host with the checkpoint the suite runs both stages and elsewhere it prints its SKIP line). zig fmt --check build.zig zig: clean. Filtered contract runs: SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 34 ms, limits 121 ms and SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 30077 ms, limits 119 ms. Lean rule: lean: 424 problems, 424 on the base, 0 new (tools/zig/lean_new.py, exit 0).
Not run, and why: nothing code-shaped is skipped by this commit; a two-Mac run is not this commit's evidence (steps 4-5 own it, and the two-Mac receipt lives there). CUDA, HIP/RDNA, M1-M3, M5 and Windows are not in this room.

Checklist

Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. — docs-only; the text describes the lane rounds a pair already executes and says the pair serves one greedy stream.
Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — docs-only; the recipe says a pair drafts nothing and keeps no prompt cache, and points at the gates that compare against a single Mac.
No precision traded for speed. If the bits change, the description says which sums change and why. — the recipe now states that each rank reduces its own output rows at full K, that pair_gather moves bf16 halves without arithmetic, and that no cross-rank partial sum and no tie policy exist.
A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. — no GPU kernel changes in this commit; the filtered contract runs in the receipt are the regression check for the pair's four records core: add the row-split pair seam and split simd_qmm over K #1432 put on main.
Prompt processing is no slower than the last release. — n/a, docs-only.
The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. — Metal on an M4 Pro (applegpu_g16) for the gates; the docs say which formats and platforms a pair takes (Metal affine, no CUDA-only formats) and do not claim a run on them.
New tests fail before the change, pass after it, and skip cleanly without their dependency. — no tests in this commit; the tf-qwen27-tp-check step it documents skips cleanly without --model: the suite run in the receipt prints SKIP qwen27-tp-check: no --model DIR, so this host has no checkpoint to check and exits 0.
Comments and docstrings are one line. No measurements or history in the source. — lean: 424 problems, 424 on the base, 0 new; the three files are prose and the rule does not apply to Markdown, and no measurement or history was added.
No personal data, machine names, internal hosts or local paths. No AI attribution lines. — the added text names no machine, host, address or local path; its commands use the docs' ~/models/qwen3.8-27b form (the RUNBOOK writes it as $HOME/models/qwen3.8-27b), and no peer host is named.

Stacked on: qwen27/rank-slice, qwen27/pair-lifecycle, qwen27/pair-tests, qwen27/pair-tools. This commit is step 6 of 6.

ashhart · 2026-10-27T09:58:24Z

known-issues.md and the recipe present the chat-256 divergence as accepted behaviour, which amounts to a new tie policy. Please drop that once the seam is exact (see core: add the row-split pair seam and split simd_qmm over K #1432).
RUNBOOK says that when rank 0 stops, rank 1 "keeps running until you end it". The code (qwen27_link.watch) exits rank 1 within about 3 s. The code matches the requirement, so please fix the text.

akol1 · 2026-10-27T22:48:26Z

Both text bugs are fixed on 0c500376:

docs/known-issues.md and the recipe no longer present the chat-256 divergence as accepted. They say the pair's logits are byte-identical to one Mac's on the recorded fixtures, and that no cross-rank partial sum exists, so there is no tie policy.
RUNBOOK.md now says rank 1 exits by itself within about three seconds when rank 0 stops, matching qwen27_link.watch, and keeps the "restart both after a link failure" instruction.

ashhart · 2026-10-27T23:59:21Z

Thanks @akol1: the chat-256 divergence is gone from the docs, and the RUNBOOK's rank-1 exit now matches qwen27_link.watch. Three corrections so the docs describe what the code does:

docs/recipes/qwen3.8-27b.md:106 says the pair takes the argmax of the 248,320 logits gathered from both halves. In the code each rank takes the argmax of its own half and pair_pick chooses between the two. No logits are gathered, and until core: add the row-split pair seam and split simd_qmm over K #1432's third point lands, a NaN or Inf is handled differently from one Mac. Please describe that.
:113 and docs/known-issues.md:42-43 say a sampled or grammar request fails. Today a sampled request gets a greedy first token and fails on the second. Please match the text to the refusal qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 adds, and list --vision with the refused flags in the RUNBOOK and known issues.
The recipe names only TensorFold/Qwen3.8-27B-MLX-4bit, but nothing refuses other widths on a pair yet. Once qwen27: rank-slice the 27B family and gather the whole embedding table #1433 settles which formats a pair takes, please say so here.

This also needs #1432's rebase.

akol1 · 2026-10-28T04:13:16Z

Refreshed and force-pushed: qwen27/pair-docs is now 660f7a35, rebased onto current main (db3fd5ab), with the body rewritten around this head. The three corrections:

docs/recipes/qwen3.8-27b.md no longer says the logits are gathered. It now says that at a decided token each rank takes the argmax over its own half of the 248,320 logits and pair_pick chooses between the two half-vocab winners, and that the pair never merges the halves into one whole-vocabulary argmax. It also says that a non-finite logit on either rank is flagged by pair_argmax and fails closed through pair_pick's sentinel as error.PairFailed — so the text now matches what one Mac and a pair do, which is core: add the row-split pair seam and split simd_qmm over K #1432's third point landed rather than a difference the recipe has to describe.
The refusal text matches qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434. The recipe, RUNBOOK.md and docs/known-issues.md now quote the refusal that exists: speed-up mode: a pair answers greedy only; send the request with sampling and grammar off, raised at the head of prefill rather than answered greedily for one token and failed on the next. --vision is listed with the refused flags in both RUNBOOK.md and docs/known-issues.md, with its own message (--vision does not run on a pair) and the reason rank 1 replays rank 0's token ids.
The formats section says which checkpoints a pair takes: the MLX affine stacks a single Mac reads on Metal — the 4-bit it is qualified on, plus the 2-, 3-, 4-, 5-, 6- and 8-bit affine stacks of groups 64 — each rank keeping its row slice while the K split stays the checkpoint matrix's, and nothing CUDA-only (no NVFP4, no EXL3). A tied embedding head is refused at load, as it is on one Mac. This is what qwen27: rank-slice the 27B family and gather the whole embedding table #1433 settles.

Gates on 660f7a35: zig build test 565/565 steps succeeded; 193/196 tests passed (3 skipped); test-kernel-contract 43/43; dev-tools 216/216; tf-qwen27-tp-check 42/42 steps succeeded; zig fmt --check build.zig zig clean; pair SUMMARY 15 kernels: 15 pass gates 1-4, 0 fail, 0 known findings; 35 controls: 35 red as planted, 0 missed; 0 limit problems; gates 280 ms, limits 110 ms; simd SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 30265 ms, limits 104 ms; lean 428 problems, 428 on the base, 0 new. The docs carry no host names, paths or addresses beyond the documented loopback ports.

ashhart · 2026-10-28T08:27:13Z

Thanks @akol1. All three corrections are in on 660f7a35: the recipe describes each rank's half-vocab argmax and pair_pick, the refusal text matches #1434 with --vision listed in the RUNBOOK and known issues, and the formats section says which checkpoints a pair reads. One small thing: please add that only the 4-bit has been checked on a pair so far, so the 2- to 8-bit stacks don't read as qualified before one of them has a stage-3 receipt. This one lands last, after the rest of the series.

akol1 · 2026-10-28T18:46:20Z

Thanks — done on 05bb79a9, the last commit of the series.

The formats paragraph no longer reads as a claim for stacks nothing has run. It now says the 4-bit TensorFold/Qwen3.8-27B-MLX-4bit stack is the one with a pair receipt — the two-Mac acceptance run below and tf-qwen27-tp-check replay one Mac's recorded 4-bit replies against the pair — and that the 2-, 3-, 5-, 6- and 8-bit affine stacks of groups 64 load through the same readers and split the same way, each rank keeping its row slice while the K split stays the checkpoint matrix's, but that no pair run has produced a receipt on them yet, so a pair serve there is unqualified. The sentence that said "the pair reads every format one Mac reads" is gone; so is the implication that the format list is a tested list. The edit against the wording you quoted is one file, +7/−5, docs/recipes/qwen3.8-27b.md, prose only.

That is also what the two-Mac run in step 5 measured: the oracle it replays is the 4-bit checkpoint at revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c, and the five recorded replies match case for case, two-turn case included.

Fresh receipts on 05bb79a9: zig build test 576/576 steps succeeded; 191/194 tests passed (3 skipped), exit 0; zig build dev-tools 216/216 steps succeeded; zig build tf-qwen27-tp-check 42/42 steps succeeded; zig build test-kernel-contract 43/43 steps succeeded; zig fmt --check build.zig zig clean; the pair filter SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 34 ms, limits 127 ms and the simd filter SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 33630 ms, limits 138 ms; lean 425 problems, 425 on the base, 0 new.

This is step 6 of 6, stacked on step 5 (qwen27/pair-tools, d2e1e62b); the series is now in the order you asked for, with this one last.

akol1 · 2026-10-29T04:44:05Z

Rebased onto the current main (7c068895) and force-pushed; the head is now 7718b1c2b54f25b31eb88faf193c626e07e3ad24. The series order is unchanged, and these are re-run numbers from the rebased head, not carried over.

zig build test: 576/576 steps succeeded; 203/206 tests passed (3 skipped); zig build test-kernel-contract: 43/43; zig build dev-tools: 216/216; zig build tf-qwen27-tp-check: 42/42; zig fmt --check build.zig zig clean; filtered pair run gates 34 ms, limits 104 ms, filtered simd run gates 29788 ms, limits 103 ms; lean rule 425 problems, 425 on the base, 0 new.

This room is one Mac now: the second Mac the pair receipts were recorded on is gone (the cluster was broken), so a fresh two-Mac pair_accept.py run is not possible here. The bodies keep that run labelled as a dated receipt from before the rebase and put the single-Mac loopback gate (tf-qwen27-tp-check) and the host tests next to it. No other question from the earlier comments is still open.

akol1 · 2026-10-30T03:08:46Z

Rebased onto main (581cbe65); qwen27/pair-docs is c30fc31395b4d092d4d1b3440f681604d4328e2d, two commits on step 5 (ac0688ce): the docs and the qualification pass over them.

The text now says, in the formats paragraph, that a pair takes Metal affine and no CUDA-only format; in the recipe, that each rank reduces its own output rows at full K, that pair_gather moves bf16 halves without arithmetic, and that no cross-rank partial sum and no tie policy exist; and docs/known-issues.md carries no chat-256 divergence. The kernel-contract lines point at the pair's four records #1432 put on main.

Numbers from this head, run here, not carried over: zig build test 580/580 steps succeeded; 210/215 tests passed (5 skipped); zig build test-qwen27-served-host 7/7 steps succeeded; 38/39 tests passed (1 skipped); zig build tf-qwen27-tp-check 42/42 steps succeeded; zig build test-kernel-contract 20/20 steps succeeded; 114/114 tests passed; zig build dev-tools 216/216 steps succeeded; zig fmt --check build.zig zig clean; filtered pair run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 36 ms, limits 127 ms; filtered simd run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 32438 ms, limits 125 ms; lean rule 425 problems, 425 on the base, 0 new.

This commit edits three Markdown files and makes no code or exactness claim of its own; the exactness and speed rows are n/a, docs-only and point at steps 4-5's gates. The suite's tf-qwen27-tp-check step prints SKIP qwen27-tp-check: no --model DIR, so this host has no checkpoint to check here, so no gate number of this commit's own is claimed.

ashhart · 2026-10-30T10:41:25Z

Thanks @akol1. Everything from the earlier rounds is answered on c30fc31: the chat-256 divergence is gone, the rank-1 exit matches the watchdog, the recipe describes the half-vocab argmax and pair_pick, the refusals quote the real messages with --vision listed, and the formats paragraph says only the 4-bit stack has a pair receipt. As planned, this one waits until #1433 to #1436 have landed.

One correction while it waits. The recipe (and the description) says a non-finite logit is flagged by pair_argmax, pair_pick answers its sentinel, "which error.PairFailed reports". The flag and the sentinel are right, but on a healthy link the code reports error.NonfiniteLogits (Session.greedy in qwen27/session.zig, and picked() in qwen27_lanes.zig). That is the same error one Mac gives. error.PairFailed is only returned when the link itself is broken. Please make the sentence say that: a non-finite logit fails the request with NonfiniteLogits, as on one Mac, and a dropped link fails it with PairFailed.

Small and optional: the docs/known-issues.md bullet has a short line break after "each rank reads the whole input K". Joining it with the next line would read better.

akol1 · 2026-10-30T16:16:31Z

Both points are in, on 3165ba36, now three commits on step 5 (e1302401), rebased onto main (40cb15b4ebbbae4aa5ef028b264363bfd5a18524), which carries the cost-rule retarget itself, so the tree under this gate is green.

The non-finite-logit sentence now names the right error. docs/recipes/qwen3.8-27b.md:124 reads: "the request fails with error.NonfiniteLogits, as on one Mac, and a dropped link fails it with error.PairFailed rather than letting a half-vocab argmax stand in for the whole vocabulary's." That matches the code you pointed at: Session.greedy returns error.PairFailed only when reduce.broken(), and error.NonfiniteLogits otherwise (qwen27/session.zig:38-46), and picked() names only error.NonfiniteLogits (qwen27_lanes.zig:265-283). The description says the same thing now.
The docs/known-issues.md break is joined. The bullet now reads "each rank reads the whole input K and reduces its rows as one Mac does", with the wrap moved to the next phrase.

Within the files this branch touches, docs/known-issues.md is byte-identical to the head you reviewed and docs/recipes/qwen3.8-27b.md differs only where main added its lane-switch paragraph (the non-finite and break edits are the ones you read, moved down six lines). Outside those files the non-main deltas are the step-3 qwen27_link.Workers move, and the retired qwen27_warm.zig; everything else is main's advance.

Numbers from this head, run here, not carried over: zig build test 580/580 steps succeeded; 8566/8618 tests passed (52 skipped); zig build test-qwen27-served-host 7/7 steps succeeded; 44/45 tests passed (1 skipped); zig build tf-qwen27-tp-check 42/42 steps succeeded (builds and skips without --model); zig build dev-tools 216/216 steps succeeded; zig build test-kernel-contract 20/20 steps succeeded; 138/138 tests passed; zig fmt --check build.zig zig clean; filtered pair run gates 36 ms, limits 126 ms; filtered simd run gates 32171 ms, limits 125 ms; lean rule 425 problems, 425 on the base, 0 new. The commit is docs only and still waits for #1433 to #1436.

akol1 · 2026-10-30T19:24:53Z

Both points are in, on 6c0f3418, now three commits on step 5 (5d02ad91), rebased onto main (40cb15b4ebbbae4aa5ef028b264363bfd5a18524), which carries the cost-rule retarget itself, so the tree under this gate is green.

The non-finite-logit sentence now names the right error. docs/recipes/qwen3.8-27b.md:124 reads: "the request fails with error.NonfiniteLogits, as on one Mac, and a dropped link fails it with error.PairFailed rather than letting a half-vocab argmax stand in for the whole vocabulary's." That matches the code you pointed at: Session.greedy returns error.PairFailed only when reduce.broken(), and error.NonfiniteLogits otherwise (qwen27/session.zig:38-46), and picked() names only error.NonfiniteLogits (qwen27_lanes.zig:265-283). The description says the same thing now.
The docs/known-issues.md break is joined. The bullet now reads "each rank reads the whole input K and reduces its rows as one Mac does", with the wrap moved to the next phrase.

Within the files this branch touches, every one is byte-identical to the head you reviewed: the non-finite and break edits are the ones you read. The only change against that head is the slotResume fix step 3 now carries (four files under zig/src/native, +15/-8, unchanged through steps 4-6), and there is no base advance.

Numbers from this head, run here, not carried over: zig build test 580/580 steps succeeded; 8566/8618 tests passed (52 skipped); zig build test-qwen27-served-host 7/7 steps succeeded; 44/45 tests passed (1 skipped); zig build tf-qwen27-tp-check 42/42 steps succeeded (builds and skips without --model); zig build dev-tools 216/216 steps succeeded; zig build test-kernel-contract 20/20 steps succeeded; 138/138 tests passed; zig fmt --check build.zig zig clean; filtered pair run gates 37 ms, limits 132 ms; filtered simd run gates 32385 ms, limits 126 ms; lean rule 425 problems, 425 on the base, 0 new. The commit is docs only and still waits for #1433 to #1436.

Step 2 of 6, stacked on step 1 (`qwen27/pair-seam`) and main (`40cb15b4`); step 3 stacks on this commit. A rank's layer program now carries one `Reduce` interface instead of assuming the whole matrix. Attention's o_proj and the MLP's down_proj seam this rank's output columns, gather the whole frame, and let the producer kernel reduce over full K on its own rank; the head seams the last row's logits and takes the pair's argmax over the whole vocab, so the picked token is the single Mac's token, not rank 0's half-vocab pick. On one Mac `reduce` is null and every seam is compiled out of the program: the single-Mac program, its digests and its tests are the ones that ran before. The rank-local K-split is fixed while the slice is. A projection's split is keyed on the checkpoint tensor's rows, not the view's rows: `Triple` carries `whole` (the matrix's row count) and `prepare` derives `tensorRows` from it, while the M5 and prompt tables keep the checkpoint rows they always keyed. `in_proj_qkv` loads as three blocks of one tensor (10,240 rows), so its gate/up-style stack keeps main's 4 K slices on 6,144 rows instead of reading the q block's 2,048 rows and splitting into 8; every width main served on one Mac keeps main's split, and host tests "a rank's row slice keeps the checkpoint matrix's K split" and "every one-Mac stack keeps main's K split" cover the eight stacks (qkv 10,240 rows: 4, q 6,144: 4, against the 2,048-row view's 8). `Part` drops its unused `k0`, `Triple` its `k`/`k0`, and `Shard`/the family's private `cols`/`attentionCols`/`gdnCols`/`downCols` and the column-window code `prepare`/`packTiles` used them for are gone, with `config_test.zig`'s column-span assertions deleted and the row fields left as they were. `simd_qmm.call` takes the rank's `whole` width and `chunks(n, whole)` picks the chunk fence from it, so a rank's K tree is its own full matrix's while a one-Mac stack passes 0 and keeps exactly main's fence (the core test pins `chunks` on 6,144/12,288 and 5,120/10,240 at 8, and on 6,144 and 5,120 alone at 16, which is the pair's own-width split the `gn` forwarding is there to avoid). `quant_gpu.strided` forwards the `gn` of the linear it serves. `q27_attn_merge` takes the rank's head count (buffer 11) and strides a row by that many heads instead of 24 (`G * 4`), so the non-prompt multi-row rounds a served chat feeds every earlier reply through merge the rank's own rows; `attention.zig` binds `g.heads`. The buffered form is only what a pair runs. One Mac has its 24 heads at compile time, and main's kernel used them there (`G * 4`, and `PG * 4` in the prompt merge), so each site now has a second instance: `q27_attn_merge_single` and `q27_attn_prompt_merge_single` carry main's body verbatim — same reduction, same float arithmetic, main's indexing, no `heads` or `kv_heads` buffer — and the encoder takes a trailing `split` and picks `io[4]`/`prompt[1]` for a pair and `io[10]`/`prompt[2]` for one Mac (`decode_round.zig` passes `r.reduce != null`). The two instances compute the same values, so only the emitted code differs. A function constant would need `MTLFunctionConstantValues`, which the runtime library never creates (`PipelineBuild` would fail the contract's first gate), so the second compiled instance is the route that needs no new Metal machinery. The config gains its shard parse (`config_test.zig` covers it), the embedding table stays whole on every rank — only the head is sliced — with `embedBound` reading the table's bound rather than the family's `vocab_size`, and the session keeps the pair's pick as the round's token and surfaces `error.PairFailed` from the head seam's sentinel rather than returning `maxInt(u32)`. The seam's `Reduce` is threaded through `forward.encodeTp`/`session`, and the prompt attention kernel's signature follows the seam's reopened encoder.

…ith the peer Step 3 of 6, stacked on steps 1-2 (`qwen27/pair-seam`, `qwen27/rank-slice`) and main (`40cb15b4`); step 4 stacks on this commit. The serving path learns the pair: a rank reads its rank-local shard, the serve loop routes every seam through the link, and the head's pick is the reply. A pair refuses what it cannot keep: vision, a rank count the shard does not match, and a drafter whose state only rank 0 holds, and it fails closed when its watchdog cannot be started. The slot accounting understands that both ranks hold the frame. A pair also keeps three serving rules. Its prefill chunk is capped at the seam window's rows (`pair.chunk`), because each chunk stages a seam and the window holds `batch_rows` of them; the host test pins that the served chunk times the widest seam fits `pairBytes()`. A sampled or grammar-constrained request is refused at the head of `prefillFn` (`error.PairGreedyOnly`) instead of being answered greedily and failing on the second token, with a name in `words()` and a host test. And a pair never resumes a slot in place: rank 1 resets the slot at each round's `begin` (`link.follow`), so `attach` passes the switch `Metal.init` read through `lb.slotResume(init, pair)`, which keeps `TF_QWEN27_SLOT_RESUME=0` off on one Mac and turns resume off for a pair (`lb.resumesInPlace`), or the ranks would part at the resumed offset; the host test pins that composition on both paths. A start-up step that fails after the watchdog is up is undone: the two threads live in `qwen27_link.Workers`, so `attach` sets their `quit`, stops following and joins the watchdog, and a host already started is stopped; `close` joins both before the pair it watched is freed, so neither thread reads a pair that `open` has already freed. `qwen27_link.zig` is the pair's link over the existing relay, and holds the watchdog and rank 1's follower in one `Workers`; `qwen27_pair.zig` holds the rank-local shard and the seam transport for the serve loop; main's `qwen27_costs.zig` times the window both ranks share; and `qwen27_served.zig`/`qwen27_slots.zig`/`qwen27_lanes.zig` the accounting a shard adds. GLM-5.3's transport is not touched: `pair.zig`/`pair.metal` are as on main, and no `glm_dsa` file changes in this series.

Step 4 of 6, stacked on steps 1-3 (`qwen27/pair-seam`, `qwen27/rank-slice`, `qwen27/pair-lifecycle`) and main (`40cb15b4`); step 5 stacks on this commit. `tf-qwen27-tp-check` is built by `zig build test` and launched by the suite. It needs `--model`, so CI's run prints `SKIP qwen27-tp-check: no --model DIR` and carries on, and a host with the checkpoint gets both stages. Stage 1 loads the model, then rank 1 through `loadShard`, and requires the same greedy ids, logits bytes and prompt-state digest; stage 2 starts two rank-local shards joined by `pair.Loopback`, a host-memory link, and requires both ranks to emit the same ids as the single Mac, with `pair_logits_byte_equal` and `pair_first_logits_difference: null` reported, and `bad_pick_at` and the seam give-up counters naming any loss. `--stage 1|2` narrows it and `--dump` prints each side's top two. `--decode-prompt` feeds the prompt through `Session.decodeChunk` in `--chunk N`-row rounds, an earlier reply's arithmetic rather than the prompt kernels: the multi-row round a served chat gives every earlier reply, which runs the tree merge (`q27_attn_merge_single` on one Mac) and not the prompt kernels' merge. Both stages use the same feed, and the receipt reports `prompt_arithmetic: "decode"` next to the chunk. `--chunk` may then reach 512, the served chunk (`model.promptRows()`); the pair's stages keep 128, the rows the seam window holds (`pair.stageBytes`), so `--chunk 512 --stage 2` is refused by name rather than panicking at the seam. Stage 1 at chunk 128 is where main's merge stride is exercised: it takes `q27_attn_merge` at `chunk` rows a round, so the single-Mac logits bytes are the reference the pair has to match. `zig/qwen27_core.zig` exports the pair and `zig/build/qwen27.zig` registers the program; `--forced-file` replays one side's picks on the other side for divergence localisation, and `--taps` hashes five layers' post-prompt residuals.

Step 5 of 6, stacked on steps 1-4 (`qwen27/pair-seam`, `qwen27/rank-slice`, `qwen27/pair-lifecycle`, `qwen27/pair-tests`) and main (`40cb15b4`); step 6 stacks on this commit. `tools/qwen27/pair_accept.py` replays the checked-in single-Mac oracle against a served pair: every case is sent twice, both replies must be identical, and their `token_sha` must equal the oracle's, with no case exempt. `tools/qwen27/pair_supervise.sh` starts rank 1 on `$PEER` and rank 0 locally, restarts both when either stops answering, stops only the pids it started (never by pattern), and verifies the peer's host key rather than turning the check off. `qwen38_27b_oracle.json` is the single-Mac, undrafted, temperature-0 record they compare against. The oracle records the GPU family it was taken on (`apple-m4` here), and `pair_accept.py` refuses to replay it against a serve that reports a different family, or one whose family it cannot read, instead of comparing tokens across two GPUs as if they were the same machine. The two-turn case (`chat-two-turn-128`) opens with the recorded `chat-128` reply and replays the oracle's second turn, so the acceptance run covers the multi-row `decodeChunk` rounds a chat takes, not just single-turn prompts; `record_oracle.py` records it the same way. `pair_supervise.sh`'s `PEER_STATE` default is now peer-relative (rank 1's own `${TMPDIR:-/tmp}/tensorfold-pair`, expanded there) rather than rank 0's local directory, so a supervision run does not need rank 0 to know rank 1's filesystem layout; a caller-set `PEER_STATE` is used as given, and a peer TMPDIR that cannot be read stops the script with exit status 1 instead of carrying an empty path into `down`. `up` returns each start-up step's failure itself (`check_binaries || return 1`, `start_rank1 || return 1`, `start_rank0 || return 1`): `watch` calls it as `if up`, and sh switches `set -e` off inside a function an `if` tests, so a reported failure alone would leave both ranks starting on different builds and a failed start sitting out the whole `READY_TIMEOUT`. Nothing in either script names a machine, a user path or an address: every location is environment-overridable, and the shim's stop path is the pidfile.

…fuse a pair serve Two review findings on the oracle and its recorder, and the pair's own side of the second one. `qwen38_27b_oracle.json` carried one top-level `runtime`, `tensorfold-native 1.1.2`, though `chat-two-turn-128` was recorded later. Each case now carries the runtime that recorded its own `token_sha`: `code-128`, `code-256`, `chat-128` and `chat-256` say `tensorfold-native 1.1.2`, and `chat-two-turn-128` says `tensorfold-native 1.1.7`, so a case names the exact binary that must reproduce it. The description says so. `pair_accept.py` compares `token_sha`, not the build, so its cases are unchanged. `record_oracle.py` reads the serve's `/health` before it sends any completion. It refuses (exit 2, a plain line on stderr) a serve whose `policy` line reports the two-Mac pair speed-up mode, because rank 0 of a pair passes the `chat-128` turn-1 check just as well, and an oracle must come from one Mac. It also refuses a serve whose `/health` is unreadable or carries no version string. The reported version becomes the new case's `runtime`, so the receipt names the build that produced it, and the turn-1 `chat-128` equality and decimal-token-id hash checks are unchanged. That refusal could not fire before this commit: `engine_api.Info.policy` existed and the server prints it under `/health`, but the qwen27 host left it empty, so a pair serve reported no policy line at all. `qwen27_served` now names the mode when it attaches a pair, and its host test pins the line, so a pair's `/health` carries "speed-up mode: a two-Mac pair, rank 0 answers greedy requests only" and the recorder refuses that serve. `pair_accept.py`'s warm case asked a pair for a resumed prompt state, but a pair keeps no prompt cache (its slots are one and its prompt states are two Macs', so `qwen27_served.attach` enables none), so that case failed by construction on any pair. It is now `cold`: it drives a second turn in the same conversation and passes when the pair reports no cached prompt tokens, pinning the documented "keeps no prompt cache" limitation instead of contradicting it.

Step 6 of 6, stacked on steps 1-5 (`qwen27/pair-seam`, `qwen27/rank-slice`, `qwen27/pair-lifecycle`, `qwen27/pair-tests`, `qwen27/pair-tools`) and main (`40cb15b4`). The recipe and the runbook now carry the two-Mac 27B case: both Macs hold the same checkpoint, each keeps its rank-local half of every row-split projection, `--parallel 1`, requests to rank 0 only, and the watchdog that ends a rank when its peer stops signalling. The known issues page says what the pair does not carry: no drafter, no prompt cache, no learning, one greedy stream, and no `pair_known` or `TF_DUMP_IDS` shortcut. The seam splits by output rows with the full K on each rank, and its gather moves bf16 halves without arithmetic, so there is no fp32 partial sum and no tie policy; the pair's logits are byte-identical to one Mac's on the recorded fixtures and `tf-qwen27-tp-check` fails on any token difference with no exemptions. The recipe says what the pair actually does at a decided token: each rank takes the argmax of its own half of the 248,320 logits and `pair_pick` chooses between the two half-vocab winners, and a non-finite logit on either rank is flagged by `pair_argmax` and fails the request through `pair_pick`'s sentinel as `error.NonfiniteLogits`, while a broken link reports `error.PairFailed` — the pair never takes a whole-vocabulary argmax over a merged buffer. A sampled or grammar-constrained request is refused at the head of prefill by name, not answered greedily for one token; `--vision` is listed with the other refused flags in the recipe and in `docs/known-issues.md`. The formats section says which checkpoints a pair takes: the MLX affine stacks a single Mac reads on Metal, each rank keeping its row slice while the K split stays the checkpoint matrix's, and nothing CUDA-only.

…eipt The recipe said a pair reads every affine stack one Mac reads, and listed 2- through 8-bit beside the 4-bit checkpoint it is qualified on. Only the 4-bit stack has a pair receipt -- the two-Mac acceptance run and tf-qwen27-tp-check replay one Mac's recorded 4-bit replies -- so the other affine widths now read as formats the pair loads through the same readers and splits the same way, with no recorded run behind them yet.

The recipe said a non-finite logit on either rank reaches the request as `error.PairFailed`. `pair_argmax`'s flag and `pair_pick`'s sentinel are right, but on a healthy link `Session.greedy` and `picked()` report `error.NonfiniteLogits`, the error one Mac gives; `error.PairFailed` is reserved for a link that broke. The sentence now says both, and the `docs/known-issues.md` bullet joins a short line break after "each rank reads the whole input K".

akol1 · 2026-10-31T02:52:02Z

Rebased onto main at 94b90c9db; the head is 6e8cd9973636de1440ed4690be8c21b6dcd1c45c, three commits on step 5 (3b6e996d5398926148f68e38688d5ae0c7282bd2), all part of step 6; they touch only RUNBOOK.md, docs/known-issues.md and docs/recipes/qwen3.8-27b.md. git range-diff against the chain you reviewed reports no hunk change in any of the three commits; the base advance only moved the docs' references to main forward.

The non-finite-logit wording now matches one Mac: docs/known-issues.md says a rank whose logit is non-finite fails the request with error.NonfiniteLogits, as a single Mac does, and a dropped link fails it with error.PairFailed rather than error.NonfiniteLogits. The recipe's format section carries the same sentence, qualified to the 4-bit stack, the one stack the pair has a receipt on. The code behind those words is unchanged from steps 2-3: pair_argmax flags a non-finite logit and pair_pick answers its sentinel, and the served path maps the sentinel to error.PairFailed only when the peer is broken.

Run here, on this head, not carried over: zig build test 584/584 steps succeeded; 8772/8825 tests passed (53 skipped) (cold local cache, so the whole suite ran; every step succeeded, exit 0); zig build test-kernel-contract 20/20 steps succeeded; 141/141 tests passed; zig build dev-tools 220/220 steps succeeded; zig build test-qwen27-served-host 7/7 steps succeeded; 49/50 tests passed (1 skipped); zig build tf-qwen27-tp-check build-only 42/42 steps succeeded; zig build test-qwen27-bank-boundary-host 7/7 steps succeeded; 4/4 tests passed; zig fmt --check build.zig zig clean; filtered pair run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 34 ms, limits 121 ms; filtered simd run SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 30077 ms, limits 119 ms; lean: 424 problems, 424 on the base, 0 new (tools/zig/lean_new.py, exit 0).

This room has M4 Pro hardware only (applegpu_g16, macOS 27.0.1, metal 32023.921, no tensor units); the M5 and M3 Ultra cells in the matrix remain yours.

The PR body is refreshed with these same receipts.

ashhart · 2026-11-01T09:27:32Z

Thanks @akol1. Both points are in: the recipe now says a non-finite logit fails with NonfiniteLogits, as on one Mac, and only a dropped link with PairFailed, which is what Session.greedy does. The known-issues bullet reads well joined. We checked the docs against steps 3 to 5's code: the refusal messages, the five-minute ready wait, the three-second watchdog and the half-vocab pick all match. One small optional addition: docs/speed-up-mode.md is still titled for Flash Next and ends "Flash Next 6-bit only", and the 27B section links there for setup. A line in that guide pointing to "Two Macs over Thunderbolt" would save a reader the doubt. This lands last, with #1434 to #1436, after our two-Mac run, so the "byte-identical on the recorded fixtures" line stands on a run at this head.

ashhart · 2026-11-01T13:28:05Z

Thanks @akol1. We ran this branch on one Mac (M5 Ultra, Qwen3.8-27B MLX 4-bit) against its main base, base/new/new/base, decode sweeps at 1 and 4 streams:

prompt	temp	streams	base tok/s	this PR	change
chat	0.0	1	161.1	161.1	+0.0%
chat	0.0	4	309.4	304.4	-1.6%
chat	1.0	1	131.6	128.4	-2.4%
chat	1.0	4	243.8	240.4	-1.4%
code	0.0	1	229.8	229.4	-0.2%
code	0.0	4	387.3	390.3	+0.8%
code	1.0	1	200.8	196.1	-2.3%
code	1.0	4	368.6	363.2	-1.5%

Both chats (thinking on and off) qualified. Each change is inside our noise threshold, so the run reads GREEN, but six of eight cells moved the same way. Before this lands we'll rerun it alternating six times each to see whether the 1-2% at 4 streams is real. That's queued after today's 1.2.1 release, so this PR isn't in 1.2.1.

ashhart · 2026-11-01T15:57:19Z

Correction to this follow-up: these six-and-six runs used copy drafts only, while the first run used the DFlash2 drafter (161 tok/s at one stream). So they show no slowdown without the drafter (every cell -0.1% to +3.5%), but they don't settle the 1-2% at 4 streams the first run showed with it. We're rerunning the same six-and-six with DFlash2 now and will post it here.

Without the drafter, base against this branch, medians:

prompt	temp	streams	base tok/s	this PR	change
chat	0.0	1	48.9	49.0	+0.2%
chat	0.0	4	183.2	183.8	+0.3%
chat	1.0	1	51.6	52.0	+0.8%
chat	1.0	4	183.9	184.4	+0.3%
code	0.0	1	48.8	50.5	+3.5%
code	0.0	4	179.7	183.1	+1.9%
code	1.0	1	51.8	51.7	-0.1%
code	1.0	4	182.2	182.5	+0.1%

ashhart · 2026-11-01T17:57:51Z

The six-and-six rerun with DFlash2 is in, on the same M5 Ultra and Qwen3.8-27B MLX 4-bit (medians):

prompt	temp	streams	base tok/s	this PR	change
chat	0.0	1	163.1	161.6	-0.9%
chat	0.0	4	319.7	309.8	-3.1%
chat	1.0	1	128.7	131.4	+2.1%
chat	1.0	4	241.9	244.8	+1.2%
code	0.0	1	234.5	233.3	-0.5%
code	0.0	4	396.5	393.6	-0.7%
code	1.0	1	197.8	197.7	-0.0%
code	1.0	4	370.3	369.1	-0.3%

Five cells are within half a percent, but greedy chat at 4 streams with the drafter is slower in both reruns (-1.6% in the first, -3.1% here). Its six runs each are base 306.1-325.4 and this PR 301.8-317.3. They overlap, but this PR's mean is 2.3% lower. We can't land a slowdown, even a small one, so before we gate it: does anything in qwen27_lanes.zig, decode_round.zig or qwen27_served.zig change what a one-Mac server does per drafted round at 4 streams (an extra check, a copy, or a pair-mode branch taken on one Mac)? If one does, making it free on one Mac should close this. If not, we'll run that one cell alternating 12 times to settle it.

akol1 force-pushed the qwen27/pair-docs branch from e6e6e96 to cb488f2 Compare October 27, 2026 03:58

akol1 marked this pull request as ready for review October 27, 2026 04:32

akol1 force-pushed the qwen27/pair-docs branch 3 times, most recently from 6011c77 to 0c50037 Compare October 27, 2026 22:38

akol1 changed the title ~~qwen27: document two-Mac 27B serving and the seam's known tie~~ qwen27: document two-Mac 27B serving and the row split's exactness Oct 27, 2026

akol1 force-pushed the qwen27/pair-docs branch 2 times, most recently from f041082 to 660f7a3 Compare October 28, 2026 04:09

ashhart mentioned this pull request Oct 28, 2026

core: add the row-split pair seam and split simd_qmm over K #1432

Merged

7 of 9 tasks

akol1 force-pushed the qwen27/pair-docs branch 4 times, most recently from 380ae40 to 05bb79a Compare October 28, 2026 18:37

akol1 force-pushed the qwen27/pair-docs branch from 05bb79a to 7718b1c Compare October 29, 2026 04:21

akol1 force-pushed the qwen27/pair-docs branch from 7718b1c to c30fc31 Compare October 30, 2026 03:05

akol1 force-pushed the qwen27/pair-docs branch from c30fc31 to e38effb Compare October 30, 2026 16:12

akol1 force-pushed the qwen27/pair-docs branch 3 times, most recently from 3165ba3 to 6c0f341 Compare October 30, 2026 19:23

akol1 force-pushed the qwen27/pair-docs branch from 6c0f341 to 48bfaa4 Compare October 31, 2026 01:45

akol1 added 8 commits October 30, 2026 18:54

akol1 force-pushed the qwen27/pair-docs branch from 48bfaa4 to 6e8cd99 Compare October 31, 2026 02:25

kieranscotney mentioned this pull request Oct 31, 2026

qwen27 DFlash: compile the drafter head's lane pipeline at load #1734

Open

7 of 9 tasks

BobClawblaw mentioned this pull request Oct 31, 2026

qwen27: serve Qwen3.5-MoE (Ornith-1.5-122B) on Metal: routed experts, its DFlash2 drafter and the publisher's 4-bit weights #1125

Open

grantoverton mentioned this pull request Nov 1, 2026

qwen27: chunk mixed-bit prompt passes at 128 rows #1822

Merged

ashhart mentioned this pull request Nov 1, 2026

qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434

Open

6 of 9 tasks

BlivionIaG mentioned this pull request Nov 1, 2026

Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B #1835

Open

This was referenced Nov 1, 2026

Qwen3.5-9B DFlash drafter in Zig engine #1851

Open

Qwen3.5-9B decisions in Zig engine #1871

Open

Qwen3.5-9B grow-as-used K/V in Zig engine #1896

Open

Qwen3.5-9B prompt rows in one product in Zig engine #1897

Open

Conversation

akol1 commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Receipt

Checklist

Uh oh!

ashhart commented Oct 27, 2026

Uh oh!

akol1 commented Oct 27, 2026

Uh oh!

ashhart commented Oct 27, 2026

Uh oh!

akol1 commented Oct 28, 2026

Uh oh!

ashhart commented Oct 28, 2026

Uh oh!

akol1 commented Oct 28, 2026

Uh oh!

akol1 commented Oct 29, 2026

Uh oh!

akol1 commented Oct 30, 2026

Uh oh!

ashhart commented Oct 30, 2026

Uh oh!

akol1 commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akol1 commented Oct 30, 2026

Uh oh!

akol1 commented Oct 31, 2026

Uh oh!

ashhart commented Nov 1, 2026

Uh oh!

ashhart commented Nov 1, 2026

Uh oh!

ashhart commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashhart commented Nov 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

akol1 commented Oct 27, 2026 •

edited

Loading

akol1 commented Oct 30, 2026 •

edited

Loading

ashhart commented Nov 1, 2026 •

edited

Loading