[linson007]

Historical source / current-main reconciliation: upstream has merged #487 and the later runtime stack at 1482f6bbc2abf12375332725302019594150b6cb. This draft preserves the original source and scoped evidence; its old runtime flags are not current-main settings. The mergeable qualification tools, updated source audit and consolidated evidence are now in #523. New-source validation and remaining awake replacement runs are in progress.

Summary

Greedy Qwen MoE decode can route experts for a draft suffix with little chance of surviving acceptance. This adds an opt-in, GPU-selected contiguous verification prefix so inactive suffix rows contribute no routed or shared expert routes, while retaining the current eight-row layout and state-management paths.

On the measured M5 Pro/Qwen3.6 GGUF configuration, threshold 0.1 improves native committed-token throughput by 6.2–7.1% at B1–B4. A 9,641-token coding prompt improves HTTP decode throughput by 6.5% with identical measured output. Short-prompt HTTP results remain inconclusive; the feature therefore remains off by default.

Expanded qualification update (October 21): #523 now retains 27 additional GGUF B1 trials over 2k/8k/31,744-token contexts and code/math/prose. Thirteen clear their noise-adjusted speed gate, three are inconclusive, and no stable trial fails the throughput-regression bound. Three 8k trials have changed output hashes (math sample 2; prose samples 0 and 2). These differences are visible in the raw reports; the initial bundle's matching hashes do not establish universal autoregressive equality. Full output budgets remain checked. The larger MLX model is being repeated with one resident server after dual residency failed at startup; B2–B4 and remaining format qualification are still running. Threshold 0 remains the default. See the expanded evidence and #508 for the current scope.

Refs #506 and #280. This refreshes the prototype in #283 against current main and adds completed real-model checks, startup/thinking-budget fixes, ordinary-command tracing and portable evidence. The original PR/branch has not been modified. Please coordinate with #283 and #487 before merging overlapping routing/acceptance changes.

Behavior and correctness boundaries

Enable explicitly:

SPLASH_LIVE_VERIFY_THRESHOLD=0.1 ./splash serve \
  --model unsloth/Qwen3.6-35B-A3B-GGUF:UD-IQ3_XXS

Unset/zero retains full verification. The environment parser and runtime setter reject nonfinite values and values outside [0, 1].

1. draft_verify_live_rows reads the selected path's candidate/unary/edge tables and computes cumulative confidence on the GPU. It does not overwrite sampled proposal probabilities. Confidence is a selector heuristic, not a trained acceptance predictor.
2. For each position, confidence is the softmax of unary + edge(previous_selected, candidate) among the selector candidates, evaluated at the actual selected candidate; prefix confidence is the cumulative product. This is not a full-vocabulary proposal probability or a calibrated target-acceptance estimate. Every eligible lane keeps its anchor and a contiguous prefix. Invalid confidence uses all eight rows. There is no CPU readback between selection and verification.
3. Live MoE grouping omits inactive routed and shared expert routes. The one-lane staged path retains its in-place row positions, avoiding an unnecessary gather. Live combine writes the residual for inactive rows without reading unwritten route entries.
4. Acceptance with L live target rows compares at most L - 1 proposals and uses the final live target row for the continuation when all eligible proposals match. Merely lowering retention would not bound the comparisons; this change explicitly bounds them.
5. Existing retained-prefix KV, pending GDN tape and draft-context paths perform state commits. The old stashed explicit-GDN-commit implementation was not transplanted.

Situation	Verification policy
Unconstrained greedy sparse-MoE lane, positive threshold	Contiguous live prefix
Sampled lane, including a mixed batch	All eight rows
Any constrained lane in the batch	Full batch fallback
Forced thinking-end transition within this cycle	Full batch fallback
Dense target or threshold zero	Existing full-width path
Invalid selector confidence	All eight rows for that lane

The sampled fallback is intentional: truncating from the selected sampled path is not established to preserve the distribution. The existing target/proposal acceptance rule remains in force for sampled requests.

Draft generation, dense projections, attention, GDN arithmetic, head work and physical storage retain their original shapes. Expert tiles can still compute padding. Eligible expert rows are not a count of all target rows executed. Existing drafted-token counters still count generated proposals; they are not redefined as verified proposals.

New metadata has a 32-byte payload per lane and is included in the decode arena memory plan, using existing arena alignment. No new per-cycle Metal allocation or verification-policy CPU synchronization is introduced.

Current-main reconciliation

The implementation preserves current main's indirect MoE grids, quantization paths, BF16 GDN state and pending-tape semantics. Two additional fixes were needed during real-model verification:

• Prepare ordinary acceptance at startup even when enabled greedy warmup takes live acceptance. Otherwise the first later sampled request can compile that pipeline after startup.
• Bypass pruning for a cycle forcing the thinking-end token. That transition is imposed by the budget and must not be delayed by treating it as a low-confidence draft prediction.

The final one-lane refinement preserves the existing staged in-place path. It was qualified across all expert formats and both real-model state formats; see the validation record for the exact order/scope of the checks.

Reviewer map

Review order	Files	What to inspect
1. Policy and ABI	runtime/metal/abi/LiveRows.h, runtime/ops/DraftSelector.*, sampling.metal	Buffer layout, candidate/edge indexing, cumulative confidence, nonfinite fallback and sampled-lane guard
2. Expert work	runtime/ops/MoE.*, shared/moe.metal, QwenTarget.*	Active route grouping, shared routes, indirect tile grids, in-place B1 layout, padding and inactive combine reads
3. Acceptance/runtime	runtime/ops/Sampling.*, runtime/model/Runtime.*	Prefix comparison bound, continuation row, retained state, thinking-cutoff fallback and startup pipeline preparation
4. Memory/tests	RuntimeArenas.*, model_execution_plan_test.cpp, live_verify_rows_metal_test.mm, gguf_moe_test.mm	Planned metadata bytes; all cutoff lengths, formats and widths; poisoned inactive data; fallback behavior
5. Measurement	BackendInstrumentation.hpp, MetalBackend.mm, decode_profile.mm, verification_sweep.py	Benchmark-only tracing, aligned host/GPU clocks, ordinary versus attributed execution, actual committed-token normalization and ABBA noise handling
6. Evidence	Results report and bundle linked below	Raw metrics, hashes, validity bounds, reproducible workload and outstanding qualification

The commit split follows the same boundaries: runtime/policy and tests, tracing and sweep tools, then the source audit/evidence and final validation record.

Measurement setup

• Machine: Apple M5 Pro, 20 GPU cores, 48 GiB; macOS 26.6.0; AC power for reported final runs.
• Compiler: Metal 32023.921. The missing Xcode Metal Toolchain component was installed before building.
• Base: be3b083c5f413673d44e44881042e875e674e333, still current upstream main when preparing this PR.
• MoE: unsloth/Qwen3.6-35B-A3B-GGUF:UD-IQ3_XXS; target revision a483e9e6cbd595906af30beda3187c2663a1118c, draft revision 51ef7b6923ad6c14cb1bb41c37a9041446496ab6.
• Dense fallback/oracles: mlx-community/Qwen3.8-27B-4bit; target revision 10c35caafbb80f7dc6a7a432cdd11af10a6d4818, draft revision 015e795645c74b1a0eeef3b570031fb62e769bc5.
• Timing formats: INT8 KV, FP32 GDN state; native profiling uses GPU execution. Model oracles explicitly disable ANE FFN and cover FP32/BF16 state.

All speed comparisons use off/on/on/off rounds. Per the existing abba.py, spread above 5% is inconclusive; the regression bound is max(2%, 2 × spread). Claimed gains must exceed that bound too. Native runs use the same instrumented binary with the policy toggled; HTTP runs additionally compare a clean-main binary with the candidate.

Native committed-token throughput

2,048-token prompt, 32 measured cycles per width, ordinary fused execution with tracing enabled symmetrically in all arms. Both policies committed the same number of tokens per cycle for each width.

Lanes	Full, tok/s	Threshold 0.1, tok/s	Gain	Spread	Eligible expert rows, on/full
1	114.09	121.12	6.16%	0.20%	6.09 / 8
2	174.37	186.45	6.93%	0.24%	12.28 / 16
3	204.75	219.21	7.06%	0.04%	18.63 / 24
4	248.35	265.22	6.79%	0.12%	25.28 / 32

All four gains exceed the 2% bound. The bundle also includes threshold-0.2 results, which were noisy and do not establish a better setting.

HTTP results, including inconclusive runs

34 GiB maximum memory, 32K maximum context, disk prefix cache disabled, greedy/no-thinking requests and 512 requested output tokens. Throughput derives from /status deltas of decode_cycle_ms / decode_output_tokens, including between-command time. It is serving decode throughput, not total request latency or TTFT.

Workload	Result
9,641-token coding prompt	203.27 → 216.53 tok/s, +6.52%, 0.66% spread; identical message hashes across all four rounds
Short code/math/prose and four concurrent requests	Inconclusive; initial spread 8.8–12.6%
Ten-second warmed rerun of short workloads/batch-four	Still inconclusive, 14–16% spread; thermal state changed from nominal to fair
Candidate disabled versus clean main	Code/math/long-code/batch-four pass the regression bound; prose passes its warmed rerun (+0.22% seconds/token, 0.54% spread)

Every HTTP series in the original evidence bundle has matching output hashes for corresponding requests. Seeded sampled probes match and strict JSON probes are valid. These are finite observations, not proof of universal output equality. No broad HTTP gain, dense speedup, optimal threshold or cross-device default is claimed.

Test results

make -j8 check: PASS (exit 0). This completes the CPU and shader-validated Metal suites, builds every native test/tool, runs full Python discovery (1,270 tests, 27 skipped), compiles Python sources, and passes Ruff lint/format. A separate make check-source after packaging the evidence also passes.

make check does not include the separate sanitizer target, which was not run locally for this publication. GitHub's native CI job runs it; CI status is reported separately in the PR checks.

Local validation details (the linked record identifies exact model/format coverage):

Check	Result
Production build, normal CLI help, CPU engine suite, architecture and whitespace checks	PASS
Shader-validated live-verify-rows, target-sampling, draft-selector, gdn-decode, metal-backend	PASS
Final GGUF MoE full FP64/numerical + bitwise-prefix suite, repeated in make check	PASS
Final gguf-moe --live-only, all supported expert formats, staged/register paths, B1–B4 and every cutoff	PASS
MoE and dense full runtime oracles, INT8 KV, FP32/BF16 state, threshold 0.1	PASS; MoE 0.1 runs precede the final in-place refinement
Final in-place MoE oracles, FP32 and BF16 state, threshold 0.2	PASS
Full Python discovery, compileall, Ruff lint/format	1,270 tests run, 27 skipped; all checks pass (158 files)
Focused benchmark/smoke Python tests	38 passed
Focused sampling/thinking/tools/partial-output Python tests	105 passed
Recompute all published native summaries from CSV and HTTP summaries from raw metric deltas	Exact match, including validity verdicts and output/probe hashes

The runtime oracle covers restore/preemption, same-graph lane equivalence, permutations, sampling, constraints and thinking budgets under its existing numerical contract. Cross-width bitwise identity is not an existing guarantee. GitHub CI for this new head is separate from the earlier PR's CI; no prior CI status is reused.

Evidence and reproduction

These links target this PR's commit rather than ignored local files:

• Detailed results and reproduction commands.
• Evidence bundle/schema, including 1,024 raw native cycles, HTTP metric deltas/usage, environments, output hashes and binary/model fingerprints.
• Validation commands and exact scope.
• Pinned DeepInfer/Splash source audit and conditional roadmap.

The evidence excludes model bytes, generated answer text, absolute workspace paths and complete local logs/traces. The HTTP workload snapshot preserves the measured requests and calculation, with only formatting and a required model-root option replacing the local default. Standard regression/release harnesses remain the rollout gates.

Why the larger changes are deferred

Expert kernels account for roughly 40–50% of attributed decode time, GDN 4–7%. Ordinary native traces show a median GPU-to-next-GPU gap around 0.50–0.53 ms. These observations favor the narrow expert-route experiment first; attribution executes dispatches separately and is not itself a fused speed prediction or full engine-loop decomposition.

This PR does not implement GDN pruning, arbitrary dense row shapes, learned confidence, online policy adaptation, prefill scheduling or device-side next-step propagation. Those require their own measured gates. DeepInfer's trained confidence head and device-side correction pipeline cannot be reproduced by copying its CPU scheduling loop.

Before any default change: repeat the HTTP result under stable clocks; qualify short workloads, other contexts, quantizations and Apple GPU families; cover more model/storage combinations; measure contended TTFT/streaming gaps. #506 tracks that work. There is no configuration migration: remove the environment variable or set it to zero to return to full verification.

[linson007]

Closing this historical runtime experiment in favor of the current-main contribution #523 and roadmap #508. Main now includes #487; the old fixed-threshold routing/acceptance patch overlaps that implementation and its environment flag is not a current-main setting.

The preserved source branch and immutable commits remain available for reproduction. The completed local prefix matrix has 72 four-arm trials across GGUF/MLX, contexts and widths: 30 gains above individual bounds, six inconclusive comparisons and zero throughput regressions beyond tolerance. Three original GGUF cases contain differing hashes, including same-policy repetitions; clean-main versus disabled-candidate exact-prompt replay matches all three in every arm. These are source-pinned historical results, not current-main or shared-default qualification.

#523 retains the complete reports, original exclusions, source/model/build hashes, reproduction helper and current all/auto qualification tools. It contains no production code from this prototype. The original #283 branch/stash and #280 issue are preserved.