[linson007]

Historical source / current-main reconciliation: upstream has merged #487 and the later runtime stack at 1482f6bbc2abf12375332725302019594150b6cb. This draft preserves the original source and scoped evidence; its old runtime flags are not current-main settings. The mergeable qualification tools, updated source audit and consolidated evidence are now in #523. New-source validation and remaining awake replacement runs are in progress.

This is the default-off scheduler experiment for step 6 of #508. A single global milliseconds-per-token estimate can underpredict long-history prefill and miss a non-leading expensive lane in a ragged batch. SPLASH_PREFILL_COST_MODEL=bucketed uses bounded prior observations by actual rows, row-weighted history, batch width and the currently available GPU/ANE split. An absent variable or ema preserves the baseline policy.

Draft: local production serving qualification is still running. No latency improvement or default change is claimed. The initial single-resident control traces establish actual prefill/decode contention, but their ABBA throughput spread is too large for a speed claim. Earlier dual-resident dense traces exercised ANE handoffs but serialized admission; they are excluded from contended-latency evidence.

Review map and dependencies

This is stacked on measurement base bc09da81 from #507/#509. Review the estimator implementation in 02a1162e, the preceding exact-geometry trace extension dd62043b, dedicated scheduler sanitizer integration d870fe7c, and subsequent analysis/docs/evidence commits. The later harness commits include matched warmups, native overlap validation, stale-status rejection, cached/prefix-resume modes and single-resident ABBA rounds; the same fixes are separately reviewed in #509. The existing MoE prefix is independently controlled and unchanged here. #510 contains the other-device maintainer package; #511 is a separate GDN experiment.

Implementation files:

• Scheduler.hpp/.cpp: fixed 96-cell cost table, conservative fallback and provisional ragged prediction; approximately 1.5 KiB per engine, no persistence or unbounded growth.
• Bootstrap.mm, Engine.hpp/.cpp, main.mm: opt-in configuration, resource-lifetime-safe path callback and exact after-cut prediction/seed/lane diagnostics.
• ragged_scheduler_test.cpp, dev/native.mk: new policy/fairness cases and ASAN/UBSAN/TSAN targets.
• prefill_prediction.py, trace_report.py, associated Python tests: prequential analysis with every lane's history; GPU/ANE and widths stay separate.
• dev/benchmarks/prefill-cost-model.md: complete contract, commands, caveats and promotion gate.
• dev/benchmarks/results/qwen-prefill-cost-m5-pro/: control reports, raw prefill-only trace excerpts with source hashes, prediction records, focused test logs and a qualification ledger. The ledger explicitly marks unfinished gates.

Behavior and invariants

The estimator has three row bands, four history bands, four widths and two execution paths. History is weighted by each lane's actual new rows; a short leading request cannot conceal a long-history second lane. Only finite positive representative text-prefill observations of at least 64 rows train a cell, with EWMA weight 0.25 and three matching prior samples required. Before that, fallback exactly preserves the baseline capacity-based global EMA, including underfilled trial budgets.

The scheduler captures the submitted path at commit: an ANE split retiring during completion does not train the future GPU cell. Runtime/model/draft/KV/state/build identity is implicit in the owning engine; there is no cross-runtime calibration cache. Resource callbacks run on the engine thread and their resources outlive it.

The existing 500 ms target, 64-row minimum, 2,048-row ceiling, priority, shortest-first order, overtaking bounds, admission and decode debt remain intact. Restore/image/auxiliary work does not train ordinary text-prefill cells. This changes only the time estimate used to halve contended row budgets. Actual submitted geometry is traced after resource/capture cuts; provisional capacity estimates are distinguished from these diagnostics.

Completed validation on M5 Pro

• Full make -j8 test-engine-cpu test-sanitizers, production/profile builds and focused engine/native-loop/ragged-scheduler tests pass for the frozen implementation used by the performance queue.
• Added scheduler tests cover prior-only learning/fallback, unseen context/width, GPU/ANE separation, path retention across split retirement, auxiliary timing exclusion, invalid geometry, and a non-leading long-history lane reducing a trained budget from 2,048 to 1,024 without changing shortest-first order.
• Existing priority/fairness/debt/overtaking cases run under both policies.
• The dedicated ragged-scheduler sanitizer targets were added after the frozen timing build. Their first local run is queued after timing, and CI will run them here; the earlier passing sanitizer suite did not yet include those new dedicated cases.
• Six offline estimator tests pass, including causality, row-weighted ragged history, width/path separation and malformed geometry. Existing HTTP lifecycle/overlap/report tests cover single-resident cleanup, serialized-admission rejection, stale status, cold/cache conditions and bounded output/resource accounting.
• Four real single-resident MoE control traces retain 74–75 eligible prefill observations each, including actual two-lane ragged commands. Prior-only bucket predictions reduce mean absolute relative error from 9.67–10.40% to 5.89–7.34%, and p95 from 24.44–27.31% to 13.95–17.92%. These are prediction diagnostics, not latency gains. Raw excerpts reproduce the calculation and deliberately omit GPU events; they cannot reconstruct a full GPU timeline.

Qualification gate still running

Production comparisons use the same frozen executable in both arms (ema versus bucketed), one resident server per full ABBA round, matched fixed warmup histories, unchanged verification policy, and native proof of actual contention. The queue covers MoE and dense targets; 2k/8k/31,744-token cold contexts; cached and RAM-prefix continuation; three trials; 512 active output tokens. It retains total workload time per output token, short/long TTFT, p95/max streaming gaps, actual output budgets, cache reuse, memory and scheduled widths.

Promotion requires lower contended latency without starvation or throughput regression outside max(2%, 2 * measured noise); noise above 5% is inconclusive. Prefix continuation is not a claim about resource-pressure suspension. Other Apple GPU families remain unexecuted. Results, failed/inconclusive cases and the final keep-off/promote decision will be appended before this draft is considered ready.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[linson007]

Closing the historical bucket-estimator prototype after completing its conditional experiment. #523 retains the design, code references, complete production reports, full trace archives, prediction analysis and scheduler sanitizer logs.

The 30 production ABBA comparisons show no qualified throughput gain (four inconclusive, zero throughput regressions beyond tolerance). At 2k, long TTFT improves 12.7-14.5% while maximum active-stream gaps regress 41.4-45.0%; a dense cached case also has a stable maximum-gap regression. The focused trace records two larger contended commands (1,024/1,536 rows) versus EMA's three (1,024/1,024/512), consistent with the latency tradeoff. Fairness/fallback cases and dedicated ASAN/UBSAN/TSAN checks pass. Better prior-only prediction does not clear the latency gate.

Keep the baseline. Current main now models fixed/per-row costs, attention pairs, streaming and passes; this old 2,048-row-only prototype is not transplanted into that scheduler. #523 changes current qualification tools and versioned evidence without applying this runtime policy.