[linson007]

Historical source / current-main reconciliation: upstream has merged #487 and the later runtime stack at 1482f6bbc2abf12375332725302019594150b6cb. This draft preserves the original source and scoped evidence; its old runtime flags are not current-main settings. The mergeable qualification tools, updated source audit and consolidated evidence are now in #523. New-source validation and remaining awake replacement runs are in progress.

This adds the serving measurements needed to evaluate the remaining Qwen-on-Mac optimization roadmap in #508. It follows #507 and changes no scheduler policy or GDN arithmetic. Production serving has an optional null trace sink; the development-only serving-profile executable records engine, mask, ANE handoff and event-split GPU timing without writing during a workload.

Review order and dependency

This branch includes #507 because its live-prefix observations are used by the new reports. Review commits after d3424cc5: 0fdb5312 adds instrumentation/native tests; bc09da81 adds HTTP/report tooling and portable diagnostics; daea4188, 14c1b4f8 and dbba6a23 add sustained warmups, cached/prefix-resume scenarios and native-overlap/single-resident qualification controls. Once #507 merges, the diff can be reduced to those commits. The regular splash binary does not read SPLASH_SERVING_TRACE.

What the measurements mean

• Engine spans cover admission, planning, preparation, submission, result consumption and application. Actual batch rows are recorded after resource/capture cuts. Committed counters come from engine application, excluding provisional cancelled output.
• Initial and verification grammar-mask roundtrips are separate from the remaining mask wait after target forward. ANE spans measure host notification/report intervals, not exact hardware execution.
• GPU command buffers split around signal/wait events retain separate GPU segments. Host scopes overlap and must not be summed. The report separates idle command gaps from gaps during the union of active requests.
• Trace storage is bounded. Overflow and incomplete/unsuccessful captures are reported and rejected by the report tool. Late ANE callbacks retain shared ownership of their sink. The null path skips trace-clock reads.
• Live-prefix confidence labels distinguish observed rejection from an unverified, censored suffix. The disabled selector emits no confidence observations.

Repeatable serving tools

serving_regression.py runs two owned resident servers and serial ABBA requests with exact tokenized contexts. Every measured request has a unique cache salt, and zero cached tokens is enforced. It retains raw arm timings, token counts, output hashes, binary/model fingerprints, memory and thermal observations. It also supports overlapping streaming/long-prefill/interactive requests, client TTFT and chunk gaps, seeded sampling, and multi-cycle strict JSON probes. Final speed comparisons use production binaries with tracing off; two-server memory/cache effects remain a documented limitation.

See dev/benchmarks/serving-measurements.md for commands, event contracts and limitations. Portable evidence is committed in dev/benchmarks/results/qwen-serving-trace-m5-pro/diagnostics.json; raw traces stay in ignored build output.

Validation on Apple M5 Pro / macOS 26.6.0

• Production and instrumented builds pass, as do architecture checks and Ruff.
• Complete CPU and ASAN/UBSAN/TSAN suites pass after instrumentation changes. Focused engine/native-loop tests pass after the final counter and initial-mask audit.
• Shader-validated Metal backend tests pass, including event-split GPU segments. Handoff success/failure/timeout/cancellation/late-report tests pass with tracing attached.
• Ten report/environment/lifecycle tests pass, alongside the previously passing 38 existing benchmark/smoke tests. Tests include bounded overflow, cancelled provisional counters, actual rows after cuts, invalid captures, overlapping request intervals and environment restoration on failure.
• Real mixed HTTP smoke completes with matching outputs and seeded-sampling/strict-JSON probes. Both server traces have zero drops, exit status zero and no command in flight at stop. The mixed smoke includes 26 verification-mask roundtrips and tail waits per arm. A final trace also exercises separate initial-mask spans.

These results validate the measurement contracts. The cold diagnostic comparison traced only the candidate, and the mixed diagnostic overlapped installation/sanitizer activity; neither qualifies a policy speedup or a latency improvement. Quiet production comparisons and real dense-model ANE traces remain tracked in #508. Only this Mac is locally available; other-device qualification will use repeatable maintainer runs and is not claimed as passed.

Overlap audit and one-resident qualification

A subsequent dense-model capture exercised 3,072 real ANE handoffs per arm, but two resident ANE servers triggered memory pressure and serialized admission. The original client-lifetime check could accept that trace even though no long prefill was submitted before the initial decode ended. That capture validates ANE execution/handoffs and does not establish contended latency.

The harness now polls native request progress and prefill counters, requires proof of long-prefill progress during native decoding, distinguishes fully cached continuations without prefill, and rejects serialized-admission false positives. --residency sequential owns one server per full ABBA round; fixed warmup counts preserve matched histories across arms. Sustained paired warmups retain equal request counts and actual durations. Per-round memory fields use the actual status schema and scheduled decode-width histograms are retained. Unit tests cover the false overlap, cache-restore-only progress and closing each server before the next starts. Actual single-resident mixed qualification remains in #508; no latency improvement is claimed here.

[linson007]

Closing this conflicting historical instrumentation stack in favor of #523. Current main has substantially newer runtime/cache/streaming/status code, so the old full-engine native hooks are preserved as an experiment rather than transplanted over those interfaces.

The useful Python HTTP/trace/prediction/calibration tools are adapted to current main in #523. Its evidence includes full CPU/GPU/mask/ANE trace archives and five display windows, per-width operator attribution, single-resident native contention controls, transient stale-poll handling with fresh-progress requirements, complete output budgets and scoped suspend detection. The complete dense trace demonstrates GPU prefill during active decoding and actual ANE handoffs elsewhere; configured ANE share is not presented as contended ANE timing.

The source audit, completion ledger and full test/results map in #523 are the review entry points. This branch remains available for reconstructing the archived trace builds; current ordinary binaries explicitly reject the historical trace hook.