[sysuTobo]

Description

Adds two load-aware levers to the Qwen3 speculative-decode path, both off by default, and the L20 measurement study that says when to arm them.

1. PEGAINFER_SPEC_MAX_BATCH=<n> — the load gate (#718). The verify pass costs batch * span target rows while a plain decode costs batch, so at high concurrency speculation can lose to plain decode outright. The obvious implementation — fall back to ExecutionPlan::Decode — is a one-way latch: a plain decode advances the sequence outside the speculative path, so the executor drops the captured draft context, and only a fresh prefill can rebuild it. The first trip would kill speculation engine-wide, the opposite of "recover when the load drops" (#718 §4).

So a gated step is a one-token verify round: no draft is proposed, the target verifies the span [current_token] alone, one token is committed per request, and the DFlash context advances by exactly the one captured row — the same bookkeeping a full round does. The capture and the draft-ready state survive, so the batch speculates again as soon as it shrinks below the threshold. The scheduler now distinguishes the two fallbacks explicitly: NotReady (a request has no captured context — plain decode, the pre-existing behaviour) versus Gated (everything is draft-ready — a one-token verify round that keeps the capture).

The verify CUDA graph needed a span dimension for this: a captured dense segment bakes its row count, so the one-token rows cannot share a graph with the full-span rows. VerifyGraphBuffers now holds one graph set per span variant, and a step replays a variant only when its row count matches that variant exactly — capture-shape ≡ replay-shape, unchanged.

2. PEGAINFER_SPEC_MAX_DRAFTS=<k> — the static draft trim. The drafter still rolls its whole block out (its cache still writes the block_size-slot in-fill tail every round — the headroom is unchanged), but only the first k candidates enter the verify span, so the target forward carries 1 + k rows per request. The verify buffers are sized to the trimmed span and the span stays uniform across the batch, so the capture-shape ≡ replay-shape invariant still holds.

Interaction and validation. Under PEGAINFER_SPEC_HEDGE_AUTO a gated round is not a hedging sample: it breaks the controller's timing window instead of being booked to C = 0, and the first resumed round re-anchors rather than charging the gated interval. Malformed knob values (-1, a typo) refuse to start with the variable's name on every engine start path (start_engine_with_offload_inner and start_engine_with_lora_control, not just launch); unset and an explicit 0 still mean off, and the readers no longer fall back to 0 after validation. The trace parser used by the gate reads request=req-<n> (the real RequestId format) and has its own weight-free unit test.

The measured result (one L20, Qwen3-4B, DFlash/DSpark block7) — docs/benchmarks/qwen3-4b-spec-decode-l20.md:

	c32	c64
plain decode	1974 tok/s	3263 tok/s
DFlash, no lever	1846 (−6.5 %)	2113 (−35 %)
DFlash + trim K=4	2014 (+9 % vs untrimmed; on par with plain)	2405 (+14 % vs untrimmed)
DFlash + gate	1860 (+0 %)	3083 (+46 %)

The two levers are the ends of one axis: the optimal verify span shrinks as the batch grows (the row cost of the target forward scales with batch × span, the accept-rate cost does not). A static K is therefore a load-adaptive lever in disguise — it needs a concurrency range, not a constant. The threshold must be armed at or above the measured crossover; below it the rows are nearly free and the lever throws the speculation win away (−16…−23 % measured). At c32 the trim's remaining gap to plain decode is +1.8 % by per-round pairing — under the study's own 2 % bar, so read that cell as parity rather than a win, and c16 K=4 changes sign across rounds (unresolved).

Cell provenance: the c32 rows come from the trim sweep (0g; the gate cell from 0c), the c64 rows from 0h with the plain cell from 0c — the doc footnotes each cell.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Evidence

• cargo test --release --workspace --lib — 1143 passed / 0 failed across 21 test binaries at this head (includes the new gate and the five new unit tests).

• cargo test --release -p pegainfer-qwen3 --test dflash_speculative_gate levers_trace_parses_real_trace_lines — passed (weight-free: feeds real-format trace lines, including request=req-0).

• cargo test --release -p pegainfer-qwen3 --test dflash_speculative_gate — needs the target + drafter weights, so CI does not run it. dflash_spec_levers_keep_losslessness (new, child-process gate, the hedged gate's pattern) re-execs the lossless suites with the knobs set:

  • MAX_BATCH=1 — every round gated, so the one-token verify path is the only path exercised;
  • MAX_DRAFTS=4 / =2 — greedy and concurrent gates stay lossless, and no round verifies more than K;
  • MAX_BATCH=3 on four staggered budgets — the batch shrinks below the threshold mid-run; the trace must show gated rounds and a request that was gated must later speculate and accept at least one draft (checked per request id), with the child's lossless assertions covering the output.
    The executor's per-round trace now carries gated= (explicit, set by the plan) plus verified_draft= and accepted_draft= and the request id. A zero verified_draft alone is ambiguous: a near-budget span is truncated to one token too.

• Gate run record (one L20, head 3b9f0a66, 86.5 s, PEGAINFER_TEST_LOG=1): 1 passed / 0 failed, with these per-child trace rows (one per request per round):

  child	env	gated rows	speculating rows	max drafts
  greedy	MAX_BATCH=1	315	0	0
  greedy	MAX_DRAFTS=4	0	116	4
  concurrent	MAX_DRAFTS=4	0	64	4
  greedy	MAX_DRAFTS=2	0	142	2
  concurrent	MAX_DRAFTS=2	0	79	2
  concurrent	MAX_BATCH=3	140	8	—

  The MAX_BATCH=3 row is the resume case (the per-request "gated then later accepts" assertion passed); the MAX_DRAFTS rows keep the gate off and never exceed K. The pod's toolkit is CUDA 12.9 while this tree references CUDA 13's cudaErrorLossyQuery, so the local build wraps that one error-mapping case in a version guard (outside the speculative path). The earlier 8142/29 c64 count came from accepted_draft, which cannot tell a gated round from a fully-rejected speculative one, and is not quoted as evidence.

• Throughput: 8 interleaved A/B sweeps on one L20 (sm_89), 64 ShareGPT first-turn prompts, greedy, 128 output tokens, --no-prefix-cache symmetric across arms. The harness's own paired noise floor is 0.3–2.2 % (measured with two identical arms) — the raw CSVs are retained next to the doc.

Scope: the #717 truncation-sweep lesson and the DSpark Phase 2 archival moved to #1185 — they record a different sweep; this PR keeps the gate, the trim and the L20 study.

Checklist

• My code follows the style guidelines of this project (see docs/conventions/coding-style.md).
• I have performed a self-review of my own code.
• I have formatted my commits according to Commitizen conventions.
• I have run the local test suite and all tests pass (see CLAUDE.md).

[github-actions]

This pull request has merge conflicts with main. Please rebase it so it can be reviewed and merged.

[FeathBow]

The one-token verify round is the right shape for the gate: on main a plain decode drops the draft context, and the gated round advances it by the one committed row like any verify round. The span variants keep capture and replay shapes equal, and the trim stays consistent with hedge spans and the DFlash2 selector.

Nothing in the repo runs the lossless gates with either knob set. The runs in the description were manual, and both values are read once from the environment, so a test cannot set them per case. Could the gate cover them, through a gate run with the variables set or with the values passed in through the launch options? PEGAINFER_SPEC_MAX_BATCH=1 also never resumes speculation, so a case where the batch drops below the threshold mid request would show the resumed rounds still accept.

Two smaller points on the study. At c32, K=4 against plain decode is +1.8, +2.2 and +0.6 per round, under the 2% median the doc itself asks for, so the data supports on par with plain decode there. At c16, K=4 against DFlash changes sign across rounds. The #717 lesson and the DSpark Phase 2 rewrite record a different sweep and would sit better in their own doc change.

[FeathBow]

The auto hedge controller needs the same skip. With the hedge on, a gated round falls back from try_execute_hedged_verify, clear_round_chains sets the chain count to 0, and record_verify_dflash_context then ticks the controller with one-token rounds booked as C=0.

[sysuTobo]

Fixed in ce7b387: the explicit gated flag is threaded through VerifyPlan into record_verify_dflash_context, which now calls AutoHedge::break_window() (resets last_tick and the window accumulators) instead of tick. The gated interval is neither booked to C=0 nor charged to the first resumed sample. Unit test: break_window_never_charges_the_gated_interval.

[FeathBow]

The study wording and the docs split are done, and the one token verify and per span graphs read right. Three things remain.

The auto hedge point from the last round is still open: record_verify_dflash_context ticks the controller for gated rounds with C=0, so under PEGAINFER_SPEC_HEDGE_AUTO the gated throughput lands in the C=0 windows and advances exploration and the commit count. Skipping the tick alone is not enough, because tick times from last_tick and the first resumed sample would carry the gated interval, so the gate boundary should also break the interval.

Both new knobs parse with .parse::<usize>().ok() and fall back to 0, so -1 or a typo turns the gate off or keeps the full span with no message. Could malformed values be rejected with the variable name, keeping unset and explicit 0 as off?

The reversibility section of the study counts accepted_draft=0 lines as gated rounds. A round with every draft rejected logs the same value, and the log is per request, so 8142 and 29 do not show that speculation resumed after gating. Could that section be corrected together with the gate below? The gate needs weights and CI does not run it, so a run record at the new head (passed count and the counts per child) would help.

[FeathBow]

This can pass without the gate ever tripping. Ordinary speculation also logs verified_draft=0 when a request has one output token left, since the span is truncated to the remaining budget, and each line is per request. Lossless output also holds whatever the drafts are. Could the trace mark gated rounds explicitly, and the test require one surviving request that is gated and later accepts drafts?

[sysuTobo]

Fixed in ce7b387: the plan now threads an explicit gated flag through VerifyPlan → StepCommand → the per-round trace (gated=), so a near-budget truncated span cannot stand in for a gated round. The resume case now asserts per request id: a request that had a gated=1 round must later show verified_draft>0 and accepted_draft>0. The MAX_DRAFTS children additionally assert gated=0 throughout.

[FeathBow]

This calls span_variant on a hand written [1] and never builds VerifyGraphBuffers, so it cannot catch a duplicate variant. It can go, or test the constructor.

[sysuTobo]

Removed in ce7b387 — it only exercised the pure selector on a hand-written [1] and could not catch a duplicate variant. The exact-match test above it stays.

[FeathBow]

The assertion above already pins GatedSpeculativeDecode for the same call, so this one adds nothing.

[sysuTobo]

Removed in ce7b387; the preceding GatedSpeculativeDecode assertion covers the same call.

[sysuTobo]

Round 2 addressed in ce7b3874 (CI 14/14):

• Auto-hedge boundary — a gated round now carries an explicit gated flag from SpeculativeDecode/GatedSpeculativeDecode through VerifyPlan → StepCommand → record_verify_dflash_context, which calls AutoHedge::break_window() instead of tick; the first resumed round re-anchors rather than charging the gated interval. Unit test added.
• Malformed env values — a new spec_env module rejects -1, typos and blanks at launch with the variable's name; unset and an explicit 0 stay off. Two readers share the parser; unit tests cover unset/blank/0/positive/malformed.
• Reversibility — the study now uses the explicit gated= trace marker (a zero verified_draft is ambiguous: a near-budget span is also one token) and states that the old 8142/29 accepted_draft count is superseded; the resume gate asserts per request that a gated request later speculates and accepts drafts.
• Inline nits — the tautological verify_graph test and the redundant scheduler assertion are removed.

The run record (per-child gated_rounds / speculating_rounds, passed count) needs the target + drafter weights, which CI does not have. The gate now prints one spec lever gate: child=… env=… gated_rounds=… speculating_rounds=… line per child; I will attach the record to the benchmark doc as soon as it runs on a GPU host.

[FeathBow]

The hedge window and the explicit gated marker are fixed, and the per request resume check asks the right question. Four things remain, all inline.

The new gate cannot parse a real trace and one new unit test expects the wrong result, so neither has run at this head, and the 1138 passed in the description comes from an earlier one. Malformed knobs still get through the public start_engine_with_offload. The docs still claim the resume from the withdrawn count.

CI runs no Qwen3 tests, so once these are fixed, could you paste the lib run and the gate run with passed counts?

[FeathBow]

RequestId displays as req-<n>, so a real line reads request=req-0. This field takes the empty digit run after request= and panics on the first trace line, before any assertion. The parser should read the id the way it is printed.

[sysuTobo]

Fixed in 3b9f0a6: each field now takes the first non-empty digit run after its name, which reads request=req-0. The parser was extracted to levers_trace_rows and has a weight-free unit test (levers_trace_parses_real_trace_lines), passed locally against real-format lines.

[FeathBow]

"+2".parse::<usize>() returns Ok(2), since core's integer parse accepts a leading + for unsigned types too, so expect_err panics on that entry. It is a valid count, so it can come out of the list.

[sysuTobo]

Fixed: +2 parses to 2 (core accepts a leading + for unsigned types), so it moved out of the malformed list and into the positive assertions — the test now pins that behaviour instead of expecting an error.

[FeathBow]

This runs only in launch. The public start_engine_with_offload also takes a draft path and goes to start_engine_with_offload_inner without it, and spec_env::read then logs the error and returns 0, so the lever is silently off again. Could the check sit in the path every start shares, and read drop its fallback to 0?

[sysuTobo]

Fixed in 3b9f0a6: validation moved to start_engine_with_offload_inner and start_engine_with_lora_control — every public start (launch, start_engine, start_engine_with_offload, LoRA) funnels through one of them. The duplicate call in launch is gone, and spec_env::read now expects the validated value instead of falling back to 0.

[FeathBow]

This still cites 8142 gated and 29 speculating rounds as proof that speculation resumes, as do docs/index.md, dflash-speculative-decoding.md and the Settled list below. The reversibility section withdraws that count, the new gate has not run, and its record slot is a literal RUN_RECORD_PLACEHOLDER. Until the record exists, these places should say the resume is pending verification.

[sysuTobo]

Fixed: the TL;DR, docs/index.md, dflash-speculative-decoding.md and the Settled list now say the gate is reversible by construction with the resume run record pending; the only remaining 8142 mention is the withdrawal note in the Reversibility section.

[sysuTobo]

The record now exists. The gate ran on one L20 at head 3b9f0a66: 1 passed / 0 failed (86.5 s, PEGAINFER_TEST_LOG=1). Per-child trace rows (one per request per round): MAX_BATCH=1 315 gated / 0 speculating; MAX_DRAFTS=4 0/116 (max 4) and 0/64 (max 4); MAX_DRAFTS=2 0/142 (max 2) and 0/79 (max 2); MAX_BATCH=3 140 gated / 8 speculating, where the per-request "gated then later accepts" assertion passed. The benchmark doc carries this table; the TL;DR, index.md, dflash-speculative-decoding.md and the Settled list now point at it instead of the withdrawn count.

[sysuTobo]

Round 3 addressed in 3b9f0a66:

• Trace parser — reads the first non-empty digit run after each field, so request=req-0 parses. Extracted to levers_trace_rows with a weight-free unit test.
• +2 — treated as the valid count it is; moved to the positive assertions.
• Shared start path — validation now sits in start_engine_with_offload_inner and start_engine_with_lora_control (all public starts funnel through one), the duplicate in launch is removed, and spec_env::read expects the validated value instead of falling back to 0.
• Docs — TL;DR / index.md / dflash-speculative-decoding.md / the Settled list now say reversible by construction and mark the resume record pending; only the withdrawal note still names 8142.

Local runs at this head (CI runs no Qwen3 tests):

• cargo test --release --workspace --lib → 1143 passed / 0 failed across 21 test binaries.
• cargo test --release -p pegainfer-qwen3 --test dflash_speculative_gate levers_trace_parses_real_trace_lines → 1 passed / 0 failed (no weights needed).

The gate itself (dflash_spec_levers_keep_losslessness) still needs the target + drafter weights. Its six children print one spec lever gate: child=… env=… gated_rounds=… speculating_rounds=… line each; I will paste the passed count and those six lines here as soon as it runs on the GPU host.

[sysuTobo]

Run record for the new gate (head 3b9f0a66), on the existing p8s L20 container:

• cargo test --release -p pegainfer-qwen3 --test dflash_speculative_gate dflash_spec_levers_keep_losslessness -- --nocapture
  → test result: ok. 1 passed; 0 failed (86.5 s), BUILD_EXIT=0, GATE_EXIT=0.

child	env	gated rows	speculating rows	max drafts
dflash_speculative_greedy_matches_plain_greedy	MAX_BATCH=1	315	0	0
dflash_speculative_greedy_matches_plain_greedy	MAX_DRAFTS=4	0	116	4
dflash_concurrent_heterogeneous_is_lossless	MAX_DRAFTS=4	0	64	4
dflash_speculative_greedy_matches_plain_greedy	MAX_DRAFTS=2	0	142	2
dflash_concurrent_heterogeneous_is_lossless	MAX_DRAFTS=2	0	79	2
dflash_concurrent_heterogeneous_is_lossless	MAX_BATCH=3	140	8	—

Rows are per request per round. The threshold-3 concurrent child is the resume case: it gates first (batch 4→3) and speculates after requests finish, and the gate's per-request assertion (a gated request later speculates and accepts) passed.

Environment note: the pod's toolkit is CUDA 12.9 while the tree references CUDA 13's cudaErrorLossyQuery; the build wraps that one error-mapping case in a CUDART_VERSION >= 13000 guard. It is in the FFI error switch, outside the speculative path, and the same source compiles unmodified under CI's CUDA 13.0.2.

Local lib runs at this head: cargo test --release --workspace --lib → 1143 passed / 0 failed across 21 binaries; levers_trace_parses_real_trace_lines → 1 passed.

[github-actions]

Commit attribution notice

This pull request contains commits attributed to an account other than the PR author @sysuTobo without a verified signature from that account:

• cbcd31b6 attributed to @FeathBow
• a8db9d42 attributed to @FeathBow

If you are mentioned above and did not take part in this pull request, please say so here. Attributing a commit to someone else normally requires that person to be involved; maintainers should treat unacknowledged attribution as a review blocker.

This check never fails the build.

[FeathBow]

We pushed two commits on top that trim what repeats. Behaviour is unchanged except one fix.

• The eight CSVs are gone. The study's tables carry the numbers, and tpot_avg_ms was empty in every row.
• Each explanation now lives in one place next to its code: the gated round keeping its capture on ExecutionPlan::GatedSpeculativeDecode, the row count rule in batch_prefill_into, the trim on effective_drafts. The index row and the model doc point to the study.
• Removed tests that checked str::parse, restated a one line builder, or repeat what the scheduler tests and the GPU gate already cover: the four spec_env tests are now one, three of the four plan tests are gone, as are the span_variant test and the hand typed trace parser test. The not ready plan test and the two scheduler routing tests stay.
• spec_gate_trips and effective_verify_span are inlined, and validate reads the environment itself.
• Fix: a knob whose value is not UTF-8 used to read as unset and turn the lever off. It now refuses to start like any other bad value.

Checked on sm_89: clippy with -D warnings over all qwen3 targets, and the spec_env and scheduler unit tests, 28 passed. The GPU gate was not rerun, since what changed there is comments and a removed unit test.

The description still mentions the CSVs and the earlier unit test count.

[FeathBow]

thanks for your contrib :)