[Bugfix][MRV2] Backport phase-aware decode graph dispatch

[Leonccaa]

Problem and fix

A real prompt chunk can contain exactly 1 + num_speculative_tokens tokens. MRV2 then mistakes its shape for uniform decode and replays a FULL speculative-decode CUDA graph even though the attention metadata describes prefill. Stateful hybrid models can produce corrupted output; a genuine decode sharing that batch can also be affected.

Adapt the phase-aware dispatch fix from vllm-project/vllm#51865 (0a94d85a66499cad8297ead86a470967de5c0212) to 1Cat's current MRV2 layout:

• Target dispatch consults the current request state before DP graph synchronization and requires every scheduled request to have finished prefill. It supersedes the single-request DFlash2-only short-tail guard.
• The Eagle/MTP draft prefill dispatcher uses the same phase-aware predicate.
• Dummy capture/profiling batches retain shape-based classification; non-uniform batches skip the request-state scan. Genuine decode still qualifies for FULL graphs.

This is a narrow semantic backport, not a wholesale import of upstream's gather_batch_req_state refactor or its PCP/autoregressive modules, which are absent in this branch. No new setting, kernel change, or cache-retention change is introduced.

Existing work

Upstream #51865 is merged but is absent from 1Cat main. 1Cat #434 addresses the older runner (gpu_model_runner.py), not MRV2 (gpu/model_runner.py). The existing DFlash2 guard does not cover MTP or mixed batches. Searches of open 1Cat PRs for 51865 in:body and prefill decode dispatch found no overlapping fix. Related upstream report: vllm-project/vllm#49918.

Validation

Unit and graph checks: 42 passed (39 new regressions + 3 existing graph-shape tests), in the candidate's Python environment:

/opt/venv/bin/python -m pytest /audit/test_prefill_dispatch.py -q --confcutdir=/audit
/opt/venv/bin/python -m pytest /audit/test_sm70_mtp_split_cudagraphs.py -q --confcutdir=/audit
pre-commit run --files vllm/v1/worker/gpu/cudagraph_utils.py \
  vllm/v1/worker/gpu/model_runner.py \
  vllm/v1/worker/gpu/spec_decode/eagle/speculator.py \
  tests/v1/worker/test_gpu_model_runner_v2_prefill_dispatch.py

For the container run, the two tests were copied to /audit and executed separately with /opt/venv/bin/python -m pytest ... --confcutdir=/audit; pre-commit ran in the source worktree. All applicable hooks, including mypy, passed.

The regressions cover fresh prompts, chunked tails, mixed batches, fully computed prompts, non-contiguous state indices, unscheduled prefills, dummy capture, non-uniform batches, and propagation of the classification into DP dispatch. Widths 1, 3, 4, 5, 7, 8 and 16 are covered. Existing MTP split/DFlash graph shapes remain valid.

GPU: 4 x V100 32 GiB, TP4, Qwen3.8 Flash-Next AWQ target + native FP8 MTP3 draft, FULL_AND_PIECEWISE, 800-token cache blocks, C2, 32 GiB RAM offload, unchanged retention interval 0.

Before the fix, a public minimal prompt (Reply with OK. padded with space token IDs to 804 tokens) produced duct!!!!!!!!!!!!!!!!!!!!!!! at temperature 1/top_p .95/top_k 20. Greedy output instead contained repeated out-of-vocabulary ID 248320. Neighboring 803/805-token controls were clean. A previous same-process diagnostic OFF/ON/OFF/ON guard experiment on an 88,804-token prompt produced bad/good/bad/good output while keeping the cache/offload patches enabled.

With this formal target+draft patch, 25/25 GPU responses passed:

Case	Result
803, 804, 805, 1,604, 88,804, 99,204 tokens; cold then warm	All OK; exact cold/warm output token parity
804 with production sampling	OK, no invalid token IDs or repeated punctuation
Concurrent 804/804, 804/1,604, 88,804/99,204	Both requests OK in all three pairs
Ongoing decode + new 804, 1,604 or 88,804-token prompt	Each original decode completed 512 tokens with the exact ordered 1..124 sequence; every inserted prompt returned OK

88,804 cold/warm completion times were 35.834/0.590 s; 99,204 were 40.750/0.596 s. Both long requests together completed in 88.795 s. These are end-to-end completion times, not a controlled performance benchmark. Long-prefill contention remains; this fix addresses correctness.

The NaN counter was zero even in the earlier broken run, so success is assessed from actual output IDs/text and sequence correctness, not that counter.

Minimal HTTP reproducer against a running model with MTP3 and 800-token block alignment:

import requests

base = "http://localhost:8000"
model = requests.get(base + "/v1/models").json()["data"][0]["id"]

def post(path, **payload):
    response = requests.post(base + path, json={"model": model, **payload})
    response.raise_for_status()
    return response.json()

ids = post("/tokenize", messages=[
    {"role": "user", "content": "Reply with OK."}
])["tokens"]
space = post("/tokenize", prompt=" ", add_special_tokens=False)["tokens"]
assert len(space) == 1 and len(ids) < 804
prompt = ids[:-4] + space * (804 - len(ids)) + ids[-4:]
print(post("/v1/completions", prompt=prompt, max_tokens=24,
           temperature=1, top_p=0.95, top_k=20, return_token_ids=True))

GPU validation uses the existing production image with only these three Python source edits applied, preserving the previously deployed cache/offload fixes. It does not claim a full rebuild or model-wide qualification of the latest 1Cat main. Original incident state and production cache are preserved outside the test environment.

AI assistance

OpenAI Codex assisted with upstream research, adaptation, regression tests and V100 validation, under the maintainer's explicit direction. The evidence and test scope above describe agent-run checks; they do not assert an independent human review.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-14T15:47:30.598353Z	1892fb2	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.