Skip to content

[Bugfix][Qwen4Exp] Read the PLE row-gather policy when the layer is built - #1004

Merged
yangzhuxinyzx merged 161 commits into
1CatAI:mainfrom
Peuqui:pr-ple-row-gather-policy
Oct 9, 2026
Merged

yangzhuxinyzx merged 161 commits into
1CatAI:mainfrom
Peuqui:pr-ple-row-gather-policy

Conversation

@Peuqui

@Peuqui Peuqui commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

Since #885, Qwen4ExpNGramEmbedding.load_weights reads kernel_config.ple_disk_row_gather through get_current_vllm_config() and writes the row-reader admission into kernel_config.ple_disk_row_readers. get_current_vllm_config() asserts outside a set_current_vllm_config() context. The engine loads inside one, but direct loads do not, and ten cases in tests/models/qwen4_exp/test_ple.py fail on main with that assertion.

This PR

  • reads the policy when the layer is built, next to ple_disk_release_pages, which __init__ already reads through get_current_vllm_config_or_none(), and records the admission only when an engine config exists (same pattern as [Bugfix][DeepSeek-V4] Read the SM70 sparse policy when the layers are built #857 for the SM70 sparse policy);
  • makes the native row gather (csrc/ple_disk_rows.cpp) report an out-of-range row id the way the Python lane and the tests do: TORCH_CHECK_INDEX (an IndexError in Python) with the Python lane's message and the offending id, instead of a RuntimeError with "row ID". Callers catching IndexError missed it on the native lane;
  • adds the two new attributes to the test helper that builds the layer without __init__.

In the engine nothing changes: the layer is built inside the config context, so it reads the same KernelConfig object as before.

Not a duplicate: no open PR touches ple_disk_row_gather or ple_disk_rows.cpp (gh pr list --state open --search "ple_disk_row_gather", "ple_disk_rows": none; "row id out of range" only finds #828, QSA prefill attention, unrelated). #903 also edits ple_layer.py, in other functions (fused n-gram ids, PLE MTP guard); this branch merges cleanly with it.

AI assistance was used for this change. I reviewed every line and ran the tests below.

Test Plan

pytest tests/models/qwen4_exp/test_ple.py tests/v1/worker/test_ple_offload_worker.py
pre-commit run --files csrc/ple_disk_rows.cpp tests/models/qwen4_exp/test_ple.py vllm/models/qwen4_exp/nvidia/ple_layer.py
pre-commit run mypy-3.10 --hook-stage manual --files tests/models/qwen4_exp/test_ple.py vllm/models/qwen4_exp/nvidia/ple_layer.py

Acceleration and benchmark contract (required for performance changes)

Not a performance change. The admission decision is the same; only where the policy is read from moves, and the native error type changes.

Test Result

  • On main bc01f28: 10 of the cases in test_ple.py fail (AssertionError: Current vLLM config is not set). With the Python part alone, 7 of them pass and the 3 cases of test_ngram_embedding_disk_gather_rejects_invalid_ids then reach the native lane and fail on the exception type (checked on 17310de; none of the three files changed since); with the native part all pass.
  • Main bc01f28 built from source (sm_70 only) with this PR: Tesla V100-PCIE-32GB test_ple.py 98 passed, test_ple_offload_worker.py 55 passed and 1 skipped; Quadro RTX 8000 95 passed and 3 skipped (the existing exact-SM70 cases), 55 passed and 1 skipped.
  • pre-commit and mypy (manual stage) pass.

🤖 Generated with Claude Code

cookys and others added 14 commits October 5, 2026 12:24
…er async scheduling

MambaManager (--mamba-cache-mode align) tracked the state block awaiting
release in a single slot, `last_state_block_idx[request_id]`.
`remove_skipped_blocks` frees it only when
`idx < cdiv(processed_computed_tokens, block_size) - 1`, and
`allocate_new_blocks` then unconditionally overwrote the slot with the newest
superseded index.

Under async scheduling the step scheduled just before the current one is still
in flight, so `processed_computed_tokens = total_computed - num_in_flight`
lags by one more step. The recorded block is then the copy source of that
in-flight step and is (correctly) not freed, but the overwrite in
`allocate_new_blocks` drops the only reference to it. When a chunk spans
several state blocks, the generic skipped-block sweep in the base class stops
at the first null block in `req_to_blocks` and never reaches it either, so the
block is held until the request finishes or is preempted: one pool block per
prefill chunk per Mamba KV-cache group.

Fix: keep a per-request list of pending state-block indices.
`allocate_new_blocks` appends instead of overwriting; `remove_skipped_blocks`
frees (and nulls in `req_to_blocks`) every pending index that satisfies the
unchanged criterion `idx < cdiv(processed, block_size) - 1` and keeps the
rest, so the in-flight copy source is still protected. `free()` already pops
the dict entry.

Steady state per group: 2 + num_speculative_blocks held after the sweep and
3 + num_speculative_blocks after allocation under async scheduling; sync
scheduling is unchanged.

No kernel or numeric change: only which block ids are recycled and when.
Measured on a 4x V100 TP4 hybrid deployment (async scheduling, 8080-token
chunks over 1616-token state blocks): a 253,731-token cold prefill went from
343 s with 6 self-preemptions to 60.3 s with none; greedy output parity was
bit-identical.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: cookys <2537196+cookys@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…uilt

load_weights read kernel_config.ple_disk_row_gather through
get_current_vllm_config(), which asserts outside a config context. The
engine loads inside one, but direct loads and the disk-tier tests do not,
so ten cases in tests/models/qwen4_exp/test_ple.py failed on main. The
layer now reads the policy, like ple_disk_release_pages next to it, when
it is built, and records the row-reader admission only when an engine
config exists.

The native row gather also reported an out-of-range row id as a
RuntimeError ('PLE disk row ID out of range') while the Python lane and
the tests expect IndexError ('PLE disk row id out of range'); it now uses
TORCH_CHECK_INDEX with the Python message and the offending id.

Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZLMmGB8mjc43VpwPxLQNn
Each stand-in or expectation predates a change on main; the code under
test is unchanged.

- test_sm70_long_attention_graphs: the hand-built GPUModelRunner lacks
  'device', which execute_model reads since the mixed-prefill timer
  (1cabc7c); it now runs on the CPU device, which skips the timer.
- test_gdn_metadata_builder: FakeLayer lacks _can_use_sm70_gdn_preprocess,
  consulted since the SM70 verifier preprocessing (deadbde); the case
  covers the existing convolution path, so it declines.
- test_sm70_modelopt_mixed_nvfp4: the skinny MoE admission now runs first
  on the automatic route and records into the engine's kernel config
  (356306b). The TurboMind-required case states its precondition (skinny
  not admitted); the TurboMind case runs inside a config context.
- test_schedule_partial_requests: about the encoder budget; the
  mixed-prefill limit (512 tokens next to a running decode) is kept out.
- test_unused_fp8_policy_preserves_nvfp4_fingerprint: the hash changed with
  new kernel-config fields (a096d6d, e03af7c); updated.
- test_release_cleanup: the SM70 FP8 workspaces moved to the linear kernel
  module (333217c); the test now fills and checks them there.

Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZLMmGB8mjc43VpwPxLQNn
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…ale-20261006-140711

[Kernel] Bundle native SM70 NVFP4 codes and scales
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…ections-20261006-005255

[Kernel] Add shared-activation GGUF projection planes on SM70
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…undary-20261007-20261006-164334

[Bugfix] Match GGUF epsilon in SM70 TP4 collective norm fusion
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
yangzhuxinyzx and others added 26 commits October 9, 2026 08:40
Original-author: Yang <153831768+yangzhuxinyzx@users.noreply.github.com>

Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…0261008-143030

[Test] Freeze Flash-V100 GPU route, token and output parity
…008-153907

[Core] Centralize dynamic Flash-V100 policy reads
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…tion

Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…g-20261008-160814

[Core] Freeze resolved Flash-V100 attention policy
…-20261008-170157

[Core] Own Flash-V100 per-layer decode cache state (A3 Step 3a)
…kspace-20261008-171509

[Core] Own Flash-V100 metadata plans and persistent buffers (A3 Step 3b)
…tor-20261008-174522

[Core] Inject Flash-V100 decode executor dependencies (A3 Step 4a)
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…dates-20261008-181623

[Core] Plan native Flash-V100 decode with ordered candidates (A3 Step 4b)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…-20261008-183224

[Core] Select outer Flash-V100 decode routes through candidates (A3 Step 4c)
…hot-20261008-190002

[Test] Freeze requested parity configuration before initialization
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…idates-20261008-191139

[Core] Execute sequence prefill through ordered candidates
…h-20261008-193103

[Core] Select batch prefill through explicit candidates
…g-20261008-195253

[Core] Observe prefix diagnostics through ordered events
…r-20261008-201608

[Core] Execute verification with explicit policy and operators
Repair stale test stand-ins and expectations after SM70 runtime integration. Retain current workspace ownership, isolate the encoder-budget scheduler case from the mixed-prefill limit, and update the NVFP4 fingerprint for current defaults. Latest-main CPU merge gate: 291 passed, 33 hardware skips; scoped pre-commit passed. Original contributor GPU evidence is retained; local GPU-only cases were not rerun. Runtime source, numerical behavior and defaults are unchanged.
Load CUDA Python bindings only when PLE GPU transport is actually used, allowing CPU model imports without the CUDA package. Preserve all GPU wait/write/copy arguments and ordering. CPU gate: 129 passed, one existing skip; scoped pre-commit passed. The fresh-process missing-CUDA regression fails before the fix and passes afterward. macOS native smoke validation is pending in CI; no local macOS or GPU transport execution is claimed.
…fix-20261009

Resolve layer-keyed context-slot dictionaries before DFlash native KV publication. Preserve tensor/list/tuple/None inputs and exact selected tensor/cache identity. Extract the two-file fix from 1CatAI#1089 without importing the dependent architecture stack. CPU gate: 84 passed, scoped pre-commit passed. No arithmetic, precision, sampling or default route changes. Full-model generation was not rerun.
…uilt

load_weights read kernel_config.ple_disk_row_gather through
get_current_vllm_config(), which asserts outside a config context. The
engine loads inside one, but direct loads and the disk-tier tests do not,
so ten cases in tests/models/qwen4_exp/test_ple.py failed on main. The
layer now reads the policy, like ple_disk_release_pages next to it, when
it is built, and records the row-reader admission only when an engine
config exists.

The native row gather also reported an out-of-range row id as a
RuntimeError ('PLE disk row ID out of range') while the Python lane and
the tests expect IndexError ('PLE disk row id out of range'); it now uses
TORCH_CHECK_INDEX with the Python message and the offending id.

Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZLMmGB8mjc43VpwPxLQNn
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
@yangzhuxinyzx

Copy link
Copy Markdown
Contributor

I have integrated this fix onto current main, preserving your signed contribution. I also updated the native row-reader regression to expect the same IndexError as the Python path. The scoped CPU gate passes 205 tests with 36 skips and pre-commit passes. Separately, compiling the production CPU gather source with an isolated unit-test registration and running the three affected suites yields 146 passes and 22 hardware skips, including all 14 native row-reader cases. This checks the new error type and unchanged byte/shard behavior without requiring a GPU. The existing contributor source-build hardware evidence remains retained; no full-model or local GPU result is claimed. AI assistance was used for integration and validation.

@yangzhuxinyzx
yangzhuxinyzx merged commit 3db99d0 into 1CatAI:main Oct 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants