Repository navigation
[Bugfix][Qwen4Exp] Read the PLE row-gather policy when the layer is built - #1004
Merged
Merged
Conversation
…er async scheduling MambaManager (--mamba-cache-mode align) tracked the state block awaiting release in a single slot, `last_state_block_idx[request_id]`. `remove_skipped_blocks` frees it only when `idx < cdiv(processed_computed_tokens, block_size) - 1`, and `allocate_new_blocks` then unconditionally overwrote the slot with the newest superseded index. Under async scheduling the step scheduled just before the current one is still in flight, so `processed_computed_tokens = total_computed - num_in_flight` lags by one more step. The recorded block is then the copy source of that in-flight step and is (correctly) not freed, but the overwrite in `allocate_new_blocks` drops the only reference to it. When a chunk spans several state blocks, the generic skipped-block sweep in the base class stops at the first null block in `req_to_blocks` and never reaches it either, so the block is held until the request finishes or is preempted: one pool block per prefill chunk per Mamba KV-cache group. Fix: keep a per-request list of pending state-block indices. `allocate_new_blocks` appends instead of overwriting; `remove_skipped_blocks` frees (and nulls in `req_to_blocks`) every pending index that satisfies the unchanged criterion `idx < cdiv(processed, block_size) - 1` and keeps the rest, so the in-flight copy source is still protected. `free()` already pops the dict entry. Steady state per group: 2 + num_speculative_blocks held after the sweep and 3 + num_speculative_blocks after allocation under async scheduling; sync scheduling is unchanged. No kernel or numeric change: only which block ids are recycled and when. Measured on a 4x V100 TP4 hybrid deployment (async scheduling, 8080-token chunks over 1616-token state blocks): a 253,731-token cold prefill went from 343 s with 6 self-preemptions to 60.3 s with none; greedy output parity was bit-identical. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: cookys <2537196+cookys@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…uilt
load_weights read kernel_config.ple_disk_row_gather through
get_current_vllm_config(), which asserts outside a config context. The
engine loads inside one, but direct loads and the disk-tier tests do not,
so ten cases in tests/models/qwen4_exp/test_ple.py failed on main. The
layer now reads the policy, like ple_disk_release_pages next to it, when
it is built, and records the row-reader admission only when an engine
config exists.
The native row gather also reported an out-of-range row id as a
RuntimeError ('PLE disk row ID out of range') while the Python lane and
the tests expect IndexError ('PLE disk row id out of range'); it now uses
TORCH_CHECK_INDEX with the Python message and the offending id.
Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZLMmGB8mjc43VpwPxLQNn
Each stand-in or expectation predates a change on main; the code under test is unchanged. - test_sm70_long_attention_graphs: the hand-built GPUModelRunner lacks 'device', which execute_model reads since the mixed-prefill timer (1cabc7c); it now runs on the CPU device, which skips the timer. - test_gdn_metadata_builder: FakeLayer lacks _can_use_sm70_gdn_preprocess, consulted since the SM70 verifier preprocessing (deadbde); the case covers the existing convolution path, so it declines. - test_sm70_modelopt_mixed_nvfp4: the skinny MoE admission now runs first on the automatic route and records into the engine's kernel config (356306b). The TurboMind-required case states its precondition (skinny not admitted); the TurboMind case runs inside a config context. - test_schedule_partial_requests: about the encoder budget; the mixed-prefill limit (512 tokens next to a running decode) is kept out. - test_unused_fp8_policy_preserves_nvfp4_fingerprint: the hash changed with new kernel-config fields (a096d6d, e03af7c); updated. - test_release_cleanup: the SM70 FP8 workspaces moved to the linear kernel module (333217c); the test now fills and checks them there. Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014ZLMmGB8mjc43VpwPxLQNn
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Assisted-by: Codex Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…ale-20261006-140711 [Kernel] Bundle native SM70 NVFP4 codes and scales
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…rm-boundary-20261007-20261006-164334
…ections-20261006-005255 [Kernel] Add shared-activation GGUF projection planes on SM70
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…undary-20261007-20261006-164334 [Bugfix] Match GGUF epsilon in SM70 TP4 collective norm fusion
…e832-20261006-175956
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Original-author: Yang <153831768+yangzhuxinyzx@users.noreply.github.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…0261008-143030 [Test] Freeze Flash-V100 GPU route, token and output parity
…008-153907 [Core] Centralize dynamic Flash-V100 policy reads
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…tion Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…g-20261008-160814 [Core] Freeze resolved Flash-V100 attention policy
…-20261008-170157 [Core] Own Flash-V100 per-layer decode cache state (A3 Step 3a)
…kspace-20261008-171509 [Core] Own Flash-V100 metadata plans and persistent buffers (A3 Step 3b)
…tor-20261008-174522 [Core] Inject Flash-V100 decode executor dependencies (A3 Step 4a)
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…dates-20261008-181623 [Core] Plan native Flash-V100 decode with ordered candidates (A3 Step 4b)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…-20261008-183224 [Core] Select outer Flash-V100 decode routes through candidates (A3 Step 4c)
…hot-20261008-190002 [Test] Freeze requested parity configuration before initialization
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…idates-20261008-191139 [Core] Execute sequence prefill through ordered candidates
…h-20261008-193103 [Core] Select batch prefill through explicit candidates
…g-20261008-195253 [Core] Observe prefix diagnostics through ordered events
…r-20261008-201608 [Core] Execute verification with explicit policy and operators
Repair stale test stand-ins and expectations after SM70 runtime integration. Retain current workspace ownership, isolate the encoder-budget scheduler case from the mixed-prefill limit, and update the NVFP4 fingerprint for current defaults. Latest-main CPU merge gate: 291 passed, 33 hardware skips; scoped pre-commit passed. Original contributor GPU evidence is retained; local GPU-only cases were not rerun. Runtime source, numerical behavior and defaults are unchanged.
Load CUDA Python bindings only when PLE GPU transport is actually used, allowing CPU model imports without the CUDA package. Preserve all GPU wait/write/copy arguments and ordering. CPU gate: 129 passed, one existing skip; scoped pre-commit passed. The fresh-process missing-CUDA regression fails before the fix and passes afterward. macOS native smoke validation is pending in CI; no local macOS or GPU transport execution is claimed.
…fix-20261009 Resolve layer-keyed context-slot dictionaries before DFlash native KV publication. Preserve tensor/list/tuple/None inputs and exact selected tensor/cache identity. Extract the two-file fix from 1CatAI#1089 without importing the dependent architecture stack. CPU gate: 84 passed, scoped pre-commit passed. No arithmetic, precision, sampling or default route changes. Full-model generation was not rerun.
…uilt
load_weights read kernel_config.ple_disk_row_gather through
get_current_vllm_config(), which asserts outside a config context. The
engine loads inside one, but direct loads and the disk-tier tests do not,
so ten cases in tests/models/qwen4_exp/test_ple.py failed on main. The
layer now reads the policy, like ple_disk_release_pages next to it, when
it is built, and records the row-reader admission only when an engine
config exists.
The native row gather also reported an out-of-range row id as a
RuntimeError ('PLE disk row ID out of range') while the Python lane and
the tests expect IndexError ('PLE disk row id out of range'); it now uses
TORCH_CHECK_INDEX with the Python message and the offending id.
Signed-off-by: Peuqui <43776522+Peuqui@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZLMmGB8mjc43VpwPxLQNn
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Contributor
|
I have integrated this fix onto current main, preserving your signed contribution. I also updated the native row-reader regression to expect the same IndexError as the Python path. The scoped CPU gate passes 205 tests with 36 skips and pre-commit passes. Separately, compiling the production CPU gather source with an isolated unit-test registration and running the three affected suites yields 146 passes and 22 hardware skips, including all 14 native row-reader cases. This checks the new error type and unchanged byte/shard behavior without requiring a GPU. The existing contributor source-build hardware evidence remains retained; no full-model or local GPU result is claimed. AI assistance was used for integration and validation. |
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Since #885,
Qwen4ExpNGramEmbedding.load_weightsreadskernel_config.ple_disk_row_gatherthroughget_current_vllm_config()and writes the row-reader admission intokernel_config.ple_disk_row_readers.get_current_vllm_config()asserts outside aset_current_vllm_config()context. The engine loads inside one, but direct loads do not, and ten cases intests/models/qwen4_exp/test_ple.pyfail on main with that assertion.This PR
ple_disk_release_pages, which__init__already reads throughget_current_vllm_config_or_none(), and records the admission only when an engine config exists (same pattern as [Bugfix][DeepSeek-V4] Read the SM70 sparse policy when the layers are built #857 for the SM70 sparse policy);csrc/ple_disk_rows.cpp) report an out-of-range row id the way the Python lane and the tests do:TORCH_CHECK_INDEX(anIndexErrorin Python) with the Python lane's message and the offending id, instead of aRuntimeErrorwith "row ID". Callers catchingIndexErrormissed it on the native lane;__init__.In the engine nothing changes: the layer is built inside the config context, so it reads the same
KernelConfigobject as before.Not a duplicate: no open PR touches
ple_disk_row_gatherorple_disk_rows.cpp(gh pr list --state open --search "ple_disk_row_gather", "ple_disk_rows": none; "row id out of range" only finds #828, QSA prefill attention, unrelated). #903 also editsple_layer.py, in other functions (fused n-gram ids, PLE MTP guard); this branch merges cleanly with it.AI assistance was used for this change. I reviewed every line and ran the tests below.
Test Plan
Acceleration and benchmark contract (required for performance changes)
Not a performance change. The admission decision is the same; only where the policy is read from moves, and the native error type changes.
Test Result
test_ple.pyfail (AssertionError: Current vLLM config is not set). With the Python part alone, 7 of them pass and the 3 cases oftest_ngram_embedding_disk_gather_rejects_invalid_idsthen reach the native lane and fail on the exception type (checked on 17310de; none of the three files changed since); with the native part all pass.test_ple.py98 passed,test_ple_offload_worker.py55 passed and 1 skipped; Quadro RTX 8000 95 passed and 3 skipped (the existing exact-SM70 cases), 55 passed and 1 skipped.🤖 Generated with Claude Code