Repository navigation
[Bugfix][PLE] Integrate split placement with padding and loading repairs (#528) - #538
Merged
Merged
Conversation
…host The pinned-host PLE path keeps the whole FP8 n-gram shard in pinned host memory and gathers every row over UVA. On a card that has room beside its layers that is the slow option, and on a box with little host RAM it does not boot at all: the Qwen3.8 Flash Next table is 51 GB, a TP2 rank pins 25 GB. Place the table by capacity instead: rows stay in device memory as long as the stage's weights, the KV cache of the requested max_model_len and a reserve for the activation peak and the graph pool still fit; only the remainder goes to pinned host memory. The budget is derived from the real headroom when the first checkpoint shard arrives (every other weight of the stage is placed by then), or fixed per rank with VLLM_QWEN4EXP_PLE_HOST_GIB. Both halves are gathered by the existing pinned-gather kernel; a row-less half is skipped without a device-to- host sync. The loader routes each shard into the half that owns its rows. This is how we serve Qwen3.8 Flash Next NVFP4 at TP2/PP2 on 2x RTX 8000 + 2x V100 with VLLM_QWEN4EXP_PLE_HOST_GIB=6 (the rest of each shard on the 48 GB cards): 51.9 tok/s with MTP k=4, coherence 8/8, measured 2026-08-28 on our fork and carried into our 1.5.0 deployment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com>
_should_use_pinned_host_ple() asked current_platform for the capability of device 0 of the visible list. On a heterogeneous pipeline the stage that owns the PLE layers is not necessarily that device: with a Turing card first the Volta stage lost its pinned-host path and hit the fp16 materialization of the generic embedding, with a Volta card first a Turing PLE stage got it although it only qualifies with the split placement. Ask for the capability of the worker's current device instead. The split placement also serves every pre-Ampere card, not only exact SM70: the generic path dequantizes the whole FP8 table to fp16 inside the compiled graph (47.7 GiB for Qwen3.8 Flash Next), which neither Volta nor Turing can hold. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com>
Every tensor-parallel rank of the stage that owns the PLE table derives its host budget from MemAvailable, and all of them draw on the same host memory. Reading it per rank double-books it: on a 30 GB host with 20 GB available, two ranks each saw room for 7 GiB, pinned 14 GiB together and pushed the engine processes, the checkpoint loading and everything else into swap. The machine was unreachable for 40 minutes. cap_host_budget_bytes bounds a rank's budget by (available - reserve) / ranks_sharing_host. The reserve is a fraction of the host rather than a fixed size, so the rule means the same on every machine; the default is a quarter, overridable with VLLM_QWEN4EXP_PLE_HOST_RESERVE_GIB. What no longer spills stays on the device, and if the requested context then does not fit, the KV allocator reports the reachable max_model_len -- host memory is the hard limit, context the negotiable one. Measured on the reporting rig: 7.09 -> 3.18 GiB per rank, boot healthy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…w-20260907-030709 Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Integrate #528 safely on current main; validate memory budgets, preserve retryable allocation state, and test graph replay with in-place reload. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Contributor
Author
|
Final integration audit: merged latest main 4791946 into the isolated validation tree. All 94 combined PLE/projection/HC tests passed on reserved V100 GPU 3; 101 config/environment/fingerprint tests previously passed. Fresh GitHub pre-commit run 34079052326 succeeded. Original #528 commits are preserved; no full-model speed or 1M-context quality claim. |
yangzhuxinyzx
marked this pull request as ready for review
September 7, 2026 03:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Integrate #528 on current public main
53199eb8b996831718330c23144c60ed9952f690, preserving its commits and automatic split PLE placement. Fix audit findings without adding decode-time branches or changing gather arithmetic.Repairs
Test Plan / Result
Performance / limitations
Preserve #528 capacity/default placement logic. These repairs are load-time; no new decode arithmetic or additional runtime copy is introduced. No fresh model-level speed or maximum-context claim. Memory planning is an estimate; hybrid allocator padding, actual graph/activation peaks and host limits can still cap the reachable context. See
docs/design/ple_split_mainline_audit.md.Ownership
Owned worktree
/home/ymzx/桌面/1cat-vllm/worktrees/v100-sep07-ple-review-20260907-030709. Logs/tmp/1cat-sep07-ple-gpu.log,/tmp/1cat-sep07-ple-retry.log,/tmp/1cat-sep07-ple-config.log. Canonical dirty checkout untouched.