Repository navigation
fix(devicemanager): stop restoring ECC capacity on GDDR7 GPUs - #755
Merged
Merged
Conversation
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s). |
NVML reports framebuffer capacity with the current ECC mode already applied. Since Turing, professional GDDR parts trade ~1/16 of user-visible memory for ECC parity when the mode is on, so the detector restored the nominal size (x16/15) on every narrow-bus ECC-enabled GPU. GDDR7 ends that trade: its ECC is built into the DRAM die (NVIDIA RTX Blackwell GPU Architecture whitepaper v1.1), and the driver already reports the nominal capacity with ECC enabled. Measured on an ECC- enabled RTX PRO 6000 Blackwell Server Edition (driver 580.173.02): compute capability 12.0, 512-bit bus, MemoryInfo total 97887 MiB, the 96 GB nominal with nothing carved out; the restore inflated it to 104412 MiB, shown as 102Gi. Gate the restore on compute capability below 12 in addition to the narrow bus and ECC-enabled conditions, and keep everything the detector cannot classify on the driver-reported total: HBM stacks (bus width >= 1024), ECC mode off or unreadable, bus width unreadable, and compute capability unreadable. L40S-class GDDR6 keeps the restore. The fixture tests drive the real DetectAccelerator and Node-label path, and reject both the old baseline (RTX PRO 6000 restored to 102Gi) and a restore-free implementation (L40S left at 45Gi). Signed-off-by: thxCode <thxcode0824@gmail.com>
thxCode
force-pushed
the
fix/nvidia-memory-ecc
branch
from
October 8, 2026 15:31
a02a4d9 to
153dcdb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
/area devicemanager
/area vendor/nvidia
What this PR does / why we need it:
On an RTX PRO 6000 Blackwell Server Edition, the DeviceManager reported 104412 MiB per GPU and the InstanceType memory label showed
102Gifor a 96 GB card.Root cause: NVML reports framebuffer capacity with the current ECC mode already applied. Since Turing, professional GDDR parts trade ~1/16 of user-visible memory for ECC parity when the mode is on, so the detector restored the nominal size (x16/15) on every narrow-bus (<1024-bit) ECC-enabled GPU. GDDR7 makes no such trade: its ECC is built into the DRAM die (NVIDIA RTX Blackwell GPU Architecture whitepaper v1.1), and the driver already reports the nominal capacity with ECC enabled.
The fix gates the restore on the GPU generation: it now applies only when the compute capability is readable and below 12.0 (RTX Blackwell is 12.0), on top of the existing narrow-bus and ECC-enabled conditions. Everything the detector cannot classify keeps the driver-reported total: HBM stacks (bus width >= 1024-bit), ECC mode off or unreadable, bus width unreadable, compute capability unreadable. L40S-class GDDR6 keeps the existing restore.
Evidence, by kind:
102Gi), matching the reported cluster output.DetectAcceleratorand Node-label construction against a simulated NVML library: the measured readings (97887 MiB ->96Gi), L40S GDDR6 (46080 -> 49152 MiB,48Gi), Turing T4, a 128-bit narrow GDDR7 counterexample, ECC disabled, HBM, the V1 memory fallback, and unreadable bus-width/ECC/compute-capability paths. RED runs confirm the suite rejects both the baseline bug (RTX PRO 6000 restored to 104412 MiB) and a restore-free implementation (L40S left at 45Gi).Which issue(s) this PR links to:
NONE
Special notes for your reviewer:
go test -race -shuffle=onon the nvidia package and on detector/... + binding/nvml + pkg/device + pkg/nodefeature + worker controllers,make lint(0 issues),go build, andgit diff --checkall exit 0.Does this PR introduce a user-facing change?