Skip to content

fix(devicemanager): stop restoring ECC capacity on GDDR7 GPUs - #755

Merged
thxCode merged 1 commit into
mainfrom
fix/nvidia-memory-ecc
Oct 8, 2026
Merged

thxCode merged 1 commit into
mainfrom
fix/nvidia-memory-ecc

Conversation

@thxCode

@thxCode thxCode commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

/kind bug
/area devicemanager
/area vendor/nvidia

What this PR does / why we need it:

On an RTX PRO 6000 Blackwell Server Edition, the DeviceManager reported 104412 MiB per GPU and the InstanceType memory label showed 102Gi for a 96 GB card.

Root cause: NVML reports framebuffer capacity with the current ECC mode already applied. Since Turing, professional GDDR parts trade ~1/16 of user-visible memory for ECC parity when the mode is on, so the detector restored the nominal size (x16/15) on every narrow-bus (<1024-bit) ECC-enabled GPU. GDDR7 makes no such trade: its ECC is built into the DRAM die (NVIDIA RTX Blackwell GPU Architecture whitepaper v1.1), and the driver already reports the nominal capacity with ECC enabled.

The fix gates the restore on the GPU generation: it now applies only when the compute capability is readable and below 12.0 (RTX Blackwell is 12.0), on top of the existing narrow-bus and ECC-enabled conditions. Everything the detector cannot classify keeps the driver-reported total: HBM stacks (bus width >= 1024-bit), ECC mode off or unreadable, bus width unreadable, compute capability unreadable. L40S-class GDDR6 keeps the existing restore.

Evidence, by kind:

  • Measured - single RTX PRO 6000 Blackwell Server Edition, driver 580.173.02, raw NVML probe (the pre-fix code path; the fixed build was not involved): compute capability rc=0 -> 12.0; bus width rc=0 -> 512; ECC mode rc=0 -> current/pending ENABLED; MemoryInfo v1 and v2 total = 97887 MiB (the 96 GB nominal, no 1/16 carve). The old formula on these inputs produces 104412 MiB (102Gi), matching the reported cluster output.
  • Regression (CPU fixture) - 12 table cases drive the real DetectAccelerator and Node-label construction against a simulated NVML library: the measured readings (97887 MiB -> 96Gi), L40S GDDR6 (46080 -> 49152 MiB, 48Gi), Turing T4, a 128-bit narrow GDDR7 counterexample, ECC disabled, HBM, the V1 memory fallback, and unreadable bus-width/ECC/compute-capability paths. RED runs confirm the suite rejects both the baseline bug (RTX PRO 6000 restored to 104412 MiB) and a restore-free implementation (L40S left at 45Gi).
  • Not executed - running the fixed image on a physical RTX PRO 6000. The measurement probed NVML directly on the pre-fix path; physical validation of the fixed build remains open.

Which issue(s) this PR links to:

NONE

Special notes for your reviewer:

  • The class rule "GDDR7 needs no restore" rests on the whitepaper's die-built ECC statement plus one measured model (RTX PRO 6000 Server); other GDDR7 members were not measured. A future generation whose GDDR memory returns to framebuffer-carved ECC while shipping compute capability >= 12 would under-report nominal capacity - the code comment records this boundary.
  • Validation: go test -race -shuffle=on on the nvidia package and on detector/... + binding/nvml + pkg/device + pkg/nodefeature + worker controllers, make lint (0 issues), go build, and git diff --check all exit 0.

Does this PR introduce a user-facing change?

NVIDIA DeviceManager now reports the nominal 96GB framebuffer for GDDR7 GPUs such as the RTX PRO 6000 Blackwell Server Edition (previously 102Gi), while keeping the ECC capacity restore for older GDDR generations such as the L40S.

@gpustack-code-review

gpustack-code-review Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

NVML reports framebuffer capacity with the current ECC mode already
applied. Since Turing, professional GDDR parts trade ~1/16 of
user-visible memory for ECC parity when the mode is on, so the detector
restored the nominal size (x16/15) on every narrow-bus ECC-enabled GPU.

GDDR7 ends that trade: its ECC is built into the DRAM die (NVIDIA RTX
Blackwell GPU Architecture whitepaper v1.1), and the driver already
reports the nominal capacity with ECC enabled. Measured on an ECC-
enabled RTX PRO 6000 Blackwell Server Edition (driver 580.173.02):
compute capability 12.0, 512-bit bus, MemoryInfo total 97887 MiB, the
96 GB nominal with nothing carved out; the restore inflated it to
104412 MiB, shown as 102Gi.

Gate the restore on compute capability below 12 in addition to the
narrow bus and ECC-enabled conditions, and keep everything the detector
cannot classify on the driver-reported total: HBM stacks (bus width
>= 1024), ECC mode off or unreadable, bus width unreadable, and compute
capability unreadable. L40S-class GDDR6 keeps the restore.

The fixture tests drive the real DetectAccelerator and Node-label path,
and reject both the old baseline (RTX PRO 6000 restored to 102Gi) and a
restore-free implementation (L40S left at 45Gi).

Signed-off-by: thxCode <thxcode0824@gmail.com>
@thxCode
thxCode force-pushed the fix/nvidia-memory-ecc branch from a02a4d9 to 153dcdb Compare October 8, 2026 15:31
@thxCode
thxCode merged commit 45d1484 into main Oct 8, 2026
6 checks passed
@thxCode
thxCode deleted the fix/nvidia-memory-ecc branch October 8, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant