Repository navigation
nixlbench: allocate VRAM on Intel GPUs through Level Zero - #19
Draft
wpower-work wants to merge 1 commit into
Draft
wpower-work wants to merge 1 commit into
wpower-work wants to merge 1 commit into
Conversation
Let nixlbench use Intel GPU memory for VRAM segments
(--initiator_seg_type/--target_seg_type VRAM), so the libfabric
backend's Level Zero support can be benchmarked.
src/utils/ze.{h,cpp} wrap the Level Zero calls nixlbench needs: device
count, allocation, free, fill and device-to-host copy. libze_loader is
loaded with dlopen, so a build with the headers runs on hosts without
it, and meson detects Level Zero from level_zero/ze_api.h alone.
Allocations are dma-buf exportable: otherwise, in a single-device
context (e.g. under ZE_AFFINITY_MASK), the driver may compress the
allocation, so the NIC moves compressed bytes, or pool it into a shared
buffer object that libfabric registers at the wrong offset. Fill and
copy run on a synchronous immediate command list of the buffer's
device.
xferBenchUtils::useZeVram() decides once per process whether Level Zero
supplies VRAM. CUDA and ROCm take precedence when they have a device,
since such hosts often also expose an integrated Intel GPU. The decision
is made from device counts, not build flags, so a CUDA build uses Level
Zero on a host without NVIDIA devices. Allocation, cleanup, segment
selection and the --check_consistency copy-back all use it; the
copy-back checks Level Zero before CUDA, which a CUDA-header build
would otherwise reach on an Intel host.
resolveVramSegment() only checked for Neuron cores when built without
CUDA or ROCm, so a build with CUDA headers on a Neuron host never used
Neuron memory. Neuron is now checked at runtime first, alongside Level
Zero.
The nvshmem link step lists the object files it links by hand. It now
also links ze.cpp.o and -ldl, and neuron.cpp.o, which utils.cpp calls
into and whose omission made nvshmem builds fail to link.
|
👋 Hi wpower-work! Thank you for contributing to intel-staging/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
There was a problem hiding this comment.
🟡 Changes recommended
Level Zero allocation can succeed on Intel-only hosts, but the NIXL LIBFABRIC backend lacks VRAM capability advertisement and FI_HMEM_ZE registration/device mapping, causing registration to fail with NIXL_ERR_NOT_SUPPORTED.
1 open issue
What changed in this PR
Adds Level Zero memory allocation to nixlbench, enabling Intel GPU VRAM selection for benchmarking, with related runtime-selection and build updates.
Changes:
- Adds a dynamically loaded Level Zero wrapper for allocation, cleanup, fill, and device-to-host copy.
- Integrates runtime-based Level Zero selection into VRAM handling and consistency checks.
- Updates Meson detection and nvshmem linking, and checks Neuron availability before compile-time VRAM branches.
| File | Description |
|---|---|
| benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp | Integrates Level Zero allocation and cleanup into VRAM handling. |
| benchmark/nixlbench/src/utils/ze.h | Declares the Level Zero wrapper API and error-checking macro. |
| benchmark/nixlbench/src/utils/ze.cpp | Implements runtime loading and device-memory operations. |
| benchmark/nixlbench/src/utils/utils.h | Declares the shared Level Zero selection helper. |
| benchmark/nixlbench/src/utils/utils.cpp | Selects the VRAM provider and supports Level Zero consistency copy-back. |
| benchmark/nixlbench/src/utils/meson.build | Adds Level Zero wrapper sources to the utility library. |
| benchmark/nixlbench/meson.build | Detects Level Zero headers and updates nvshmem link inputs. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What?
Lets nixlbench use Intel GPU memory for VRAM segments
(
--initiator_seg_type/--target_seg_type VRAM) through Level Zero.It also fixes two nixlbench bugs found along the way:
neuron.cpp.owas missing from the link step.Why?
The libfabric backend supports Level Zero memory, but nixlbench could only allocate
VRAM through CUDA, ROCm or Neuron, so that support couldn't be benchmarked.
The bug fixes:
neuron.cpp.o,which
utils.cppcalls into.resolveVramSegment()only checked for Neuron cores when built without CUDA or ROCm.How?
Level Zero wrapper:
src/utils/ze.{h,cpp}wrap the Level Zero calls nixlbench needs:device count, allocation, free, fill and device-to-host copy.
libze_loaderis loaded withdlopen, so a build with the headers still runs onhosts without it. Meson detects Level Zero from
level_zero/ze_api.halone.ZE_AFFINITY_MASK), the driver may:Choosing the VRAM provider:
xferBenchUtils::useZeVram()decides once per processwhether Level Zero supplies VRAM.
expose an integrated Intel GPU.
Zero on a host without NVIDIA devices.
--check_consistencycopy-back all useit. The copy-back checks Level Zero before CUDA, which a CUDA-header build would
otherwise reach on an Intel host.
Neuron:
resolveVramSegment()now checks for Neuron at runtime first, alongsideLevel Zero, whatever the build flags.
nvshmem link step: now also links
ze.cpp.o,-ldlandneuron.cpp.o.