Skip to content

nixlbench: allocate VRAM on Intel GPUs through Level Zero - #19

Draft
wpower-work wants to merge 1 commit into
mainfrom
feat/add-intel-xe-5
Draft

wpower-work wants to merge 1 commit into
mainfrom
feat/add-intel-xe-5

Conversation

@wpower-work

Copy link
Copy Markdown

What?

Lets nixlbench use Intel GPU memory for VRAM segments
(--initiator_seg_type/--target_seg_type VRAM) through Level Zero.

It also fixes two nixlbench bugs found along the way:

  • nvshmem builds failed to link because neuron.cpp.o was missing from the link step.
  • A build with CUDA headers on a Neuron host never used Neuron memory.

Why?

The libfabric backend supports Level Zero memory, but nixlbench could only allocate
VRAM through CUDA, ROCm or Neuron, so that support couldn't be benchmarked.

The bug fixes:

  • The nvshmem link step lists its object files by hand and left out neuron.cpp.o,
    which utils.cpp calls into.
  • resolveVramSegment() only checked for Neuron cores when built without CUDA or ROCm.

How?

Level Zero wrapper: src/utils/ze.{h,cpp} wrap the Level Zero calls nixlbench needs:
device count, allocation, free, fill and device-to-host copy.

  • libze_loader is loaded with dlopen, so a build with the headers still runs on
    hosts without it. Meson detects Level Zero from level_zero/ze_api.h alone.
  • Allocations are dma-buf exportable. Otherwise, in a single-device context (e.g. under
    ZE_AFFINITY_MASK), the driver may:
    • compress the allocation, so the NIC moves compressed bytes, or
    • pool it into a shared buffer object that libfabric registers at the wrong offset.
  • Fill and copy run on a synchronous immediate command list of the buffer's device.

Choosing the VRAM provider: xferBenchUtils::useZeVram() decides once per process
whether Level Zero supplies VRAM.

  • CUDA and ROCm take precedence when they have a device, since such hosts often also
    expose an integrated Intel GPU.
  • The decision is made from device counts, not build flags, so a CUDA build uses Level
    Zero on a host without NVIDIA devices.
  • Allocation, cleanup, segment selection and the --check_consistency copy-back all use
    it. The copy-back checks Level Zero before CUDA, which a CUDA-header build would
    otherwise reach on an Intel host.

Neuron: resolveVramSegment() now checks for Neuron at runtime first, alongside
Level Zero, whatever the build flags.

nvshmem link step: now also links ze.cpp.o, -ldl and neuron.cpp.o.

Let nixlbench use Intel GPU memory for VRAM segments
(--initiator_seg_type/--target_seg_type VRAM), so the libfabric
backend's Level Zero support can be benchmarked.

src/utils/ze.{h,cpp} wrap the Level Zero calls nixlbench needs: device
count, allocation, free, fill and device-to-host copy. libze_loader is
loaded with dlopen, so a build with the headers runs on hosts without
it, and meson detects Level Zero from level_zero/ze_api.h alone.
Allocations are dma-buf exportable: otherwise, in a single-device
context (e.g. under ZE_AFFINITY_MASK), the driver may compress the
allocation, so the NIC moves compressed bytes, or pool it into a shared
buffer object that libfabric registers at the wrong offset. Fill and
copy run on a synchronous immediate command list of the buffer's
device.

xferBenchUtils::useZeVram() decides once per process whether Level Zero
supplies VRAM. CUDA and ROCm take precedence when they have a device,
since such hosts often also expose an integrated Intel GPU. The decision
is made from device counts, not build flags, so a CUDA build uses Level
Zero on a host without NVIDIA devices. Allocation, cleanup, segment
selection and the --check_consistency copy-back all use it; the
copy-back checks Level Zero before CUDA, which a CUDA-header build
would otherwise reach on an Intel host.

resolveVramSegment() only checked for Neuron cores when built without
CUDA or ROCm, so a build with CUDA headers on a Neuron host never used
Neuron memory. Neuron is now checked at runtime first, alongside Level
Zero.

The nvshmem link step lists the object files it links by hand. It now
also links ze.cpp.o and -ldl, and neuron.cpp.o, which utils.cpp calls
into and whose omission made nvshmem builds fail to link.
@wpower-work
wpower-work requested a balanced review from Copilot October 30, 2026 00:05
@github-actions

Copy link
Copy Markdown

👋 Hi wpower-work! Thank you for contributing to intel-staging/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Level Zero allocation can succeed on Intel-only hosts, but the NIXL LIBFABRIC backend lacks VRAM capability advertisement and FI_HMEM_ZE registration/device mapping, causing registration to fail with NIXL_ERR_NOT_SUPPORTED.

1 open issue
What changed in this PR

Adds Level Zero memory allocation to nixlbench, enabling Intel GPU VRAM selection for benchmarking, with related runtime-selection and build updates.

Changes:

  • Adds a dynamically loaded Level Zero wrapper for allocation, cleanup, fill, and device-to-host copy.
  • Integrates runtime-based Level Zero selection into VRAM handling and consistency checks.
  • Updates Meson detection and nvshmem linking, and checks Neuron availability before compile-time VRAM branches.
File Description
benchmark/​nixlbench/​src/​worker/​nixl/​nixl_worker.cpp Integrates Level Zero allocation and cleanup into VRAM handling.
benchmark/​nixlbench/​src/​utils/​ze.h Declares the Level Zero wrapper API and error-checking macro.
benchmark/​nixlbench/​src/​utils/​ze.cpp Implements runtime loading and device-memory operations.
benchmark/​nixlbench/​src/​utils/​utils.h Declares the shared Level Zero selection helper.
benchmark/​nixlbench/​src/​utils/​utils.cpp Selects the VRAM provider and supports Level Zero consistency copy-back.
benchmark/​nixlbench/​src/​utils/​meson.build Adds Level Zero wrapper sources to the utility library.
benchmark/​nixlbench/​meson.build Detects Level Zero headers and updates nvshmem link inputs.

Review effort: Balanced

@wpower-work
wpower-work requested a review from tsg- October 30, 2026 15:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants