HRX opens the NPU device (/dev/accel/accel0) in GPU-only processes

HRX opens the NPU device in GPU-only processes

GPU-only processes on the HRX backend open /dev/accel/accel0: llama-bench -dev HRX0 and the llama-server that 1bit serve starts each hold the device with a 64 MiB mapping, although they never use the NPU. HRX enumerates the amdxdna device at startup.

Why it matters

NPU work in another process waits on, or competes with, GPU runs that do not need the NPU. The amdxdna firmware schedules contexts temporally across all columns, and one stuck context starves the rest.
A GPU process dying while it holds the NPU fd goes through the amdxdna client-cleanup path.
Tools that check whether the NPU is idle (fuser /dev/accel/accel0) see GPU jobs as NPU users.

Expected

A process that uses only HRX0 (GPU) does not open the NPU device. Open it only when an NPU device is selected, or provide an environment switch that turns off amdxdna enumeration.

🤖 Generated with Claude Code

Status, with one more data point and the safety context.

It affects every GPU process, not only llama-bench. On 2026-10-11 the PR-Agent serve and the canonical 1bit serve --device vulkan on strixhalo also held /dev/accel/accel0 while idle, and so did a llama-cli at 22:11. Any process that loads the HRX (or ggml) backend set enumerates amdxdna.
The kernel-panic risk in "Why it matters" is real on the stock driver. On 2026-10-11 the stock out-of-tree amdxdna (0.17.0 @342187206) panicked three times in amdxdna_client_cleanup when a process holding the device died by a signal. strixhalo now runs our patched module (srcversion 402ED31D0BF4EEF186795AC; the disclosure to AMD is pending), so a GPU process dying there is safe now, but any box on the stock driver is still exposed.
Fix direction is as the issue says: open the NPU only when an NPU device is selected, plus an env switch that disables amdxdna enumeration. For the switch, a narrow first step is to skip the HRX amdxdna backend's device probe when every -dev the user passed is HRX0/GPU.

The device is opened by `endpoint_open()`, not by enumeration

I traced the path in hrx-system (libamdf/src/platform/linux/endpoint.c). The
stated cause — "HRX enumerates the amdxdna device at startup" — is not what opens
the device:

amdf_platform_endpoint_enumerate() (endpoint.c:170) never opens the device.
It only reads sysfs: amdf_linux_read_pci_attribute() for the PCI ids, and
readlinkat(directory, "device/driver") to classify the engine kind. That path is
read-only and harmless.

The open() is at endpoint.c:362, in a different function:

static amdf_status_t amdf_linux_open_device_file(
    const amdf_endpoint_info_t* info, const char* path, int* out_descriptor) {
  int descriptor = open(path, O_RDWR | O_CLOEXEC);
  ...
}

and it is reached only through amdf_platform_endpoint_open() (endpoint.c:378).
endpoint_open() also qualifies the device file with DRM_IOCTL_VERSION, checking
the driver name against amdxdna_accel_driver for an XDNA endpoint — which matches
the observed 64 MiB mapping, since that is a real device handle, not a sysfs read.

So a GPU-only HRX process is calling endpoint_open() on the XDNA endpoint.

Why this changes the proposed fix

The issue proposes "open it only when an NPU device is selected, or provide an
environment switch that turns off amdxdna enumeration." A switch on enumeration
would not help: enumeration is sysfs-only and does not hold the fd. The switch has
to gate amdf_platform_endpoint_open() for AMDF_ENGINE_KIND_XDNA — i.e. HRX
must not open the XDNA endpoint unless an NPU device was selected.

The rest of the issue stands: a GPU process holding the NPU fd can starve NPU work
(amdxdna schedules contexts temporally across all columns), can enter the amdxdna
client-cleanup path on death, and makes fuser /dev/accel/accel0 report GPU jobs
as NPU users.

Not verified: that the 64 MiB mapping in the report comes from this specific
endpoint_open() call rather than a later context/BO creation on the same fd — the
open is confirmed as the only open() on that path, but I did not attribute the
mapping to it with a trace.

amdgpu: keep libhsa from opening the NPU (/dev/accel) in GPU processe…

Fixed in `libhsa`, and the opener is not `endpoint.c` after all

Merged on 1bit-MONSTER/hrx-system 1bit/main: PR 1bit-MONSTER/hrx-system#4 (branch 1bit/hrx-401-npu-gate-v2, commit e8a7c9e16, rebased onto 4ba76c18). A candidate gate had been sitting on branch 1bit/hrx-401-npu-gate (637c2845, 1688 commits behind); it cherry-picked cleanly.

What actually holds the fd

The issue says HRX "enumerates the amdxdna device at startup", and the follow-up analysis traced amdf_platform_endpoint_open() -> open() at libamdf/src/platform/linux/endpoint.c:362. Measured on strixhalo, that is not the opener for this process class:

llama-bench -dev HRX0 (base build) holds fd 4 -> /dev/accel/accel0, maps ... rw-s 10b533000 ... /dev/accel/accel0 (64 MiB), and appears in fuser -v /dev/accel/accel0.
/proc/<pid>/maps for that process lists libhrx + libloomc + libhsa — no libamdf at all, in every variant (base, gated, gate + escape hatch). The same holds for the llama-server child of 1bit serve --device hrx.

The opener is libhsa (ROCr/TheRock): XdnaDriver::DiscoverDriver opens /dev/accel/accel0 during hsa_init, maps the 64 MiB heap and registers the aie2p agent, reached from runtime/src/iree/hal/drivers/amdgpu/util/libhsa.c. So the fix is a gate there, not in libamdf.

Result

process	accel fds	64 MiB map	in `fuser`
base (`4ba76c18`)	1	yes	yes
gated (`e8a7c9e16`)	0	no	no
gated + `HRX_ENABLE_XDNA=1`	1	yes	yes
`1bit serve --device hrx` child, base -> gated	1 -> 0	yes -> no	yes -> no

The gate prints libhsa NPU gate: 1 open slot(s) filtered; HRX_ENABLE_XDNA=1 is the escape hatch and restores the device, so NPU device selection is unaffected. Throughput, 3 interleaved rounds, contended (Tctl 81-89 C, load 10-12, other agents' Vulkan services resident): pp512 median 16879 (base) vs 17278 (gate), tg128 297.2 vs 294.5 — within noise.

One observed, unexplained wrinkle is recorded rather than folded into the claim: the 1bit serve text comparison gave 3/3 identical hashes on the gated build and a different third answer on base, with the server's default cache_prompt.

Not yet in engine builds

The fix is on the fork line; third_party/hrx-system is held at 98d05d94 by config/hrx-system-hold.config. Two measured reasons make the move non-trivial, and neither is this fix: after 98d05d94 AMD's tree carries the long-context attention miscompile (ROCm/hrx-system#1535), and the llama.cpp fork tip's kernel corpus uses the Loom predicate mul( where the newer hrx-system's Loom only knows multiple_of — while 98d05d94's own loom-link parses all 123 corpus files. So the engine pair (llama.cpp ad18320a + hrx-system 98d05d94) builds, and (ad18320a + 4ba76c18) does not. Ship route is being costed separately; this issue's code defect is fixed.

Evidence: team/evidence/401-npu-gate.md and team/evidence/401-raw/{cap,ab}/.

HRX: pin hrx-system to the held-pin backport carrying #409, #410 and #…

Shipped: the engine now pins the fix

The NPU-gate fix is in engine builds: PR #423 moved third_party/hrx-system to 1bit/held-98d05d94-3fixes (3319e5361, merged as 7d44c99). The hold itself is unchanged, so this did not drag in the 1688-commit core move.

Verified at the pinned line on strixhalo (gfx1151, -DONEBIT_HRX=ON build exit 0):

GPU-only llama-server and test-backend-ops hold 0 /dev/accel/accel0 fds; fuser -v /dev/accel/accel0 never lists them (before: fd 4 plus the 64 MiB rw-s mapping, and present in fuser).
HRX_ENABLE_XDNA=1 restores the device and mapping, so NPU device selection is unaffected.
Op gate test-backend-ops -b HRX0: 1141/1141 passed.

The attribution correction from the fix comment stands and is now the record: it is libhsa, not libamdf, that opens the device in these processes — /proc/<pid>/maps shows libhrx + libloomc + libhsa and no libamdf, so the issue thread's endpoint.c:362 path is not what holds the fd here.

Closing as completed.

Move HRX pins: hrx-system to AMD's e3e6a976 line, llama.cpp 8d9ce2c1a (…

Navigation Menu

Repository navigation

HRX opens the NPU device (/dev/accel/accel0) in GPU-only processes #401

HRX opens the NPU device in GPU-only processes

Why it matters

Expected

The device is opened by `endpoint_open()`, not by enumeration

Why this changes the proposed fix

Fixed in `libhsa`, and the opener is not `endpoint.c` after all

What actually holds the fd

Result

Not yet in engine builds

Shipped: the engine now pins the fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX opens the NPU device (/dev/accel/accel0) in GPU-only processes #401

Description

HRX opens the NPU device in GPU-only processes

Why it matters

Expected

Activity

bong-water-water-bong commented on Oct 12, 2026

bong-water-water-bong commented on Oct 12, 2026

The device is opened by endpoint_open(), not by enumeration

Why this changes the proposed fix

bong-water-water-bong commented on Oct 16, 2026

Fixed in libhsa, and the opener is not endpoint.c after all

What actually holds the fd

Result

Not yet in engine builds

bong-water-water-bong commented on Oct 16, 2026

Shipped: the engine now pins the fix

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The device is opened by `endpoint_open()`, not by enumeration

Fixed in `libhsa`, and the opener is not `endpoint.c` after all