fix: check cudaHostRegister return code and clear CUDA sticky error by jooho-XCENA · Pull Request #65 · xcena-dev/maru

jooho-XCENA · 2026-07-15T03:43:22Z

Problem

DaxMapper.map_region() ignores the return code of torch.cuda.cudart().cudaHostRegister(...) — torch's cudart binding reports errors via cudaError_t, it does not raise. Two consequences:

False success: a failed pin still logs CUDA pinned region ..., so the region silently runs unpinned (pageable-copy GPU transfers).
Misattributed crash: the failed call latches CUDA's per-thread sticky error, and the next error-checked CUDA call on that thread — typically an unrelated kernel launch — raises torch.AcceleratorError: CUDA error: out of memory. Reproduced with pool_size: "520": vLLM EngineCore dies at startup in flashinfer autotune (sm.fill_), pointing at an innocent op.

The concrete trigger today is the NVIDIA r580+ driver's per-registration limit: a single cudaHostRegister of >= 2^39 bytes (512 GiB) fails with cudaErrorMemoryAllocation because the driver's per-page bookkeeping table (16 B / 4 KiB page, one flat kvzalloc) hits the kernel's INT_MAX cap — NVRM: failed to allocate page table in dmesg. Confirmed on two hosts (580.126.20, 580.159.03); drivers <= 575.x are unaffected.

Change (minimal, no behavior change on the success path)

Check the cudaHostRegister return code. Success: same log as before, and the region is marked pinned. Failure: consume the sticky error via the process's own libcudart (cudaGetLastError) and log a warning — the region stays usable, GPU transfers fall back to pageable copies, and nothing crashes later.
unmap_region()/close() only call cudaHostUnregister for regions that actually pinned, and check that return code too.

Verification

2 new unit tests (rc failure -> warning + no unregister; rc success -> pinned + single unregister); mapper suite 30 passed.
Diagnosis aid: with this branch, an oversized pool logs cudaHostRegister failed for region N (...): rc=2 — region stays unpinned instead of crashing with a fake OOM.

Upstream driver fix

The underlying >= 512 GiB failure is an r580 driver regression, and the fix (vzalloc fallback for the oversized page table) is already pending upstream: NVIDIA/open-gpu-kernel-modules#1234 (we reproduced the bug on two hosts and confirmed the same fix compiles against 580.126.20; see the supporting comment there). Once that lands and a fixed driver ships, large single-call registrations simply work again.

So the plan is: merge this rc-check fix (worth having regardless of the driver — a failed pin should warn instead of lying in the logs and crashing an unrelated kernel launch later), and wait for the driver fix for the >= 512 GiB case itself. Until a fixed driver ships, oversized pools run unpinned with a clear warning (pageable-copy performance). The chunked-registration workaround was #64, now closed; the branch is kept in case the upstream fix stalls.

Minimal diagnostic fix (no chunking): DaxMapper ignored the return code of torch's cudaHostRegister binding, which reports errors via cudaError_t instead of raising. A failed pin — e.g. the NVIDIA r580+ per-registration 512GiB limit ("NVRM: failed to allocate page table", per-page table kvzalloc > INT_MAX) — was logged as "CUDA pinned region" and left CUDA's per-thread sticky error latched, crashing an unrelated later kernel launch with a misattributed "CUDA error: out of memory". - Check rc: success sets MappedRegion._cuda_pinned and keeps the old log; failure clears the sticky error via the process's own libcudart and warns that the region stays unpinned (pageable-copy fallback). - unmap/close only cudaHostUnregister regions that actually pinned, and check that rc too. The chunked-registration fix that makes >=512GiB pools actually pin is feat/chunked-cuda-pin (PR #64); this branch is the rc-check subset for minimal-risk deployment/diagnosis.

Builds on the rc-check fix (#65) to make >= 512GiB pools actually pin. A single cudaHostRegister call of >= 2^39 bytes (512GiB) fails on NVIDIA r580+ drivers (per-registration page-table kvzalloc > INT_MAX, "NVRM: failed to allocate page table"); the cap is per call, not cumulative, so large regions are registered in chunks. - Register regions in MARU_PIN_CHUNK_GB chunks (default 128GiB, clamped to 256GiB; 0 = single call), recording (addr, size) per pinned chunk on MappedRegion and unregistering per chunk on unmap/close (replaces the single _cuda_pinned flag from #65). - MARU_PIN_MODE=eager (default) pins synchronously inside map_region(), preserving the fully-pinned-on-return contract; MARU_PIN_MODE=lazy pins from a background thread. A per-region pin lock + stop event serialize the pin loop against unmap-time unpinning, bounding teardown to one in-flight chunk. Verified: unit suite 766 passed; E2E on gaia (/dev/dax0.0) with pool_size=520 — previously crashed at startup, now pins 5 chunks in 4.45s and the benchmark passes 20/20 with cache-hit TTFT 1.8s.

github-actions · 2026-07-15T03:44:10Z

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	6	0	100%
__main__.py	3	3	0%	5, 7–8
allocation_manager.py	129	11	91%	30–31, 44–45, 52, 237–238, 268–271
client.py	187	8	96%	88, 130, 147–148, 311–313, 319
config.py	46	1	98%	72
constants.py	9	0	100%
device_scanner.py	94	31	67%	25, 100, 102–104, 106, 114–123, 125, 127, 129, 134–143, 145, 147
handler.py	557	86	85%	153, 168, 179–186, 194–196, 211, 217–219, 227, 238–243, 248, 273, 300–301, 305, 345, 349–350, 355, 377, 384–385, 389–395, 398, 402–404, 420, 422–423, 432, 486, 507, 517–518, 624–626, 633, 755–758, 761, 772–775, 781, 867, 971, 1134–1138, 1144, 1155–1159, 1165, 1244, 1249, 1291
ipc.py	275	2	99%	365, 441
kv_manager.py	108	0	100%
logging_setup.py	19	0	100%
plugin.py	59	3	95%	65–67
protocol.py	233	0	100%
resource_manager_installer.py	103	13	87%	80–86, 167, 169–172, 187
rpc_async_client.py	190	0	100%
rpc_async_server.py	111	0	100%
rpc_client.py	71	0	100%
rpc_client_base.py	106	10	91%	185, 221–222, 233–234, 306–307, 311–312, 375
rpc_handler_mixin.py	106	19	82%	154–156, 159–161, 219–222, 227, 231–235, 256–257, 263
rpc_server.py	64	0	100%
serializer.py	81	0	100%
server.py	170	13	92%	64–65, 73, 168, 172, 243, 247, 265–266, 357–359, 444
stats_manager.py	95	0	100%
types.py	60	1	98%	145
uds_helpers.py	13	0	100%
memory
__init__.py	5	0	100%
allocator.py	55	0	100%
mapper.py	164	17	90%	65–74, 77–80, 293, 342, 376
owned_region_manager.py	101	1	99%	212
types.py	63	0	100%
TOTAL	3304	219	93%

Tests	Skipped	Failures	Errors	Time
713	4 💤	0 ❌	0 🔥	8.204s ⏱️

seohui-XCENA

LGTM overall — clean, minimal fix. Inline comments below.

Summary: please take a look at the libcudart-matching comment before merge — it can defeat the fix on some torch builds. The log-level and DRY ones are optional cleanups. The multi-host one doesn't block this PR; let's discuss it separately (happy to file an issue).

seohui-XCENA · 2026-07-23T07:38:51Z

+        path = None
+        with open("/proc/self/maps") as f:
+            for line in f:
+                if "libcudart.so" in line:


torch builds that bundle cudart name it libcudart-<hash>.so.X, which this substring doesn't match — path stays None and the sticky error stays latched. Matching "libcudart" is safer.

Also, multiple cudart instances can be loaded (pip nvidia package + system CUDA), and the last-error state is per instance — clear every match, not just the first.

Nit: the maps path can carry an " (deleted)" suffix, which breaks CDLL.

Good catch — stock torch wheels do bundle it as libcudart-.so.X, so the old match would have missed it entirely. Fixed in a565bc1: match on the basename containing "libcudart", strip the " (deleted)" suffix, and clear every distinct mapped copy (added parser tests for the hashed name, deleted suffix, and dedupe cases).

seohui-XCENA · 2026-07-23T07:38:51Z

+            lib.cudaGetLastError.restype = ctypes.c_int
+            lib.cudaGetLastError()
+    except (OSError, IndexError, AttributeError):
+        logger.debug("could not clear CUDA sticky error", exc_info=True)


If the clear fails, a misattributed crash can still happen later — that deserves warning, not debug.

Changed — it warns now whenever nothing could be cleared.

seohui-XCENA · 2026-07-23T07:38:51Z

-                        region_id,
-                        handle.length,
-                    )
+                    rc = int(rc[0]) if isinstance(rc, tuple) else int(rc)


This normalization appears 3×, and the unregister block in unmap_region()/close() is nearly identical — worth extracting helpers. A test for the unregister-failure path could come with it.

Extracted _cuda_rc() and _cuda_unpin_region(); unmap_region()/close() share the unregister path now, and the unregister-failure rc test is in.

seohui-XCENA · 2026-07-23T07:38:51Z

+                        logger.warning(
+                            "cudaHostRegister failed for region %d "
+                            "(%d bytes): rc=%d — region stays unpinned, "
+                            "GPU transfers fall back to pageable copies",


The pageable fallback routes transfers through the CPU, so the CXL region is no longer touched by GPU DMA only. This could be a problem for multi-host sharing — probably worth a follow-up discussion.

Agreed, and out of scope here — worth its own thread. Note the driver-side fix is pending upstream (NVIDIA/open-gpu-kernel-modules#1234), which removes the >=512GiB trigger, but partial pin failure in general still leaves the pageable path so the multi-host question stands.

…in path - _find_libcudart_paths(): match on basename containing "libcudart" so torch wheels' bundled libcudart-<hash>.so.X is found, strip the " (deleted)" maps suffix, and return every distinct mapped copy — the sticky-error state is per instance, so clear all of them. - Warn (not debug) when no libcudart could be cleared. - Extract _cuda_rc() and _cuda_unpin_region(); unmap_region()/close() now share the unregister path. - Tests: maps-parser cases (hashed name, deleted suffix, dedupe, anonymous rows) and the unregister-failure rc path.

seohui-XCENA

a565bc1 addresses everything — the parser tests are a nice touch. Thanks for the quick turnaround!

jooho-XCENA mentioned this pull request Jul 15, 2026

feat: chunked CUDA host registration with eager/lazy pin modes #64

Closed

jooho-XCENA requested a review from a team July 15, 2026 04:37

style: ruff format

3743de0

This comment was marked as duplicate.

Sign in to view

seohui-XCENA reviewed Jul 23, 2026

View reviewed changes

jooho-XCENA and others added 2 commits July 23, 2026 16:41

Merge branch 'main' into fix/pin-rc-check

42ee811

seohui-XCENA approved these changes Jul 23, 2026

View reviewed changes

jooho-XCENA merged commit 42f5ae6 into main Jul 23, 2026
3 checks passed

jooho-XCENA deleted the fix/pin-rc-check branch July 23, 2026 08:06

Uh oh!

Conversation

jooho-XCENA commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change (minimal, no behavior change on the success path)

Verification

Upstream driver fix

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

seohui-XCENA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seohui-XCENA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jooho-XCENA commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading