CUDA: full test-backend-ops sweep segfaults in ggml_backend_cuda_graph_compute at ROLL on GB10 (passes with graphs disabled)

[TheTom]

Summary

The full test-backend-ops sweep on a GB10 (sm_121) segfaults inside ggml_backend_cuda_graph_compute when it reaches the first ROLL case. It is deterministic (3 of 3 runs, two different builds), it is on the base branch, and it goes away with GGML_CUDA_DISABLE_GRAPHS=1, which then passes the whole sweep. ROLL on its own passes, so the crash depends on graph state accumulated by earlier cases.

Environment

• feature/turboquant-kv-cache at c26baf1 (also reproduced at 80be9a7)
• NVIDIA GB10, driver 580.173.02, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121, Release
• gcc + nvcc from /usr/local/cuda

Repro

./build/bin/test-backend-ops test -b CUDA0
  ...
  PAD_REFLECT_1D(type=f32,ne_a=[3000,384,4,1],pad_0=10,pad_1=9): OK
  ROLL(shift0=3,shift1=-2,shift3=1,shift4=-1):
Segmentation fault (core dumped)

9857 cases pass before the crash. Backtrace (base build, no debug symbols):

#0  ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*)  libggml-cuda.so.0
#1  ggml_backend_graph_compute
#2  ggml_backend_compare_graph_backend
#3  test_case::eval(...)

Control:

./build/bin/test-backend-ops test -b CUDA0 -o ROLL
  1/1 tests passed          (three runs)

GGML_CUDA_DISABLE_GRAPHS=1 ./build/bin/test-backend-ops test -b CUDA0
  18408/18408 tests passed

Notes

• Crash is in the graph-compute wrapper, not a kernel, and only after thousands of prior graphs, so it smells like the captured-graph cache (keyed by first-node pointer in this fork; upstream models: Qwen3.8-Flash-Next MTP ggml-org/llama.cpp#28243 changed that to key by shape). qwen4exp: upstream catch-up, sparse-fa, and NextN/MTP support #340 lists that cherry-pick in its description but does not contain it.
• Found while gating cuda: shared-quantize cache, residual and elementwise-chain fusions (generic half of #338) #343 on a full CUDA sweep; the crash is identical on that PR's head, so it is not caused by it.
• Every hosted CI job for this fork only builds CUDA, so nothing upstream of a GB10 would have caught this.

[TheTom]

Data point: #340's head (69499d4) carries the upstream ggml-org#28243 change (captured graphs keyed by shape instead of first-node address) and the full sweep still segfaults at the same ROLL case on the GB10, so the keying is not the cause. Everything before ROLL passes and GGML_CUDA_DISABLE_GRAPHS=1 still passes the whole sweep.

[tmark00]

Reproduced on a second platform, so this is not GB10-specific.

RTX 5090 Laptop GPU (sm_120), driver 616.56, Windows 11, MSVC 14.51, Ninja Release,
CMAKE_CUDA_ARCHITECTURES=120-real, BUILD_SHARED_LIBS=ON, branch at fb2cc35ab.

Both of your observations hold here:

Default sweep dies at the same case. test-backend-ops test -b CUDA0 stops with no verdict
printed for the ROLL line and exits 5:

  PAD_REFLECT_1D(type=f32,ne_a=[3000,384,4,1],pad_0=10,pad_1=9): OK
  ROLL(shift0=3,shift1=-2,shift3=1,shift4=-1):

ROLL on its own passes. test-backend-ops test -b CUDA0 -o ROLL → 1/1 tests passed,
Backend CUDA0: OK, exit 0.

Graphs disabled fixes it. With GGML_CUDA_DISABLE_GRAPHS=1 the same full sweep runs to
completion: ROLL(...) reports OK, execution continues through ARANGE,
TIMESTEP_EMBEDDING and the rest, and it ends at 18348/18396. The 48 failures are unrelated
pre-existing ones — iq1_s, iq2_s and iq3_s in MUL_MAT / MUL_MAT_ID (15 each) plus three
ROUND(type=f16) tolerance cases — and they reproduce identically on df7f54729, so they are not
connected to this crash.

One practical side effect worth noting for anyone using the sweep as a gate: because the crash
lands before FLASH_ATTN_EXT, a default run never reaches the flash-attention cases at all. Run
separately, test-backend-ops test -b CUDA0 -o FLASH_ATTN_EXT passes 7658/7658 here, including
754 turbo4 and 752 turbo3 cases — but only if you know to ask for it. I initially misread my own
truncated log as "the sweep does not cover FA".

Different OS, different compiler, different compute capability, same failure and same workaround,
so it looks like the accumulated-graph-state issue you suspected rather than anything specific to
GB10 or to sm_121.