Skip to content

Give the A40 device GEMM a named functor (#815) - #841

Merged
jtravs merged 4 commits into
mainfrom
fix/radial-device-815
Nov 15, 2026
Merged

jtravs merged 4 commits into
mainfrom
fix/radial-device-815

Conversation

@jtravs

@jtravs jtravs commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

On a Release CUDA build for compute capability 8.6 (AMPERE86, such as the NVIDIA A40), every radial native run segfaulted in the session's first advance. That held for every stepper, envelope or field grid, linear or Kerr, and with or without a collar or statistics.

Cause. device_dense_gemm's kernel was an extended __host__ __device__ lambda.

  • nvcc keeps such a lambda's copy, call and delete functions in per-file static variables, set where the lambda is constructed.
  • Kokkos's dispatch, which copies the lambda, is an inline template shared between files.
  • Three source files instantiated it with the same types (dispersive.cpp, statistics.cpp, trial.cpp), and the link kept dispersive.cpp's dispatch.
  • The radial route constructs the lambda in trial.cpp or statistics.cpp, so its first step called dispersive.cpp's copy slot, which was still null. A SIGSEGV handler reported rip=0 just after the call through that slot.

Other device targets are unaffected: a HOPPER90 build compiles no device_dense_gemm.

Fix. The kernel is now a named functor (DeviceDenseGemm), whose copy is its own. The arithmetic is unchanged.

Guard.

  • New: native/ci/check_lambda_tags.py fails a CUDA build's object files when more than one of them holds the same extended lambda. Kokkos-FFT's never-constructed normalize_impl lambdas are allowed.
  • It runs in the CUDA device-compile lane. That lane builds HOPPER90, so it cannot see this particular lambda; against AMPERE86 objects it flags main and passes this branch.
  • A design note in docs/dev/backends.md states the named-functor rule.

GPU lane, NVIDIA A40, CUDA 12.8.1, Release AMPERE86, cuBLAS on:

main (63b5f5a) this branch
six tests named in #815 segfault (exit 139) 6 passed
poe test-gpu segfault 312 passed, 1 skipped (by design)
pytest -m gpu -p no:randomly tests segfault 312 passed, 1 skipped
plain radial runs, each in a fresh process all crash all run, fields finite

Closes #815

🤖 Generated with Claude Code

https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH

jtravs and others added 4 commits November 15, 2026 18:03
nvcc keeps an extended host-device lambda's copy, call and delete
functions in per-file statics set where the lambda is constructed, while
Kokkos's dispatch that copies it is an inline template shared between
files. device_dense_gemm's views are the same in three files, so a
Release link kept one file's dispatch, and a radial run that had built
the lambda only in another file called a null copy function in advance.
A named functor's copy is its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH
A static check over the extension's object files, run in the CUDA
device-compile lane: it fails when an extended lambda's helper statics
appear in more than one file, the condition behind #815, with Kokkos-FFT's
never-constructed normalisation lambda allowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH
codespell reads "statics" as a misspelling of "statistics"; the comments
now say static members or variables. The check sees only the kernels its
build's device target instantiates: the CUDA device-compile lane builds
for compute capability 9.0, which compiles no device_dense_gemm, so the
lambda behind #815 shows only in a build for 8.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH
On a Release CUDA build for compute capability 8.6 every radial run
crashed in its first step, envelope or field, with or without a collar or
statistics, with every stepper; other device targets compile no such
GEMM. The changelog entry says so, and the design page states the rule
that a shared header's kernel instantiated alike in several files is a
named functor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPwwkUUG9KSAwneHjgH
@jtravs
jtravs marked this pull request as ready for review November 15, 2026 18:32
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T18:34:36.792996Z cdd021e Draft marked ready
🔒 Security Review ✅ Completed 2026-11-15T18:36:05.819930Z cdd021e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 solprop | 🛠️ Build #35910210 | 📁 Comparing cdd021e against latest (63b5f5a)

  🔍 Preview build  

1 file changed
± dev/backends.html

@codecov

codecov Bot commented Nov 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jtravs
jtravs merged commit c6c6290 into main Nov 15, 2026
72 checks passed
@jtravs
jtravs deleted the fix/radial-device-815 branch November 15, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native radial route segfaults in session advance on a CUDA device

1 participant