CI: vet PRs and master pushes with the routing acceptance gates - #37
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
apt has no cuda-nvcc-12-4 / cuda-cudart-dev-12-4 / cuda-nvrtc-dev-12-4 packages under this CUDA 12.4.1 network repo, so pinning sub-packages makes the install step fail outright in both jobs. Drop sub-packages for a full toolkit install (slower, but reliable per the CI bring-up triage table). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
Even the full cuda-12-4 metapackage 404s on this runner's NVIDIA apt mirror (E: Unable to locate package cuda-12-4). Per the triage table's next step, drop Jimver/cuda-toolkit entirely and install the distro's nvidia-cuda-toolkit via apt, which puts nvcc on PATH without extra env vars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
Verified directly against NVIDIA's ubuntu2404 apt repo (curl the Packages index): it only carries cuda-toolkit-12-5 through 13-3, never 12-4 — CUDA 12.4 targeted jammy/22.04, not noble. That's why the full cuda-12-4 metapackage 404'd. The apt fallback (nvidia-cuda-toolkit) does install cleanly, but Ubuntu's package pins CUDA 12.0, whose cuSPARSE lacks cusparseSpSV_updateMatrix/cusparseSpSVUpdate_t (src/sparse/cusparse.rs), added in cuSPARSE 12.4. Go back to Jimver/cuda-toolkit (full install, no sub-packages) requesting 12.6.3, the nearest version noble actually offers with that API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
Requesting the full toolkit (no sub-packages) at 12.6.3 pulls in nvidia-driver-560-open / nvidia-dkms-560-open, and their DKMS kernel-module postinst fails on the GitHub-hosted runner (no matching kernel headers/GPU): "Bad return status for module build on kernel 6.17.0-1022-azure". Restore sub-packages (nvcc, cudart-dev, nvrtc-dev — compile-only, no driver) now that 12.6.3 is a version that actually exists in NVIDIA's noble repo, combining both prior fixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
The blanket `/fixtures/` .gitignore rule silently swallowed
fixtures/sandbox/, even though src/sandbox.rs::load_embedded() embeds
its 5 CSVs at compile time via include_str! (unconditionally compiled,
not feature-gated) and tests/ddr_sandbox_match.rs's own doc comment
already declares the directory "committed" ("a missing file means a
broken checkout, not a skippable test"). This is not a CI artifact —
any fresh clone would fail to build the lib the same way; CI is just
the first thing to actually try a clean checkout. The CI design spec
had assumed this was already tracked.
Carve fixtures/sandbox/ out of the ignore rule and commit its 6 files
(44KB): the 5 CSVs loaded by include_str!, plus ddr_discharge_rapid2.csv
(DDR's reference output, read at runtime by both
compare_ddr_sandbox.rs and ddr_sandbox_match.rs). The much larger ad
hoc fixtures/gradcheck, fixtures/sp4, fixtures/sp5 dirs are untouched
(sp4/sp5 were already tracked from before the ignore rule existed;
gradcheck stays ignored — nothing reads it at compile time).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
…s_mask
tests/training_step_layer_{b,c,d}.rs (4 struct literals total) never
picked up the impervious_mask: Option<Tensor<I, 1>> field that
src/routing/mmc.rs's SpatialParameters carries — every other call site
in the codebase (src/sandbox.rs, src/training/forward.rs,
tests/mmc.rs, tests/common.rs, tests/leakance_off_parity.rs, etc., 14+
in total) already sets it to None. Genuine compile break, not a CI
artifact: `cargo check --tests` fails identically on any checkout.
Verified locally: `cargo check --tests --features fixtures` and
`cargo check --release --example compare_ddr_sandbox --test
juniata_acceptance` both compile clean after this fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
test job failed mid-build with "No space left on device" writing a fingerprint for one of the example binaries (cargo test compiles every example, and the burn/cubecl/netcdf-static tree is heavy). Free the preinstalled Android/.NET/GHC SDKs this repo never uses as the first step of both jobs, per the triage table's disk-exhaustion fix (originally scoped to acceptance only, but applying to both since test hit it first here). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
Ten test targets failed identically: "Unable to dynamically load the
'cuda' shared library" — a panic inside cudarc's own dlopen, not a
Result our code's existing guards (src/cli/system.rs::probe(),
tests/cusparse_ptr_spike.rs's catch_unwind) can catch, because our
sub-packages (nvcc, cudart-dev, nvrtc-dev) never install libcuda.so
itself — only the full CUDA metapackage's driver deps do, and those
pull in the DKMS kernel-module build that already failed on this
runner. libnvidia-compute-560 is the driver's userspace library with
no DKMS dependency (Depends: libc6/libgcc-s1/libnvidia-decode-560
only, verified against NVIDIA's repo). Installing it makes libcuda.so
present, so cuInit() fails gracefully with "no device" instead of
dlopen panicking — the graceful path every affected test already
expects (this also covers the 4 spike/gradcheck tests whose panic
propagated asynchronously off a worker thread as a channel RecvError,
past their own catch_unwind guards).
Separately, tests/sparse_gradcheck.rs panicked reading a fixture
(fixtures/gradcheck/a_values.csv) that was correctly left out of the
prior fixtures/sandbox carve-out — nothing embeds it at compile time,
and it's produced by scripts/dump_solver_gradcheck.py against DDR,
same as the leakance/subdivision gradcheck fixtures. Add the
repo-standard skip guard (tests/training_verification.rs:79-100
model): Path::exists, eprintln!("skipping: ..."), return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
…uard Two more genuine, non-CI-specific bugs surfaced now that the dlopen crash is fixed (previous commit) and real cuInit()/cuInit-failure errors surface instead: 1. tests/cli_run_preflight.rs::run_train_requires_gpu_when_none_probed expects "requires a CUDA GPU" from the GPU pre-flight check, but its inline ddrs.yaml fixture has no experiment: block, so config validation now fails first with "experiment: missing" (src/cli/plan.rs:391-394, compute_summary requires it for training mode) — a stale fixture, not a CI artifact. Add the minimal experiment: block (matches tests/disagg_enabled.rs's fixture). Applied to both tests in the file since they share the same fixture string. 2. tests/cusparse_ptr_spike.rs (3 fns) and tests/sparse_cusparse_v5.rs (3 fns) gate on `catch_unwind(|| Default::default())`, but burn_cuda::Cuda's Default::default() just spawns a background cubecl worker thread and returns — the actual "no CUDA device" DriverError panics asynchronously inside that worker (cubecl-cuda/runtime.rs:53), which catch_unwind on the calling thread never sees; it only surfaces later as a channel RecvError once real work is submitted. Replace with the same ddrs::cli::system::probe()-based check already used successfully in tests/cli_run_preflight.rs, which detects "no GPU" via a separate, synchronous cudarc path with no worker thread involved. Verified locally (this machine has a real GPU): all 4 touched test files still exercise and pass the real CUDA branch — the probe() gate doesn't regress GPU-present behavior, it just makes the no-GPU skip actually work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
Owner
Author
|
Both checks are green (cold, uncached run — run 33749635266):
This is after 8 fix-iteration commits on the CUDA-toolkit/fixture-tracking/compile/test-guard fronts (see commit history on this branch). Warm-cache validation run to follow in a separate comment. |
Owner
Author
|
Warm-cache validation dispatch: 33755052570 — both jobs green again.
Caveat: this did not actually exercise a Practical upshot: this specific validation method (manual dispatch) can't demonstrate the warm path — the acceptance job's ~20% delta (51m2s → 40m44s) is most likely runner/network variance, not caching. The cache will be exercised for its intended use case — a subsequent commit pushed to this PR — since that reuses the |
Run 33755052570's step timings show the Juniata acceptance step spent 30m12s in the build (a single 'Compiling ddrs' line followed by the thin-LTO link of the release test binary on 2 cores) while the test itself ran in 26.72s — all 30 training epochs took ~13s. Shortening the training (fewer epochs/mini-batches) would save seconds; the link is the cost. CARGO_PROFILE_RELEASE_LTO=false (cargo's default crate-local thin LTO) keeps opt-level 3 and debug_assertions off, so juniata_acceptance still runs with its full metric floors, and FP semantics are unchanged. Env override only — Cargo.toml and local builds keep lto = "thin". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements docs/superpowers/specs/2026-09-03-ci-vetting-design.md:
test(debug suite + KAN fixtures) and
acceptance(release sandbox parity +Juniata metric floors) on every PR and master push. This PR is the bring-up
vehicle — CUDA-install iteration commits land here until both jobs are green.
🤖 Generated with Claude Code
https://claude.ai/code/session_01RHusoRV4Th4tio8bh2oPca