Skip to content

fix(v4): rebuild backend_cuda_dsv4.o when the nvcc command changes - #1731

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
bokiko:fix/v4-cuda-arch-stamp
Sep 24, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
bokiko:fix/v4-cuda-arch-stamp

Conversation

@bokiko

@bokiko bokiko commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

backend_cuda_dsv4.o is named after its source, but what it contains comes from
$(NVCC) $(V4_NVCCFLAGS): -arch=$(CUDA_ARCH) (or the -gencode preset), -DCOLI_DSV4_NO_TC,
and the DeepGEMM defines. None of that is a prerequisite, so make goes on timestamps alone:

$ make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=sm_86
$ make -f Makefile.deepseek-v4 deepseek-v4 CUDA=1 CUDA_ARCH=sm_80
$ # no output at all, exit 0

The engine asked for sm_80 links the sm_86 object. This is the CUDA_ARCH / backend_cuda_dsv4.o
case #1707 listed as out of scope. Follows #1707 / #1702.

Measured on dev at a2e578c, the merge commit of #1707 — Ubuntu 24.04 container, real gcc 13.3,
real nvcc 13.0, cuobjdump, GNU Make 4.3, no GPU, no network, one tree throughout:

tree step CUDA_ARCH nvcc calls for the object object arch (cuobjdump --list-elf) what the build printed
a2e578c 1 sm_86 1 sm_86 the nvcc and link lines
a2e578c 2 sm_80 0 sm_86 nothing at all, exit 0
a2e578c 3 sm_80 0 sm_86 nothing at all, exit 0
+ this patch 1 sm_86 1 sm_86 the nvcc and link lines
+ this patch 2 sm_80 1 sm_80 deepseek-v4: CUDA build flags changed, rebuilding backend_cuda_dsv4.o, then the nvcc and link lines
+ this patch 3 sm_80 0 sm_80 nothing at all, exit 0 — the guard holds

A genuinely up-to-date build prints nothing at all, in both trees, with and without -j: byte
count 0, exit 0. So the a2e578c rows are the whole failure mode — the arch changes, make exits
successfully, and the engine silently links the object built for the previous target.

The same three failures reproduce on arm64 macOS with a stubbed nvcc and no toolkit at all.

make -n and deepseek-v4-clean write nothing, and the clean target removes the stamp whether or
not CUDA=1 is passed. Verified with a whole-tree manifest (path, mtime, sha256) taken before and
after each call, on GNU Make 4.3 (container) and GNU Make 3.81 (arm64 macOS, stubbed toolchain):
make -n deepseek-v4 on a tree with no stamp leaves the stamp absent, make -n deepseek-v4-clean
leaves the tree byte-identical, and a real clean removes the stamp. The 3.81 run and its per-step
logs are kept with the rest of the evidence. This is a consequence of the stamp being written by a
recipe rather than at parse time.

The change

  • c/Makefile.deepseek-v4: record $(NVCC) $(V4_NVCCFLAGS) in deepseek_v4.cudaflags and rewrite
    it only when it changes — the same recipe-guarded stamp fix(v4): rebuild the unit objects when the build flags change #1707 added for $(CC) $(CFLAGS), kept in
    its own file so that an arch change does not recompile the C units. backend_cuda_dsv4.o depends
    on it, and deepseek-v4-clean removes it.
  • .gitignore, c/tools/clean.py: the new stamp, like the other build artifacts.
  • c/tests/test_deepseek_v4_cuda_arch_flags.py: same shape as fix(v4): rebuild the unit objects when the build flags change #1707's
    test_deepseek_v4_build_flags.py — stand-in nvcc, sources copied to a temp dir with pinned
    timestamps, no toolchain and no GPU required. Seven tests: CUDA_ARCH, the -gencode preset,
    NO_TC, the "unchanged command does not rebuild" guard, make -n writing nothing, clean
    writing nothing, and clean removing the stamp. Against dev three of the seven fail (arch,
    preset, NO_TC); with the patch all seven pass, on GNU Make 3.81 and 4.3.

The parent c/Makefile keeps this kind of state for exactly this reason, CUDA_ARCH included, in
.build-config (#306); this is the standalone build's equivalent, using the recipe-guarded form
rather than .build-config's parse-time $(file ...) write.

Not covered by these runs: make -C c cuda-test on a real GPU (no GPU is free here), Windows, and
macOS CUDA builds. make check on the patched tree is green on both hosts (Ubuntu 24.04 container
and arm64 macOS). #1707's objects and stamp are untouched, so the two changes compose in either
order.

No runtime behaviour changes: the object is only rebuilt when the command that produces it moved.
Rollback: drop the patch, delete c/deepseek_v4.cudaflags.

The object is named after its source, but its contents come from
$(NVCC) $(V4_NVCCFLAGS): -arch=$(CUDA_ARCH) (or the -gencode preset),
-DCOLI_DSV4_NO_TC and the DeepGEMM defines. None of that is a
prerequisite, so make decides on timestamps alone: asking for
CUDA_ARCH=sm_80 after an sm_86 build recompiles nothing, exits 0, and the
engine links the sm_86 object.

Record the nvcc command in deepseek_v4.cudaflags and make
backend_cuda_dsv4.o and deepseek-v4-clean depend on it. Same
recipe-guarded stamp JustVugg#1707 added for $(CC) $(CFLAGS) (JustVugg#1702), kept in its
own file so an arch change does not recompile the C units. The stamp is
written by a recipe, so make -n writes nothing and clean removes it.

Regression test mirrors JustVugg#1707's test_deepseek_v4_build_flags.py: stand-in
nvcc, no toolchain, no GPU. Three of seven cases fail on dev; all seven
pass with the fix, on GNU Make 3.81 and 4.3.

Follows JustVugg#1707 / JustVugg#1702.
@JustVugg
JustVugg merged commit 2516472 into JustVugg:dev Sep 24, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants