Skip to content

vulkan: keep the VK_TEST harness clean under synchronization validation - #1603

Merged
JustVugg merged 3 commits into
JustVugg:devfrom
crichalchemist:vulkan-syncval-harness
Sep 18, 2026
Merged

JustVugg merged 3 commits into
JustVugg:devfrom
crichalchemist:vulkan-syncval-harness

Conversation

@crichalchemist

Copy link
Copy Markdown
Contributor

Ran the Khronos validation layer's synchronization validation (validate_sync, SDK 1.4.357.1) over backend_vulkan.c on a Radeon Pro 580 through MoltenVK (macOS 13, x86_64), with duplicate suppression off so nothing past the first message ID was hidden. The production paths came back with zero hazards — matmul, fused gate_up, the expert group both sync and issue/take, the matmul pair, attn_qprep, absorb — and so did the qwen36 tier tests and four token-exact oracle runs through the tier on #1338. What the layer did report is confined to the VK_TEST harness, and that is what this PR fixes, so the harness can be a clean regression tool under the layer rather than 300 lines of noise.

c/backend_vulkan.c, harness throughput loops — bench_batched, bench_gu_batched and bench_experts_fair record 64 dispatches into one command buffer, each rewriting the same output, with a SHADER_WRITE → SHADER_READ barrier between them. That orders the reads, not the next write, so the layer reports a write-after-write hazard on every dispatch after the first: 315 in all (63 + 63 for the two matmul benches, 63 for the fused bench, 126 for the two fair-cycling benches). Harmless for a timing loop — nothing reads those outputs — but the fix is one bit: the destination mask now also carries SHADER_WRITE. The production dispatch loops (coli_vk_expert_group, the second-device path) bind distinct slices per expert and were not flagged.

c/backend_vulkan.c, coli_vk_shutdown — at vkDestroyDevice the layer lists 6 VkBuffer + 6 VkDeviceMemory the harness never freed: the two qprep scratch buffers G.qp1 / G.qp2 and the per-layer resident norm weights G.lnbuf[] / G.lnmem[] that coli_vk_attn_qprep uploads once per layer. None were destroyed at shutdown; they are now, next to the other scratch buffers. Eight more objects remain on dev — the rmsnorm pipeline set, its layout, shader module and two descriptor pools — and those are already fixed in #1338 (cb70b7a), so this PR leaves them to it rather than carry the same hunk twice; the two touch different lines of the function.

c/backend_vulkan.c, harness — only shaders/qmatmul.spv is tracked; the sibling .spv files come from make … VK=1. In a fresh checkout without them coli_vk_init succeeds, run_gate_up reports the missing shader and returns, but the two fused benches then bind G.pipe_gu == VK_NULL_HANDLE directly — the layer flags VUID-vkCmdBindPipeline-pipeline-parameter and MoltenVK segfaults on the next descriptor bind. Both benches now return −1 when the pipeline is absent, matching what run_gate_up already does.

docs/vulkan.md — one bullet under Correctness: how to enable the core and synchronization checks from the loader (nothing in the backend enables layers), the fact that a settings file is only read from a directory holding a file named exactly vk_layer_settings.txt, and a known layer defect: with submit-time synchronization validation on, this SDK segfaults inside the layer at vkDeviceWaitIdle during shutdown — after all work is done, but inside an atexit handler, so buffered stdout is lost with it. The workaround (disable that one setting, or run under a pty) is in the docs because it cost an hour to see through.

Verified on the Radeon Pro 580 via MoltenVK, harness built with clang -O3 -DVK_TEST against /opt/local:

run before (dev @ cc756a6) after
harness under core + sync validation, duplicates unlimited 315 SYNC-HAZARD-WRITE-AFTER-WRITE, 20 leaked objects 0 hazards, 8 leaked objects (the #1338 set)
harness, no layer 50 cases, same values 50 cases, same values to the printed digits
harness with qmatmul_gate_up.spv moved away SIGSEGV in vkCmdBindDescriptorSets completes, reports gate_up failed and the fused benches as −1
make colibri VK=1 — 0 warnings
make check — exit 0, Python suite OK

Two things this did not do. It ran only on MoltenVK, not RADV, and the VK_KHR_portability_subset error the layer also reports on MoltenVK is #1338's. And the harness still ends in FAIL on this box before and after: qprep fmt=1 S=11 reports a kv maxrel of 2.9e-3 against the 1e-3 gate, deterministic across runs and unchanged with MoltenVK's fast-math disabled, while the S=1 and S=2 cases pass. I have not chased it; if the RX 580 rows in the docs passed that case, it is MoltenVK-specific and worth a line in the docs, and if they did not it is a gate calibration question. Separately, the fresh device-local-block nondeterminism the docs describe on the RX 580 does not reproduce here: staged and mapped are byte-identical on all 50 cases over three runs, with the fill-once workaround and with it compiled out.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NJcbVYkasdfXg5DEsHdSCt

coli_vk_attn_qprep reserves G.qp1 and G.qp2 and uploads each layer's norm
weights once into G.lnbuf[layer]; coli_vk_shutdown destroyed every other
scratch buffer but not these. Under the Khronos validation layer the
harness ends with VUID-vkDestroyDevice-device-05137 listing exactly them:
six VkBuffer and six VkDeviceMemory for its four qprep cases plus the two
scratch slots. Destroy them with the rest.
The three throughput loops (bench_batched, bench_gu_batched,
bench_experts_fair) record 64 dispatches that each rewrite the same
output, separated by a SHADER_WRITE -> SHADER_READ barrier. That orders
the reads and not the next write, so synchronization validation reports
a write-after-write hazard on every dispatch after the first, 315 in a
run. Nothing consumes those outputs, but the barrier is wrong as written:
the destination mask now also carries SHADER_WRITE. The production
dispatch loops bind distinct slices per expert and were not flagged.

Only shaders/qmatmul.spv is tracked; without the sibling .spv files
run_gate_up reports the missing shader and returns, but the two fused
benches then bind G.pipe_gu == VK_NULL_HANDLE (the layer flags
VUID-vkCmdBindPipeline-pipeline-parameter, MoltenVK segfaults on the
next descriptor bind). They return -1 now, as run_gate_up already fails.
… defect

Nothing in the backend enables layers, so the loader has to: the env
vars for the core and synchronization checks, and the fact that a
settings file is only read from a directory holding a file named exactly
vk_layer_settings.txt. Also the known defect in SDK 1.4.357.1: with
submit-time synchronization validation on, the layer segfaults at
vkDeviceWaitIdle during shutdown, inside an atexit handler, taking the
unflushed stdout with it.
Copilot AI lite review requested due to automatic review settings September 18, 2026 01:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JustVugg
JustVugg merged commit e9a650b into JustVugg:dev Sep 18, 2026
28 checks passed
@crichalchemist
crichalchemist deleted the vulkan-syncval-harness branch September 23, 2026 19:24
crichalchemist added a commit to crichalchemist/colibri that referenced this pull request Sep 24, 2026
Conflicts, all where the backend shim meets dev's CUDA tier work:

- qt_shutdown: dev's JustVugg#1678 now drains open groups and frees the resident
  experts and G.slot. That supersedes this branch's release loop, which ran
  after free(G.slot) and would have dereferenced NULL. Dev's release is kept
  and routed through be_take/be_free; G.ybuf is now freed there and on the
  qt_init failure path (JustVugg#1683's goto chain).
- qt_take: dev's two-phase drain (no partial contribution when a device
  fails) with be_take in place of coli_cuda_expert_group_take.
- qt_dense_matmul_batch / qt_dnproj_matmul_batch (JustVugg#1674, JustVugg#1677):
  be_trunk_matmul takes the row count S; Vulkan still refuses.
- Clean merges that bypassed the shim: the uploader's partial-upload free
  (JustVugg#1680's reclaim) and dense_free_all now use be_free.
- The "[CUDA] mode: routed experts" literal (JustVugg#1533) is CUDA-only, so a
  Vulkan binary is not read as a CUDA build by doctor.py.
- qwen36.c: the mixed-layout refusal also names COLI_VULKAN=1.
- Tests: dev's shutdown checks read !G.slot, so the abandoned-swap pins
  keep only the upload counter; the fake-Vulkan gate now pins that
  shutdown frees every resident expert through coli_vk_tensor_free.
- Makefile: dev's prerequisites, with $(VK_OBJ) on every target that
  links the tier source, including dev's new qwen36 test targets.
- backend_vulkan.c: both shutdown blocks (staging/upload handles here,
  JustVugg#1603's qprep and norm buffers from dev).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants