vulkan: keep the VK_TEST harness clean under synchronization validation - #1603
Merged
Merged
Conversation
coli_vk_attn_qprep reserves G.qp1 and G.qp2 and uploads each layer's norm weights once into G.lnbuf[layer]; coli_vk_shutdown destroyed every other scratch buffer but not these. Under the Khronos validation layer the harness ends with VUID-vkDestroyDevice-device-05137 listing exactly them: six VkBuffer and six VkDeviceMemory for its four qprep cases plus the two scratch slots. Destroy them with the rest.
The three throughput loops (bench_batched, bench_gu_batched, bench_experts_fair) record 64 dispatches that each rewrite the same output, separated by a SHADER_WRITE -> SHADER_READ barrier. That orders the reads and not the next write, so synchronization validation reports a write-after-write hazard on every dispatch after the first, 315 in a run. Nothing consumes those outputs, but the barrier is wrong as written: the destination mask now also carries SHADER_WRITE. The production dispatch loops bind distinct slices per expert and were not flagged. Only shaders/qmatmul.spv is tracked; without the sibling .spv files run_gate_up reports the missing shader and returns, but the two fused benches then bind G.pipe_gu == VK_NULL_HANDLE (the layer flags VUID-vkCmdBindPipeline-pipeline-parameter, MoltenVK segfaults on the next descriptor bind). They return -1 now, as run_gate_up already fails.
… defect Nothing in the backend enables layers, so the loader has to: the env vars for the core and synchronization checks, and the fact that a settings file is only read from a directory holding a file named exactly vk_layer_settings.txt. Also the known defect in SDK 1.4.357.1: with submit-time synchronization validation on, the layer segfaults at vkDeviceWaitIdle during shutdown, inside an atexit handler, taking the unflushed stdout with it.
crichalchemist
added a commit
to crichalchemist/colibri
that referenced
this pull request
Sep 24, 2026
Conflicts, all where the backend shim meets dev's CUDA tier work: - qt_shutdown: dev's JustVugg#1678 now drains open groups and frees the resident experts and G.slot. That supersedes this branch's release loop, which ran after free(G.slot) and would have dereferenced NULL. Dev's release is kept and routed through be_take/be_free; G.ybuf is now freed there and on the qt_init failure path (JustVugg#1683's goto chain). - qt_take: dev's two-phase drain (no partial contribution when a device fails) with be_take in place of coli_cuda_expert_group_take. - qt_dense_matmul_batch / qt_dnproj_matmul_batch (JustVugg#1674, JustVugg#1677): be_trunk_matmul takes the row count S; Vulkan still refuses. - Clean merges that bypassed the shim: the uploader's partial-upload free (JustVugg#1680's reclaim) and dense_free_all now use be_free. - The "[CUDA] mode: routed experts" literal (JustVugg#1533) is CUDA-only, so a Vulkan binary is not read as a CUDA build by doctor.py. - qwen36.c: the mixed-layout refusal also names COLI_VULKAN=1. - Tests: dev's shutdown checks read !G.slot, so the abandoned-swap pins keep only the upload counter; the fake-Vulkan gate now pins that shutdown frees every resident expert through coli_vk_tensor_free. - Makefile: dev's prerequisites, with $(VK_OBJ) on every target that links the tier source, including dev's new qwen36 test targets. - backend_vulkan.c: both shutdown blocks (staging/upload handles here, JustVugg#1603's qprep and norm buffers from dev).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ran the Khronos validation layer's synchronization validation (
validate_sync, SDK 1.4.357.1) overbackend_vulkan.con a Radeon Pro 580 through MoltenVK (macOS 13, x86_64), with duplicate suppression off so nothing past the first message ID was hidden. The production paths came back with zero hazards — matmul, fused gate_up, the expert group both sync and issue/take, the matmul pair, attn_qprep, absorb — and so did the qwen36 tier tests and four token-exact oracle runs through the tier on #1338. What the layer did report is confined to theVK_TESTharness, and that is what this PR fixes, so the harness can be a clean regression tool under the layer rather than 300 lines of noise.c/backend_vulkan.c, harness throughput loops —bench_batched,bench_gu_batchedandbench_experts_fairrecord 64 dispatches into one command buffer, each rewriting the same output, with aSHADER_WRITE → SHADER_READbarrier between them. That orders the reads, not the next write, so the layer reports a write-after-write hazard on every dispatch after the first: 315 in all (63 + 63 for the two matmul benches, 63 for the fused bench, 126 for the two fair-cycling benches). Harmless for a timing loop — nothing reads those outputs — but the fix is one bit: the destination mask now also carriesSHADER_WRITE. The production dispatch loops (coli_vk_expert_group, the second-device path) bind distinct slices per expert and were not flagged.c/backend_vulkan.c,coli_vk_shutdown— atvkDestroyDevicethe layer lists 6VkBuffer+ 6VkDeviceMemorythe harness never freed: the two qprep scratch buffersG.qp1/G.qp2and the per-layer resident norm weightsG.lnbuf[]/G.lnmem[]thatcoli_vk_attn_qprepuploads once per layer. None were destroyed at shutdown; they are now, next to the other scratch buffers. Eight more objects remain ondev— the rmsnorm pipeline set, its layout, shader module and two descriptor pools — and those are already fixed in #1338 (cb70b7a), so this PR leaves them to it rather than carry the same hunk twice; the two touch different lines of the function.c/backend_vulkan.c, harness — onlyshaders/qmatmul.spvis tracked; the sibling.spvfiles come frommake … VK=1. In a fresh checkout without themcoli_vk_initsucceeds,run_gate_upreports the missing shader and returns, but the two fused benches then bindG.pipe_gu == VK_NULL_HANDLEdirectly — the layer flagsVUID-vkCmdBindPipeline-pipeline-parameterand MoltenVK segfaults on the next descriptor bind. Both benches now return −1 when the pipeline is absent, matching whatrun_gate_upalready does.docs/vulkan.md— one bullet under Correctness: how to enable the core and synchronization checks from the loader (nothing in the backend enables layers), the fact that a settings file is only read from a directory holding a file named exactlyvk_layer_settings.txt, and a known layer defect: with submit-time synchronization validation on, this SDK segfaults inside the layer atvkDeviceWaitIdleduring shutdown — after all work is done, but inside anatexithandler, so buffered stdout is lost with it. The workaround (disable that one setting, or run under a pty) is in the docs because it cost an hour to see through.Verified on the Radeon Pro 580 via MoltenVK, harness built with
clang -O3 -DVK_TESTagainst/opt/local:dev@ cc756a6)SYNC-HAZARD-WRITE-AFTER-WRITE, 20 leaked objectsqmatmul_gate_up.spvmoved awayvkCmdBindDescriptorSetsgate_up failedand the fused benches as −1make colibri VK=1make checkTwo things this did not do. It ran only on MoltenVK, not RADV, and the
VK_KHR_portability_subseterror the layer also reports on MoltenVK is #1338's. And the harness still ends inFAILon this box before and after:qprep fmt=1 S=11reports a kv maxrel of 2.9e-3 against the 1e-3 gate, deterministic across runs and unchanged with MoltenVK's fast-math disabled, while the S=1 and S=2 cases pass. I have not chased it; if the RX 580 rows in the docs passed that case, it is MoltenVK-specific and worth a line in the docs, and if they did not it is a gate calibration question. Separately, the fresh device-local-block nondeterminism the docs describe on the RX 580 does not reproduce here: staged and mapped are byte-identical on all 50 cases over three runs, with the fill-once workaround and with it compiled out.🤖 Generated with Claude Code
https://claude.ai/code/session_01NJcbVYkasdfXg5DEsHdSCt