Skip to content

fix: build Android Vulkan shaders with a coopmat-capable host glslc - #92

Merged
leehack merged 7 commits into
mainfrom
fix/android-vulkan-host-glslc
Sep 24, 2026
Merged

leehack merged 7 commits into
mainfrom
fix/android-vulkan-host-glslc

Conversation

@leehack

@leehack leehack commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

#91 is merged; this branch now targets main and contains main up to it.

Fixes the two Android Vulkan failures in the scheduled v0.5.0 release run 35976011426 (build-android (x86_64, x64, vulkan, …) and (arm64-v8a, arm64, vulkan, …)). The cause is an upstream v0.5.0 bug that our NDK glslc exposes. No upstream source is patched here.

Cause

-- GL_KHR_cooperative_matrix not supported by glslc
...
glslc -fshader-stage=compute --target-env=vulkan1.2 …/flash_attn_decode_phase_1.comp -o …/fa_decode_ph1_cm1.spv
flash_attn_decode_phase_1.comp:115: error: 'coopmat' : undeclared identifier
flash_attn_decode_phase_2.comp:118: error: 'coopmat' : undeclared identifier

Upstream's CMake feature test correctly reports no coopmat. However, 4ceb17191 (ggml-org/llama.cpp#24406, Intel Xe flash-attention kernels, new in v0.5.0) added two shaders that bypass that check:

  • vulkan-shaders-gen.cpp:926-927 builds fa_decode_ph1/fa_decode_ph2 with coopmat=true unconditionally.
  • ggml-vulkan.cpp:3018-3019 references fa_decode_ph*_cm1_data without the GGML_VULKAN_COOPMAT_GLSLC_SUPPORT guard that every other _cm1 shader uses.

Master (6b790a9c29) is unchanged. Filed as ggml-org/llama.cpp#29373; no earlier report existed. Upstream CI never catches this: its Android jobs build without Vulkan, and its desktop Vulkan jobs use SDK or distro glslc builds that have the extension.

Reproduced locally with the exact shader invocation. NDK r28 28.2.13676358 and r29 29.0.14206865 both ship shaderc v2022.3 and fail the same way. A newer NDK therefore wouldn't help, and only the glslc can change without patching upstream.

Change

  • tools/build.py: new ANDROID_VULKAN_GLSLC override for the Android Vulkan glslc. It fails closed if the path is not a file. Without it, the NDK glslc is used as before.
  • native_release.yml Android Vulkan lanes: apt-get install glslc (the same Ubuntu package the Linux Vulkan lanes install), log glslc --version, and export the override. build-android now runs on ubuntu-24.04 instead of ubuntu-latest. noble's glslc is fixed at shaderc 2023.8-1build1, so the Android shader feature set cannot widen silently when ubuntu-latest moves to 26.04 in November.
  • validate_wrapper.yml: new android-vulkan-shaders lane, upstream: [pinned, v0.5.0]. It builds the release arm64-v8a ggml-vulkan target through build.py's real configure path with that glslc. It asserts that the override reached CMake, that coopmat was detected, and that libggml-vulkan.so exists. It is added to the wrapper-validation aggregate and tools/ci_scope.py's job set. LLAMADART_V050_QUALIFICATION_SHA is used for qualification only and moves no pin.
  • tests/test_android_vulkan_glslc.py covers override precedence, fail-closed, and the release-workflow wiring.
  • docs/platform_backend_strategy.md documents the requirement.

Runtime implication (maintainer decision)

Upstream turns shader features on from the glslc, so the choice of glslc changes what Android ships. Shader feature detection in the same release run:

lane coopmat (KHR) integer dot bf16 coopmat2 e2m1/e4m3
Android, NDK glslc (before) no no no no no
Linux x64/arm64, Ubuntu glslc yes no no no no
Windows x64, LunarG SDK yes yes yes yes no

With this change Android matches Linux, a one-feature step. This applies to every Android release built after this merges, including a wrapper-only rebuild of the current v0.4.1 pin, not only v0.5.0. It is the smallest glslc change that builds v0.5.0. The extra KHR-coopmat mul_mm/flash-attention shaders run only on devices whose driver reports VK_KHR_cooperative_matrix. For vendors other than Intel and AMD, upstream's ggml_vk_khr_cooperative_matrix_support() defaults to allowing it. Devices without the extension keep today's pipelines. GGML_VK_DISABLE_COOPMAT=1 disables it at runtime. No physical Android device has run the coopmat path here.

Validation

  • Hosted android-vulkan-shaders (v0.5.0) log: HEAD is now at 7fe450e, shaderc 2023.8-1build1, -DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc, GL_KHR_cooperative_matrix supported by glslc (all other extensions not supported), Generate vulkan shaders for flash_attn_decode_phase_1.comp succeeded, and libggml-vulkan.so linked. The pinned row (v0.4.1, which does not have the bug) shows the new glslc regresses nothing. Release run 35976011426 is the negative control with the NDK glslc.

  • validate_wrapper → android-vulkan-shaders (pinned) and (v0.5.0).

  • python3 -m unittest discover -s tests: 159 tests OK.

  • actionlint: the new lane is clean, and native_release.yml has the same findings as main (pre-existing PowerShell steps misread by shellcheck).

Out of scope

  • The Android arm64 OpenCL failure in the same run is the CPU ISA fingerprint gate, handled in a separate PR.
  • Remove the override once upstream gates the shaders (#29373) and a pinned release contains the fix. The tools/build.py comment says so. The NDK glslc can then be used again if matching the old feature set matters.

Independent review

A fresh reviewer re-derived the upstream cause from v0.5.0 source and both release logs. It confirmed the override reaches every Android configure path (arm64 primary, CPU variants, x86_64) and leaves OpenCL untouched, confirmed the feature table, and confirmed the lane cannot pass vacuously. It found no blockers. Its should-fix items were: the floating glslc (fixed by the ubuntu-24.04 pin), the stacked head (disclosed at the top), and a removal-plan note (added in code). A nit remains: a local build.py android of v0.5.0 without ANDROID_VULKAN_GLSLC still fails with the upstream shader error. That is left as is, since the docs name the variable.

GitHub is migrating the windows-11-arm label to the Windows 11 Arm64 with
Visual Studio 2026 image between 2026-09-21 and 2026-09-30
(actions/runner-images#14602). That image has no VS 2022 instance, so the
windows-arm64-full preset's hardcoded "Visual Studio 17 2022" generator
fails at configure. Native release run 35976011426 lost both arm64 lanes
(blas, vulkan) this way; the same lanes passed on the VS 2022 image in run
35850331498 on the same commit.

Switch the preset to "Visual Studio 18 2026" and pin the arm64 jobs to
windows-11-vs2026-arm so the image cannot flip mid-rollout. ClangCL, the
ARM64 and x64 MSVC tools, and CMake 4.4 are all present on that image.
llama.cpp v0.5.0 (4ceb17191, #24406) compiles the new fa_decode_ph1/ph2
shaders as coopmat variants unconditionally, and ggml-vulkan.cpp references
them without the GGML_VULKAN_COOPMAT_GLSLC_SUPPORT guard every other _cm1
shader has. NDK glslc (shaderc v2022.3, through r29) lacks
GL_KHR_cooperative_matrix, so both Android Vulkan lanes of native release run
35976011426 failed in vulkan-shaders-gen. Reported upstream as
ggml-org/llama.cpp#29373; no upstream source is patched here.

tools/build.py now accepts ANDROID_VULKAN_GLSLC to replace the NDK glslc,
and the release Android Vulkan lanes set it to the Ubuntu glslc package the
Linux Vulkan lanes already use. That glslc adds exactly one detected shader
feature over the NDK one (KHR coopmat), so Android now ships the same shader
feature set as Linux.

validate_wrapper gains android-vulkan-shaders, which builds the release
arm64 ggml-vulkan target with that glslc on the pinned upstream and on
v0.5.0.
@leehack
leehack changed the base branch from main to fix/windows-arm64-vs2026 September 24, 2026 12:17
@leehack
leehack changed the base branch from fix/windows-arm64-vs2026 to main September 24, 2026 12:47
@leehack
leehack merged commit b21fd83 into main Sep 24, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant