fix: build Android Vulkan shaders with a coopmat-capable host glslc - #92
Merged
Merged
Conversation
GitHub is migrating the windows-11-arm label to the Windows 11 Arm64 with Visual Studio 2026 image between 2026-09-21 and 2026-09-30 (actions/runner-images#14602). That image has no VS 2022 instance, so the windows-arm64-full preset's hardcoded "Visual Studio 17 2022" generator fails at configure. Native release run 35976011426 lost both arm64 lanes (blas, vulkan) this way; the same lanes passed on the VS 2022 image in run 35850331498 on the same commit. Switch the preset to "Visual Studio 18 2026" and pin the arm64 jobs to windows-11-vs2026-arm so the image cannot flip mid-rollout. ClangCL, the ARM64 and x64 MSVC tools, and CMake 4.4 are all present on that image.
llama.cpp v0.5.0 (4ceb17191, #24406) compiles the new fa_decode_ph1/ph2 shaders as coopmat variants unconditionally, and ggml-vulkan.cpp references them without the GGML_VULKAN_COOPMAT_GLSLC_SUPPORT guard every other _cm1 shader has. NDK glslc (shaderc v2022.3, through r29) lacks GL_KHR_cooperative_matrix, so both Android Vulkan lanes of native release run 35976011426 failed in vulkan-shaders-gen. Reported upstream as ggml-org/llama.cpp#29373; no upstream source is patched here. tools/build.py now accepts ANDROID_VULKAN_GLSLC to replace the NDK glslc, and the release Android Vulkan lanes set it to the Ubuntu glslc package the Linux Vulkan lanes already use. That glslc adds exactly one detected shader feature over the NDK one (KHR coopmat), so Android now ships the same shader feature set as Linux. validate_wrapper gains android-vulkan-shaders, which builds the release arm64 ggml-vulkan target with that glslc on the pinned upstream and on v0.5.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two Android Vulkan failures in the scheduled v0.5.0 release run 35976011426 (
build-android (x86_64, x64, vulkan, …)and(arm64-v8a, arm64, vulkan, …)). The cause is an upstream v0.5.0 bug that our NDK glslc exposes. No upstream source is patched here.Cause
Upstream's CMake feature test correctly reports no coopmat. However, 4ceb17191 (ggml-org/llama.cpp#24406, Intel Xe flash-attention kernels, new in v0.5.0) added two shaders that bypass that check:
vulkan-shaders-gen.cpp:926-927buildsfa_decode_ph1/fa_decode_ph2withcoopmat=trueunconditionally.ggml-vulkan.cpp:3018-3019referencesfa_decode_ph*_cm1_datawithout theGGML_VULKAN_COOPMAT_GLSLC_SUPPORTguard that every other_cm1shader uses.Master (
6b790a9c29) is unchanged. Filed as ggml-org/llama.cpp#29373; no earlier report existed. Upstream CI never catches this: its Android jobs build without Vulkan, and its desktop Vulkan jobs use SDK or distro glslc builds that have the extension.Reproduced locally with the exact shader invocation. NDK r28
28.2.13676358and r2929.0.14206865both shipshaderc v2022.3and fail the same way. A newer NDK therefore wouldn't help, and only the glslc can change without patching upstream.Change
tools/build.py: newANDROID_VULKAN_GLSLCoverride for the Android Vulkan glslc. It fails closed if the path is not a file. Without it, the NDK glslc is used as before.native_release.ymlAndroid Vulkan lanes:apt-get install glslc(the same Ubuntu package the Linux Vulkan lanes install), logglslc --version, and export the override.build-androidnow runs onubuntu-24.04instead ofubuntu-latest. noble'sglslcis fixed at shaderc2023.8-1build1, so the Android shader feature set cannot widen silently whenubuntu-latestmoves to 26.04 in November.validate_wrapper.yml: newandroid-vulkan-shaderslane,upstream: [pinned, v0.5.0]. It builds the release arm64-v8aggml-vulkantarget throughbuild.py's real configure path with that glslc. It asserts that the override reached CMake, that coopmat was detected, and thatlibggml-vulkan.soexists. It is added to thewrapper-validationaggregate andtools/ci_scope.py's job set.LLAMADART_V050_QUALIFICATION_SHAis used for qualification only and moves no pin.tests/test_android_vulkan_glslc.pycovers override precedence, fail-closed, and the release-workflow wiring.docs/platform_backend_strategy.mddocuments the requirement.Runtime implication (maintainer decision)
Upstream turns shader features on from the glslc, so the choice of glslc changes what Android ships. Shader feature detection in the same release run:
With this change Android matches Linux, a one-feature step. This applies to every Android release built after this merges, including a wrapper-only rebuild of the current v0.4.1 pin, not only v0.5.0. It is the smallest glslc change that builds v0.5.0. The extra KHR-coopmat
mul_mm/flash-attention shaders run only on devices whose driver reportsVK_KHR_cooperative_matrix. For vendors other than Intel and AMD, upstream'sggml_vk_khr_cooperative_matrix_support()defaults to allowing it. Devices without the extension keep today's pipelines.GGML_VK_DISABLE_COOPMAT=1disables it at runtime. No physical Android device has run the coopmat path here.Validation
Hosted
android-vulkan-shaders (v0.5.0)log:HEAD is now at 7fe450e,shaderc 2023.8-1build1,-DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc,GL_KHR_cooperative_matrix supported by glslc(all other extensions not supported),Generate vulkan shaders for flash_attn_decode_phase_1.compsucceeded, andlibggml-vulkan.solinked. Thepinnedrow (v0.4.1, which does not have the bug) shows the new glslc regresses nothing. Release run 35976011426 is the negative control with the NDK glslc.validate_wrapper→android-vulkan-shaders (pinned)and(v0.5.0).python3 -m unittest discover -s tests: 159 tests OK.actionlint: the new lane is clean, andnative_release.ymlhas the same findings asmain(pre-existing PowerShell steps misread by shellcheck).Out of scope
tools/build.pycomment says so. The NDK glslc can then be used again if matching the old feature set matters.Independent review
A fresh reviewer re-derived the upstream cause from v0.5.0 source and both release logs. It confirmed the override reaches every Android configure path (arm64 primary, CPU variants, x86_64) and leaves OpenCL untouched, confirmed the feature table, and confirmed the lane cannot pass vacuously. It found no blockers. Its should-fix items were: the floating glslc (fixed by the
ubuntu-24.04pin), the stacked head (disclosed at the top), and a removal-plan note (added in code). A nit remains: a localbuild.py androidof v0.5.0 withoutANDROID_VULKAN_GLSLCstill fails with the upstream shader error. That is left as is, since the docs name the variable.