From 869b8071e7b48ecc7c00aef298bee5f2caafd310 Mon Sep 17 00:00:00 2001 From: Local Dev Date: Fri, 18 Sep 2026 17:49:35 +1000 Subject: [PATCH 1/2] =?UTF-8?q?compat:=20905=20=E2=80=94=20restore=20RESIZ?= =?UTF-8?q?E=5FALGO=5FBILINEAR=20for=20gemma4=20and=20qwen3vl?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit b10864 switched these projectors from bilinear to bicubic in the same hunk that raised gemma4's ceiling from (40,280) to (70,1120), and separately deleted the naive resamplers so every algo now routes through the Pillow-parity path. Measured on gfx1151, gemma4:31b-it-q4_K_M, budget pinned with run_budget_sweep.sh, name_bbox_mean_iou: at 280 and 560 the two builds agree (0.612/0.691, 0.713/0.714), at 1120 it goes 0.728 -> 0.000. prompt_eval_count is identical between builds on every test, so the grid and token count are unchanged and only the pixels differ. 1120 is the only rung that enlarges a 1920x1080 source. Changes the resampler ONLY, leaving (70,1120) alone, so a re-run at 1120 isolates it. Verified: the full series 001/002/004/005/801/903/905 applies to a clean b10864 with plain git apply. Co-Authored-By: Claude Opus 5 --- .../905-restore-bilinear-vision-resize.patch | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 llama/compat/905-restore-bilinear-vision-resize.patch diff --git a/llama/compat/905-restore-bilinear-vision-resize.patch b/llama/compat/905-restore-bilinear-vision-resize.patch new file mode 100644 index 00000000000..b269f4a436e --- /dev/null +++ b/llama/compat/905-restore-bilinear-vision-resize.patch @@ -0,0 +1,50 @@ +MAXUSAI 905: restore RESIZE_ALGO_BILINEAR for gemma4 and qwen3vl. + +b10864 changed the vision resampler for these projectors in the same hunk that +raised gemma4's token ceiling from (40,280) to (70,1120): + + - hparams.image_resize_algo = RESIZE_ALGO_BILINEAR; + + hparams.image_resize_algo = RESIZE_ALGO_BICUBIC; + - hparams.set_limit_image_tokens(40, 280); + + hparams.set_limit_image_tokens(70, 1120); + +Separately, the naive resamplers (resize_bilinear, resize_bicubic) were deleted +and every algo now routes through the Pillow-parity path, so BICUBIC also means +something different than it did: antialiased, with half-pixel centres. + +MEASURED ON gfx1151 (Radeon 8060S), gemma4:31b-it-q4_K_M, budget pinned via +run_budget_sweep.sh, name_bbox_mean_iou (vertical localization): + + budget 0.32.1/b9888 0.34.1/b10864 + 280 0.612 0.691 + 560 0.713 0.714 + 1120 0.728 0.000 <-- total loss + +The grid is unchanged: prompt_eval_count is identical between builds on every +test (1684 on scene_single), so the token count and tokenization are the same +and only the PIXELS differ. The failure is confined to the 1120 rung, which is +the only rung that enlarges a 1920x1080 source (1120 * 48^2 = 2580480 px), and +it is the vertical axis that breaks -- the same axis and the same rung as the +coordinate defect ADR 0007 mitigated and ADR 0008 declared fixed by patch 004. +diff --git a/tools/mtmd/clip.cpp b/tools/mtmd/clip.cpp +index b4f07c4..0db1971 100644 +--- a/tools/mtmd/clip.cpp ++++ b/tools/mtmd/clip.cpp +@@ -1684,7 +1684,7 @@ struct clip_model_loader { + { + hparams.rope_theta = 100.0f; + hparams.n_merge = 3; // pooling_kernel_size +- hparams.image_resize_algo = RESIZE_ALGO_BICUBIC; ++ hparams.image_resize_algo = RESIZE_ALGO_BILINEAR; + get_u32(KEY_PROJ_SCALE_FACTOR, hparams.n_merge, false); + if (model.proj_type == PROJECTOR_TYPE_GEMMA4UV) { + // for "unified" variant, we directly use a bigger patch size, because the "token merging" is done directly on conv layer +@@ -1720,7 +1720,7 @@ struct clip_model_loader { + case PROJECTOR_TYPE_QWEN3VL: + { + hparams.n_merge = 2; // default value for Qwen 2 and 2.5 +- hparams.image_resize_algo = RESIZE_ALGO_BICUBIC; ++ hparams.image_resize_algo = RESIZE_ALGO_BILINEAR; + get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false); + get_u32(KEY_WIN_ATTN_PATTERN, hparams.n_wa_pattern, model.proj_type == PROJECTOR_TYPE_QWEN25VL); // only 2.5 requires it + // ref: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/blob/main/preprocessor_config.json From 968762ca8a5c06f7e92850d7c394ee370231ee72 Mon Sep 17 00:00:00 2001 From: Local Dev Date: Fri, 18 Sep 2026 18:22:49 +1000 Subject: [PATCH 2/2] =?UTF-8?q?compat:=20906=20=E2=80=94=20carry=20upstrea?= =?UTF-8?q?m's=20revert=20of=20prop.integrated=20on=20HIP=20builds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream PR #28604 (d4389a4dd92) reverts c7d8722922a (#24233). Our payload b10864 is 5d806aa25 at 2026-09-08T13:01:03Z; the revert landed at 14:19:53Z the same day, so v0.34.1 carries the defect and not the fix -- we missed it by 78 minutes. Upstream #28211 root-causes it on gfx115x as an MMQ tile-barrier race that produces wrong output with no crash once a decode exceeds n_ubatch, and #27419 reports our exact symptom on our exact silicon: upper image content lost on HIP while Vulkan and CPU are correct. Measured here on gemma4:31b with the budget pinned, name_bbox_mean_iou is 0.728 -> 0.000 at the 1120 rung only; 280 and 560 are equivalent between builds. prompt_eval_count is identical, so the pixels reaching the model are unchanged. Replaces 905, which reverted the b10864 resize-algo flip and is dropped: it was built, verified present in-binary by output-hash diff, and did not fix anything. Co-Authored-By: Claude Opus 5 --- .../905-restore-bilinear-vision-resize.patch | 50 ------------------- .../906-revert-hip-integrated-flag.patch | 50 +++++++++++++++++++ 2 files changed, 50 insertions(+), 50 deletions(-) delete mode 100644 llama/compat/905-restore-bilinear-vision-resize.patch create mode 100644 llama/compat/906-revert-hip-integrated-flag.patch diff --git a/llama/compat/905-restore-bilinear-vision-resize.patch b/llama/compat/905-restore-bilinear-vision-resize.patch deleted file mode 100644 index b269f4a436e..00000000000 --- a/llama/compat/905-restore-bilinear-vision-resize.patch +++ /dev/null @@ -1,50 +0,0 @@ -MAXUSAI 905: restore RESIZE_ALGO_BILINEAR for gemma4 and qwen3vl. - -b10864 changed the vision resampler for these projectors in the same hunk that -raised gemma4's token ceiling from (40,280) to (70,1120): - - - hparams.image_resize_algo = RESIZE_ALGO_BILINEAR; - + hparams.image_resize_algo = RESIZE_ALGO_BICUBIC; - - hparams.set_limit_image_tokens(40, 280); - + hparams.set_limit_image_tokens(70, 1120); - -Separately, the naive resamplers (resize_bilinear, resize_bicubic) were deleted -and every algo now routes through the Pillow-parity path, so BICUBIC also means -something different than it did: antialiased, with half-pixel centres. - -MEASURED ON gfx1151 (Radeon 8060S), gemma4:31b-it-q4_K_M, budget pinned via -run_budget_sweep.sh, name_bbox_mean_iou (vertical localization): - - budget 0.32.1/b9888 0.34.1/b10864 - 280 0.612 0.691 - 560 0.713 0.714 - 1120 0.728 0.000 <-- total loss - -The grid is unchanged: prompt_eval_count is identical between builds on every -test (1684 on scene_single), so the token count and tokenization are the same -and only the PIXELS differ. The failure is confined to the 1120 rung, which is -the only rung that enlarges a 1920x1080 source (1120 * 48^2 = 2580480 px), and -it is the vertical axis that breaks -- the same axis and the same rung as the -coordinate defect ADR 0007 mitigated and ADR 0008 declared fixed by patch 004. -diff --git a/tools/mtmd/clip.cpp b/tools/mtmd/clip.cpp -index b4f07c4..0db1971 100644 ---- a/tools/mtmd/clip.cpp -+++ b/tools/mtmd/clip.cpp -@@ -1684,7 +1684,7 @@ struct clip_model_loader { - { - hparams.rope_theta = 100.0f; - hparams.n_merge = 3; // pooling_kernel_size -- hparams.image_resize_algo = RESIZE_ALGO_BICUBIC; -+ hparams.image_resize_algo = RESIZE_ALGO_BILINEAR; - get_u32(KEY_PROJ_SCALE_FACTOR, hparams.n_merge, false); - if (model.proj_type == PROJECTOR_TYPE_GEMMA4UV) { - // for "unified" variant, we directly use a bigger patch size, because the "token merging" is done directly on conv layer -@@ -1720,7 +1720,7 @@ struct clip_model_loader { - case PROJECTOR_TYPE_QWEN3VL: - { - hparams.n_merge = 2; // default value for Qwen 2 and 2.5 -- hparams.image_resize_algo = RESIZE_ALGO_BICUBIC; -+ hparams.image_resize_algo = RESIZE_ALGO_BILINEAR; - get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false); - get_u32(KEY_WIN_ATTN_PATTERN, hparams.n_wa_pattern, model.proj_type == PROJECTOR_TYPE_QWEN25VL); // only 2.5 requires it - // ref: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/blob/main/preprocessor_config.json diff --git a/llama/compat/906-revert-hip-integrated-flag.patch b/llama/compat/906-revert-hip-integrated-flag.patch new file mode 100644 index 00000000000..b39097fe8fb --- /dev/null +++ b/llama/compat/906-revert-hip-integrated-flag.patch @@ -0,0 +1,50 @@ +MAXUSAI 906: carry upstream's revert of "restore prop.integrated on HIP builds". + +Upstream llama.cpp PR #28604 (d4389a4dd92, 2026-09-08T14:19:53Z) reverts +c7d8722922a (#24233), which set info.devices[id].integrated = prop.integrated on +HIP builds. On an integrated/UMA device that flag changes which buffer types the +HIP backend claims, and upstream issue #28211 root-causes the result as an MMQ +tile-barrier race on gfx115x that produces WRONG OUTPUT with no crash and no +warning, once a decode exceeds n_ubatch. + +WE MISSED THE FIX BY 78 MINUTES. Our payload b10864 is 5d806aa25, 2026-09-08 +13:01:03Z. The revert landed at 14:19:53Z the same day, so ollama v0.34.1 has the +defect and not the fix. + +MEASURED HERE (gfx1151, Radeon 8060S, ROCm 7.2.x), gemma4:31b-it-q4_K_M with the +image-token budget pinned per cell, name_bbox_mean_iou: + + budget 0.32.1/b9888 0.34.1/b10864 + 280 0.612 0.691 + 560 0.713 0.714 + 1120 0.728 0.000 <-- total loss + +Only the top rung fails, and only on the newer payload. prompt_eval_count is +identical between builds on every test, so the grid and token count are unchanged +and the pixels reaching the model are the same -- consistent with corruption +downstream of preprocessing rather than in it. Related upstream reports on this +exact silicon: #27419 (upper image content lost on HIP, Vulkan and CPU correct) +and #28211 (wrong logits past n_ubatch). + +GGML_CUDA_NO_PINNED=1 is NOT a workaround: it reaches the runner and the runner +aborts (signal: aborted, core dumped) on model load. + +Drop this patch when the payload advances past d4389a4dd92. + +Files: ggml/src/ggml-cuda/ggml-cuda.cu +diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu +index 0e6601f..38bd4c9 100644 +--- a/ggml/src/ggml-cuda/ggml-cuda.cu ++++ b/ggml/src/ggml-cuda/ggml-cuda.cu +@@ -304,11 +304,7 @@ static ggml_cuda_device_info ggml_cuda_init() { + + info.default_tensor_split[id] = total_vram; + total_vram += device_vram; +-#if defined(GGML_USE_HIP) +- info.devices[id].integrated = prop.integrated; +-#else + info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034) +-#endif + info.devices[id].nsm = prop.multiProcessorCount; + info.devices[id].smpb = prop.sharedMemPerBlock; + info.devices[id].warp_size = prop.warpSize;