fix: harden Lunar Lake Gemma 4 attention and quantized CPU copies - #172
Draft
marksverdhei wants to merge 21 commits into
Draft
fix: harden Lunar Lake Gemma 4 attention and quantized CPU copies#172marksverdhei wants to merge 21 commits into
marksverdhei wants to merge 21 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hybrid Gemma 4 attention could route a transposed quantized V cache to a CPU copy that wrote beyond its output allocation. Reject unsupported quantized copy layouts before scheduling or direct/planned CPU execution, and correct row-byte counts for valid reshaped copies.
Restrict the hybrid experiment to Mesa Vulkan on the validated Lunar Lake PCI device (8086:64a0), full layer/KV offload, F16 K/V, one sequence, and context up to 2048. Only the exact environment value
1enables eligibility;0disables it, and FA auto-probing retains its fused graph. Limit the added Q4_0 MMVQ exception to the same device/driver. Add pinned model artifacts, bounded serving presets, CPU-logit/KV-reuse tests, interleaved depth benchmarks, and a one-hour server soak runner.Validation completed:
GG_BUILD_LOW_PERF=1): Debug 45/45 and Release 47/47 pass.LLAMA_FATAL_WARNINGS=OFFwas needed for a GCC 16 warning in unchanged vocabulary code.Full-model preset checks pass on both targets: FA-off, bounded hybrid, automatic hybrid, and strict KV reuse. The complete 26B sweep still fails one plain-FA-on control (KL 0.00991866 at 16 continuation tokens, limit 0.005); that failure remains visible. Plain FA-on is not an offered serving preset. Full-model metrics now compare token distributions with explicit bounds; raw NMSE remains reported and KV reuse retains its original strict threshold. Tests use the GGUF chat template and check answer content.
An isolated original-branch build produces byte-identical raw-prompt logits for both models in FA-off and hybrid, confirming the initial raw-prompt discrepancies predate these fixes. The validation collector now records transient firmware telemetry errors and preserves interrupted-run evidence; three infrastructure regression tests pass.
All 11 corrected lifecycle profiles pass, including tools, thinking, cache reuse, cancellation after generated content, and MTP target-token agreement. The final-budget one-hour soak passed all four phases (887 requests), including four concurrent 26B slots, with flat GPU residency and no recorded validation or telemetry errors. Serving presets use soft budgets of 1 GiB for prompt cache and 512 MiB per slot for checkpoints; measured memory use is recorded separately.
Draft: five-run depth benchmarks are running. See
docs/backend/VULKAN-GEMMA4-INTEL-XE2-VALIDATION.mdfor measured results and limitations. No fresh speedup or deployment-readiness claim yet. The historical Xe2 artifact hashes were missing; the manifest matches documented filenames and establishes a checksummed fresh baseline.