Skip to content

fix: harden Lunar Lake Gemma 4 attention and quantized CPU copies - #172

Draft
marksverdhei wants to merge 21 commits into
htfrom
fix/xe2-attention-safety
Draft

fix: harden Lunar Lake Gemma 4 attention and quantized CPU copies#172
marksverdhei wants to merge 21 commits into
htfrom
fix/xe2-attention-safety

Conversation

@marksverdhei

@marksverdhei marksverdhei commented Sep 5, 2026

Copy link
Copy Markdown

Hybrid Gemma 4 attention could route a transposed quantized V cache to a CPU copy that wrote beyond its output allocation. Reject unsupported quantized copy layouts before scheduling or direct/planned CPU execution, and correct row-byte counts for valid reshaped copies.

Restrict the hybrid experiment to Mesa Vulkan on the validated Lunar Lake PCI device (8086:64a0), full layer/KV offload, F16 K/V, one sequence, and context up to 2048. Only the exact environment value 1 enables eligibility; 0 disables it, and FA auto-probing retains its fused graph. Limit the added Q4_0 MMVQ exception to the same device/driver. Add pinned model artifacts, bounded serving presets, CPU-logit/KV-reuse tests, interleaved depth benchmarks, and a one-hour server soak runner.

Validation completed:

  • CPU regression tests and ASan/UBSan pass, including rejection through direct/planned execution, one-block views, and valid reshaped copies.
  • Vulkan: 280 copy/contiguous comparisons, 29 Q4_0 matmul comparisons, and 247 Gemma attention comparisons pass.
  • Local CPU CI (GG_BUILD_LOW_PERF=1): Debug 45/45 and Release 47/47 pass. LLAMA_FATAL_WARNINGS=OFF was needed for a GCC 16 warning in unchanged vocabulary code.

Full-model preset checks pass on both targets: FA-off, bounded hybrid, automatic hybrid, and strict KV reuse. The complete 26B sweep still fails one plain-FA-on control (KL 0.00991866 at 16 continuation tokens, limit 0.005); that failure remains visible. Plain FA-on is not an offered serving preset. Full-model metrics now compare token distributions with explicit bounds; raw NMSE remains reported and KV reuse retains its original strict threshold. Tests use the GGUF chat template and check answer content.

An isolated original-branch build produces byte-identical raw-prompt logits for both models in FA-off and hybrid, confirming the initial raw-prompt discrepancies predate these fixes. The validation collector now records transient firmware telemetry errors and preserves interrupted-run evidence; three infrastructure regression tests pass.

All 11 corrected lifecycle profiles pass, including tools, thinking, cache reuse, cancellation after generated content, and MTP target-token agreement. The final-budget one-hour soak passed all four phases (887 requests), including four concurrent 26B slots, with flat GPU residency and no recorded validation or telemetry errors. Serving presets use soft budgets of 1 GiB for prompt cache and 512 MiB per slot for checkpoints; measured memory use is recorded separately.

Draft: five-run depth benchmarks are running. See docs/backend/VULKAN-GEMMA4-INTEL-XE2-VALIDATION.md for measured results and limitations. No fresh speedup or deployment-readiness claim yet. The historical Xe2 artifact hashes were missing; the manifest matches documented filenames and establishes a checksummed fresh baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant