Skip to content

qwen4exp: port selected upstream performance paths - #44

Open
edwinbrowwn wants to merge 3 commits into
masterfrom
perf/qwen-upstream-optimizations
Open

edwinbrowwn wants to merge 3 commits into
masterfrom
perf/qwen-upstream-optimizations

Conversation

@edwinbrowwn

@edwinbrowwn edwinbrowwn commented Sep 7, 2026

Copy link
Copy Markdown
Owner

What changed

  • Use the KV sequence-position index for n-gram history lookups.
  • Reduce Qwen4Exp indexer scoring copies.
  • Add opt-in QSA gather for supported flash-attention decode, preserving selection budgets, masked padding, multi-stream bias strides and cache-layout guards.
  • Add opt-in Qwen4Exp PLE explicit reads with lazy-mmap/resident fallback, file-span validation and safe reader ownership.
  • Port upstream models: fix GDN normalization from max to rsqrt ggml-org/llama.cpp#28068 GDN normalization correctness fix across affected architectures; RWKV normalization is unchanged.
  • Keep existing rocPRIM TOP_K unchanged: the temporary radix experiment did not establish a valid kernel-level comparison.

Validation

  • CPU/HIP Release builds and argument-parser tests pass.
  • Qwen4Exp replay passes with gather enabled/disabled, including individual V620 GPUs and TP4 F16/Q8 configurations. TP roundtrip remains skipped by the harness.
  • Independent GDN scalar-reference tests cover widths 4/32/128, epsilons 1e-2/1e-6, ordinary/tiny/epsilon-scale/zero rows on CPU and HIP devices.
  • PLE row tests cover F32/F16/Q8_0/Q4_0, shuffled/duplicate rows and multiple workers; F32 bounds/truncation tests pass. Temporary paths are cleaned on early returns and exceptions.
  • A compatible synthetic Qwen4Exp GGUF fixture, not a production model, matched direct-vs-lazy-mmap logits exactly. Resident fallback with load-mode none also matched exactly.
  • DFlash controller, sidecar-cap, speculative-content and sidecar-artifact tests pass. Server/sidecar builds pass.
  • git diff --check passes. Local logs: /home/edwin/.ralph/merge-review-.log and final-.log.

Limits and merge gate

QSA gather and PLE direct reads remain opt-in. No throughput or memory improvement is claimed. Production Qwen4Exp PLE benchmarking remains blocked by available compatible model artifacts. MTP gather, QSA input sharing, PLE graph splitting and incremental block-key caching are separate work.

The prior CI run (34130448354) has ARM64 failures in files unchanged from fork/master: llama-bench.cpp unused repetitions parameter with -Werror, and Windows ARM64 test-recurrent-state-rollback missing llm_arch_supports_rs_rollback at link time. New-head CI must be reviewed before merge; local CPU/HIP success is not a claim that cross-platform CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant