[AaronStGeorge]

Kernel support added

Add the HRX operator coverage needed by the Qwen3.8-27B text model and its vision encoder/projector for images and video. The implementation adds reusable Loom motifs and extends existing matmul and Flash Attention kernels. The validated model pair is the Q4_K_M text GGUF and BF16 projector.

New operator coverage

Operation	Required by	Support added
Interleaved multimodal RoPE (mode 40)	Text model	Position-stream selection and dispatch for text MRoPE, using the shared F32 rotary arithmetic. Existing quantized matmul, embedding, normalization, recurrent, and text-attention kernels are reused.
Vision RoPE	Vision encoder	Vision-specific position selection and strided tensor handling, using the same shared rotary arithmetic.
2D IM2COL	Image patch extraction	Generic patch gathering with stride, padding, and dilation; F32 input with F16 or F32 output variants.
Bilinear UPSCALE	Vision positional embeddings	F32 bilinear interpolation with the required coordinate mapping.
NORM	Vision layer normalization	F32 mean-and-variance normalization. Row-statistics and normalization-application motifs are separately composable for fusion.
Strided CONT / CPY and casts	Vision tensor layouts and attention inputs	Broaden F32 strided copies and add strided F32-to-F16 and F16 copy variants, all sharing a source-indexing motif. Internal self-target casts are treated as write destinations, while externally visible outputs and ordinary target copies retain their storage requirements.

Existing kernels extended and reused

• BF16 projection matmul: extend the existing tiled matmul to supported input widths divisible by 16, and process long image-token sequences through consecutive launches of at most 2,048 tokens. BF16 weight loading and matrix arithmetic reuse the existing kernel corpus.
• Vision Flash Attention: compose the existing WMMA attention core with tile loaders and output stores for logical head width 72 and strided views. Loaders pad Q/K and V tiles and handle boundaries while preserving logical dimensions and attention scaling. Preserve F16 WMMA operands, F32 QK accumulation, and F16 PV accumulation.
• F16 QKV and rotary handoffs: the existing BF16 projection matmul plus bias can publish F16 QKV. Vision RoPE accepts that F16 storage, uses the shared F32 rotary arithmetic, and publishes F16 for the tiled attention loader. Eligible internal K/V casts disappear; observable outputs and mixed consumers retain F32 or dual publication. Backend-owned alternates carry the F16 storage while GGML-visible tensor types retain their contract. Thin typed entry points reuse the existing matrix and rotary motifs.
• F16 attention-to-projection handoff: publish only F16 on eligible transient edges whose BF16 projection consumers accept that representation. Reuse the projection's existing F16 input support and preserve existing F32 and dual-publication consumers.
• Shared rotary arithmetic: ordinary RoPE, text MRoPE, vision RoPE, and fused Qwen attention postprocessing call the same rotary motif. Typed copy kernels likewise share coordinate indexing.

Backend integration registers the operators and their parameters, advertises supported IM2COL to GPU placement, and models cast destination self-reference correctly throughout graph indexing, traversal, dependency checks, and allocation. Internal cast results can use transient storage without inventing a read dependency on their own destination. The superseded grouped matmul, scalar attention fallback, and standalone masked softmax paths are removed.

The supported scope is full GPU transformer layers and projector execution with Flash Attention on/auto. Explicit Flash Attention off, partial layer offload, and the removed fallback's extra irregular or aliased layouts are outside this scope; the README describes the attention boundary.

Validation

Validated source: 9f97957b48a1b18e3eaafc8fea9e2ad25552f49c; HRX: 7a7f1fe53f14805cd8a9d5753279508e9eb6d9bb.

• Final remote workflow passed, with only qwen3.8-27b selected, tests and all benchmarks enabled: run 41360912282, benchmark/video report artifact.
• All four HRX suites passed: 494 cases locally on gfx1100 and 494 packaged cases remotely on gfx1151, with zero failures or skips. The separate local PR160 build passed 525 cases.
• The fused F16 handoff exactly matches staged HRX execution with explicit F16 roundtrips in the 17-token test on both GPU targets. CPU comparison remains a separate check because the existing WMMA arithmetic differs from F32 CPU accumulation. Graph tests cover observable outputs, mixed consumers, explicit copy destinations, and the existing GPU paths at short and long token counts.
• Local and remote image/video each passed 24/24 answers across dog identification, first video color, and motion direction on HRX and Vulkan. Logs confirm 66/66 text layers on GPU, projector on HRX0, and one projector graph split.
• Actual 256- and 704-token vision graphs drop from 453 to 399 dispatches, removing all 54 standalone K/V casts. Warm encoder medians were 58.75 → 59.66 ms for 256-token frames and 165.22 → 165.18 ms for the 704-token dog image; these samples do not establish a speedup. Reported transient arenas increase from 11,906,048 → 14,543,872 bytes and 32,693,248 → 39,947,264 bytes respectively. Arena accounting excludes external allocations, so these figures are not total GPU memory; this PR makes no peak-memory reduction claim.
• Remote WikiText perplexity: HRX 5.5995 / 6.0650, Vulkan 5.6097 / 6.0613 for prefill/decode regimes. HRX exactly matches the prior validated baseline. The manifest classifies HRX text rows as report-only; the numeric values and successful exits were inspected separately.

Shared regressions cover all 31 local text models with paired pre-change/candidate throughput and prefill/decode perplexity. All 62 reported PPL comparisons match exactly, with no compute errors. Five initial timing differences were repeated in reversed order with five samples; no slowdown beyond the observed sample variation remained. Mistral 24B and Qwen 27B used identical PR160 residency changes on both sides.

Local 24GB validation with PR160

The complete series is also applied locally on top of PR160, yielding test commit 6df3ce019. Use explicit --load-mode mmap to exercise PR160's imported-weight residency. The model and projector fit the 24GB gfx1100 device with this configuration. PR160 is not included in this PR's branch; the requested hrx-graph-develop-v2 base is retained.

Local WikiText perplexity is HRX 5.5995 / 6.0650 versus Vulkan 5.6120 / 6.0614 for the same prefill/decode regimes. The paired pre-change HRX results are unchanged.

Freeform dog and Apollo 11 newspaper checks return identical answers on HRX and Vulkan. Additional paired projector checks retain the same pre-existing results: MiniCPM-V 4.6 and its Thinking variant reject the F16×F16 patch matmul with reduction width 588; Mistral Small rejects its F16×F32 patch matmul at width 588; Gemma 4's encoder executes but its image answers remain unusable on both revisions. These other projectors are not newly supported by this PR. Their text-only prefill/decode PPL is unchanged in the 31-model matrix above.

Automation companion draft: ROCm/ggml-staging-automation#118. This PR remains a draft.

[AaronStGeorge]

Final validation for b9d99ceeb is complete; the PR remains a draft.

• Qwen-only remote workflow passed: all HRX suites, 24/24 image/video answers, 66/66 text layers on GPU, and the projector on HRX without the old IM2COL CPU fallback. Download the benchmark/video report.
• The separate local PR160 rebase fits the full Qwen model and projector on the 24GB GPU with explicit --load-mode mmap. All 513 combined test cases pass. HRX/Vulkan PPL is 5.5995/5.6120 for prefill and 6.0650/6.0614 for decode; freeform dog and newspaper checks also pass.
• The staged 31-text-model regression coverage found unchanged PPL and no repeatable performance decrease beyond timing variation. Mistral's previous memory gap was retested with PR160 applied to both comparison revisions.

The PR description now records the design, exact validation revisions, measured limits, and existing failures in the additional MiniCPM/Mistral/Gemma projector checks. Those failures are not counted as passing image tests. PR160 remains a separate local testing base and is not included in this PR.

[AaronStGeorge]

Local full-model testing uses a separate branch based on PR #160 (c3bf46aac), with --load-mode mmap to exercise demand-based weight residency. The draft PR's base remains hrx-graph-develop-v2.

--load-mode mmap should not be required, this was a bug in #160 (c3bf46aac) that has now been fixed.