Skip to content

hrx: keep contiguous F32 duplicates on the device - #164

Merged
AaronStGeorge merged 2 commits into
hrx-graph-develop-v2from
users/astgeorg/hrx-contiguous-f32-dup-20261030
Oct 30, 2026
Merged

AaronStGeorge merged 2 commits into
hrx-graph-develop-v2from
users/astgeorg/hrx-contiguous-f32-dup-20261030

Conversation

@AaronStGeorge

Copy link
Copy Markdown
Collaborator

Summary

Keep contiguous F32 DUP operations on HRX by reusing the existing copy dispatch. The mixed-embedding input duplicate otherwise runs on CPU and introduces a backend boundary. Advertise support only for nonempty contiguous F32 tensors with matching shapes, a distinct contiguous output, and at most 2^30 elements. Extend the existing CPU-reference test to execute and compare the duplicate.

This is one commit on hrx-graph-develop-v2 at 2120f7434eeb8d958cd0c0fab3be581e12a8f29b (after #162 and #163), cherry-picked from 73f49408113923d48bcb7df7602cf43075dc2900.

Performance against the current base

Local before/after measurements used the Radeon RX 7900 XTX (gfx1100), Qwen3-0.6B Q4_0, context 4096, one slot, 16 CPU threads, and a 17-token prompt. Both builds used sources/build.py and the same HRX revision, 82214dfb029975205d1c3a59164b678e2e21fc41. The only source difference for this comparison is the DUP patch. Each server handled three 20-token requests followed by three warm 256-token requests.

The before-DUP source tree at 760fe39af is identical to the current target branch at 2120f7434. The after-DUP tree is identical to this PR.

256-token generation Base, ms Base + DUP, ms
Warm request 1 715.798 669.302
Warm request 2 622.058 596.952
Warm request 3 608.892 585.599
Mean 648.916 617.284

This is 5.12% higher throughput, or 4.87% less generation time, for these warm requests. Generated text was byte-identical before and after. Scheduler diagnostics showed three partitions (HRX / CPU / HRX) becoming one HRX partition, removing the CPU F32 DUP (4 KiB for a single-token input).

These are three sequential warm requests from one server session per version. The result is a limited local measurement, without repeated alternating sessions or a confidence interval. The separate compiler optimizations are not included; their previously reported 2.71x gain does not quantify this change.

Validation

  • Local build.py build passed on pinned gfx1100.
  • Local test-backend-ops F32 DUP comparison passed against CPU; unsupported types and noncontiguous inputs remained rejected.
  • The existing dynamic_generic_ops.cpu_reference test now checks HRX placement, copy-kernel selection, and exact numerical agreement for the active DUP.
  • Completed single-model CI: 41059144662, SUCCESS on gfx1151. Qwen3-0.6B chat checks, prefill/decode perplexity comparisons, and all four packaged HRX suites passed. Perplexity was HRX 20.4920 / Vulkan 20.4695 for prefill and HRX 19.2819 / Vulkan 19.2796 for decode.
  • Completed full-tier CI: 41061540987, SUCCESS: all 31 HRX chat cases and 30 perplexity models passed their underlying checks, plus all four packaged HRX suites.

Both CI runs used staging c6985eaf0ad6f254f973ed122d46e6faec610eb6, llama.cpp 73f49408113923d48bcb7df7602cf43075dc2900, and unchanged HRX 82214dfb029975205d1c3a59164b678e2e21fc41. This PR head 6945a7fe669a84c68b842a894e7759f1773848e0 and the tested llama.cpp revision have exactly the same Git source tree, a1356018805a971e51fa8de997987117e083ad94; only commit history differs. The existing single-model run therefore covers the proposed source state, and no duplicate run was dispatched. CI timings are not used to claim an isolated DUP speedup.

Assisted-by: Codex

Reuse the copy dispatch for DUP and constrain placement to the supported contiguous F32 case. Cover the active copy in the existing CPU reference test.

Assisted-by: Codex
(cherry picked from commit 73f4940)
@AaronStGeorge
AaronStGeorge merged commit 76707b4 into hrx-graph-develop-v2 Oct 30, 2026
4 of 10 checks passed
@AaronStGeorge
AaronStGeorge deleted the users/astgeorg/hrx-contiguous-f32-dup-20261030 branch October 30, 2026 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant