Skip to content

ggml-hrx: add opt-in compilation diagnostics - #165

Merged
AaronStGeorge merged 1 commit into
hrx-graph-develop-v2from
users/astgeorg/hrx-compilation-diagnostics
Nov 1, 2026
Merged

AaronStGeorge merged 1 commit into
hrx-graph-develop-v2from
users/astgeorg/hrx-compilation-diagnostics

Conversation

@AaronStGeorge

Copy link
Copy Markdown
Collaborator

Overview

Add GGML_HRX_TRACE_COMPILE=1 to diagnose runtime kernel compilation during prefill and decode. It emits JSON records to stderr with graph execution and cache IDs, kernel keys and specialization parameters, compiler timing, failures, and unresolved-compilation waits. Asynchronous compiler events retain their originating execution ID. Graph-end records include zero compilation counts so repeated cycles can show cache reuse.

The caller chooses the workload and repetitions. Records are informational; backend graph IDs do not claim to identify token steps or prefill/decode phases. Existing asynchronous compilation and graph replay remain intact. Disabled tracing adds cached-state checks without timestamps, JSON construction, logging locks, or file I/O.

All changes are inside ggml/src/ggml-hrx. The README documents activation, events, timing limits, and JSON extraction.

Validation

Built llama-bench, hrx-backend-test, and hrx-loom-jit-test with HRX 82214dfb029975205d1c3a59164b678e2e21fc41 on a pinned gfx1100 GPU.

  • Llama-3.2-1B-Instruct UD-Q4_K_XL, -p 0 -n 0 -pg 64,16 -b 64 -ub 64 -r 2 --no-warmup --progress: async and sync JIT both produced 68 backend graph executions and 43 initial compilation requests. Counts for the first four graphs were [1, 23, 1, 18]; all remaining graphs, including the entire second cycle, requested zero compilations.
  • Parsed records to reconcile graph counts and match request/begin/end identities, timestamps, originating executions, and wait attribution. Sync and async compiled identical keys. Async recorded 16 waits; sync recorded none, as expected.
  • Existing backend cases add_f32, scale_f32, and both same_uid_distinct_graph_reuses_graph_program variants: 4 passed with tracing disabled and 4 passed with tracing enabled. Disabled tracing emitted no trace records.
  • Deliberately invalid GGML_HRX_LOOM_SANITIZER produced a matching failed compile-end record with compiler_started=false, the initialization error, and a failed graph-end record.
  • Modified translation-unit syntax checks and git diff --check passed. The full backend suite was not run.

The existing compatibility floor is stale and is being corrected separately. Validation used the compatible HRX revision above; this PR does not change that floor or migrate kernel syntax for newer HRX revisions.

Requirements

  • Reviewed the contributing guidance; this submission targets the AMD-Ecosystem fork.
  • AI usage disclosure: YES. Codex implemented, reviewed, and tested the change and prepared this PR under user direction.

@AaronStGeorge
AaronStGeorge merged commit f6d215d into hrx-graph-develop-v2 Nov 1, 2026
4 of 24 checks passed
@AaronStGeorge
AaronStGeorge deleted the users/astgeorg/hrx-compilation-diagnostics branch November 1, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant