bump-hrx: the scheduled HRX pin sync is failing

The bump-hrx run 37470341222 failed: https://github.com/1bit-MONSTER/engine/actions/runs/37470341222

Look at the sync steps ('Sync the fork and merge AMD's pin into our line', 'Merge AMD's hrx-system pin into our line'). If one is a merge conflict, the conflicting files are named in the log; resolve them by hand on 1bit/hrx-vulkan-patched or on 1bit-MONSTER/hrx-system 1bit/main, then re-run this workflow.

Why this step exists: bump-hrx failed on every scheduled run from 2026-10-01 to 2026-10-05 without anyone noticing, and the HRX pin silently froze.

Diagnosis, and the resolution this needs

What fails: the step "Sync the fork and merge AMD's pin into our line" exits 1 at the merge — 15 conflicting files, all under ggml/src/ggml-hrx/. (The "1bit/hrx-vulkan has commits that AMD's pin does not contain; rebase them by hand" guard fires for the same reason.) The later announcement step then files this issue.

Why it conflicts every time: both lines rewrote ggml/src/ggml-hrx/ — AMD's refactor (tiled/skinny/vector kernel families, new dispatchers) against our backend. The content genuinely diverges, so it will conflict on every scheduled run until one side's ggml-hrx is chosen once and committed as the merge resolution. It is not a stale branch.

Which side to resolve toward — measured. Fresh build directories, the engine's own ExternalProject arguments (cmake/hrx.cmake: fresh -B, Release, amdclang, -DHRX_SOURCE_DIR=…), GLM-4.7-Flash Q4_K_M 269-token request at -ub 512, three requests:

build	full gate `test-backend-ops -b HRX0`	model
AMD's core + AMD's `ggml-hrx`	1019/1071 — 54 FAIL	18 NaN events, reply `??????`
AMD's core + our `ggml-hrx`	1073/1073 — 0 FAIL	0 NaN, reply `' Paris.'`

So the merge resolution should take ours for ggml/src/ggml-hrx/. Resolving toward AMD's side would import engine#315's all-NaN logits into the pinned line, and keep the 54 gate failures (40 of them MUL_MAT, in iq1_s/iq1_m/iq3_xxs/mxfp4/pq2_0/ptq1_0/q2_*).

Artifact and the one thing I could not do. Branch 1bit/amd-core-our-hrx (3d8dbcd7a) on my clone is the pinned core (51e538fa lineage) plus our ggml-hrx, with hrx-system staying at 98d05d94 — our loom, because AMD's does not export loomc_amdgpu_runtime_global_flags_t, which our loom-jit.cpp needs; that is the second pin this fix moves. I could not push the branch: this credential is rejected with "Unable to determine if workflow can be created or updated due to timeout; workflows scope may be required", even for a single commit built on the remote's own head, and the commit touches no workflow file.

A patch of that branch is in the working notes at port-fused-mul-mat/amd-core-our-hrx.patch. With a workflows-scoped token the branch can go up as 1bit/hrx-vulkan-patched (with the ggml-hrx side taken from ours), which is the resolution this workflow is waiting for.

The failing step is the merge, and a ready resolution — blocked on the same scope

The guard is not what fails. The run's annotation points at .github/workflows/bump-hrx.yml#51, which is just cancel-in-progress: false — unhelpful. I checked the ancestry guard by hand instead: 1bit/hrx-vulkan is f1a0aca141, AMD's current pin is b802a507b, git merge-base --is-ancestor says yes, and there are 0 commits on hrx-vulkan that AMD's pin lacks. So the guard at lines 108-111 passes and the failure is the merge — which is what the workflow header already describes: 15 conflicting files, all under ggml/src/ggml-hrx/.

A resolution, on branch fix/bump-hrx-resolve-ours (commit 430c5ce). It resolves conflicts inside ggml/src/ggml-hrx/ toward the branch being merged into (our patched line), keeps AMD's side everywhere else, emits a ::notice:: naming the files it took, and still aborts loudly if any conflict falls outside that path so a real divergence cannot be auto-resolved by accident. I simulated both cases against a scratch repo: only-ggml-hrx conflict → k.loom=ours, other/x.txt=amd-changed, real merge commit; mixed conflict → abort. (A first simulation of mine made both files conflict, which only exercised the abort path — worth saying, because it is exactly the kind of thing that would have looked like a passing test.)

Taking ours is measured, not assumed — fresh builds, this workflow's own ExternalProject arguments, 269-token prompt at -ub 512:

build	full gate `test-backend-ops -b HRX0`	model
AMD core + our `ggml-hrx`	1073/1073 — 0 FAIL	0 NaN, reply `' Paris.'`
AMD core + AMD's `ggml-hrx`	1019/1071 — 54 FAIL	18 NaN, reply `??????`

Resolving these 15 files toward AMD's side is what puts engine#315's NaN into the pinned line.

Blocked on the same scope as the other branch, now with an exact message:

refusing to allow an OAuth App to create or update workflow `.github/workflows/bump-hrx.yml` without `workflow` scope

This credential can push ordinary files — I pushed fix/registry-reviewed-classes (registry + docs) today — but refuses anything touching .github/workflows/. So this fix and 1bit/amd-core-our-hrx both need one token with the workflow scope, and then they can go up. Until then the scheduled run keeps failing and filing issues, which is at least no longer silent.

Landed the resolution as #335 — the workflow scope is present on the local credential, so the blocker in the comments above is gone.

#335 resolves conflicts ggml/src/ggml-hrx/ toward 1bit/hrx-vulkan-patched (our patched line), takes AMD's side everywhere else, keeps the out-of-path conflict abort, and emits a ::notice::. It applies cleanly on current main (ef04beb, workflow blob 90e36c0) and validates as YAML.

Once it merges, the scheduled sync should stop failing: the merge will commit toward our ggml-hrx instead of aborting, which is the measured-correct side (AMD core + our ggml-hrx = 1073/1073, 0 NaN; AMD core + AMD ggml-hrx = 1019/1071, 54 FAIL, 18 NaN).

The two branches named in the diagnosis are now pushed — the local credential has the workflow scope, so the blocker is gone:

fix/bump-hrx-resolve-ours → engine#335 (the workflow fix; YAML-valid, applies cleanly on current main).
1bit/amd-core-our-hrx = f2099e9b7 on 1bit-MONSTER/llama.cpp — the corrected pin. Verified by tree: ggml/src/ggml-hrx is byte-identical to 522dab47's (tree 919cd999f), and everything outside it is byte-identical to 56c3c8a3.

Root cause: the bump token lacks the `workflows` scope

Reproduced by dispatching this workflow on 2026-10-07 (run 37640512284). It fails in "Sync the fork and merge AMD's pin into our line" after ~5 seconds with:

gh: Unable to determine if workflow can be created or updated due to timeout; `workflows` scope ...
##[error]Process completed with exit code 1.

HRX_BUMP_TOKEN is a fine-grained token with Contents read/write on llama.cpp and engine and Pull requests read/write on engine (per the workflow header). Syncing master and AMD's pin into the fork pushes .github/workflows/*, which requires the workflows permission (classic PATs call it the workflow scope). Without it the push is refused and the step exits 1 — before any of the merge logic runs.

What it is not: the merge. I reproduced the merge locally against the same inputs on 2026-10-07:

git merge-tree --write-tree 26330cc20490 b802a507b57c5a5019c38f3b4498c1b7a14dec08 → exit 0, no conflicts at all (not even the known 15-file ggml/src/ggml-hrx/ set);
the ancestor guard passes: our 1bit/hrx-vulkan head f1a0aca141de is 70 commits behind AMD's pin b802a507b57c and fully contained in it (behind_by=0).

So the conflict-resolution code, the guard, and the pinned-vs-AMD relationship are all healthy. The only blocker is the token scope.

Fix: add the workflows permission to HRX_BUMP_TOKEN (Settings → Developer settings → Fine-grained tokens → the token this workflow uses) and re-run. I can't change repository secrets from here.

Two follow-ups worth doing at the same time:

The failure-reporting step ("Report a failed sync so it cannot break silently") also failed in the same run — which is why this could fail on 12 consecutive scheduled runs with no visible signal while this issue sat closed. Its token needs the same attention.
The hrx-system half is untested by my reproduction — I only reproduced the llama.cpp merge. Once the token is fixed, watch the hrx-system merge step; the workflow's own header says a conflict there "still fails the run" and must be resolved by hand.

2. A second, independent blocker: the `hrx-system` merge conflicts

The other half of the sync — the step "Merge AMD's hrx-system pin into our line" — was reproduced with a
full (non-shallow) clone of 1bit-MONSTER/hrx-system and AMD's pin cdabb9ac35cb from
ROCm/hrx-system:

merge-base(98d05d94a9f9, cdabb9ac35cb) = 51b1739ae5fdcfe05ba915bed0f62304193d401c
git merge-tree --write-tree 98d05d94a9f9 cdabb9ac35cb  ->  exit 1, 3 content conflicts:
  loom/py/loom/gen/target/arch/amdgpu/amdgpu_target_info.py
  loom/py/loom/target/arch/amdgpu/target_info.py
  loom/src/loom/target/arch/amdgpu/target_info_defs.h

All three are AMDGPU target-info files in loom — a different repository from llama.cpp — and the
workflow's own rule is that "a conflict outside ggml/src/ggml-hrx, or in hrx-system, still fails the run".
So fixing the token will not by itself make the workflow succeed; the hrx-system merge needs a hand
resolution too, consistent with .gitmodules: "hrx-system is rebased by hand".

3. Why it broke silently for 12 days

The run's second failing step is "Report a failed sync so it cannot break silently" — and it failed
too. Whatever token/permission issue breaks the sync appears to break the reporting path as well, which is
how 12 consecutive scheduled failures (last success 2026-09-24) accumulated while this issue sat closed.

Method note

My first attempt at (2) used a shallow clone and reported fatal: refusing to merge unrelated histories. That is an artifact of truncated history, not a conflict; the full clone shows the merge base
above. Recorded because the shallow result looked exactly like a real blocker.

4. The hrx-system conflicts are benign — resolution prep

Both sides of the merge added the same AMDGPU scheduling bit, and AMD's side adds one more. From the
merge base 51b1739a:

ours:  + LOOM_AMDGPU_PROCESSOR_SCHEDULING_VALU_MASK_WRITE_DEPCTR = 1u << 9
amd:   + LOOM_AMDGPU_PROCESSOR_SCHEDULING_TENSOR_ISSUE_DRAIN    = 1u << 8
       + LOOM_AMDGPU_PROCESSOR_SCHEDULING_VALU_MASK_WRITE_DEPCTR = 1u << 9

So the resolution for loom/src/loom/target/arch/amdgpu/target_info_defs.h — and, by the same shape, for the
two generated Python mirrors — is take AMD's side: it is a strict superset of ours for these hunks. That
should be verified per hunk rather than applied blind (the same bit got the same value on both sides, which is
what makes the superset reading safe here), and the JSON/py mirrors must end up agreeing with the header, or
loom's target-info generation will detect the mismatch at build time.

This is the step that will still be needed after the token is fixed, because the workflow's own rule is that
a conflict in hrx-system fails the run. Nothing here has been pushed: it is a prepared resolution, not an
applied one.

Follow-up: the hrx-system blocker now has a single applicable patch — and it does not need the `workflows` scope

issue-332-hrx-union-merge.patch is the net change from our pin 98d05d94a9f9 to the resolved union state (444 lines, 3 files). Verified against a fresh worktree of our pin:

git worktree add --detach /tmp/vw 98d05d94a9f9
git -C /tmp/vw apply --check issue-332-hrx-union-merge.patch   -> APPLIES CLEANLY
after apply: TENSOR_ISSUE_DRAIN present, VALU_MASK_WRITE_DEPCTR present, 0 conflict markers

Because the patch touches no .github/workflows/* file, the missing workflows permission does not apply to it — so it can be pushed with the token that exists today. The two blockers are therefore separable:

blocker	fix	who
sync step fails (`workflows` scope)	add the permission to `HRX_BUMP_TOKEN`	repo owner
hrx-system merge conflicts	apply this patch to `1bit/main`	prepared and verified

Alternative route if the workflow should perform the merge itself once the token is fixed: run the merge, then issue-332-hrx-union-resolve.py, then commit (the original recipe).

Root cause of the silent failures: the same token lacks Issues write

The workflow header documents the intended scopes:

HRX_BUMP_TOKEN: Contents read/write on 1bit-MONSTER/llama.cpp and 1bit-MONSTER/engine,
and Pull requests read/write on 1bit-MONSTER/engine.

Absent from that list are Workflows (needed to push .github/workflows/* — exactly what the sync step does) and Issues (needed by the reporting step):

- name: Report a failed sync so it cannot break silently
  if: failure()
  env: { GH_TOKEN: ${{ secrets.HRX_BUMP_TOKEN }} }
  run: |
    existing=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search ...)
    if [ -n "$existing" ]; then gh issue comment ... ; else gh issue create ... ; fi

gh issue list/comment/create all need Issues read/write, which the token does not have — so the anti-silence mechanism is silenced by the same under-scoped token. That is why 12 consecutive scheduled failures went unreported while this issue sat closed.

One change fixes both symptoms. Give HRX_BUMP_TOKEN:

permission	why
Workflows: read/write	the sync step pushes `.github/workflows/*`
Issues: read/write	the reporting step files/comments the failure issue

with Contents R/W and Pull requests R/W as already documented (gh pr create needs the latter, which the token has, so the PR step is unaffected).

Reproduce the first symptom in ~5 s with gh workflow run bump-hrx.yml: it fails in the sync step with "Unable to determine if workflow can be created or updated due to timeout; workflows scope."

Fixed and verified: first green `bump-hrx` run since 2026-09-24

Run 37699244582 — completed: success, every step passing and the "report a failed sync" step correctly skipped:

success  Require the token
success  Read AMD's tested pair and ours
success  Sync the fork and merge AMD's pin into our line
success  Merge AMD's hrx-system pin into our line
success  Open the bump PR
skipped  Report a failed sync so it cannot break silently

It opened #357 — Bump HRX: llama.cpp 02f2880c9f42 + hrx-system 4ba76c18eafe — the first pin-bump PR from this workflow since 2026-09-24. All substantive checks pass, including build, which runs ctest registry_pins:

build pass 2m27s | Analyze (actions, c-cpp, python, rust) pass | CodeQL pass | pr_agent pending
state=OPEN  mergeable=MERGEABLE

#357 carries exactly the three expected files — third_party/llama.cpp, third_party/hrx-system, registry/architectures.json (+1/−1) — so the workflow now runs scripts/registry-regen.sh itself (line 233), the step I had to do by hand for #347.

What the 12 failures actually were — three independent causes, each reproduced locally:

Workflows scope on HRX_BUMP_TOKEN. The sync step pushes .github/workflows/*, and the token's documented scopes were Contents + Pull requests only. Reproduced in ~5 s by dispatch (failed with "…workflows scope"). Now working: the fork's 1bit/hrx-vulkan sits at AMD's current pin 02f2880c9f42.
The hrx-system merge conflicting in three loom AMDGPU target-info files. The workflow's own rule is that a conflict outside ggml/src/ggml-hrx, or in hrx-system, fails the run — and it correctly refuses to auto-resolve there. Now resolved on 1bit/main: AMD's pin c0b135a778cc is an ancestor, the merge is a clean no-op, and I verified the landed resolution independently — 0 conflict markers, both sides' members present (TENSOR_ISSUE_DRAIN, VALU_MASK_WRITE_DEPCTR, our DESCRIPTOR_SET_INFO_FLAG/LDS additions), and target_info_test.py + amdgpu_config_tables_test.py = 42 passed on that tree.
Why the 12 failures were silent: the anti-silence step itself used the same PAT and needed Issues, which it did not have. Fixed in bump-hrx: fix the output-key mismatch and modify/delete conflicts #354 by switching it to github.token with job-level permissions: issues: write — a better fix than granting the scope to the PAT.

Also fixed while this was open: the re-run failure fatal: tag 'main-4ba76c18eafe' already exists → git tag -f, and modify/delete conflicts in the resolution loop (both #354).

Credit: I diagnosed the three causes and verified the outcome end to end; the fix commits are #354 (merged), the hrx-system resolution now on 1bit/main, and the registry step. #356 documents the token permission separately.

Closing this issue: the reporting step now works, so a future regression will file its own issue rather than going silent for twelve days — "closed and silent" is no longer the failure mode.

Navigation Menu

Repository navigation

bump-hrx: the scheduled HRX pin sync is failing #332

Diagnosis, and the resolution this needs

The failing step is the merge, and a ready resolution — blocked on the same scope

Root cause: the bump token lacks the `workflows` scope

2. A second, independent blocker: the `hrx-system` merge conflicts

3. Why it broke silently for 12 days

Method note

4. The hrx-system conflicts are benign — resolution prep

Follow-up: the hrx-system blocker now has a single applicable patch — and it does not need the `workflows` scope

Root cause of the silent failures: the same token lacks Issues write

Fixed and verified: first green `bump-hrx` run since 2026-09-24

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bump-hrx: the scheduled HRX pin sync is failing #332

Description

Activity

bong-water-water-bong commented on Oct 6, 2026

Diagnosis, and the resolution this needs

bong-water-water-bong commented on Oct 6, 2026

The failing step is the merge, and a ready resolution — blocked on the same scope

bong-water-water-bong commented on Oct 6, 2026

bong-water-water-bong commented on Oct 6, 2026

bong-water-water-bong commented on Oct 7, 2026

Root cause: the bump token lacks the workflows scope

bong-water-water-bong commented on Oct 7, 2026

2. A second, independent blocker: the hrx-system merge conflicts

3. Why it broke silently for 12 days

Method note

4. The hrx-system conflicts are benign — resolution prep

bong-water-water-bong commented on Oct 7, 2026

Follow-up: the hrx-system blocker now has a single applicable patch — and it does not need the workflows scope

bong-water-water-bong commented on Oct 7, 2026

Root cause of the silent failures: the same token lacks Issues write

bong-water-water-bong commented on Oct 7, 2026

Fixed and verified: first green bump-hrx run since 2026-09-24

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Root cause: the bump token lacks the `workflows` scope

2. A second, independent blocker: the `hrx-system` merge conflicts

Follow-up: the hrx-system blocker now has a single applicable patch — and it does not need the `workflows` scope

Fixed and verified: first green `bump-hrx` run since 2026-09-24