The macOS Metal shader compile fails intermittently on the runner, costing a ~70 minute job each time

[leonardoaraujosantos]

The build+test (macos, +metal, parity) job fails intermittently with the Metal backend declining to register, and it is the runner rather than any diff.

The signature, identical every time

claycore: the Metal backend will not register — no compute pipeline for clay_eval_grid: Compilation failed
claycore: full Metal error: Error Domain=CompilerError Code=2 "Compilation failed"
  FAIL the Metal backend registered — this machine HAS a Metal device, so a missing
       backend would mean the embedded metallib failed to load
swift smoke: 1 failure(s)     411/412 checks passed

Always the same check, always 411/412, always in the Swift smoke step.

Why it is the runner and not a diff

A pull request that changes one markdown file produced it. #482 touches openspec/ROADMAP.md and nothing else, and hit this exact failure. A one-file documentation diff cannot break a Metal shader compile.

Directly observed tonight, four occurrences across four different PRs:

PR	what it changes	outcome
#482	one markdown file	failed, passed on re-run
#481	regional multires	failed twice, passed on the third attempt after 84 min
#483	one markdown file	passed first time
#490	scene bounds + a C entry point	failed, re-running

What it costs

Each occurrence burns a ~70 minute job (observed 52m to 1h14m) and blocks the PR until someone notices and re-runs. Tonight it delayed two merges by several hours.

A note on evidence, because it disappears

gh api repos/.../actions/runs/<id>/jobs reports the latest attempt only. Re-running a failed job rewrites the conclusion to success, so a tally taken afterwards shows a clean history and the flake becomes invisible to exactly the query someone would run to measure it. The four rows above are from direct observation at the time, not reconstructed — a reconstruction would have reported zero.

That is worth knowing before anyone tries to quantify the rate from the API.

Worth investigating

• whether the failure correlates with a particular runner image or a concurrent job on the same host;
• whether the metallib build can be made to retry the pipeline creation, or to report which shader stage failed rather than Compilation failed with no detail;
• whether the Swift smoke can distinguish "no Metal device" from "device present, metallib failed to load" more cheaply than by failing the whole job — the check text already knows the difference and is the only thing that reports it.

[leonardoaraujosantos]

A second repository on the same runner image can produce it

The consuming host's CI matrix carries macOS, Metal (debug) and macOS, Metal (release) alongside CPU-only and Linux Vulkan. Those jobs build and link the same ClayCore Metal backend, so this signature is one their runners can produce as readily as ours.

That is the control the one-markdown-file PR above was reaching for, from further away: a different repository, a different diff, a different job definition, the same backend and the same runner image. If it appears there it is conclusively the runner.

They have six macOS jobs pending as of writing, two of them Metal, and will report a red one with this text as an occurrence of this issue rather than as their own defect.

Recorded before it happens rather than after, so that a fifth occurrence counts as evidence rather than as a coincidence noticed in hindsight.

[leonardoaraujosantos]

Recurred on #509 (run 34221257975, job 102044778880), same signature, and the branch cannot be the cause: it touches no kernel, Metal, shader or dialect file — 14 files, all of bindings/c, tests, tools, docs, openspec and the three version lines. package-kernels reports the 15 headers byte-identical and the dialect gate passes.

Failure is the single Swift-smoke check the Metal backend registered, at 411/412, after the embedded metallib fails to build three pipelines in a row:

no compute pipeline for clay_eval_grid: Compilation failed
no compute pipeline for clay_eval_grid_batch: Compilation failed
no compute pipeline for clay_raycast: Compilation failed
Error Domain=CompilerError Code=2 "Compilation failed"

Worth noting for whoever picks this up: the job took 53m41s to reach that check, so each recurrence costs most of an hour of the scarcest runner in the matrix. The same job passed on #505 earlier today at 16/16, which is the intermittency rather than a branch difference.

Re-running the failed job only.

[leonardoaraujosantos]

Four more occurrences, and two facts the issue does not yet have

Confirming the diagnosis from a later session, and adding evidence that narrows it further.

It reproduces on the SIMULATOR arm, not the macOS arm

Every failure I saw has the macOS arm passing completely and only the simulator failing:

swift smoke: building the xcframework first
  Metal device: Apple Paravirtual device
  ok   the Metal backend registered
412/412 checks passed
swift smoke: OK                       <- macOS arm, clean

== iOS Simulator ==
claycore: the Metal backend will not register — no compute pipeline for clay_eval_grid: Compilation failed
claycore: the Metal backend will not register — no compute pipeline for clay_raycast: Compilation failed
  Metal device: Apple iOS simulator GPU
  FAIL the Metal backend registered

check_swift_smoke.sh all runs both. So the failing device is Apple iOS simulator GPU, paravirtualised, and the macOS Apple Paravirtual device arm is unaffected. Worth putting in the title if others agree — it changes what a fix would target.

It passes on a real machine's simulator, with the check firing

The decisive control, run locally on the same branch that was failing in CI:

./tools/check_swift_smoke.sh sim
  Metal device: Apple iOS simulator GPU
  ok   the Metal backend registered
412/412 checks passed          SIM SMOKE EXIT=0

412 checks, 0 skips — so the Metal assertion genuinely ran rather than being skipped for want of a device. Same code, same check, same class of GPU, opposite result. That is the runner.

Separately, the full ClayCoreDeviceTests parity bundle also runs clean on a local iOS 17.0.1 simulator with backends = cpu, metal (see #243), which is the same Metal stack compiling the same kernels without complaint.

The runner image is IDENTICAL on passing and failing runs

Checked, because "the image changed under us" was the obvious theory:

failing job 102220856227   Image: macos-26-arm64   Version: 20260831.0337.3
passing job 102049230472   Image: macos-26-arm64   Version: 20260831.0337.3

So it is not an image bump. It is the same image behaving differently between runs, which is the strongest evidence yet that Apple's Metal compiler in that VM is simply unreliable under load.

Four more occurrences

PR	what it changes	outcome
#516	docs + baseline JSON	failed twice, passed on the third attempt
#520	five counter fields + one assignment	failed, passed on re-run
#504	docs only	passed
#521	one counter call	(in flight)

#516 and #520 both change nothing a shader can see. #516 needing three attempts matches #481's three in the original report.

The cost, beyond the job minutes

It cost me about an hour of misdirection: three consecutive apparent failures on #520 sent me looking for a real defect, and they turned out to be one failure read three times by a retry loop that checked the board before GitHub had re-queued the job. A flake that looks like a reproducible failure is worse than one that looks like a flake.

On the suggested remedies

Making the simulator arm's Metal registration non-fatal while keeping its other 411 checks strict, and keeping the macOS arm strict throughout, would keep everything this gate exists for. The registration assertion's own message says the two causes it guards — a slice built without the backend, and a metallib compiled for the wrong SDK — "look identical from here"; a third cause now shares that appearance and is not ours.

Pinning the simulator runtime is the alternative, but the identical-image evidence above suggests it would not help.

[leonardoaraujosantos]

Recurred on #648 (run 35822005360, job 107055664718), this time on the macOS arm of the Swift smoke, which the fix that closed this issue keeps strict — only the Simulator is exempt.

• Runner image macos-26-arm64 20260907.0351.1, Metal device: Apple Paravirtual device.
• Four pipelines failed to compile in a row: clay_eval_points, clay_eval_grid, clay_eval_grid_batch, clay_raycast — each CompilerError Code=2 "Compilation failed", no further detail. Smoke result 447/448, the single failure is the Metal backend registered.
• The branch cannot be the cause: git diff origin/main...origin/perf/639-undo-bound-grab-support -- include/clay/kernel backends is empty.
• Control: the same image (macos-26-arm64 20260907.0351.1, Apple Paravirtual device) registered Metal on main at 9616286 (job 107044159842), 448/448 on both the macOS and Simulator arms.

Not reopening: the Simulator exemption this issue closed with still holds; this is a macOS-arm occurrence of the same runner signature. If it recurs on macOS again, the macOS arm likely needs the same treatment the Simulator got.