fix(pytest-matrix): pin uv, because the per-runner cache fix cannot fire - #24
Merged
Merged
Conversation
The RUNNER_TOOL_CACHE override on the setup-uv step has never taken
effect, and cannot. Measured, not inferred.
WHAT THE OVERRIDE WAS FOR, and the diagnosis in its comment is correct:
~80 runners share one mutable tool cache, @actions/tool-cache's writer
rmRF's <ver>/<arch> before refilling it, and its reader treats
dir + .complete as a HIT without checking the binary is inside. So
concurrent jobs delete the tree each other is reading or filling, and
the reader then puts a binary-less directory on PATH.
WHY IT CANNOT FIRE. scitex-cards PR #667, run 30568910438: all three
matrix legs -- including the py3.12 leg that PASSED -- resolved
/home/ywatanabe/.runner-toolcache, not the per-runner path. A probe
(scitex-cards branch probe/toolcache-env, run 30571092482) then asked
the same question of a shell step and of the JS action with distinct
sentinel paths:
runner.name = spartan-pooled-cpu-01
RUNNER_TOOL_CACHE = /home/ywatanabe/.runner-toolcache
AGENT_TOOLSDIRECTORY = /home/ywatanabe/.runner-toolcache
shell step, env: RUNNER_TOOL_CACHE=/tmp/probe-shell/<runner>
observed inside the step: /home/ywatanabe/.runner-toolcache
JS action, env: RUNNER_TOOL_CACHE=/tmp/probe-jsaction/<runner>
setup-uv resolved: /home/ywatanabe/.runner-toolcache
The override fails even for a plain shell step, so this is not a
setup-uv bug and not a JS-action quirk: the runner re-injects
RUNNER_TOOL_CACHE from AGENT_TOOLSDIRECTORY when it builds each step's
environment, after merging the user's env:. No workflow-layer override
of that variable can win. The existing fix is configured, visible,
believed to work, and inert -- which is worse than absent, because its
presence stopped anyone looking further.
WHAT THIS COMMIT DOES. It does not try to fix the variable again. It
removes the TRIGGER instead. The destructive writer runs only on a cache
MISS; on a HIT the action calls find() + addPath and writes nothing.
Unpinned, `version` resolves to `latest`, so an upstream release turns
every concurrent job into a simultaneous miss -- exactly the storm the
original comment describes. Pinning makes every job a hit.
The shared cache is healthy right now (probe: `which uv` ->
/home/ywatanabe/.runner-toolcache/uv/0.12.0/x86_64/uv, uv 0.12.0), so
pinning to 0.12.0 pins to a good tree rather than freezing a broken one.
This is a mitigation and is labelled as one in the comment. The HAZARD
-- one mutable directory shared by every runner -- is only fixable where
the value is set, in the runner supervisor, by giving each runner its
own AGENT_TOOLSDIRECTORY. That is filed for scitex-hpc, who own the
runners. Both belong in place; neither replaces the other.
Independently justified: an unpinned toolchain silently changes what CI
runs whenever an upstream release lands, which is the opposite of a
reproducible build.
Bumping the pin re-opens the window for one run, so the comment says to
warm the new version on the pool first, or bump while it is idle.
This was referenced Jul 31, 2026
ywatanabe1989
added a commit
that referenced
this pull request
Jul 31, 2026
…day's pool (#26) Operator ruling 2026-07-31: GitHub-hosted runners are the DEFAULT CHOICE for new work. Most scitex repositories are public, so hosted minutes are free and the cost objection that justified blanket self-hosting mostly does not apply. Self-hosted stays for jobs where our own machines are genuinely faster -- long builds, GPU, large data -- chosen because it wins, not by policy. BACKWARD COMPATIBLE BY CONSTRUCTION. The new `runs_on` input defaults to '["self-hosted","Linux","X64","spartan-cpu"]', byte-identical to the value that was hardcoded, so all 28 callers keep exactly today's behaviour. Only a repo that passes the input moves. WHY A CALLER WANTS OUT TODAY. Every runner on the Spartan lease inherits one mutable ~/.runner-toolcache on GPFS, so runners contend for a single directory. Jobs fail BOTH reading it (`uv: No such file` -> exit 127) and writing it (ENOENT during copyfile INTO the cache) -- one bug seen from opposite ends. Measured on scitex-cards 2026-07-31: seven failures across five pull requests in one afternoon, four differently-named runners, every one cleared by a plain rerun. This workflow's own comments already describe the hazard, and #24 records that the per-runner fix "cannot fire". The durable fix is still a per-runner AGENT_TOOLSDIRECTORY in the supervisor. This input is not that fix; it is an escape hatch for callers that cannot absorb random reruns while that work is pending. THE FORK GUARD NEEDED A REAL FIX, NOT A TEST UPDATE. Parameterising `runs-on` made the guard's selector stop recognising this job: it matched the literal label "self-hosted", and an expression is not that string. The job silently left the guarded set, so the fork guard would have become unenforced AND untested in the same moment -- the suite would have stayed green while the protection disappeared. The tests caught it. _may_run_self_hosted now answers YES for an expression-valued `runs-on`, because such a job CAN resolve to self-hosted and its default here does. Erring towards guarded is the only safe direction: a false yes costs one harmless step on a hosted runner, a false no removes a security boundary quietly. 74 passed, up from 72.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(pytest-matrix): pin uv, because the per-runner cache fix cannot fire
The RUNNER_TOOL_CACHE override on the setup-uv step has never taken
effect, and cannot. Measured, not inferred.
WHAT THE OVERRIDE WAS FOR, and the diagnosis in its comment is correct:
~80 runners share one mutable tool cache, @actions/tool-cache's writer
rmRF's / before refilling it, and its reader treats
dir + .complete as a HIT without checking the binary is inside. So
concurrent jobs delete the tree each other is reading or filling, and
the reader then puts a binary-less directory on PATH.
WHY IT CANNOT FIRE. scitex-cards PR #667, run 30568910438: all three
matrix legs -- including the py3.12 leg that PASSED -- resolved
/home/ywatanabe/.runner-toolcache, not the per-runner path. A probe
(scitex-cards branch probe/toolcache-env, run 30571092482) then asked
the same question of a shell step and of the JS action with distinct
sentinel paths:
runner.name = spartan-pooled-cpu-01
RUNNER_TOOL_CACHE = /home/ywatanabe/.runner-toolcache
AGENT_TOOLSDIRECTORY = /home/ywatanabe/.runner-toolcache
shell step, env: RUNNER_TOOL_CACHE=/tmp/probe-shell/
observed inside the step: /home/ywatanabe/.runner-toolcache
JS action, env: RUNNER_TOOL_CACHE=/tmp/probe-jsaction/
setup-uv resolved: /home/ywatanabe/.runner-toolcache
The override fails even for a plain shell step, so this is not a
setup-uv bug and not a JS-action quirk: the runner re-injects
RUNNER_TOOL_CACHE from AGENT_TOOLSDIRECTORY when it builds each step's
environment, after merging the user's env:. No workflow-layer override
of that variable can win. The existing fix is configured, visible,
believed to work, and inert -- which is worse than absent, because its
presence stopped anyone looking further.
WHAT THIS COMMIT DOES. It does not try to fix the variable again. It
removes the TRIGGER instead. The destructive writer runs only on a cache
MISS; on a HIT the action calls find() + addPath and writes nothing.
Unpinned,
versionresolves tolatest, so an upstream release turnsevery concurrent job into a simultaneous miss -- exactly the storm the
original comment describes. Pinning makes every job a hit.
The shared cache is healthy right now (probe:
which uv->/home/ywatanabe/.runner-toolcache/uv/0.12.0/x86_64/uv, uv 0.12.0), so
pinning to 0.12.0 pins to a good tree rather than freezing a broken one.
This is a mitigation and is labelled as one in the comment. The HAZARD
-- one mutable directory shared by every runner -- is only fixable where
the value is set, in the runner supervisor, by giving each runner its
own AGENT_TOOLSDIRECTORY. That is filed for scitex-hpc, who own the
runners. Both belong in place; neither replaces the other.
Independently justified: an unpinned toolchain silently changes what CI
runs whenever an upstream release lands, which is the opposite of a
reproducible build.
Bumping the pin re-opens the window for one run, so the comment says to
warm the new version on the pool first, or bump while it is idle.
Blast radius: every caller of this reusable workflow (~58 repos) gets a pinned uv instead of
latest. They are all resolving to 0.12.0 today anyway, so the immediate behaviour is unchanged; what changes is that a future upstream release no longer lands on all of them simultaneously.Reversible: delete the
version:line.Follow-up, filed separately for the runner owner:
ci-agent-toolsdirectory-must-be-per-runner-20260731— setAGENT_TOOLSDIRECTORYper runner in the supervisor. That is the only place the shared-directory hazard can actually be removed; this PR only removes the trigger.Regression test:
scitex-cardsbranchprobe/toolcache-env(one job, installs nothing extra). After the supervisor change it should showAGENT_TOOLSDIRECTORYdiffering per runner.🤖 Generated with Claude Code
https://claude.ai/code/session_01QQBFHZzKnEAa54hpZia3Vc