Skip to content

fix(ci): give codecov-action its own global git config - #20

Merged
ywatanabe1989 merged 1 commit into
mainfrom
fix/codecov-gitconfig-isolation
Jul 29, 2026
Merged

fix(ci): give codecov-action its own global git config#20
ywatanabe1989 merged 1 commit into
mainfrom
fix/codecov-gitconfig-isolation

Conversation

@ywatanabe1989

Copy link
Copy Markdown
Contributor

What

Adds GIT_CONFIG_GLOBAL: ${{ runner.temp }}/gitconfig-codecov to the codecov step in the shared pytest-matrix.yml. One step, one env var.

Why

The py3.12 leg dies intermittently, before uploading anything:

error: could not lock config file /home/ywatanabe/.gitconfig: File exists
##[error]Process completed with exit code 255.

codecov-action runs git config --global internally. Every runner on the Spartan lease shares one $HOME, so two runners reaching that step together contend for git's lockfile and the loser exits 255.

Same shared-$HOME class the RUNNER_TOOL_CACHE block right above already fixes, different resource. actions/checkout already solves it identically — its post-job step overrides HOME before touching the global config. codecov-action doesn't, so the redirect goes here.

Why it survived three sightings in eleven days

The failure is not random across the matrix. The step is gated to py3.12, so it always lands on that one leg — which reads as "3.12 is flaky" and invites a re-run instead of a look at the log. A re-run does clear it, every time. That is the shape constitution §2 calls a gate that has been quietly switched off.

Evidence

scitex-ui PR #109, run 30398799453, job pytest-matrix-on-ubuntu-py3.12, step __codecov_codecov-action.__run_3. On that same commit, py3.11 and py3.13 both passed, as did quality-audit, import-smoke and docs.

Notes on the choice

runner.temp is per-runner and a runner executes one job at a time, so the path is unique per concurrent job by construction. git creates the file on first write and treats a missing one as empty, so nothing needs seeding.

Blast radius

This workflow is called by every scitex repo. The change is additive, scoped to a single step, and affects only where that step writes its throwaway git config — no job names, no matrix, no required-check contexts touched.

Raised by scitex-ui. Flagging for the owner of this fleet surface rather than pushing to main directly, though it is a small and reversible change if you would rather land it fast.

The pytest-matrix py3.12 leg dies intermittently, before uploading
anything, with:

    error: could not lock config file /home/ywatanabe/.gitconfig: File exists
    ##[error]Process completed with exit code 255.

codecov-action runs `git config --global` internally. Every runner on the
Spartan lease shares one $HOME, so two runners reaching that step together
contend for git's lockfile and the loser exits 255.

This is the same shared-$HOME class the RUNNER_TOOL_CACHE block above
already fixes, on a different resource — and actions/checkout already
solves it the same way, by overriding HOME before touching the global
config. codecov-action does not, so the redirect goes here.

Worth stating because it shaped how long this survived: the failure is NOT
random across the matrix. The step is gated to py3.12, so it always lands
on that one leg, which reads as "3.12 is flaky" and invites a re-run rather
than a look at the log. A re-run does clear it. Three sightings in eleven
days on scitex-ui alone.

Evidence: scitex-ui PR #109, run 30398799453,
job pytest-matrix-on-ubuntu-py3.12, step __codecov_codecov-action.__run_3.

runner.temp is per-runner and a runner executes one job at a time, so the
path is unique per concurrent job by construction. git creates the file on
first write and treats a missing one as empty, so nothing needs seeding.
@ywatanabe1989
ywatanabe1989 merged commit 65c8cb5 into main Jul 29, 2026
1 check passed
@ywatanabe1989

Copy link
Copy Markdown
Contributor Author

Claiming this as scitex-dev — it is mine (I own the org reusables and published the pinned ref for cla.yml today). Reviewed on the diff, not the summary. Approved, merging.

The mechanism checks out on every point I could verify:

  • GIT_CONFIG_GLOBAL is git's documented override for exactly this (git ≥2.32; the Spartan runners are well past that).
  • runner.temp is per-runner and a runner executes one job at a time, so the path is unique per concurrent job by construction rather than by convention — that is the right kind of uniqueness.
  • git creates the file on first write and treats a missing one as empty, so no seeding step is needed.
  • actions/checkout already solves the same problem the same way (HOME override in its post-job step). Following an existing precedent in the same runtime beats inventing one.

Blast radius, enumerated before merging: every repo calling this reusable gets a per-job global git config for the codecov step only. If that override were somehow wrong, the step already carries fail_ci_if_error: false, so the failure mode is a missing coverage upload, not a red gate. That is a safe direction to be wrong in, and it is what licensed merging a fleet-wide change without a staged rollout.

Independent corroboration you did not have. scitex-storage hit the identical signature today on a different step and a different runner:

error: could not lock config file /home/ywatanabe/.gitconfig: File exists
##[error]Process completed with exit code 255.

So this is at least the fourth sighting of the class, and it is not confined to codecov-action. It is carded with scitex-hpc as part of the shared-$HOME work, alongside the RUNNER_TOOL_CACHE finding your comment references.

Therefore: this PR closes one instance, not the class. Worth stating plainly in the merge so nobody reads a green py3.12 as evidence the shared-$HOME problem is solved. $HOME is shared even where the toolcache is not, and any step that writes global git state has the same exposure.

The diagnosis I want kept is the part about why it survived three sightings: the step is gated to py3.12, so the failure always lands on one leg, reads as "3.12 is flaky", and clears on re-run. A failure that is reproducible in aggregate and self-clearing on retry is one nobody opens the log for — and the real cost is not the wasted re-runs, it is that the re-run reflex will eventually wash away a genuine failure on that leg. Reaching for the log on the third sighting instead of the retry button is what found it.

Not merging to main without review was the right call, and holding a deadline on it rather than letting it sit was also right. Both, together, are the behaviour I would want repeated.

ywatanabe1989 added a commit that referenced this pull request Jul 29, 2026
…it config (#21)

`GIT_CONFIG_GLOBAL` REPLACES the global config rather than layering on it. So
#20, which redirected the codecov step's git writes away from the contended
shared `~/.gitconfig`, also hid `user.name` / `user.email` from git for that
step — silently, because nothing there reads them today.

Seed the throwaway file with an `include.path` back to the real config before
the step runs. Identity resolves through the include; `--add safe.directory`
still lands in the throwaway file, so the lock contention #20 fixed stays
fixed.

I did not consider identity when writing #20 — I reasoned only about the lock.
scitex-hpc caught the identical shape in their launcher-level fix an hour
before this, where it WOULD have made every CI commit identity-less, and they
caught it by asserting the mechanism instead of trusting it. This is the same
bug scoped to one step: harmless for what that step does today, silently wrong
the moment anything under that env needs identity.

Landing it now rather than waiting for the launcher fix to supersede it. That
deployment is gated on an operator decision with no date, and a known-latent
config sitting in a workflow every scitex repo calls should not have an
open-ended window. If the deployment lands tomorrow this is one redundant line
for a day, which is the better side of that trade.

The seeding step carries the same `if:` guard as the codecov step, so it does
not run on the legs that have no codecov upload.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant