Skip to content

Runner workdir keeps a stale submodule index.lock, so checkout fails on private-cf463-01-7 #1661

Closed

Description

@d-morrison

On 2026-11-07 22:46 PST, lint-changed-files on epi204#417 (run 42456055290, job 128655195566, runner private-cf463-01-7 on cf463-01) failed in actions/checkout@v7 during git submodule update --init --force --depth=1 --recursive:

fatal: Unable to create '/tmp/gha-work/private-cf463-01-7/epi204/epi204/.git/modules/rme/index.lock': File exists.
fatal: Unable to checkout 'c5498c4...' in submodule path 'rme'

The workdir was reused from a prior epi204 job (a previous merge ref was still checked out), and that job left a lock file in the submodule gitdir. prune-runner-workdir.sh freed 0 MB, so it did not clear it.

Wanted: the job-started hook (or prune-runner-workdir.sh) removes stale *.lock files under .git/modules when no git process is running in that workdir, so one crashed or cancelled job cannot fail the next job on the same runner. The failed job was re-run once.


Posted by Claude Code (AI agent) --- not written by a human.
Generated by Claude Code

Activity

  1. d-morrison commented on Nov 8, 2026

    @d-morrison
    CollaboratorAuthor

    Recurrence, 2026-11-07 22:47 PST: render-solution on epi204#417 (job 128655195766) hit the identical .git/modules/rme/index.lock: File exists on the same runner private-cf463-01-7, about a minute after the lint job. So the lock is persistent, not transient, and re-running keeps failing there until it is removed. Every epi204 job that lands on that runner is blocked. Quickest unblock: delete /tmp/gha-work/private-cf463-01-7/epi204/epi204/.git/modules/rme/index.lock (or the whole private-cf463-01-7/epi204 workdir) on cf463-01 once no git process is using it. I cannot do that from this session.

    Posted by Claude Code (AI agent) --- not written by a human.


    Generated by Claude Code

  2. d-morrison commented on Nov 8, 2026

    @d-morrison
    CollaboratorAuthor

    Removed /tmp/gha-work/private-cf463-01-7/epi204/epi204/.git/modules/rme/index.lock on cf463-01 (2026-11-07 22:5x PST). Before removal: 0 bytes, mtime 22:44:40 and unchanged for over 11 minutes, no open handle (lsof), and no git process with a cwd under that epi204 workdir. Runner -7's active git processes were in its lds checkout.

    Three other locks were left alone because each had a live git process in its directory: private-cf463-01-2 (lds, gh-pages worktree), private-cf463-01-6 (lds) and private-cf463-01-8 (epi204, rme submodule).

    Still open: a pre-checkout cleanup step. The safe signal is a lock with no git process in that workdir and an mtime older than a few minutes. It must be cheap and time-bounded, since cf463-01 runs at load average about 17.


    Posted by Claude Code (AI agent) --- not written by a human.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions