Skip to content

troubleshooting.md

Latest commit

 

History

History
154 lines (113 loc) · 5.73 KB

troubleshooting.md

File metadata and controls

154 lines (113 loc) · 5.73 KB
Raw

Troubleshooting

Symptoms and causes specific to a hand-rolled, persistent, emulated runner. For how the pieces fit together see architecture.md.

Stale "offline" runner in GitHub

The runner still appears in Settings → Actions → Runners after docker compose down.

Deregistration did not complete. Check docker compose logs for WARNING: config.sh remove failed. Usual causes:

  • Killed before the trap ran. A job was in flight and exceeded stop_grace_period (2m), so Docker sent SIGKILL. Raise it in docker-compose.yml.
  • PAT expired or lost Administration: read & write. The removal token is minted fresh at shutdown, so an expired PAT breaks teardown even though startup worked earlier.
  • Container removed with docker rm -f, which skips SIGTERM entirely.

Harmless to fix by hand: delete the entry in the GitHub UI. --replace on the next start also reclaims a runner of the same name.

Registration fails at startup

Log line Cause
ERROR: REPO_URL must be an https:// URL An ssh remote in .env. Use the https://github.com/owner/repo form
curl … 404 from token.sh Wrong REPO_URL, or the PAT is not scoped to that repository
curl … 403 from token.sh PAT lacks Administration: Read and write
jq: error … null GitHub returned a response without a token — usually the 403 above
Http response code: NotFound from 'POST …/actions/runner-registration' Registration token expired before config.sh used it. Restart

Test the PAT directly:

curl -fsS -X POST \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/OWNER/REPO/actions/runners/registration-token

Gradle build daemon disappeared unexpectedly

The job fails with DaemonDisappearedException and a daemon log that stops mid-build. This is almost always the kernel OOM-killer, not a crash. The tell is that Gradle never printed java.lang.OutOfMemoryError — the JVM did not exhaust its own heap, it was SIGKILLed from outside.

Confirm it against the Docker VM's kernel log:

docker run --rm --privileged --pid=host alpine \
  nsenter -t 1 -m -u -n -i dmesg | grep -iE "killed process|out of memory"

A hit naming task=java and this container's ID is proof:

Out of memory: Killed process 6187 (java) … anon-rss:3291992kB
task_memcg=/docker/f656a5eb52ba…   global_oom

global_oom means the whole VM ran out, not a per-container limit — /sys/fs/cgroup/memory.max is max here, so the container is only bounded by Docker Desktop's allocation.

Compare what Gradle asks for against what exists:

docker compose exec runner free -m                          # VM total
grep jvmargs ../utilityvault/gradle.properties              # what the build wants

Observed 2026-07-29: -Xmx4g requested inside a 3.83 GB VM on an 8 GB Mac. Unwinnable. org.gradle.parallel=true and org.gradle.workers.max=4 make it worse by forking further JVMs alongside the daemon.

Fixes, in order of leverage:

  • Cap the heap for the self-hosted path only — in ci.yml, guarded by inputs.runner, never by editing the target repo's gradle.properties (4g is correct for local development on the same machine).
  • Raise Docker Desktop → Resources → swap before raising memory. Swap absorbs a short peak; on an 8 GB host there is little RAM left to give.
  • Give the runner a cheaper job. See decisions.md § memory ceiling.

Builds are extremely slow

Expected to some degree — the image runs under x86_64 emulation on Apple Silicon (see decisions.md). Check Docker Desktop → General → "Use Rosetta for x86_64/amd64 emulation" is enabled; without it, emulation falls back to QEMU and is far slower.

Also verify the Gradle cache is actually persisting:

docker compose exec runner du -sh /cache/gradle

A few hundred MB after the first build is normal. Near-zero means GRADLE_USER_HOME is not landing on the volume.

Disk filling up

docker compose exec runner du -sh /_work /cache/gradle /opt/hostedtoolcache
docker system df

hooks/post-job.sh prunes the workspace only above BUILD_CACHE_MAX_MB (default 5000) — lower it in .env. Note it deliberately never touches /cache/gradle, which is the expensive cache worth keeping.

gh: command not found in a workflow

The gh CLI is installed by the Dockerfile, so this means the image is stale relative to it:

docker compose up -d --build

Same applies to any tool added to the Dockerfile after the image was last built.

Job fails with a missing SDK component

The baked SDK covers platforms;android-36.1 and build-tools;36.0.0. Note the .1: compileSdk's minorApiLevel selects a different package, so a pin of 36 looks right and still makes AGP download 36.1 mid-build. If Gradle wants something else, add it to the sdkmanager --install line in the Dockerfile and rebuild — see decisions.md.

Workflow queues forever, never picks up

runs-on: labels do not match the runner. Compare the workflow's runs-on: list against RUNNER_LABELS in .env and the labels shown in the GitHub UI. Every label in runs-on: must be present on the runner.

For ci.yml specifically, the more likely cause is that the run was not dispatched with runner=self-hosted — PRs always resolve to a hosted runner by design, and there is no push trigger at all. See workflows.md.

Verifying the scripts after an edit

./test_token.sh
bash -n entrypoint.sh token.sh hooks/post-job.sh