You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
odroid stage-qa host root filesystem at 100% — stage QA fails with ENOSPC until caches are manually cleared #248
During stage-qa of #246 (2026-11-07 08:39Z, run 42356356024, runner odroid-esb-qa, host 65.21.40.51) the run failed with OSError Errno 28 No space left on device in stage-qa-browser.py run_smoke. Root fs /dev/mapper/ubuntu--vg--ubuntu--lv: 249G total, 238G used, 2.5M free (100%). Manual freeing (npm cache clean --force ~260MB) let the next run pass (42357873301), i.e. the stage QA path is one tempfile away from random ENOSPC failures. Guard-blocked cleanup that a human should review: uv cache clean (shared package-store for venvs) and stale agent sandboxes ~3.5GB dated Oct 22–Nov 2 under the system temp dir. Needs: identify the real space consumers (docker images? old deploy checkouts? never-delete log retention volume?), a monitoring alert before 95%, and a safe automated cleanup for QA sandboxes.
Why (motivation)
Stage QA is a required per-PR check; a full disk turns it into random infra failures that block every PR — this exact failure blocked #246 until caches were cleared by hand.
User impact
Any PR can randomly fail its required stage-qa check; agents burn time diagnosing infra instead of code.
Cost of inaction
Recurring ENOSPC failures will keep blocking PRs unpredictably.
Acceptance criteria
Root cause of the 238G identified with the top consumers listed and evidence
Sustained free space >= 20G on the odroid root fs after cleanup
An alert fires before the filesystem reaches 95% again
Screenshots
(none)
Deploy
Deployed: no
Links
(none)
Skipped gates
msgref-quote: Observed directly in CI run logs and host df output on 2026-11-07 while shipping the sync-import fix
screenshots: CLI/infra evidence: failing run 42356356024 vs passing 42357873301, df output quoted in the body
Read-only scan from the bot session (2026-11-07): df on the stage host shows 249G/237G, 470M free (100%). The system temp dir alone holds 9.3G, including ~3.5G of stale agent/QA sandboxes dated Oct 22–Nov 2 (agent deletion was guard-blocked; an owner rm on those sandbox dirs is the quick win). No passwordless sudo on the host, and du over home or root times out (>60s per top dir) under current IO pressure, so the full consumer list needs an owner shell. Suggested first pass:
sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20
docker system df
then docker images/build cache, journald vacuum (journalctl --vacuum-size), and old deploy checkouts; the never-delete log retention dirs should be measured explicitly before any policy change.
Update 2026-11-07 ~16:2xZ: free space is now 6.5G (was 470M) — stage-qa is no longer blocked, so this ticket is no longer urgent for PR flow. Breakdown of the temp-dir junk found (36747 top-level entries): ~28900 dirs older than 2 days, dominated by one-shot stage-qa browser artifacts (playwright-artifacts-, playwright_chromiumdev_profile-) and 275+ stale zero-byte .hm markers. Agent-side remote rm is removal-guard-blocked (no bypass), so the exact one-liner (find -mtime +2 on the playwright dirs + stale .hm markers + df) was handed to the owner in Telegram (tg#28147).
Remaining for acceptance: top-consumer audit of the other ~230G (sudo du -xh --max-depth=1 /), the >=20G sustained-free criterion, and the 95% alert.
Activity
alex-mextner commentedon Nov 7, 2026
Read-only scan from the bot session (2026-11-07):
dfon the stage host shows 249G/237G, 470M free (100%). The system temp dir alone holds 9.3G, including ~3.5G of stale agent/QA sandboxes dated Oct 22–Nov 2 (agent deletion was guard-blocked; an ownerrmon those sandbox dirs is the quick win). No passwordless sudo on the host, andduover home or root times out (>60s per top dir) under current IO pressure, so the full consumer list needs an owner shell. Suggested first pass:then docker images/build cache, journald vacuum (
journalctl --vacuum-size), and old deploy checkouts; the never-delete log retention dirs should be measured explicitly before any policy change.alex-mextner commentedon Nov 7, 2026
Update 2026-11-07 ~16:2xZ: free space is now 6.5G (was 470M) — stage-qa is no longer blocked, so this ticket is no longer urgent for PR flow. Breakdown of the temp-dir junk found (36747 top-level entries): ~28900 dirs older than 2 days, dominated by one-shot stage-qa browser artifacts (playwright-artifacts-, playwright_chromiumdev_profile-) and 275+ stale zero-byte .hm markers. Agent-side remote rm is removal-guard-blocked (no bypass), so the exact one-liner (find -mtime +2 on the playwright dirs + stale .hm markers + df) was handed to the owner in Telegram (tg#28147).
Remaining for acceptance: top-consumer audit of the other ~230G (
sudo du -xh --max-depth=1 /), the >=20G sustained-free criterion, and the 95% alert.