Skip to content

odroid stage-qa host root filesystem at 100% — stage QA fails with ENOSPC until caches are manually cleared #248

Open

Description

@alex-mextner

What

During stage-qa of #246 (2026-11-07 08:39Z, run 42356356024, runner odroid-esb-qa, host 65.21.40.51) the run failed with OSError Errno 28 No space left on device in stage-qa-browser.py run_smoke. Root fs /dev/mapper/ubuntu--vg--ubuntu--lv: 249G total, 238G used, 2.5M free (100%). Manual freeing (npm cache clean --force ~260MB) let the next run pass (42357873301), i.e. the stage QA path is one tempfile away from random ENOSPC failures. Guard-blocked cleanup that a human should review: uv cache clean (shared package-store for venvs) and stale agent sandboxes ~3.5GB dated Oct 22–Nov 2 under the system temp dir. Needs: identify the real space consumers (docker images? old deploy checkouts? never-delete log retention volume?), a monitoring alert before 95%, and a safe automated cleanup for QA sandboxes.

Why (motivation)

Stage QA is a required per-PR check; a full disk turns it into random infra failures that block every PR — this exact failure blocked #246 until caches were cleared by hand.

User impact

Any PR can randomly fail its required stage-qa check; agents burn time diagnosing infra instead of code.

Cost of inaction

Recurring ENOSPC failures will keep blocking PRs unpredictably.

Acceptance criteria

  • Root cause of the 238G identified with the top consumers listed and evidence
  • Sustained free space >= 20G on the odroid root fs after cleanup
  • An alert fires before the filesystem reaches 95% again

Screenshots

(none)

Deploy

Deployed: no

Links

  • (none)

Skipped gates

  • msgref-quote: Observed directly in CI run logs and host df output on 2026-11-07 while shipping the sync-import fix
  • screenshots: CLI/infra evidence: failing run 42356356024 vs passing 42357873301, df output quoted in the body

Activity

  1. alex-mextner commented on Nov 7, 2026

    @alex-mextner
    OwnerAuthor

    Read-only scan from the bot session (2026-11-07): df on the stage host shows 249G/237G, 470M free (100%). The system temp dir alone holds 9.3G, including ~3.5G of stale agent/QA sandboxes dated Oct 22–Nov 2 (agent deletion was guard-blocked; an owner rm on those sandbox dirs is the quick win). No passwordless sudo on the host, and du over home or root times out (>60s per top dir) under current IO pressure, so the full consumer list needs an owner shell. Suggested first pass:

    sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20
    docker system df
    

    then docker images/build cache, journald vacuum (journalctl --vacuum-size), and old deploy checkouts; the never-delete log retention dirs should be measured explicitly before any policy change.

  2. alex-mextner commented on Nov 7, 2026

    @alex-mextner
    OwnerAuthor

    Update 2026-11-07 ~16:2xZ: free space is now 6.5G (was 470M) — stage-qa is no longer blocked, so this ticket is no longer urgent for PR flow. Breakdown of the temp-dir junk found (36747 top-level entries): ~28900 dirs older than 2 days, dominated by one-shot stage-qa browser artifacts (playwright-artifacts-, playwright_chromiumdev_profile-) and 275+ stale zero-byte .hm markers. Agent-side remote rm is removal-guard-blocked (no bypass), so the exact one-liner (find -mtime +2 on the playwright dirs + stale .hm markers + df) was handed to the owner in Telegram (tg#28147).

    Remaining for acceptance: top-consumer audit of the other ~230G (sudo du -xh --max-depth=1 /), the >=20G sustained-free criterion, and the 95% alert.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions