Skip to content

CI: fall back from gVisor's hanging systrap platform; bound hung tests - #50

Draft
Deep-Sixed wants to merge 1 commit into
mainfrom
claude/modest-fermi-d4f8dl
Draft

Deep-Sixed wants to merge 1 commit into
mainfrom
claude/modest-fermi-d4f8dl

Conversation

@Deep-Sixed

Copy link
Copy Markdown
Owner

Fixes the intermittent linux-containment hang in the oci-runsc-docker tests. It hit #27 once and #45 twice: the job stalled in a docker process until it was cancelled, after up to an hour.

Root cause

I reproduced the hang on a host where it happens.

  • gVisor's default platform, systrap, hangs. The sandbox starts, but the container's first process never runs: even runsc --network=none --platform=systrap do /bin/true hangs, while runc runs the same image in under a second.
  • docker rm -f can't remove the stuck sandbox. Each gVisor test hits its timeout, and Stele's cleanup correctly reports ContainmentCleanupError. That was the three Fs in the CI log. The container is left running, and the run crawls on until the job is cancelled.
  • The same host is fine on ptrace. With runsc registered with --platform=ptrace, the full suite passes: 908 passed, including every gVisor test.
  • Why it was intermittent: which runner machine a job lands on decides whether systrap works. On this host, systrap worked before the environment restarted and hung after it.

Changes

  • Pick the platform per runner (.github/workflows/test.yml, "Register gVisor with Docker"):
    • Probe sudo timeout -s KILL 60 runsc --network=none --platform=systrap do /bin/true.
    • If it finishes, register runsc with systrap as before.
    • If not, register it with --platform=ptrace and emit a ::warning::, so the fallback shows in the run summary. ptrace is slower but keeps the same isolation boundary.
  • Bound any future hang:
    • pytest-timeout is added to the dev extra, and uv.lock is regenerated with uv lock.
    • The containment job runs pytest with --timeout=120. The slowest test takes about 5 s, so a stuck test fails by name with every thread's stack instead of stalling the run.
    • --durations=15 puts the slowest tests in the log.
    • timeout-minutes: 30 on the job. A normal run takes about 4 minutes, and GitHub's default limit is 6 hours.
  • Docs (docs/containment.md): how to check a host for the systrap hang and register ptrace instead.

Checks

  • Full suite on the affected host, with gVisor on ptrace and the new pytest flags: 908 passed, 33 skipped. The skips are the usual GPU, podman-not-installed and root-only ones.
  • The workflow YAML parses, with timeout-minutes set to 30.
  • The skip check in the step is unchanged, and --timeout and --durations don't change what -rs prints.

Review points

  • The probe can fall back for other reasons. If runsc do fails on a runner for some other reason (for example a sandboxing restriction), the job also falls back to ptrace. It still tests gVisor, and the warning says so.
  • Not version-specific: the runner's runsc is the Ubuntu 24.04 package (0.0~20230807). Newer gVisor releases may have fixed systrap on these hosts, but the probe makes the job independent of that.

🤖 Generated with Claude Code

https://claude.ai/code/session_013P1MkGppjFZfBUKsToHzez


Generated by Claude Code

The linux-containment job intermittently hung for up to an hour in the
oci-runsc-docker tests. Reproduced on a host where it happens: gVisor's
default platform, systrap, never runs the container's first process (even
`runsc do /bin/true` hangs), and `docker rm -f` cannot remove the sandbox,
so each gVisor test fails with ContainmentCleanupError and leaves its
container behind. The same host runs every gVisor test with
--platform=ptrace (908 passed). Which hosted runner a job lands on decides
it, which is why it was intermittent.

- Register runsc with systrap only if `runsc do /bin/true` finishes on the
  runner; otherwise with ptrace, and say so with a workflow warning.
- pytest-timeout (dev dependency): --timeout=120 per test (the slowest
  takes about 5 s), so a future hang fails the stuck test by name with every
  thread's stack instead of stalling the run; --durations=15 in the log.
- timeout-minutes: 30 on the job (a normal run takes about 4 minutes).
- docs/containment.md: how to check for the systrap hang and register
  ptrace instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013P1MkGppjFZfBUKsToHzez
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants