Repository navigation
CI: fall back from gVisor's hanging systrap platform; bound hung tests - #50
Draft
Deep-Sixed wants to merge 1 commit into
Draft
Deep-Sixed wants to merge 1 commit into
Deep-Sixed wants to merge 1 commit into
Conversation
The linux-containment job intermittently hung for up to an hour in the oci-runsc-docker tests. Reproduced on a host where it happens: gVisor's default platform, systrap, never runs the container's first process (even `runsc do /bin/true` hangs), and `docker rm -f` cannot remove the sandbox, so each gVisor test fails with ContainmentCleanupError and leaves its container behind. The same host runs every gVisor test with --platform=ptrace (908 passed). Which hosted runner a job lands on decides it, which is why it was intermittent. - Register runsc with systrap only if `runsc do /bin/true` finishes on the runner; otherwise with ptrace, and say so with a workflow warning. - pytest-timeout (dev dependency): --timeout=120 per test (the slowest takes about 5 s), so a future hang fails the stuck test by name with every thread's stack instead of stalling the run; --durations=15 in the log. - timeout-minutes: 30 on the job (a normal run takes about 4 minutes). - docs/containment.md: how to check for the systrap hang and register ptrace instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013P1MkGppjFZfBUKsToHzez
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the intermittent
linux-containmenthang in theoci-runsc-dockertests. It hit #27 once and #45 twice: the job stalled in adockerprocess until it was cancelled, after up to an hour.Root cause
I reproduced the hang on a host where it happens.
systrap, hangs. The sandbox starts, but the container's first process never runs: evenrunsc --network=none --platform=systrap do /bin/truehangs, whileruncruns the same image in under a second.docker rm -fcan't remove the stuck sandbox. Each gVisor test hits its timeout, and Stele's cleanup correctly reportsContainmentCleanupError. That was the threeFs in the CI log. The container is left running, and the run crawls on until the job is cancelled.ptrace. Withrunscregistered with--platform=ptrace, the full suite passes: 908 passed, including every gVisor test.systrapworks. On this host,systrapworked before the environment restarted and hung after it.Changes
.github/workflows/test.yml, "Register gVisor with Docker"):sudo timeout -s KILL 60 runsc --network=none --platform=systrap do /bin/true.runscwithsystrapas before.--platform=ptraceand emit a::warning::, so the fallback shows in the run summary.ptraceis slower but keeps the same isolation boundary.pytest-timeoutis added to thedevextra, anduv.lockis regenerated withuv lock.--timeout=120. The slowest test takes about 5 s, so a stuck test fails by name with every thread's stack instead of stalling the run.--durations=15puts the slowest tests in the log.timeout-minutes: 30on the job. A normal run takes about 4 minutes, and GitHub's default limit is 6 hours.docs/containment.md): how to check a host for thesystraphang and registerptraceinstead.Checks
ptraceand the new pytest flags: 908 passed, 33 skipped. The skips are the usual GPU, podman-not-installed and root-only ones.timeout-minutesset to 30.--timeoutand--durationsdon't change what-rsprints.Review points
runsc dofails on a runner for some other reason (for example a sandboxing restriction), the job also falls back toptrace. It still tests gVisor, and the warning says so.runscis the Ubuntu 24.04 package (0.0~20230807). Newer gVisor releases may have fixedsystrapon these hosts, but the probe makes the job independent of that.🤖 Generated with Claude Code
https://claude.ai/code/session_013P1MkGppjFZfBUKsToHzez
Generated by Claude Code