ops(dellsrv): control experiment for the exec EAGAIN outage - #21
Draft
zenprocess wants to merge 2 commits into
Draft
ops(dellsrv): control experiment for the exec EAGAIN outage#21zenprocess wants to merge 2 commits into
zenprocess wants to merge 2 commits into
Conversation
POST /v1/sandboxes/{id}/exec has been returning HTTP 500 "Resource
temporarily unavailable" (EAGAIN) on every uniforme gate run since
2026-07-25T23:44Z (182 occurrences, zero PASS). sandbox-create still
succeeds -- only the exec response read fails.
exec-eagain-control.sh runs one trivial `echo hi` exec against the
gate's actual snapshot_tag to split the hypothesis space: EAGAIN on a
trivial command means the fault is deterministic/global (a code
defect, not a workload issue); a PASS keeps the workload-dependent
theories alive. Also times the call -- sub-second failure implies a
non-blocking fd with no retry loop, ~120s implies a proxy timeout.
Flags a specific lead found while investigating: ~/fabro-run/forkd-shim.py
(the reverse proxy in front of the controller) gives sandbox-create
extra attempts on a known transient-restore signature, but every other
request shape -- including exec, the call that is failing -- gets
exactly one attempt by explicit design (see that file's `max_attempts`
logic). This script's result determines whether extending retries to
exec is the right fix or a wasted one.
--diagnose adds a strictly read-only one-pass capture (controller/proxy
state, restart detection via /v1/snapshots, orphan sandboxes, fd/proc
limits, dmesg OOM signatures, disk/memory, per-child netns) so the
operator does not need a round trip per command on the host.
Reuses gate-health-probe.sh's conventions in this directory: token
never leaves the forkd container (docker exec heredoc), sandbox id
scraped before any die() so the EXIT trap can always tear down, same
scrape_field jq/regex-fallback contract.
Read-only proof: no systemctl start/stop/restart/enable/disable, no
docker restart/rm/stop, no forkd snapshot registration anywhere in the
script (verified by a negated grep excluding comment lines; the guard
was checked against a scratch copy with an injected `docker restart`
line to confirm it actually fails before trusting it against the real
file). EXIT-trap-fires-on-failure verified by simulating the same
trap/SANDBOX_ID pattern under a forced non-zero exit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… before host restart
Drain-and-pause ahead of an operator host restart + AO update. No workers
were running (all five dispatched this session were killed earlier after
the MiniMax/qwen/cclocal routes all failed); this commit preserves the
orchestrator's own artifacts so nothing is stranded in the worktree.
Contents:
STATE.md - full session ledger: the EAGAIN ->
restore_many investigation, four
superseded analyses with their
refutations kept visible, and the
morning-digest asks.
DELLSRV-COMPLETION-RUNBOOK.md - operator runbook. PART 00 is the
executable list (read-only forensics
first, mutating steps gated on them);
PART 0 the outage; PART A/B the
original completion mandate.
.briefs/ - the four worker task briefs authored
this session (T1-T4), reusable if the
dispatch routes recover.
.salvage/fabro-74-referee/ - partial referee run-registration work
(register.rs + Cargo.toml + diff)
salvaged before killing fabro-74, so a
future worker does not restart from zero.
HANDOFF-...-fabro-67.md - inherited predecessor context.
Deliberately NOT included: four modified .claude/ agent-config files that
were already dirty on entry and are not this session's work.
Nothing here is a code change to fabro itself. The one code deliverable
this session is PR #21 (exec-eagain-control.sh), already pushed and open
as a draft.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Active outage since 2026-07-25T23:44Z:
POST /v1/sandboxes/{id}/execreturnsHTTP 500 "exec: read response: Resource temporarily unavailable"(EAGAIN) onevery gate run for the affected repo. 182 occurrences, 218 INFRA / 10 FAIL /
zero PASS. sandbox-create still succeeds — only the exec response read fails.
What this adds
exec-eagain-control.sh— the control experiment that splits the hypothesisspace before anyone reaches for a fix:
echo hi→ deterministic/global fault, workloadirrelevant, retry/backoff is the right fix.
echo hi→ workload-dependent, sizing/config is the right fix.Times the exec call: sub-second failure implies a non-blocking fd with no
poll/retry loop; ~120s implies a proxy timeout.
--diagnoseadds a strictly read-only diagnostic capture (controller/proxystate,
/v1/snapshotsfor restart detection, orphan sandboxes, fd/proc limits,dmesg, disk/memory, per-child netns) so the operator gets everything in one pass.
A specific lead this investigation surfaced
~/fabro-run/forkd-shim.py(the reverse proxy in front of the controller)gives
sandbox-createextra attempts on a known transient-restore signature.Every other request shape — including exec, the call that's failing — gets
exactly one attempt, by explicit design (
max_attemptslogic in that file).This script's result determines whether extending retries to exec is the right
fix.
Verification
Read-only proof: negated grep for mutating verbs (systemctl start/stop/restart/
enable/disable, docker restart/rm/stop, forkd snapshot registration), excluding
comment lines — finds nothing in the real script. Verified the guard actually
bites by injecting
docker restart forkdinto a scratch copy and confirming thesame check fails there.
EXIT-trap-fires-on-failure verified by simulating the same trap/SANDBOX_ID
pattern under a forced non-zero exit — teardown runs before the process exits,
not just on the success path.
Not in scope
This is a diagnostic, not a fix. No live dellsrv execution — I don't have host
access (sandbox blocks
dellsrv.zp.digital). An operator runs this on the gatehost; the result determines the next PR.
🤖 Generated with Claude Code