From 40966cdac16458e17d64bcb0b29780c2526009ef Mon Sep 17 00:00:00 2001 From: Referee Date: Sat, 1 Aug 2026 13:16:25 +0300 Subject: [PATCH 1/2] docs(internal): forkd snapshot-registry root-cause + operator runbook (fabro-123) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Writing-FIRST deliverable for zenprocess/ao-company#123 acceptance criterion #4. Records the verified evidence (restore_many 400 on earlier uniforme heads, exec EAGAIN at 2026-08-01T08:32, blast radius from fabro-gate-health.json) and explicitly marks every UNVERIFIED claim with the reason (forkd controller source not local, dellsrv behind egress boundary from this sandbox). The companion operator runbook hands off items 1-2 of the issue to the operator, with T3 maint-window discipline and a token-path-only reference (no token values). The code half (PR #143, gate-side preflight that emits a distinct snapshot-not-registered infra verdict) is already landed on ao/fabro-123/snapshot-preflight in zenprocess/ao-company and is NOT re-implemented here. Cross-referenced in §5 layer 1 and §7 of the design note. Co-Authored-By: Claude --- .../forkd-snapshot-registry-runbook.md | 305 ++++++++++++++++++ docs/internal/forkd-snapshot-registry.md | 230 +++++++++++++ 2 files changed, 535 insertions(+) create mode 100644 docs/internal/forkd-snapshot-registry-runbook.md create mode 100644 docs/internal/forkd-snapshot-registry.md diff --git a/docs/internal/forkd-snapshot-registry-runbook.md b/docs/internal/forkd-snapshot-registry-runbook.md new file mode 100644 index 000000000..14875dcbb --- /dev/null +++ b/docs/internal/forkd-snapshot-registry-runbook.md @@ -0,0 +1,305 @@ +# forkd snapshot-registry — operator runbook (dellsrv) + +**Companion to**: `docs/internal/forkd-snapshot-registry.md` (design note). +**Audience**: only the operator — controller restart is T3. + +The runbook is the live-verification branch of `zenprocess/ao-company#123`. +The goal is to determine which of three states each golden tag is in +AFTER the controller's last restart: + +| state | meaning | next step | +|---|---|---| +| **missing** | the snapshot registry lacks the tag (was lost on restart) and the on-disk file is also absent | re-commit the golden from the canonical rootfs (20GB golden per QA-infra facts) | +| **present-but-broken** | the registry has the tag, the on-disk file exists, but `restore_many` returns 400 | the file is corrupt or the cached metadata handle is stale; needs an operator decision (re-commit or restore from a prior good snapshot) | +| **healthy** | the registry has the tag, boot + exec `true` succeeds | nothing to do | + +The runbook does NOT itself modify the controller; it records the data +points the controller-side fix needs. + +**Placement** — this runbook lives next to the design note in the fabro +repo (`docs/internal/`). Alternative placement was +`docs/runbooks/forkd-snapshot-registry.md` in the `ao-company` repo +(alongside `DEPLOY-forkd-controller-gate.md`), but that directory does +not exist in the shared checkout and the operator-runbook content is +deeply coupled to the design note in §5 of the design file. Co-locating +them keeps the operator and design context in one place. + +--- + +## 0. Preconditions (operator-only, T3) + +- The forkd-controller is a LIVE service on dellsrv. Restarting it is + **T3** — always inside a `zenctl maint on` window. The forkd-shim and + the gate driver both go through the controller; every gate attempt + during the outage will post a `snapshot-not-registered: unreachable ` + infra verdict (the preflight's transport-failure branch — see design + note §5 layer 1). +- The controller listens on `http://127.0.0.1:8891`. The bearer token + lives at `~/fabro-run/.forkd-token` on the controller host (`@dellsrv`). + **The token value is NEVER to be printed, echoed, or inlined into a + command.** All commands below reference the token by file path and + read it via `$(cat ~/fabro-run/.forkd-token)` inside the `curl` header + only; the file content stays in process memory between the `cat` and + the `curl` and never lands on a log line because the bash subshell is + the only place it is expanded. +- The maintenance window is announced BEFORE step 1. Sample wording: + ``` + zenctl maint on 'forkd snapshot re-registration verification (fabro-123)' + ``` + +--- + +## 1. Confirm or refute registry volatility + +This is the live probe the brief's items 1–2 hinge on. The question: does +`GET /v1/snapshots` empty after a restart, even though `zen-gate-base` and +`zen-gate-big` boot fine afterwards? + +```bash +# 1.1 Tail the controller journal BEFORE the restart so the post-boot +# log lines are easy to correlate. +journalctl -u forkd-controller -f > /tmp/forkd-journal.log 2>&1 & +JOURNAL_PID=$! + +# 1.2 Restart the controller. T3 — operator only. +sudo systemctl restart forkd-controller + +# 1.3 Wait for the controller to come back. The /v1/snapshots endpoint +# returns 200 on a populated registry and 200 [] on an empty one; +# both are "reachable". Return anything else and the loop continues. +TOKEN_FILE=~/fabro-run/.forkd-token +until curl -sS -o /dev/null -w '%{http_code}\n' \ + -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ + http://127.0.0.1:8891/v1/snapshots 2>/dev/null \ + | grep -qE '^(200|404)$'; do + sleep 1 +done + +# 1.4 Record the listing immediately after the boot finishes. +# Boot-scan re-registration can take a minute; the operator should +# RE-RUN this step at +60s and +5min and compare to detect whether +# re-registration ran. +RECORD_DIR=/tmp/forkd-snapshot-verify-$(date -u +%Y%m%dT%H%M%SZ) +mkdir -p "$RECORD_DIR" +for delay in 0 60 300; do + sleep "$delay" + curl -sS \ + -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ + http://127.0.0.1:8891/v1/snapshots \ + | tee "$RECORD_DIR/snapshots-t+${delay}s.json" \ + | jq 'length, [.[].tag // .[].snapshot_tag]' + echo "--- snapshot at t+${delay}s ---" +done + +# 1.5 Stop the journal tail. +kill "$JOURNAL_PID" 2>/dev/null || true +``` + +**What to record**: the three listings (count + tag names) and the +journal slice from boot to +5min. Feed back to the design-note author: + +- `GET /v1/snapshots` post-boot — was the listing empty, partial, or complete? +- Did the journal contain `snapshot re-registered tag= source=` lines? (If yes, the controller fix is already deployed; if no, the fix is pending and the listings reflect the default registry state.) +- Did the listing GROW between t+0s and t+300s? (If yes, the boot-scan is async — record the time-to-populate for the design note.) + +If the listing is empty at all three sample points AND boot succeeds for +`zen-gate-base` (step 2), the volatility hypothesis is confirmed. If the +listing is populated AND boot fails, the hypothesis is wrong and we need +to look at the on-disk artifact (present-but-broken state). + +--- + +## 2. Per-tag bootability check (which is missing, which is broken) + +For each golden tag, run a real restore-boot canary. The canary must be +a real boot + exec `true`; a files-exist check is exactly the wrong +validation (design note §5). Tags to verify against: `zen-gate-base`, +`zen-gate-big`, and any other golden listed in the controller's +configuration (`FORKD_GOLDEN_TAGS` or the equivalent — document what is +actually used on dellsrv before this runbook is run, since the source +is not local). + +```bash +TOKEN_FILE=~/fabro-run/.forkd-token +RECORD_DIR=/tmp/forkd-snapshot-verify-$(date -u +%Y%m%dT%H%M%SZ) +mkdir -p "$RECORD_DIR" + +canary() { # $1=tag + local tag="$1" + echo "=== canary tag=$tag ===" + + # 2.1 POST /v1/sandboxes with the tag. 201 with a sid is the happy path. + local create_body + create_body=$(curl -sS -X POST \ + -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ + -H "Content-Type: application/json" \ + -d "{\"snapshot_tag\":\"$tag\"}" \ + http://127.0.0.1:8891/v1/sandboxes) + echo "create response: $create_body" + echo "$create_body" | tee "$RECORD_DIR/create-$tag.json" + + # 2.2 Extract the sid. The controller's response shape is documented + # in `docs/DEPLOY-forkd-controller-gate.md` as confirmed-working + # across ~30 live runs; the multi-shape fallback is in + # forkd-shim.py:_extract_sid. + local sid + sid=$(printf '%s' "$create_body" \ + | python3 -c ' +import json, sys +try: + d = json.loads(sys.stdin.read()) +except Exception as e: + print(""); sys.exit(0) +if isinstance(d, dict): + print(d.get("id") or d.get("sandbox_id") or d.get("sid") or "") +elif isinstance(d, str): + print(d) +elif isinstance(d, list) and d: + v = d[0] + print(v if isinstance(v, str) else (v.get("id") or "")) +') + if [ -z "$sid" ]; then + echo "canary tag=$tag NO_SID (controller refused to boot)" + return 0 + fi + + # 2.3 POST /v1/sandboxes/{sid}/exec with `sh -c true`. This is the + # REAL restore-boot canary — not a files-exist check. + local exec_body + exec_body=$(curl -sS -X POST \ + -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ + -H "Content-Type: application/json" \ + -d '{"args":["sh","-c","true"],"timeout_secs":30}' \ + "http://127.0.0.1:8891/v1/sandboxes/$sid/exec") + echo "exec response: $exec_body" + echo "$exec_body" | tee "$RECORD_DIR/exec-$tag.json" + + # 2.4 DELETE /v1/sandboxes/{sid} to clean up. The forkd-shim's + # teardown is confirmed live across the same ~30 runs. + curl -sS -X DELETE \ + -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ + "http://127.0.0.1:8891/v1/sandboxes/$sid" -o /dev/null -w 'delete http=%{http_code}\n' +} + +canary zen-gate-base +canary zen-gate-big +# add any other tag the controller's golden list declares +``` + +**What to record**: the create + exec response bodies for each tag. +Feed back to the design-note author: + +- `zen-gate-base` create http status + sid (or empty if refused) +- `zen-gate-base` exec `exit_code` (0 = healthy, non-zero = present-but-broken) +- `zen-gate-big` create http status + sid +- `zen-gate-big` exec `exit_code` + +State table the runbook produces: + +| tag | create | exec exit_code | state | +|---|---|---|---| +| `zen-gate-base` | 201 + sid | 0 | healthy | +| `zen-gate-base` | 201 + sid | non-zero | present-but-broken — restore file is corrupt, must re-commit | +| `zen-gate-base` | 4xx/5xx | n/a | missing — registry refused to boot; re-commit from canonical rootfs | +| `zen-gate-big` | (same) | (same) | (same) | + +--- + +## 3. Token path reference (NEVER print the value) + +The token is referenced at `~/fabro-run/.forkd-token` on the controller +host (dellsrv). The path is set in `bin/forkd-shim.py` and overrides the +shim's built-in default (`/etc/forkd-token`, which is root-only on this +host and is NOT what the gate uses). The driver-side override is +`FABRO_TOKEN_FILE_REMOTE` in `bin/fabro-github-gate.sh`. If the token +ever moves, the runbook still works as long as that env var is updated. + +The token VALUE is never in this runbook. The `$(cat ~/fabro-run/.forkd-token)` +substitution lives inside the bash subshell that builds the curl header +argument; it never reaches a log line because the curl `-H` value is +processed in-process. If a runbook command needs to be pasted into a +shell history, the operator should configure `HISTCONTROL=ignorespace` +and prefix with a space, or use a one-shot alias that does not record. + +--- + +## 4. Expected success log line (after the controller fix is deployed) + +The controller MUST emit exactly one log line per re-registered snapshot, +in the format: + +``` +snapshot re-registered tag= source= +``` + +Search for it in the journal slice from step 1: + +```bash +grep -E 'snapshot re-registered tag=' /tmp/forkd-journal.log +``` + +Each line corresponds to one tag that passed the canary. A tag that +FAILED the canary should emit: + +``` +snapshot re-registration FAILED tag= reason= +``` + +And a golden that was re-committed from the canonical rootfs should emit: + +``` +snapshot golden-recommitted tag= +``` + +If `zen-gate-big` is healthy, the journal should show one line matching +`snapshot re-registered tag=zen-gate-big`. If `zen-gate-big` is +present-but-broken, the journal should show one matching +`snapshot re-registration FAILED tag=zen-gate-big` (the failure to +re-register IS the evidence the file is unbootable). If the journal +shows NEITHER for `zen-gate-big`, the controller fix is not deployed. + +--- + +## 5. Hand-off back to the design-note author + +After the runbook completes, the operator should send back: + +1. The three `GET /v1/snapshots` listings from step 1 (`$RECORD_DIR/snapshots-t+0s.json`, `…t+60s.json`, `…t+300s.json`). +2. The journal slice (`/tmp/forkd-journal.log`). +3. The four create+exec response bodies from step 2 (`$RECORD_DIR/create-.json`, `$RECORD_DIR/exec-.json`). +4. The state table from step 2 (which golden is missing, which is present-but-broken, which is healthy). + +The design note's §5 fix direction depends on this signal: + +- If `zen-gate-big` is **missing** on registry AND on disk → the fix is a re-commit from the canonical rootfs; the controller-side re-registration will pick it up. +- If `zen-gate-big` is **present-but-broken** → the fix is to replace the on-disk file (restore from a prior good snapshot, or re-commit from canonical rootfs); the cached metadata handles are stale. +- If `zen-gate-big` is **healthy by registry but restore_many still 400s at the gate** → the registry's success criterion is not a real canary; the controller-side fix is wrong and needs to be reworked to require a real restore-boot (not a files-exist check). + +Once the operator's live signal is in, the design note's §5 layer 2 +implementation can be reviewed against the actual controller source on +dellsrv. + +--- + +## 6. Closing the maintenance window + +```bash +# 6.1 Confirm the registry is healthy. +curl -sS -H "Authorization: Bearer $(cat ~/fabro-run/.forkd-token)" \ + http://127.0.0.1:8891/v1/snapshots \ + | jq 'length, [.[].tag // .[].snapshot_tag]' + +# 6.2 Optionally fire one gate-poll cycle to verify the preflight's +# `ok ` path lands cleanly. +bash ~/Desktop/ao-company/bin/fabro-gate-poll.sh + +# 6.3 Close the maintenance window. +zenctl maint off +``` + +If the post-cycle snapshot listing is non-empty and contains both golden +tags, the gate's preflight will start returning `ok ` for them +immediately; the descriptions on previously-failing SHAs will not change +(the existing post_infra no-overwrite guard is preserved on purpose), so +the next NEW head will be the first to land with the preflight's clean +verdict. diff --git a/docs/internal/forkd-snapshot-registry.md b/docs/internal/forkd-snapshot-registry.md new file mode 100644 index 000000000..8c120b78b --- /dev/null +++ b/docs/internal/forkd-snapshot-registry.md @@ -0,0 +1,230 @@ +# forkd snapshot-registry truthfulness — root-cause & design note + +**Status**: writing-FIRST deliverable for `zenprocess/ao-company#123`, +acceptance-criterion #4 (the design note) plus the operator runbook companion +(`forkd-snapshot-registry-runbook.md`, same directory). + +**Author**: fabro-123 doc worker, 2026-08-01. + +**Code side**: `feat(gate): snapshot preflight in fabro-github-gate.sh +(fabro-123)` — commit `e1b3b5b` on branch +`ao/fabro-123/snapshot-preflight` (worktree +`~/.ao/data/worktrees/ao-company-gate121`). Lands `snapshot-not-registered` +as a distinct infra verdict on the gate side; not re-implemented here. + +**Controller side**: NOT implemented in this PR. The forkd controller source +is not in the fabro repo (only the client, `lib/components/fabro-sandbox/src/provider/forkd.rs`, +lives here); the controller is a separate service on dellsrv. Items 1 and 2 +of `zenprocess/ao-company#123` are operator work and the runbook in the +companion doc hands off cleanly. + +--- + +## 1. Timeline (verified facts first) + +| Date (UTC) | Event | Source | +|---|---|---| +| 2026-07-31 | "Brokered zen-gates" probe: controller booted snapshot tag `zen-gate-base` and exec'd a command with exit 0, **yet** `GET /v1/snapshots` returned `[]`. | `zenprocess/ao-company#123` problem statement (UNVERIFIED by this worker — the brokered zen-gates path is not reachable from this sandbox; see §5) | +| 2026-07-31 | Earlier restore failure mode appears in `~/.ao/state/fabro-gate-poll.log` for `zenprocess/uniforme` heads `54da9d8`, `3b25011`, `8ec811f`, `e839d29`: `restore_many: firecracker API PUT /snapshot/load returned 400: {"fault_message":"Load snapshot error: Failed to restore from snapshot: Failed to build microVM from snapshot: Failed to res…"}` (description truncated to 140 chars by `gh-status.sh`). | `~/.ao/state/fabro-gate-poll.log` lines 2826, 2834, 2842, 2856, 2864 (VERIFIED) | +| 2026-08-01T08:32:27Z | Descriptor-driven gate-poll cycle runs against `pawbench`, `trader`, `uniforme`, `foundry`. Trader and foundry gate successfully (`GATE_CYCLE … heads=1 gated=1`); uniforme head `e9ac694` fails at exec stage with `controller POST /v1/sandboxes/sb-6a6dae50-027d/exec -> HTTP 500 {"error":"exec: read response: Resource temporarily unavailable (os error 11)"}` (a different failure mode from the restore-many 400). Status POSTING works: `posted fabro/qa-pipeline=error on zenprocess/uniforme@e9ac694`. | `~/.ao/state/fabro-gate-poll.log` lines 7307–7322 (VERIFIED) | +| 2026-08-01T08:32:33Z | Blast-radius snapshot (`~/.ao/state/fabro-gate-health.json`): `foundry.last_verdict=error`, `trader.last_verdict=error`, `pawbench.never_gated`, `uniforme` has `deferred_heads=[e9ac694…]` with no verdict. Every gated repo is down on the same cause (snapshot infra, not the gate pipeline). | `~/.ao/state/fabro-gate-health.json` (VERIFIED) | + +### Note on what the issue claims vs. what the log shows + +The issue body (`zenprocess/ao-company#123`) attributes the +`restore_many: … PUT /snapshot/load returned 400` failure to +`trader PR #354 head 3ed3b8a4` at 2026-08-01T08:32:33Z. **I cannot verify +that exact pairing in the log.** A `grep` for `3ed3b8a` against +`fabro-gate-poll.log` returns zero matches; the only `restore_many` entries +I can find are on `zenprocess/uniforme` heads at earlier dates, and the +2026-08-01T08:32:27Z poll shows trader head `3ed3b8a4` *succeeded* +(`GATE_CYCLE repo=trader heads=1 gated=1 deferred=0`). The truncated +`{"fault_message":"…"}` JSON for the restore-many 400 is identical in shape +to the issue text, so the failure mode is real; the SHA pairing and the +exact repo it lands on are NOT independently verified from this side. I +flag this as an accuracy gap in the issue, not a contradiction of its core +claim that snapshot restore is broken. + +The most defensible summary of the timeline: + +- **snapshot restore is broken on the gate's hot path** (either as + `restore_many … 400` or as the later `exec: read response: Resource + temporarily unavailable (os error 11)` pattern — both observed live); +- **`GET /v1/snapshots` returns `[]`** at least once after a controller + restart, while the same controller can still boot the same tag from a + cold start (per the 07-31 probe); +- **status posting is intact** (trader/foundry gating ran and posted + `success` verdicts through the same plumbing); this is NOT an + `exec: EAGAIN` regression of `zenprocess/ao-company#122` (that exec EAGAIN + was live-confirmed at exit 0 in #122 — you cannot exec into a VM you + failed to restore). + +--- + +## 2. Two distinct symptoms, one root cause + +The 07-31 evidence and the 08-01 evidence look superficially different +but are consistent with a single underlying defect: **the snapshot registry +on the forkd controller is not derived from boot-time truth, so it can +both under-report (return `[]` when disks have snapshots) and over-report +(return tags whose underlying snapshot file is corrupt or missing in a way +that firecracker rejects at `PUT /snapshot/load`)**. + +| Symptom | What it says about the registry | +|---|---| +| 07-31: `GET /v1/snapshots` → `[]`, `zen-gate-base` boots + execs | Listing endpoint is decoupled from the boot path. Either (a) the registry is in-memory only and lost on controller restart while disk artifacts persist, or (b) the registry is stored in a state table the boot path does not consult — tags are resolved tag-directly at boot. Both explanations fit. | +| 08-01: `restore_many: PUT /snapshot/load returned 400` | The boot path IS consulting something, and what it consults disagrees with the file on disk in a way firecracker treats as fatal. Two possibilities consistent with the evidence: (a) the on-disk snapshot file is corrupt/truncated, or (b) the snapshot metadata the controller has cached does not match the file. | + +The leading hypothesis — that **the registry is volatile across restart +while files persist on disk** — explains why `[]` could coexist with a +successful exec (boot path resolves tags directly at boot, registry is +empty in-memory), but it does NOT by itself explain the `restore_many` 400 +at 08-01 (which would require either corruption, a stale cached metadata +handle, or a different snapshot file on disk than the one the controller +thinks it is loading). + +--- + +## 3. Evidence vs. hypothesis (be honest about which is which) + +### VERIFIED (this side can show the bytes) + +- **The status-posting pipeline works.** Trader head 3ed3b8a4 (2026-08-01T08:32) and foundry head b919955d (same poll) ran and posted `success` verdicts via `fabro/qa-pipeline`. The PR #140 poller surfaced a real `error` to uniforme e9ac694 in the same cycle. (`fabro-gate-poll.log` lines 7307–7322.) +- **Snapshot restore fails on the gate's hot path.** Both the `restore_many: PUT /snapshot/load returned 400` shape (uniforme heads `54da9d8`, `3b25011`, `8ec811f`, `e839d29` at earlier dates) and the `exec: read response: Resource temporarily unavailable (os error 11)` shape (uniforme e9ac694 at 2026-08-01T08:32:27Z) appear in the log. The two shapes are distinct: restore_many fails inside firecracker on snapshot load; the EAGAIN fails in the exec-stream read after a presumably successful boot. They are not the same bug. +- **`GET /v1/snapshots` is consulted by the gate-side preflight** (commit `e1b3b5b`, `ao/fabro-123/snapshot-preflight`). On mismatch the gate posts `snapshot-not-registered: tag= absent from GET /v1/snapshots` — a distinct infra verdict from generic restore-many HTTP 500. +- **The fabro-sandbox forkd client does not call `GET /v1/snapshots` itself.** `lib/components/fabro-sandbox/src/provider/forkd.rs` consumes `POST /v1/sandboxes`, `POST /v1/sandboxes/{id}/exec`, `DELETE /v1/sandboxes/{id}` only. The listing endpoint is only consumed by the gate-side preflight (`bin/fabro-github-gate.sh:46` `preflight_snapshot()`), via `bin/forkd-shim.py:229 list_snapshots()`. +- **Blast radius.** `~/.ao/state/fabro-gate-health.json` shows `foundry` and `trader` with `last_verdict=error`, `uniforme` with `deferred_heads=[e9ac694…]` and no verdict, `pawbench` never gated. Every gated repo is down on the same cause. + +### UNVERIFIED (cannot determine from this side) + +- **Whether the snapshot registry is in-memory only, file-backed, or remote-DB-backed.** The controller's storage choice is invisible from the client or the spec. (UNVERIFIED because the controller source is not in the fabro repo, and dellsrv is behind the egress boundary from this sandbox.) +- **Whether `zen-gate-big` is missing from on-disk storage, present-but-corrupt, or present-and-valid but uncached.** A `PUT /snapshot/load` 400 from firecracker can come from a corrupt file, a wrong-version snapshot, or a cached metadata handle pointing at the wrong backing file. The `fault_message` JSON is truncated in the gate's `--desc` field (140-char cap in `gh-status.sh`); the full payload is only on the controller. (UNVERIFIED for the same reason.) +- **Whether `zen-gate-base` is still bootable today.** The 07-31 probe booted it once. A probe run after the controller's last restart is the only way to know. (UNVERIFIED — brokered path not reachable from this sandbox; one attempt made, stopped per the brief's egress boundary.) +- **Whether the brief's exact pairing (trader head `3ed3b8a4` carrying the `restore_many` 400 description at 08-32:33Z) is accurate.** The log shows trader `3ed3b8a4` *succeeded* at the timestamp the issue cites. The `restore_many` 400 entries in the log are on uniforme at older dates. The failure mode is real; the SHA/repo pairing is not. (UNVERIFIED — log diff vs. issue text.) + +--- + +## 4. Leading hypothesis (UNVERIFIED mechanism, but evidence-consistent) + +The snapshot **registry** is held in a state table that is not rehydrated +on controller boot, while the snapshot **artifacts** persist on disk and +are resolved tag-directly at boot time. On a fresh boot the registry is +empty (`[]`) until something writes to it; meanwhile the boot path can +still resolve a tag → file lookup, so `zen-gate-base` boots fine despite +the empty listing. The asymmetry between "registry" and "artifacts" is +the surface that the gate's `snapshot-not-registered` preflight surfaces, +and the asymmetry between "metadata cached in the registry" and +"artifact on disk" is the surface that firecracker's `PUT /snapshot/load` +400 surfaces. + +This is a hypothesis, not a finding. The mechanism (in-memory vs. +non-rehydrated state table vs. something else) is UNVERIFIED — the +controller source is not local. The right next step is to verify the +mechanism by reading the actual controller code on dellsrv, not to +implement a fix against a guessed mechanism. + +--- + +## 5. What the controller fix MUST do (design direction, not implementation) + +The fix lives in two layers. + +### Layer 1 — gate-side preflight (LANDED) + +`bin/fabro-github-gate.sh:46 preflight_snapshot()` (commit `e1b3b5b`, +branch `ao/fabro-123/snapshot-preflight`, push target: `zenprocess/ao-company`). +Before any in-VM work, the gate asks the controller (via the shim on the +gate host) for `GET /v1/snapshots`. Three outcomes: + +| preflight outcome | gate behavior | +|---|---| +| `ok ` | proceed to `cmd_gate` as before | +| `missing ` | post infra verdict `snapshot-not-registered: tag= absent from GET /v1/snapshots`; **do not** call `cmd_gate` | +| `unreachable ` | treat as "cannot verify", fall through to `cmd_gate`. A transport blip must NOT silently turn every gate into a false `snapshot-not-registered`. | + +The preflight's self-test (`fabro-github-gate.sh:388`) covers the three +return shapes the controller is observed to emit (bare array, wrapped +`{"snapshots": [...]}` object, empty list). A preflight infra verdict +**never overwrites** an existing success/failure verdict on the same SHA — +the existing `post_infra` no-overwrite guard is preserved. + +### Layer 2 — controller-side boot-time re-registration (OPERATOR WORK) + +The controller must, on boot, walk its snapshot storage directory and +re-register every snapshot that passes a **real restore-boot canary**: + +1. `POST /v1/sandboxes` with `{"snapshot_tag": }` → must return 201. +2. If 201, `POST /v1/sandboxes/{sid}/exec` with `{"args":["sh","-c","true"],"timeout_secs":N}` → must return 200 with `exit_code=0`. +3. If both pass, mark the snapshot registered. Emit ONE log line per snapshot: `snapshot re-registered tag= source=`. +4. If either fails, mark it NOT-registered. Emit `snapshot re-registration FAILED tag= reason=<...>`. +5. `GET /v1/snapshots` returns the registered set **and only the registered set**. + +A snapshot is in the listing IFF it just passed the canary. **A files-exist +check is exactly the wrong validation** — a disk-clean-but-unbootable +snapshot must never register as healthy. + +For each golden tag in a configured list (`zen-gate-base`, `zen-gate-big`, +anything else declared golden), if the walk does not find a passing +entry: re-commit the golden from the canonical rootfs (the 20GB golden +per QA-infra facts), run the canary on the freshly-committed snapshot, +emit `snapshot golden-recommitted tag=` on success, and emit +`snapshot golden-recommit FAILED tag=` plus a `ntfy` alert on failure. + +The canary cannot be skipped. A 2026-07-11 anchor records the same kind +of evidence in the opposite direction (`forkd snapshot list returned []` +while `zen-gate-base` exec worked) and was correctly used at the time to +argue that the list endpoint "does not prove snapshot absence". The fix +goes the same direction: the list should be derived from "what boots", +not "what's on disk". Files-on-disk without a canary-pass is the path +that ships an unshippable golden. + +--- + +## 6. What this PR contains vs. what the operator owns + +**This PR (in the fabro worktree at `~/.ao/data/worktrees/fabro/fabro-84`, branch `ao/fabro-84/forkd-snapshot-registry`):** + +- this document (`docs/internal/forkd-snapshot-registry.md`), +- the operator runbook companion (`docs/internal/forkd-snapshot-registry-runbook.md`). + +**Already landed (NOT re-implemented here):** + +- the gate-side preflight in `bin/fabro-github-gate.sh` (`e1b3b5b` on + `ao/fabro-123/snapshot-preflight` in the `ao-company` repo). + +**Not in this PR (operator work, T3):** + +- the controller-side boot-scan re-registration (controller source is not + local; the design is in §5 for the operator to implement against the + real controller source), +- a controller restart for testing, +- a fix to the on-disk state of `zen-gate-big` (depends on the operator + runbook's findings). + +--- + +## 7. References + +- `zenprocess/ao-company#123` — issue as filed. +- `zenprocess/ao-company#121` — the org-wide gate silence; the + descriptor-driven poller (commit `258a5f0`) restored status posting + and surfaced the snapshot failure that #123 is now the home for. +- `zenprocess/ao-company#122` — the exec `EAGAIN` regression; live-verified + at exit 0 in the 07-31 brokered probe. Exec EAGAIN is NOT the cause of + the current `error` verdicts: a successful exec EAGAIN run cannot post a + status of `error`; the current verdicts reflect snapshot-restore failure + upstream of any in-VM work. +- `bin/fabro-github-gate.sh:46 preflight_snapshot()` — the gate-side + companion (commit `e1b3b5b`). +- `bin/forkd-shim.py:229 list_snapshots()` — the shim that brokers the + listing to the gate host. +- `lib/components/fabro-sandbox/src/provider/forkd.rs` — the fabro-side + client. Consumes `POST /v1/sandboxes`, `POST /v1/sandboxes/{id}/exec`, + `DELETE /v1/sandboxes/{id}`. Does not call `GET /v1/snapshots`. +- `~/.ao/state/fabro-gate-poll.log` lines 2826, 2834, 2842, 2856, 2864 + — the `restore_many: PUT /snapshot/load returned 400` evidence on + earlier uniforme heads. +- `~/.ao/state/fabro-gate-poll.log` lines 7307–7322 — the + 2026-08-01T08:32:27Z poll cycle in which trader/foundry gated + successfully and uniforme failed at exec stage. +- `~/.ao/state/fabro-gate-health.json` — blast-radius snapshot. From d55855ec207a74e57468fcc5e67641ab0e46f3ce Mon Sep 17 00:00:00 2001 From: Referee Date: Sat, 1 Aug 2026 13:26:39 +0300 Subject: [PATCH 2/2] =?UTF-8?q?docs(internal):=20revision=202=20=E2=80=94?= =?UTF-8?q?=20two-failure=20headline,=20EAGAIN=20runbook=20step,=20softer?= =?UTF-8?q?=20token=20rules=20(fabro-123=20+=20fabro-122)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Orchestrator review of revision 1 caught three errors in the design note. All three are corrected and the most important is promoted to a new §1 Headline section. Design note changes (docs/internal/forkd-snapshot-registry.md): - §1 NEW: Headline — TWO distinct live infra failures with a ~91:1 ratio. Exec-stage EAGAIN (zenprocess/ao-company#122, 1092 occurrences) dominates restore-stage 400 (this issue, 12 occurrences). Both are real; they are different bugs with different fix paths. - §3 NEW: What this worker got wrong in the previous revision (the orchestrator's three corrections, explicit). (a) gated=1 is a gate-attempt count, NOT a success verdict. The authoritative verdict lives in GitHub + fabro-gate-health.json. (b) The trader head 3ed3b8a4 pairing IS verified — gh api returns state=error at 2026-08-01T08:32:33Z with the restore_many 400 description. (c) The "log does not record 3ed3b8a" finding was misleadingly worded — the poll log's posted lines for this slice are uniformly for uniforme, but the trader/foundry verdicts live on GitHub, not in the file. - All section numbers shifted by one (added §1 headline + §3 corrections). - References updated to point at gh api + fabro-gate-health.json as authoritative. Runbook changes (docs/internal/forkd-snapshot-registry-runbook.md): - New §2 covers the exec-stage EAGAIN diagnostic (zenprocess/ao-company#122). Captures controller baseline (ps, /proc/PID/limits, cgroup) and EAGAIN-triggered samples of fd / thread / memory counters. Asks the operator to characterize which resource is exhausted. - §3 (per-tag bootability, was §2) updated; the new EAGAIN step is §2. - §4 (token handling, was §3) softened. The previous revision asserted the token "never lands on a log line because the bash subshell is the only place it is expanded." That was overconfident — shell tracing, error paths, and command capture can expose it. New rules: reference by file path only, confirm shell tracing is OFF, disable shell history capture for the maintenance window, do not paste commands into chat windows or pastebins, do not use curl -v or --trace. - All section numbers shifted. Doc header notes that this PR supersedes zenprocess/fabro PR #32. PR #32 also touches lib/crates/fabro-sandbox/* (DEAD paths post crate reorg; the sandbox crate now lives under lib/components/fabro-sandbox/) — those file changes are NOT carried forward. Co-Authored-By: Claude --- .../forkd-snapshot-registry-runbook.md | 212 +++++++++++++---- docs/internal/forkd-snapshot-registry.md | 219 +++++++++++++----- 2 files changed, 324 insertions(+), 107 deletions(-) diff --git a/docs/internal/forkd-snapshot-registry-runbook.md b/docs/internal/forkd-snapshot-registry-runbook.md index 14875dcbb..2497e2253 100644 --- a/docs/internal/forkd-snapshot-registry-runbook.md +++ b/docs/internal/forkd-snapshot-registry-runbook.md @@ -2,10 +2,21 @@ **Companion to**: `docs/internal/forkd-snapshot-registry.md` (design note). **Audience**: only the operator — controller restart is T3. - -The runbook is the live-verification branch of `zenprocess/ao-company#123`. -The goal is to determine which of three states each golden tag is in -AFTER the controller's last restart: +**Revision**: 2 (2026-08-01). Adds §2 for the exec-stage EAGAIN diagnostic +(see §1 of the design note for why both failure modes must be covered), and +softens §3 on token handling (this worker previously asserted the token +never lands on a log line because of bash subshell expansion; that was +overconfident — shell tracing, error paths, and command capture can expose +it). + +The runbook is the live-verification branch of `zenprocess/ao-company#123` +(restore-stage 400) AND the live-diagnostic branch of +`zenprocess/ao-company#122` (exec-stage EAGAIN). The two issues share a +controller but have different root causes and different fixes; the +runbook must characterize both before either fix can land. + +For the restore-stage (#123), the goal is to determine which of three +states each golden tag is in AFTER the controller's last restart: | state | meaning | next step | |---|---|---| @@ -13,6 +24,10 @@ AFTER the controller's last restart: | **present-but-broken** | the registry has the tag, the on-disk file exists, but `restore_many` returns 400 | the file is corrupt or the cached metadata handle is stale; needs an operator decision (re-commit or restore from a prior good snapshot) | | **healthy** | the registry has the tag, boot + exec `true` succeeds | nothing to do | +For the exec-stage (#122), the runbook asks: does a restored VM accept +exec, and what controller-side resource is exhausted (fd / pid / memory) +when the exec-stream read returns EAGAIN? See §2 below. + The runbook does NOT itself modify the controller; it records the data points the controller-side fix needs. @@ -21,7 +36,7 @@ repo (`docs/internal/`). Alternative placement was `docs/runbooks/forkd-snapshot-registry.md` in the `ao-company` repo (alongside `DEPLOY-forkd-controller-gate.md`), but that directory does not exist in the shared checkout and the operator-runbook content is -deeply coupled to the design note in §5 of the design file. Co-locating +deeply coupled to the design note in §7 of the design file. Co-locating them keeps the operator and design context in one place. --- @@ -33,23 +48,20 @@ them keeps the operator and design context in one place. the gate driver both go through the controller; every gate attempt during the outage will post a `snapshot-not-registered: unreachable ` infra verdict (the preflight's transport-failure branch — see design - note §5 layer 1). + note §7 layer 1). - The controller listens on `http://127.0.0.1:8891`. The bearer token lives at `~/fabro-run/.forkd-token` on the controller host (`@dellsrv`). - **The token value is NEVER to be printed, echoed, or inlined into a - command.** All commands below reference the token by file path and - read it via `$(cat ~/fabro-run/.forkd-token)` inside the `curl` header - only; the file content stays in process memory between the `cat` and - the `curl` and never lands on a log line because the bash subshell is - the only place it is expanded. + **Reference the token by its file path only.** Never `echo`, `cat`, + `print`, or otherwise surface the token VALUE on stdout, stderr, or in + any file that might be journaled. See §3 for the strict handling rules. - The maintenance window is announced BEFORE step 1. Sample wording: ``` - zenctl maint on 'forkd snapshot re-registration verification (fabro-123)' + zenctl maint on 'forkd snapshot + exec diagnostics (fabro-123 + fabro-122)' ``` --- -## 1. Confirm or refute registry volatility +## 1. Confirm or refute registry volatility (the snapshot side) This is the live probe the brief's items 1–2 hinge on. The question: does `GET /v1/snapshots` empty after a restart, even though `zen-gate-base` and @@ -103,21 +115,103 @@ journal slice from boot to +5min. Feed back to the design-note author: - Did the listing GROW between t+0s and t+300s? (If yes, the boot-scan is async — record the time-to-populate for the design note.) If the listing is empty at all three sample points AND boot succeeds for -`zen-gate-base` (step 2), the volatility hypothesis is confirmed. If the -listing is populated AND boot fails, the hypothesis is wrong and we need -to look at the on-disk artifact (present-but-broken state). +`zen-gate-base` (step 3 below), the volatility hypothesis is confirmed. +If the listing is populated AND boot fails, the hypothesis is wrong and +we need to look at the on-disk artifact (present-but-broken state). + +--- + +## 2. Diagnose exec-stage EAGAIN (os error 11) — the dominant failure + +The git evidence shows exec-stage EAGAIN dominates restore-stage 400 by +~91:1 in the live log (1092 vs 12 occurrences). It is a different bug +with a different fix path. This section asks the operator to characterize +it well enough that the controller-side fix for `zenprocess/ao-company#122` +can be sized correctly. + +The question is: when the in-VM exec stream returns EAGAIN (`os error 11`, +`Resource temporarily unavailable`), which resource is exhausted on the +controller host or in the guest? The three leading candidates are file +descriptors, process IDs, and (resident) memory. + +```bash +# 2.1 Capture the controller process baseline BEFORE any gate load. +# Run this ON dellsrv as the controller's runtime user; if the +# controller is in its own cgroup or systemd unit, prefer the +# cgroup path (replace $UNIT with the actual unit name). +UNIT=forkd-controller.service +echo "=== controller baseline ===" +ps -o pid,ppid,user,rss,vsz,cmd -C forkd-controller 2>/dev/null \ + | tee "$RECORD_DIR/baseline-ps.txt" +cat /proc/$(pgrep -f forkd-controller | head -1)/limits 2>/dev/null \ + | tee "$RECORD_DIR/baseline-limits.txt" +systemctl show "$UNIT" \ + --property=LimitNOFILE,LimitNPROC,MemoryMax,MemoryHigh,TasksMax \ + | tee "$RECORD_DIR/baseline-cgroup.txt" + +# 2.2 For each fd/pid/memory hypothesis, capture the live state at the +# moment an EAGAIN 500 fires. The trigger is the poll log line +# "exec: read response: Resource temporarily unavailable (os error 11)". +# The cleanest way is to watch the journal and snapshot the relevant +# counters on each occurrence. +journalctl -u forkd-controller -f -n 0 \ + | awk '/exec: read response: Resource temporarily unavailable/ { + system("date -u +%Y-%m-%dT%H:%M:%SZ > /tmp/eagain-trigger"); + system("ls /proc/$(pgrep -f forkd-controller | head -1)/fd 2>/dev/null | wc -l > /tmp/eagain-fds"); + system("ps -o pid,nlwp --no-headers -C forkd-controller 2>/dev/null > /tmp/eagain-threads"); + system("ps -o pid,rss,vsz --no-headers -C forkd-controller 2>/dev/null > /tmp/eagain-mem"); + }' & +EAGAIN_TAP_PID=$! + +# 2.3 Generate a small amount of gate load so EAGAIN fires (run only +# if the live queue has no pending SHAs). The exact trigger is +# environment-specific; the operator may instead drive load by +# running fabro-gate-poll.sh a few times back-to-back. The goal is +# to capture ≥3 EAGAIN-triggered samples, then STOP and look at the +# samples. Do NOT drive load longer than needed. +bash ~/Desktop/ao-company/bin/fabro-gate-poll.sh + +# 2.4 Stop the journal tap and dump the captured samples. +kill "$EAGAIN_TAP_PID" 2>/dev/null || true +echo "=== EAGAIN-triggered snapshots ===" +for f in /tmp/eagain-fds /tmp/eagain-threads /tmp/eagain-mem /tmp/eagain-trigger; do + [ -f "$f" ] && echo "$f:" && cat "$f" && echo +done | tee "$RECORD_DIR/eagain-samples.txt" +``` + +**What to record**: the controller baseline (process info, rlimit/cgroup +limits) and the EAGAIN-triggered snapshots of fd count, thread count, RSS. +Feed back to the design-note author: + +- **File descriptors**: open-fd count at the moment of EAGAIN vs the + controller's `LimitNOFILE`. If `open_fds / LimitNOFILE > 0.8`, fd + exhaustion is the dominant cause. +- **Process / thread count**: `nlwp` (number of light-weight processes) at + the moment of EAGAIN vs `LimitNPROC` / `TasksMax`. If `nlwp` is at or + near the limit, PID exhaustion is the dominant cause. +- **Memory**: `rss` / `vsz` at the moment of EAGAIN vs `MemoryMax` / + `MemoryHigh`. If rss is at or near the limit, memory exhaustion is + the dominant cause. +- **Pattern across EAGAIN samples**: is the resource monotonically + growing across samples (cumulative leak) or stable (just hitting a + fixed ceiling)? The fix shape depends on this — a leak needs + identification + close; a fixed ceiling needs raising or pool sizing. + +The design note's §5 (UNVERIFIED items) calls out this exact question; +the live answer closes that gap and lets the operator file a sized fix +against `zenprocess/ao-company#122`. --- -## 2. Per-tag bootability check (which is missing, which is broken) +## 3. Per-tag bootability check (which golden is missing, which is broken) For each golden tag, run a real restore-boot canary. The canary must be a real boot + exec `true`; a files-exist check is exactly the wrong -validation (design note §5). Tags to verify against: `zen-gate-base`, -`zen-gate-big`, and any other golden listed in the controller's -configuration (`FORKD_GOLDEN_TAGS` or the equivalent — document what is -actually used on dellsrv before this runbook is run, since the source -is not local). +validation (design note §7 layer 2). Tags to verify against: +`zen-gate-base`, `zen-gate-big`, and any other golden listed in the +controller's configuration (`FORKD_GOLDEN_TAGS` or the equivalent — +document what is actually used on dellsrv before this runbook is run, +since the source is not local). ```bash TOKEN_FILE=~/fabro-run/.forkd-token @@ -128,7 +222,7 @@ canary() { # $1=tag local tag="$1" echo "=== canary tag=$tag ===" - # 2.1 POST /v1/sandboxes with the tag. 201 with a sid is the happy path. + # 3.1 POST /v1/sandboxes with the tag. 201 with a sid is the happy path. local create_body create_body=$(curl -sS -X POST \ -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ @@ -138,7 +232,7 @@ canary() { # $1=tag echo "create response: $create_body" echo "$create_body" | tee "$RECORD_DIR/create-$tag.json" - # 2.2 Extract the sid. The controller's response shape is documented + # 3.2 Extract the sid. The controller's response shape is documented # in `docs/DEPLOY-forkd-controller-gate.md` as confirmed-working # across ~30 live runs; the multi-shape fallback is in # forkd-shim.py:_extract_sid. @@ -163,7 +257,7 @@ elif isinstance(d, list) and d: return 0 fi - # 2.3 POST /v1/sandboxes/{sid}/exec with `sh -c true`. This is the + # 3.3 POST /v1/sandboxes/{sid}/exec with `sh -c true`. This is the # REAL restore-boot canary — not a files-exist check. local exec_body exec_body=$(curl -sS -X POST \ @@ -174,7 +268,7 @@ elif isinstance(d, list) and d: echo "exec response: $exec_body" echo "$exec_body" | tee "$RECORD_DIR/exec-$tag.json" - # 2.4 DELETE /v1/sandboxes/{sid} to clean up. The forkd-shim's + # 3.4 DELETE /v1/sandboxes/{sid} to clean up. The forkd-shim's # teardown is confirmed live across the same ~30 runs. curl -sS -X DELETE \ -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ @@ -205,7 +299,7 @@ State table the runbook produces: --- -## 3. Token path reference (NEVER print the value) +## 4. Token path reference (handling rules — softened from revision 1) The token is referenced at `~/fabro-run/.forkd-token` on the controller host (dellsrv). The path is set in `bin/forkd-shim.py` and overrides the @@ -214,16 +308,40 @@ host and is NOT what the gate uses). The driver-side override is `FABRO_TOKEN_FILE_REMOTE` in `bin/fabro-github-gate.sh`. If the token ever moves, the runbook still works as long as that env var is updated. -The token VALUE is never in this runbook. The `$(cat ~/fabro-run/.forkd-token)` -substitution lives inside the bash subshell that builds the curl header -argument; it never reaches a log line because the curl `-H` value is -processed in-process. If a runbook command needs to be pasted into a -shell history, the operator should configure `HISTCONTROL=ignorespace` -and prefix with a space, or use a one-shot alias that does not record. +### Handling rules + +- **Reference the token by file path only.** Every command in this + runbook uses `$(cat "$TOKEN_FILE")` inside the curl `-H` argument + and nowhere else. The token VALUE is never typed, echoed, or inlined + into a shell snippet, a script, a config, or a log line. +- **Before starting the maintenance window, confirm shell tracing is + OFF.** Run `set +x` and check `$SHELLOPTS` does not contain `xtrace`. + The previous revision of this runbook asserted that "the bash + subshell is the only place it is expanded" — that was overconfident. + A shell with `set -x`, a `PS4` that prints expanded commands, a + process tracer (`strace`, `dtrace`), or a wrapping utility that + captures argv can expose the expanded token. The operator must + verify none of these are in play before the runbook commands run. +- **Disable shell history capture for the maintenance window.** + `HISTCONTROL=ignorespace` plus a leading space on each command, or + a one-shot `env HISTFILE=/dev/null bash --noprofile --norc` for the + whole runbook session. +- **Do not paste runbook commands into a chat window, terminal + capture, or pastebin.** The expanded `$(cat ...)` shows up in the + rendered command. Operators running this from a tmux session with + `capture-pane` or from a screen recording should disable capture + for the duration. +- **Do not redirect curl output containing the token** (e.g. via `-v`, + `-w`, or `--trace`). The default `curl -sS` is fine because the + `-H` value is processed in-process and never reappears in the + output body; `-v` and `--trace` violate this. + +The token VALUE is never in this runbook. Following the rules above is +what keeps it out of any artifact the operator might keep. --- -## 4. Expected success log line (after the controller fix is deployed) +## 5. Expected success log line (after the controller fix is deployed) The controller MUST emit exactly one log line per re-registered snapshot, in the format: @@ -260,40 +378,42 @@ shows NEITHER for `zen-gate-big`, the controller fix is not deployed. --- -## 5. Hand-off back to the design-note author +## 6. Hand-off back to the design-note author After the runbook completes, the operator should send back: 1. The three `GET /v1/snapshots` listings from step 1 (`$RECORD_DIR/snapshots-t+0s.json`, `…t+60s.json`, `…t+300s.json`). 2. The journal slice (`/tmp/forkd-journal.log`). -3. The four create+exec response bodies from step 2 (`$RECORD_DIR/create-.json`, `$RECORD_DIR/exec-.json`). -4. The state table from step 2 (which golden is missing, which is present-but-broken, which is healthy). +3. The four create+exec response bodies from step 3 (`$RECORD_DIR/create-.json`, `$RECORD_DIR/exec-.json`). +4. The state table from step 3 (which golden is missing, which is present-but-broken, which is healthy). +5. The controller baseline + EAGAIN-triggered samples from step 2 (`$RECORD_DIR/baseline-*.txt`, `$RECORD_DIR/eagain-samples.txt`) and a one-line summary of which resource (fd / pid / memory) is exhausted when EAGAIN fires. -The design note's §5 fix direction depends on this signal: +The design note's §7 fix direction depends on signals 1–4: - If `zen-gate-big` is **missing** on registry AND on disk → the fix is a re-commit from the canonical rootfs; the controller-side re-registration will pick it up. - If `zen-gate-big` is **present-but-broken** → the fix is to replace the on-disk file (restore from a prior good snapshot, or re-commit from canonical rootfs); the cached metadata handles are stale. - If `zen-gate-big` is **healthy by registry but restore_many still 400s at the gate** → the registry's success criterion is not a real canary; the controller-side fix is wrong and needs to be reworked to require a real restore-boot (not a files-exist check). -Once the operator's live signal is in, the design note's §5 layer 2 -implementation can be reviewed against the actual controller source on -dellsrv. +Signal 5 closes `zenprocess/ao-company#122`'s UNVERIFIED item (which +resource is exhausted). Once the operator's live signals are in, the +controller-side fixes for both issues can be reviewed against the actual +controller source on dellsrv. --- -## 6. Closing the maintenance window +## 7. Closing the maintenance window ```bash -# 6.1 Confirm the registry is healthy. +# 7.1 Confirm the registry is healthy. curl -sS -H "Authorization: Bearer $(cat ~/fabro-run/.forkd-token)" \ http://127.0.0.1:8891/v1/snapshots \ | jq 'length, [.[].tag // .[].snapshot_tag]' -# 6.2 Optionally fire one gate-poll cycle to verify the preflight's +# 7.2 Optionally fire one gate-poll cycle to verify the preflight's # `ok ` path lands cleanly. bash ~/Desktop/ao-company/bin/fabro-gate-poll.sh -# 6.3 Close the maintenance window. +# 7.3 Close the maintenance window. zenctl maint off ``` diff --git a/docs/internal/forkd-snapshot-registry.md b/docs/internal/forkd-snapshot-registry.md index 8c120b78b..080d8b3aa 100644 --- a/docs/internal/forkd-snapshot-registry.md +++ b/docs/internal/forkd-snapshot-registry.md @@ -4,7 +4,9 @@ acceptance-criterion #4 (the design note) plus the operator runbook companion (`forkd-snapshot-registry-runbook.md`, same directory). -**Author**: fabro-123 doc worker, 2026-08-01. +**Author**: fabro-123 doc worker, 2026-08-01. (Revision 2 — supersedes the +2026-08-01 first revision on this branch. See §3 for what was wrong and +why. The orchestrator's review caught three errors in the first revision.) **Code side**: `feat(gate): snapshot preflight in fabro-github-gate.sh (fabro-123)` — commit `e1b3b5b` on branch @@ -18,93 +20,180 @@ lives here); the controller is a separate service on dellsrv. Items 1 and 2 of `zenprocess/ao-company#123` are operator work and the runbook in the companion doc hands off cleanly. +**Supersedes**: this PR (https://github.com/zenprocess/fabro/pull/33) +supersedes https://github.com/zenprocess/fabro/pull/32. The orchestrator +flagged #32 as a duplicate and asked this worker not to merge or rebase +it. The content of #32's docs file (`docs/internal/forkd-snapshot-truthfulness.md`, +347 lines) shares the timeline structure but DOES NOT carry the §1 +two-failure-mode headline, DOES NOT use `gh api` + `fabro-gate-health.json` +as the source of truth for posted verdicts, and DOES NOT correct the +"gated=1 means success" error that this revision explicitly walks back. +#32 also touches `lib/crates/fabro-sandbox/*` files; those paths are +DEAD after the fabro crate reorg (the sandbox crate now lives under +`lib/components/fabro-sandbox/`) and those file changes are NOT carried +forward into this PR. + +--- + +## 1. Headline: TWO distinct live infra failures + +The git evidence shows the gate is hitting **two different infra failures +on the controller**, NOT one. The dominant one — by ~91:1 in the live +log — is the **exec-stage EAGAIN** of `zenprocess/ao-company#122`: +`controller POST /v1/sandboxes//exec -> HTTP 500 {"error":"exec: read +response: Resource temporarily unavailable (os error 11)"}`. The rarer +one is the **restore-stage 400** of `zenprocess/ao-company#123` (this +design note's subject): +`controller POST /v1/sandboxes -> HTTP 500 {"error":"restore_many: +firecracker API PUT /snapshot/load returned 400: {\"fault_message\":...}"}`. + +| failure mode | occurrences in `fabro-gate-poll.log` | endpoint that 500s | stage | +|---|---|---|---| +| exec EAGAIN (os error 11) | **1092** | `POST /v1/sandboxes//exec` | exec — restore succeeded, exec read failed | +| restore_many 400 | **12** | `POST /v1/sandboxes` (no sid) | boot — firecracker refused to load the snapshot | + +These are not the same bug and they do not have the same fix. The +restore-stage 400 is what this design note addresses (make `GET +/v1/snapshots` truthful so the gate's preflight can distinguish "missing +tag" from "generic infra 500"). The exec-stage EAGAIN is a separate root +cause — most likely a controller-side resource limit (fd / pid / memory) +being exhausted on the in-VM exec stream after enough concurrent +sandboxes — and is tracked in `zenprocess/ao-company#122`. The +orchestrator's brief to this worker asserted "you cannot exec into a VM +you failed to restore" — that reasoning was plausible but the log +refutes it: 1092 of 1104 (≈99%) of the live 500s in the log are at the +exec stage, which means restore succeeded and the failure is downstream. +The orchestrator's correction is the headline of this note. + --- -## 1. Timeline (verified facts first) +## 2. Timeline (verified facts first) | Date (UTC) | Event | Source | |---|---|---| -| 2026-07-31 | "Brokered zen-gates" probe: controller booted snapshot tag `zen-gate-base` and exec'd a command with exit 0, **yet** `GET /v1/snapshots` returned `[]`. | `zenprocess/ao-company#123` problem statement (UNVERIFIED by this worker — the brokered zen-gates path is not reachable from this sandbox; see §5) | +| 2026-07-31 | "Brokered zen-gates" probe: controller booted snapshot tag `zen-gate-base` and exec'd a command with exit 0, **yet** `GET /v1/snapshots` returned `[]`. | `zenprocess/ao-company#123` problem statement (UNVERIFIED by this worker — the brokered zen-gates path is not reachable from this sandbox; see §6) | | 2026-07-31 | Earlier restore failure mode appears in `~/.ao/state/fabro-gate-poll.log` for `zenprocess/uniforme` heads `54da9d8`, `3b25011`, `8ec811f`, `e839d29`: `restore_many: firecracker API PUT /snapshot/load returned 400: {"fault_message":"Load snapshot error: Failed to restore from snapshot: Failed to build microVM from snapshot: Failed to res…"}` (description truncated to 140 chars by `gh-status.sh`). | `~/.ao/state/fabro-gate-poll.log` lines 2826, 2834, 2842, 2856, 2864 (VERIFIED) | -| 2026-08-01T08:32:27Z | Descriptor-driven gate-poll cycle runs against `pawbench`, `trader`, `uniforme`, `foundry`. Trader and foundry gate successfully (`GATE_CYCLE … heads=1 gated=1`); uniforme head `e9ac694` fails at exec stage with `controller POST /v1/sandboxes/sb-6a6dae50-027d/exec -> HTTP 500 {"error":"exec: read response: Resource temporarily unavailable (os error 11)"}` (a different failure mode from the restore-many 400). Status POSTING works: `posted fabro/qa-pipeline=error on zenprocess/uniforme@e9ac694`. | `~/.ao/state/fabro-gate-poll.log` lines 7307–7322 (VERIFIED) | -| 2026-08-01T08:32:33Z | Blast-radius snapshot (`~/.ao/state/fabro-gate-health.json`): `foundry.last_verdict=error`, `trader.last_verdict=error`, `pawbench.never_gated`, `uniforme` has `deferred_heads=[e9ac694…]` with no verdict. Every gated repo is down on the same cause (snapshot infra, not the gate pipeline). | `~/.ao/state/fabro-gate-health.json` (VERIFIED) | - -### Note on what the issue claims vs. what the log shows - -The issue body (`zenprocess/ao-company#123`) attributes the -`restore_many: … PUT /snapshot/load returned 400` failure to -`trader PR #354 head 3ed3b8a4` at 2026-08-01T08:32:33Z. **I cannot verify -that exact pairing in the log.** A `grep` for `3ed3b8a` against -`fabro-gate-poll.log` returns zero matches; the only `restore_many` entries -I can find are on `zenprocess/uniforme` heads at earlier dates, and the -2026-08-01T08:32:27Z poll shows trader head `3ed3b8a4` *succeeded* -(`GATE_CYCLE repo=trader heads=1 gated=1 deferred=0`). The truncated -`{"fault_message":"…"}` JSON for the restore-many 400 is identical in shape -to the issue text, so the failure mode is real; the SHA pairing and the -exact repo it lands on are NOT independently verified from this side. I -flag this as an accuracy gap in the issue, not a contradiction of its core -claim that snapshot restore is broken. +| 2026-08-01T08:32:27Z | Descriptor-driven gate-poll cycle runs against `pawbench`, `trader`, `uniforme`, `foundry`. The poll log records `GATE_CYCLE repo=trader heads=1 gated=1 deferred=0` and `GATE_CYCLE repo=foundry heads=1 gated=1 deferred=0`. **`gated=1` is a gate-attempt count, NOT a verdict** (a gate run that returns HTTP 500 still increments `gated`). The authoritative verdict for trader head `3ed3b8a4…` is on GitHub (see next row). The poll log in this slice shows one `posted fabro/qa-pipeline=error on zenprocess/uniforme@e9ac694` (exec-stage EAGAIN); the trader/foundry `posted` lines for this cycle are not in this log slice — all `posted fabro/qa-pipeline=` lines in this file happen to be for `uniforme`, which is the repo this cycle's poller happened to surface as a posted-failure line (the other verdicts live on GitHub, not the file). | `~/.ao/state/fabro-gate-poll.log` lines 7307–7322 (VERIFIED counts; SHA/verdict inference is the orchestrator's correction, see §3) | +| 2026-08-01T08:32:33Z | `zenprocess/trader` PR #354 head `3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c` carries `fabro/qa-pipeline` status with `state=error`, `description="infrastructure: controller POST /v1/sandboxes -> HTTP 500 {\"error\":\"restore_many: firecracker API PUT /snapshot/load returned 400: {\"fault_..."`, `created_at=2026-08-01T08:32:33Z`. GitHub is the source of truth for what status was posted when; the poll log does not always capture the per-SHA posted line for every cycle. | `gh api repos/zenprocess/trader/commits/3ed3b8a4…/status` (VERIFIED) | +| 2026-08-01T08:32:27Z | Blast-radius snapshot (`~/.ao/state/fabro-gate-health.json`): `foundry.last_verdict=error`, `trader.last_verdict=error`, `trader.last_gated_sha=3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c`, `trader.last_success_at=null`, `pawbench.never_gated`, `uniforme` has `deferred_heads=[e9ac694…]` with no verdict. Every gated repo is down on the same controller-side cause. | `~/.ao/state/fabro-gate-health.json` (VERIFIED) | + +--- + +## 3. What this worker got wrong in the previous revision (and why) + +This is the most important correction in this document. In the previous +revision of this note (commit `40966cda` on the same branch), this worker +wrote: + +> the 2026-08-01T08:32:27Z poll shows trader head `3ed3b8a4` *succeeded* +> (`GATE_CYCLE repo=trader heads=1 gated=1 deferred=0`). The truncated +> `{"fault_message":"…"}` JSON for the restore-many 400 is identical in +> shape to the issue text, so the failure mode is real; the SHA pairing +> and the exact repo it lands on are NOT independently verified from +> this side. + +That was wrong on three counts: + +1. **`gated=1` is not a success verdict — it is a gate-attempt count.** + A gate run that returns HTTP 500 still increments `gated`. The + authoritative verdict lives in (a) the GitHub commit status and (b) + `fabro-gate-health.json`, not in the `GATE_CYCLE` summary line. The + worker conflated "the gate ran" with "the gate passed". +2. **The SHA pairing IS verified.** GitHub is the source of truth for + what status was posted when, and `gh api repos/zenprocess/trader/ + commits/3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c/status` returns + `state=error` with the restore_many 400 description at + `2026-08-01T08:32:33Z`. `fabro-gate-health.json` corroborates: + `trader.last_gated_sha=3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c`, + `trader.last_verdict=error`, `trader.last_success_at=null`. The + worker asserted "the SHA pairing is not independently verified" when + in fact two authoritative sources agreed on it. +3. **The "log doesn't record 3ed3b8a" finding was misleadingly worded.** + The poll log's `posted fabro/qa-pipeline=` lines in this slice happen + to be only for `uniforme` — not because the trader/foundry status + wasn't posted, but because the log file as captured shows the + most-recent cycle's posted line and this cycle's posted-failure line + was for `uniforme@e9ac694`. The correct finding is: this poll log + slice does not include the trader/foundry `posted` lines for this + cycle; the GitHub status endpoint and `fabro-gate-health.json` are + the authoritative record for what was posted. + +The orchestrator's review caught all three. The headline of this note +(promoted to §1) is the correction: there are TWO distinct live infra +failures, the dominant one is exec-stage EAGAIN (#122), and the rarer +one is restore-stage 400 (#123, the subject of this design note). The +log is one source of truth; GitHub + `fabro-gate-health.json` are the +source of truth for what status was actually posted. The worker +conflated the two sources and asserted the absence of evidence as the +evidence of absence. The most defensible summary of the timeline: -- **snapshot restore is broken on the gate's hot path** (either as - `restore_many … 400` or as the later `exec: read response: Resource - temporarily unavailable (os error 11)` pattern — both observed live); +- **snapshot restore is broken on the gate's hot path** (the + `restore_many … 400` pattern, VERIFIED on trader head + `3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c` via GitHub + + `fabro-gate-health.json`, plus 11 older log lines on uniforme heads); +- **exec-stage EAGAIN is ALSO broken on the gate's hot path** (the + `exec: read response: Resource temporarily unavailable (os error 11)` + pattern, 1092 log lines, dominated by `uniforme` heads — different + root cause, different fix path, tracked in + `zenprocess/ao-company#122`); - **`GET /v1/snapshots` returns `[]`** at least once after a controller restart, while the same controller can still boot the same tag from a cold start (per the 07-31 probe); -- **status posting is intact** (trader/foundry gating ran and posted - `success` verdicts through the same plumbing); this is NOT an - `exec: EAGAIN` regression of `zenprocess/ao-company#122` (that exec EAGAIN - was live-confirmed at exit 0 in #122 — you cannot exec into a VM you - failed to restore). +- **status posting is intact** for both failure modes (trader head + `3ed3b8a4…` carries a posted `error` from this poll cycle; + uniforme `e9ac694` carries a posted `error` from this poll cycle). --- -## 2. Two distinct symptoms, one root cause +## 4. Two distinct symptoms, one root cause (for the restore-stage 400) The 07-31 evidence and the 08-01 evidence look superficially different -but are consistent with a single underlying defect: **the snapshot registry -on the forkd controller is not derived from boot-time truth, so it can -both under-report (return `[]` when disks have snapshots) and over-report -(return tags whose underlying snapshot file is corrupt or missing in a way -that firecracker rejects at `PUT /snapshot/load`)**. +but are consistent with a single underlying defect on the **restore path** +(this note's subject): **the snapshot registry on the forkd controller is +not derived from boot-time truth, so it can both under-report (return `[]` +when disks have snapshots) and over-report (return tags whose underlying +snapshot file is corrupt or missing in a way firecracker rejects at +`PUT /snapshot/load`)**. | Symptom | What it says about the registry | |---|---| | 07-31: `GET /v1/snapshots` → `[]`, `zen-gate-base` boots + execs | Listing endpoint is decoupled from the boot path. Either (a) the registry is in-memory only and lost on controller restart while disk artifacts persist, or (b) the registry is stored in a state table the boot path does not consult — tags are resolved tag-directly at boot. Both explanations fit. | -| 08-01: `restore_many: PUT /snapshot/load returned 400` | The boot path IS consulting something, and what it consults disagrees with the file on disk in a way firecracker treats as fatal. Two possibilities consistent with the evidence: (a) the on-disk snapshot file is corrupt/truncated, or (b) the snapshot metadata the controller has cached does not match the file. | +| 08-01: `restore_many: PUT /snapshot/load returned 400` (trader head `3ed3b8a4…`) | The boot path IS consulting something, and what it consults disagrees with the file on disk in a way firecracker treats as fatal. Two possibilities consistent with the evidence: (a) the on-disk snapshot file is corrupt/truncated, or (b) the snapshot metadata the controller has cached does not match the file. | The leading hypothesis — that **the registry is volatile across restart -while files persist on disk** — explains why `[]` could coexist with a -successful exec (boot path resolves tags directly at boot, registry is -empty in-memory), but it does NOT by itself explain the `restore_many` 400 -at 08-01 (which would require either corruption, a stale cached metadata -handle, or a different snapshot file on disk than the one the controller -thinks it is loading). +while files persist on disk** — explains why `[]` could coexist with +a successful exec (boot path resolves tags directly at boot, registry +is empty in-memory), but it does NOT by itself explain the +`restore_many` 400 at 08-01 (which would require either corruption, a +stale cached metadata handle, or a different snapshot file on disk than +the one the controller thinks it is loading). --- -## 3. Evidence vs. hypothesis (be honest about which is which) +## 5. Evidence vs. hypothesis (be honest about which is which) ### VERIFIED (this side can show the bytes) -- **The status-posting pipeline works.** Trader head 3ed3b8a4 (2026-08-01T08:32) and foundry head b919955d (same poll) ran and posted `success` verdicts via `fabro/qa-pipeline`. The PR #140 poller surfaced a real `error` to uniforme e9ac694 in the same cycle. (`fabro-gate-poll.log` lines 7307–7322.) -- **Snapshot restore fails on the gate's hot path.** Both the `restore_many: PUT /snapshot/load returned 400` shape (uniforme heads `54da9d8`, `3b25011`, `8ec811f`, `e839d29` at earlier dates) and the `exec: read response: Resource temporarily unavailable (os error 11)` shape (uniforme e9ac694 at 2026-08-01T08:32:27Z) appear in the log. The two shapes are distinct: restore_many fails inside firecracker on snapshot load; the EAGAIN fails in the exec-stream read after a presumably successful boot. They are not the same bug. +- **Two failure modes with a ~91:1 ratio.** `grep -c 'exec: read response: Resource temporarily unavailable' ~/.ao/state/fabro-gate-poll.log` → 1092. `grep -c 'restore_many' ~/.ao/state/fabro-gate-poll.log` → 12. The exec-stage EAGAIN dominates; restore-stage 400 is real but rarer. +- **Trader head `3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c` carries `fabro/qa-pipeline=error` at 2026-08-01T08:32:33Z with the `restore_many` 400 description.** Verified via `gh api repos/zenprocess/trader/commits/3ed3b8a4…/status`. `fabro-gate-health.json` corroborates: `trader.last_gated_sha=3ed3b8a4…`, `trader.last_verdict=error`, `trader.last_success_at=null`. +- **The status-posting pipeline works.** Trader head `3ed3b8a4…` and foundry head `b919955d…` posted `error` verdicts via `fabro/qa-pipeline`; the poller surfaced a real `error` to uniforme `e9ac694` in the same cycle. +- **Snapshot restore fails on the gate's hot path.** Both the `restore_many: PUT /snapshot/load returned 400` shape (VERIFIED on trader head `3ed3b8a4…` via GitHub, plus 11 older log lines on uniforme heads `54da9d8`, `3b25011`, `8ec811f`, `e839d29` at earlier dates) and the `exec: read response: Resource temporarily unavailable (os error 11)` shape (uniforme e9ac694 at 2026-08-01T08:32:27Z) appear in the log. The two shapes are distinct: restore_many fails inside firecracker on snapshot load; the EAGAIN fails in the exec-stream read after a presumably successful boot. - **`GET /v1/snapshots` is consulted by the gate-side preflight** (commit `e1b3b5b`, `ao/fabro-123/snapshot-preflight`). On mismatch the gate posts `snapshot-not-registered: tag= absent from GET /v1/snapshots` — a distinct infra verdict from generic restore-many HTTP 500. - **The fabro-sandbox forkd client does not call `GET /v1/snapshots` itself.** `lib/components/fabro-sandbox/src/provider/forkd.rs` consumes `POST /v1/sandboxes`, `POST /v1/sandboxes/{id}/exec`, `DELETE /v1/sandboxes/{id}` only. The listing endpoint is only consumed by the gate-side preflight (`bin/fabro-github-gate.sh:46` `preflight_snapshot()`), via `bin/forkd-shim.py:229 list_snapshots()`. -- **Blast radius.** `~/.ao/state/fabro-gate-health.json` shows `foundry` and `trader` with `last_verdict=error`, `uniforme` with `deferred_heads=[e9ac694…]` and no verdict, `pawbench` never gated. Every gated repo is down on the same cause. +- **Blast radius.** `~/.ao/state/fabro-gate-health.json` shows `foundry` and `trader` with `last_verdict=error`, `uniforme` with `deferred_heads=[e9ac694…]` and no verdict, `pawbench` never gated. Every gated repo is down on the same controller-side cause. ### UNVERIFIED (cannot determine from this side) - **Whether the snapshot registry is in-memory only, file-backed, or remote-DB-backed.** The controller's storage choice is invisible from the client or the spec. (UNVERIFIED because the controller source is not in the fabro repo, and dellsrv is behind the egress boundary from this sandbox.) - **Whether `zen-gate-big` is missing from on-disk storage, present-but-corrupt, or present-and-valid but uncached.** A `PUT /snapshot/load` 400 from firecracker can come from a corrupt file, a wrong-version snapshot, or a cached metadata handle pointing at the wrong backing file. The `fault_message` JSON is truncated in the gate's `--desc` field (140-char cap in `gh-status.sh`); the full payload is only on the controller. (UNVERIFIED for the same reason.) - **Whether `zen-gate-base` is still bootable today.** The 07-31 probe booted it once. A probe run after the controller's last restart is the only way to know. (UNVERIFIED — brokered path not reachable from this sandbox; one attempt made, stopped per the brief's egress boundary.) -- **Whether the brief's exact pairing (trader head `3ed3b8a4` carrying the `restore_many` 400 description at 08-32:33Z) is accurate.** The log shows trader `3ed3b8a4` *succeeded* at the timestamp the issue cites. The `restore_many` 400 entries in the log are on uniforme at older dates. The failure mode is real; the SHA/repo pairing is not. (UNVERIFIED — log diff vs. issue text.) +- **What controller-side resource is exhausted when exec EAGAIN (os error 11) fires.** The exec-stream read returning EAGAIN is consistent with fd/pid/memory exhaustion on the controller host or in the guest. The actual exhaustion point is observable only on the controller. (UNVERIFIED — same egress boundary; this is also part of what the runbook companion asks the operator to characterize.) --- -## 4. Leading hypothesis (UNVERIFIED mechanism, but evidence-consistent) +## 6. Leading hypothesis (UNVERIFIED mechanism, but evidence-consistent) The snapshot **registry** is held in a state table that is not rehydrated on controller boot, while the snapshot **artifacts** persist on disk and @@ -125,7 +214,7 @@ implement a fix against a guessed mechanism. --- -## 5. What the controller fix MUST do (design direction, not implementation) +## 7. What the controller fix MUST do (design direction, not implementation) The fix lives in two layers. @@ -180,7 +269,7 @@ that ships an unshippable golden. --- -## 6. What this PR contains vs. what the operator owns +## 8. What this PR contains vs. what the operator owns **This PR (in the fabro worktree at `~/.ao/data/worktrees/fabro/fabro-84`, branch `ao/fabro-84/forkd-snapshot-registry`):** @@ -195,25 +284,26 @@ that ships an unshippable golden. **Not in this PR (operator work, T3):** - the controller-side boot-scan re-registration (controller source is not - local; the design is in §5 for the operator to implement against the + local; the design is in §7 for the operator to implement against the real controller source), - a controller restart for testing, - a fix to the on-disk state of `zen-gate-big` (depends on the operator - runbook's findings). + runbook's findings), +- the exec-stage EAGAIN diagnostic in `zenprocess/ao-company#122` (the + runbook companion §2 captures the question for the operator, but the + fix path is a separate issue). --- -## 7. References +## 9. References - `zenprocess/ao-company#123` — issue as filed. - `zenprocess/ao-company#121` — the org-wide gate silence; the descriptor-driven poller (commit `258a5f0`) restored status posting and surfaced the snapshot failure that #123 is now the home for. -- `zenprocess/ao-company#122` — the exec `EAGAIN` regression; live-verified - at exit 0 in the 07-31 brokered probe. Exec EAGAIN is NOT the cause of - the current `error` verdicts: a successful exec EAGAIN run cannot post a - status of `error`; the current verdicts reflect snapshot-restore failure - upstream of any in-VM work. +- `zenprocess/ao-company#122` — the exec `EAGAIN` regression; the + dominant live failure mode (1092 occurrences vs 12 for restore_many + 400). Different root cause, different fix; tracked separately. - `bin/fabro-github-gate.sh:46 preflight_snapshot()` — the gate-side companion (commit `e1b3b5b`). - `bin/forkd-shim.py:229 list_snapshots()` — the shim that brokers the @@ -221,10 +311,17 @@ that ships an unshippable golden. - `lib/components/fabro-sandbox/src/provider/forkd.rs` — the fabro-side client. Consumes `POST /v1/sandboxes`, `POST /v1/sandboxes/{id}/exec`, `DELETE /v1/sandboxes/{id}`. Does not call `GET /v1/snapshots`. +- `gh api repos/zenprocess/trader/commits/3ed3b8a4da3cc4e7f41371f6ab0ace779e8a155c/status` + — the authoritative record that trader head `3ed3b8a4…` carries + `fabro/qa-pipeline=error` at 2026-08-01T08:32:33Z with the + `restore_many: PUT /snapshot/load returned 400` description. - `~/.ao/state/fabro-gate-poll.log` lines 2826, 2834, 2842, 2856, 2864 — the `restore_many: PUT /snapshot/load returned 400` evidence on earlier uniforme heads. - `~/.ao/state/fabro-gate-poll.log` lines 7307–7322 — the - 2026-08-01T08:32:27Z poll cycle in which trader/foundry gated - successfully and uniforme failed at exec stage. -- `~/.ao/state/fabro-gate-health.json` — blast-radius snapshot. + 2026-08-01T08:32:27Z poll cycle in which trader/foundry gated and + posted `error` (per GitHub) and uniforme failed at exec stage. +- `~/.ao/state/fabro-gate-health.json` — blast-radius snapshot and the + authoritative verdict ledger. +- https://github.com/zenprocess/fabro/pull/32 — superseded by this PR; + see the doc header for the differences.