diff --git a/.briefs/T1-dellsrv-runbook.md b/.briefs/T1-dellsrv-runbook.md new file mode 100644 index 000000000..49cb7c09a --- /dev/null +++ b/.briefs/T1-dellsrv-runbook.md @@ -0,0 +1,102 @@ +# TASK T1 — consolidated dellsrv T3 runbook + read-only verification checklist + +**Docs deliverable. NO live actions.** You are writing the single consolidated T3 +execution runbook for the fabro/dellsrv completion campaign, for operator Val to run +BY HAND in the morning. + +You MUST NOT ssh to dellsrv, run systemctl, run forkd, or touch any live service. The +sandbox cannot reach `dellsrv.zp.digital` anyway — DNS denial there is the egress +boundary working as designed. Do NOT attempt any workaround (no IP literals, no +tunneling, no alternate hostnames). This is a pure authoring task. + +## Deliverable + +New file `docs/internal/dellsrv-completion-runbook.md` in the fabro repo. Branch off +`origin/main`, commit, push, open a **DRAFT** PR against `zenprocess/fabro`. + +## Sources — read these first + +1. `~/.ao/data/aofactory/FABRO-COMPLETION-RUNBOOK.md` — the operator mandate (9 numbered + items). Authority on WHAT must happen. +2. `scripts/ops/dellsrv-forkd-supervision/README.md` on `origin/main` — the already-merged + gate-health-probe runbook (PR #17). Its deploy section is the model for tone and + precision. Do NOT duplicate its content; REFERENCE it. +3. `git log origin/main` plus `gh pr view 15`, `16`, `17`, `19` `--repo zenprocess/fabro` + — the commit/PR trail showing what already landed. + +## Content — exactly two parts + +### PART A — READ-ONLY VERIFICATION CHECKLIST (put this FIRST; it is the more important half) + +Numbered, copy-pasteable, strictly NON-MUTATING commands Val runs on dellsrv to establish +ground truth. Each entry needs: (a) the exact command, (b) what output means DONE, (c) what +output means NOT DONE. Cover at minimum: + +- **forkd-controller binary**: exists at `/usr/local/bin/forkd-controller`, `--version` + reports v0.5.2, and is NOT a deleted inode (compare against `/proc//exe` of the + running controller). +- **Which unit supervises forkd**: `systemctl cat forkd-ec.service`, `is-enabled`, + `is-active`, and whether `forkd-ec-boot-dellsrv.sh` references the `/usr/local/bin` + binary or something ephemeral. +- **Golden option-2 tags**: list snapshots INSIDE the forkd container; confirm BOTH + `zen-gate-base` AND `zen-gate-big` exist; confirm each `memory.bin` size + (`zen-gate-base` 1024 MiB = 1073741824 bytes, `zen-gate-big` 4096 MiB = 4294967296 + bytes). The pre-existing baseline was **536870912** bytes (512 MB) — if you see that, + option-2 did NOT happen. +- **gate-health-probe**: `systemctl list-timers gate-health-probe.timer`, `is-enabled`, + and `journalctl -t gate-health-probe --since -24h | grep -c FORKD-GATE-ALERT`. +- **Posting/poller state**: how to determine whether gate verdict posting is ON or OFF. + **CRITICAL FACT you must encode**: the fabro gate posts to the **Forgejo** forge + (contexts `fabro/qa-pipeline`, `fabro/qwen-review`), NOT github.com. `gh api .../statuses` + is ALWAYS empty and is NOT a valid health check. Say so explicitly so nobody re-derives + that wrong conclusion. + +### PART B — MUTATING STEPS, gated on Part A results + +Each remaining item in dependency order, each with a stated PRECONDITION (which Part A +check must have FAILED for this step to be needed) and a ROLLBACK. Cover: + +- **Golden tag registration** via + `forkd snapshot --tag --kernel --rootfs --tap --boot-wait-secs --mem-size-mib `. + Option 2 = `zen-gate-base` at 1024 and `zen-gate-big` at 4096, off the SAME existing + 20 GB golden rootfs — this is a **RE-REGISTER, not a re-bake**. Rollback = deregister + the new tag. +- **gate-health-probe timer enable** — commands already exist in the PR #17 README; + reference them, do not copy. +- **Re-enabling posting + poller.** + +## Honesty requirements (non-negotiable — the whole point of the document) + +- Anywhere live state is UNKNOWN to us, write it as **UNKNOWN** with the Part A check that + resolves it. Do NOT assert live dellsrv state as fact. Everything we have is inferred + from commit messages, and one commit (`6c3ed8bac`) merely *claims* "golden option-2 is + live on dellsrv" without proof. +- Explicitly record that **PR #16** (a `forkd-controller.service` supervision unit) was + CLOSED-not-merged because a new unit would DUPLICATE `forkd-ec.service`'s ownership of + the boot script. Any reader tempted to re-add such a unit must hit that warning. The + durability fix is canonicalizing the recovered v0.5.2 binary that forkd-ec already + launches, NOT a competing unit. +- No invented paths or flag names. If you do not know a value (kernel path, tap name, + boot-wait-secs), write `` rather than guessing. + +## Acceptance command (must pass; paste the real output in your report) + +``` +test -f docs/internal/dellsrv-completion-runbook.md \ + && grep -qi "forgejo" docs/internal/dellsrv-completion-runbook.md \ + && grep -q "UNKNOWN" docs/internal/dellsrv-completion-runbook.md \ + && grep -q "536870912" docs/internal/dellsrv-completion-runbook.md \ + && grep -qi "PR #16" docs/internal/dellsrv-completion-runbook.md \ + && echo ACCEPT +``` + +## Adversarial check (also required) + +Pick the two most load-bearing claims in your Part A checklist and try to prove them +WRONG from the repo/PR trail. If a check would pass even when the thing it checks is +broken, it is a worthless check — fix it and say so. Report which checks you strengthened. + +## Report back + +File path, draft PR number, real acceptance output, adversarial findings. Do NOT mark +done without the pasted acceptance output. diff --git a/.briefs/T2-state-capture.md b/.briefs/T2-state-capture.md new file mode 100644 index 000000000..33c42fef4 --- /dev/null +++ b/.briefs/T2-state-capture.md @@ -0,0 +1,94 @@ +# TASK T2 — read-only dellsrv state-capture script (code-side only) + +**NO live actions.** You are authoring a script that the operator will run on dellsrv +by hand. You must NOT ssh to dellsrv, run systemctl, or touch any live service. The +sandbox cannot reach `dellsrv.zp.digital` — DNS denial there is the egress boundary +working as designed. Do NOT attempt any workaround. Author + test locally only. + +## Why this exists + +We cannot see live dellsrv state. Four campaign items (forkd-controller durability, +golden option-2 tags, gate-health-probe deploy, posting/poller state) are each blocked +on ground truth we don't have. This script captures ALL of it in one paste-able run so +the operator spends 30 seconds, not 20 minutes, and so the output is machine-comparable +next time. + +## Deliverable + +`scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh` plus a new section in +that directory's existing `README.md`. Branch off `origin/main`, commit, push, open a +**DRAFT** PR against `zenprocess/fabro`. + +## Hard requirements + +- **STRICTLY READ-ONLY.** No `systemctl start|stop|restart|enable|disable`, no + `forkd snapshot` registration, no `docker restart`, no file writes outside a + `mktemp` output file, no DELETE/POST to the controller. A reviewer must be able to + confirm read-only-ness by reading the script once. Put a comment block at the top + stating this contract, and make it true. +- **Never print secret VALUES.** The forkd token lives at `/etc/forkd-token` INSIDE the + `forkd` docker container. If you need it for a GET, follow the existing pattern in + `gate-health-probe.sh` (`docker exec -i forkd sh` with a heredoc whose first action is + `TOKEN=$(cat /etc/forkd-token)`) so the value never crosses the host argv or the + process table. Report only presence/absence, never content. +- **Every probe must be independently failable.** Wrap each check so one missing binary + or one absent unit does not abort the rest — the operator needs the WHOLE picture from + one run. Use `|| true` guards deliberately, but never in a way that turns a real + failure into a silent pass; each check must print an explicit `OK` / `MISSING` / + `UNKNOWN` verdict line. +- **Output must be diffable**: a stable, greppable line format, one fact per line, e.g. + `CHECK `. Print a summary count at the end. + +## Checks the script must capture + +1. `forkd-controller` binary: path, `--version`, sha256, and whether the RUNNING + controller's `/proc//exe` is a deleted inode or resolves to the on-disk file. + (The 2026-07-25 emergency was exactly a deleted-inode process; this check is the + durability question.) +2. `forkd-ec.service`: `systemctl cat`, `is-enabled`, `is-active`, and whether the boot + script it owns references `/usr/local/bin/forkd-controller` or an ephemeral path. +3. Golden snapshot tags inside the forkd container: which tags exist, and the byte size + of each tag's `memory.bin`. Known baseline to flag loudly: **536870912** bytes + (512 MB) means golden option-2 did NOT happen. Targets are 1073741824 (1024 MiB, + `zen-gate-base`) and 4294967296 (4096 MiB, `zen-gate-big`). +4. `gate-health-probe`: installed? `is-enabled`? timer scheduled? count of + `FORKD-GATE-ALERT` and `FORKD-GATE-HEAL` lines in the last 24h of + `journalctl -t gate-health-probe`. +5. Per-child netns presence (`/var/run/netns/forkd-child-*`) — the 2026-07-22 three-day + silent outage was these dying. +6. Posting/poller: whether the verdict poster is running, and where it posts. + **CRITICAL FACT to encode in a comment**: the gate posts to the **Forgejo** forge + (contexts `fabro/qa-pipeline`, `fabro/qwen-review`), NOT github.com. A `gh api + .../statuses` check is always empty and is NOT valid evidence of gate health. Do not + write a github.com-based check. + +## Acceptance command (must pass; paste the real output in your report) + +``` +bash -n scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \ + && shellcheck -S warning scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \ + && ! grep -nE '(systemctl (start|stop|restart|enable|disable))|(docker (restart|rm|stop))|(forkd snapshot )|(-X (POST|DELETE|PUT))' scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \ + && grep -q 536870912 scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \ + && echo ACCEPT +``` + +(The negated grep is the read-only proof — it must find NOTHING. If shellcheck is not +installed, install it or state clearly that you could not run it; do not silently drop +it from the acceptance chain.) + +## Adversarial check (required — do not skip) + +The acceptance command above is a **negative control**: it passes when the script +contains no mutating verbs. Prove it actually bites. Temporarily insert a line like +`systemctl restart forkd-ec.service` into a **scratch copy** of the script, re-run the +acceptance chain, and confirm it FAILS. Then remove it and confirm it passes again. +Report both outputs. A guard you never saw fail is not a guard. + +Also: pick the check you judge weakest (most likely to report OK while the underlying +thing is broken) and either strengthen it or state plainly why it cannot be strengthened +without live access. + +## Report back + +File path, draft PR number, real acceptance output, and the negative-control proof +(the FAIL output and the restored PASS output). Do NOT mark done without both pasted. diff --git a/.briefs/T3-referee-run-registration.md b/.briefs/T3-referee-run-registration.md new file mode 100644 index 000000000..b665b1b60 --- /dev/null +++ b/.briefs/T3-referee-run-registration.md @@ -0,0 +1,108 @@ +# TASK T3 — register referee scores as fabro server runs (code-side only, no live calls) + +Rust implementation task in `lib/components/fabro-referee`. Branch off `origin/main`, +commit, push, open a **DRAFT** PR against `zenprocess/fabro`. + +## Why + +Campaign item 8: `fabro.zp.digital/runs` shows nothing since Jul 10 because gate execs go +through forkd and referee runs write only the Mac JSONL sink — neither becomes a fabro +server run. PR #19 already merged the receiving endpoint +(`POST /api/v1/runs/registrations`, operationId `registerExternalRun`). The missing half +is the CALLER. + +## Read first + +- `docs/public/api-reference/fabro-api.yaml` — the `/api/v1/runs/registrations` path + (around line 1316) and the `RunRegistrationRequest` schema (around line 11757). **This + spec is the source of truth for the wire contract.** Do not invent field names. +- `lib/components/fabro-referee/src/emit.rs` — the existing sink emitter. Note its + idempotency design (re-running the same `(run_id, route)` overwrites rather than + double-appends). Your registration must respect the same replay semantics: the endpoint + returns 201 on create and **200 on idempotent re-registration** — treat BOTH as success. +- `lib/components/fabro-referee/src/types.rs` — `RunRow`, `Verdict`. +- Repo conventions in `AGENTS.md`/`CLAUDE.md`: Rust import style (types by name, functions + via parent module, no glob imports), the API type-ownership rules, and the test-support + boundary rules (`#[cfg(any(test, feature = "test-support"))]`, never in default features). + +## Scope — what to build + +Emit the registration alongside the existing sink write, **opt-in and off by default**: + +- Gated behind an explicit env var (e.g. `FABRO_REFEREE_REGISTER_RUNS=1` plus a base-URL + var). **Default OFF** — a fresh checkout must not attempt any network call. +- Map the verdict exactly as the spec documents: `pass` → Succeeded/Completed, + `fail` → Failed/WorkflowError, `inconclusive` → Failed/TransientInfra. +- Populate `origin.details` with real provenance (gate backend, route/tier, commit SHA). +- **Truthfulness rule from the spec, enforce it**: there is no "register a placeholder" + path. If the source cannot produce a real verdict it MUST send `inconclusive` — never + fabricate `pass`. Encode this so a future refactor cannot quietly default to `pass`. +- **Registration failure must never corrupt or block the sink write.** The JSONL sink is + the authoritative artifact; a server that is down must degrade to a logged warning, not + a lost score and not a crash. +- The spec rejects a ref NAME where a SHA is required — validate before sending and fail + loudly with a clear message rather than sending garbage. + +Keep the diff tight: roughly a new `register.rs`, its wiring in `lib.rs`/the emit path, +and a test file. Do not refactor unrelated referee code. + +## Testing — mandatory, and no live network + +Use `httpmock` (already a dev-dependency in this workspace — see PR #20's usage) to assert +against a mock HTTP responder. **No test may contact a real fabro server.** Cover at +minimum: + +1. Request body matches the OpenAPI `RunRegistrationRequest` shape (field names verified + against the YAML, not from memory). +2. All three verdict mappings, including `inconclusive` → TransientInfra. +3. **200 idempotent re-registration is treated as success**, not as an error. +4. Server 5xx / connection failure → sink write still succeeds, warning logged, no panic. +5. Feature is OFF by default: with the env var unset, no HTTP request is made at all + (assert the mock received zero hits). + +If you add shared test helpers, they go behind the `test-support` feature per the repo's +test-support boundary rules — never exported from a production module, never in default +features. + +## Acceptance command (must pass; paste the real output in your report) + +``` +cargo nextest run -p fabro-referee \ + && cargo +nightly-2026-04-14 fmt --check -p fabro-referee \ + && cargo +nightly-2026-04-14 clippy -p fabro-referee --no-deps --tests -- -D warnings \ + && echo ACCEPT +``` + +Note: a full-workspace `cargo build` is hook-blocked on this Mac; crate-scoped commands +above are the authoritative local gate. Also confirm the root `Cargo.lock` is committed if +your change adds or changes any dependency — a prior PR in this repo (#20) shipped without +it and every contributor regenerated it as uncommitted drift. Run `git status` after +testing and check. + +## Adversarial check (required — do not skip) + +Run at least **4 mutations**, one at a time, each on a scratch copy, each reverted after, +and report a table of `mutation → failing test → tests executed`: + +1. Change the `inconclusive` mapping to `pass` (this is the campaign's core failure class: + misreporting infra as a real verdict — it MUST be caught). +2. Treat the 200 idempotent response as an error. +3. Make a registration failure propagate and abort the sink write. +4. Your own choice — target the test you judge WEAKEST, not an easy one. + +**Collapse check**: the executed test count must stay constant across every mutation. A +mutation that yields `0 passed; 0 failed`, a reduced count, or a compile error only is +INCONCLUSIVE, not a pass — redo it as a clean edit. Before trusting any "not caught" +result, `diff` the mutated file to confirm the mutation actually applied (a silently +non-applied mutation looks exactly like a vacuous test). + +If a mutation is NOT caught, that is the finding — add the missing test, re-run the +identical mutation, and prove it now fails. + +## Report back + +Draft PR number, real acceptance output, the mutation table with executed counts, and +confirmation that `git diff` vs origin is clean of mutations afterward. Verify your push +by checking the REMOTE tip SHA, not by trusting an echoed "pushed" — a `git push origin +` can succeed as a no-op while your commit sits on a detached HEAD. Do NOT mark +done without pasted acceptance output. diff --git a/.briefs/T4-control-experiment.md b/.briefs/T4-control-experiment.md new file mode 100644 index 000000000..2a287c9b8 --- /dev/null +++ b/.briefs/T4-control-experiment.md @@ -0,0 +1,111 @@ +# TASK T4 — author the CONTROL EXPERIMENT + one-pass diagnostic script + +**Authoring only. NO live execution.** You cannot reach `dellsrv.zp.digital` (DNS +denial = egress boundary working as designed). Do NOT attempt any workaround, do not +disable the sandbox. Someone with host access runs what you write. + +## Why this is the critical path + +An active production outage: every fabro gate run dies at +`POST /v1/sandboxes//exec -> HTTP 500 "exec: read response: Resource temporarily +unavailable (os error 11)"` (EAGAIN). 182 occurrences, 218 INFRA / 10 FAIL / **ZERO +PASS**. + +The hypothesis space splits cleanly in half and **one experiment resolves it**: + +- If a **trivial** exec (`/bin/true`) on the **same snapshot** ALSO EAGAINs → + the fault is **deterministic/global**. Workload, memory size, Chromium, and npm are + all irrelevant, and retry/backoff would be useless (it would just fail N times). +- If the trivial exec **passes** → the fault is **workload/output-size/duration + dependent**, and retry+backoff at the exec hop becomes the leading fix. + +Everything downstream depends on this single result. Your script must produce it +unambiguously. + +## Deliverable + +`scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh` in the fabro repo, +plus a section in that directory's existing `README.md`. Branch off `origin/main`, +commit, push, open a **DRAFT** PR against `zenprocess/fabro`. + +## Part 1 — the control experiment (the point of the script) + +Run the SAME create → exec → delete cycle the gate uses, but with a trivial command: + +1. `POST /v1/sandboxes` with `{"snapshot_tag": ""}` — take the tag as `--tag`, + defaulting to `zen-gate-base`. Also support `--tag zen-gate-big` so both can be + tested (uniforme pins `zen-gate-big`). +2. `POST /v1/sandboxes/{id}/exec` with `{"args":["/bin/true"]}` and a short + `timeout_secs`. +3. `DELETE /v1/sandboxes/{id}` — **always**, via an EXIT trap, even on failure. + A prior probe leaked a live microVM on every failed run; do not repeat that. +4. Print an unmistakable verdict line: + - `CONTROL-RESULT: PASS` — trivial exec succeeded → fault is workload-dependent + - `CONTROL-RESULT: EAGAIN` — trivial exec hit the same error → fault is global + - `CONTROL-RESULT: OTHER ` — anything else; do not force it into a bucket + +**Capture the exit code correctly.** Do NOT pipe the script's own invocation — +`script.sh | tee log` reports `tee`'s status. Note this in the README. + +**Timing matters and is evidence**: record and print how long the exec call took +before failing. An EAGAIN returned in <1s implies a non-blocking fd with no poll/retry +loop (a code defect); one returned after ~120s implies a timeout. Print +`EXEC-ELAPSED: `. + +## Part 2 — one-pass diagnostic capture (so there is no round-trip per command) + +Strictly read-only. Each check prints `CHECK `, each +independently failable so one missing binary does not abort the rest: + +- `systemctl is-active/is-enabled forkd-ec.service`; `journalctl -u forkd-ec.service -n 100` +- **Was the controller restarted?** Process start time vs. snapshot registry state. + In-memory snapshots are lost on restart and must be rebuilt — check + `GET /v1/snapshots` and list what is actually registered. +- `GET /v1/sandboxes` — orphan/leak count. +- fd and process limits: `ulimit -n`, `pids.current` vs `pids.max` (cgroup), `ps -eLf | wc -l` +- `dmesg -T | grep -iE 'oom|firecracker|cannot allocate'` +- `df -h` on the VM store / snapshot root — **disk exhaustion also surfaces as EAGAIN-adjacent failures** +- `free -g`, `docker stats --no-stream` +- per-child netns presence: `/var/run/netns/forkd-child-*` +- **The proxy hop**: is `~/fabro-run/forkd-shim.py` running, on which port, and what is + `FORKD_SHIM_FORWARD_TIMEOUT_S` set to in its environment? + +## Secrets discipline + +The controller token lives at `/etc/forkd-token` **inside the `forkd` container**. +Follow the existing pattern in `gate-health-probe.sh`: wrap calls in +`docker exec -i forkd sh` with a heredoc whose first action is +`TOKEN=$(cat /etc/forkd-token)`, so the value never crosses the host argv or process +table. Report presence/absence only — **never print a token value**. + +## Acceptance command (must pass; paste real output) + +``` +bash -n scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh \ + && shellcheck -S warning scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh \ + && grep -q 'CONTROL-RESULT' scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh \ + && grep -q 'EXEC-ELAPSED' scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh \ + && ! grep -nE '(systemctl (start|stop|restart|enable|disable))|(docker (restart|rm|stop))' scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh \ + && echo ACCEPT +``` + +The negated grep is the read-only proof for Part 2 — it must find NOTHING. (The +control experiment's own create/exec/DELETE calls are intentional and are HTTP, not +those verbs.) + +## Adversarial check (required) + +Prove the acceptance chain actually bites: on a **scratch copy**, inject +`systemctl restart forkd-ec.service`, re-run the chain, confirm it FAILS; remove it, +confirm PASS. Paste both outputs. A guard you never saw fail is not a guard. + +Then: verify your EXIT trap actually fires on the failure path — simulate an exec +failure and confirm the DELETE still runs. A cleanup path that only works on success +is the leak bug all over again. + +## Report back + +File path, draft PR number, real acceptance output, and the two adversarial proofs. +Verify your push by checking the **remote tip SHA**, not an echoed "pushed" — a +`git push origin ` can succeed as a no-op while your commit sits on a detached +HEAD. Do NOT report done without pasted output. diff --git a/.salvage/fabro-74-referee/Cargo.toml b/.salvage/fabro-74-referee/Cargo.toml new file mode 100644 index 000000000..307b993a6 --- /dev/null +++ b/.salvage/fabro-74-referee/Cargo.toml @@ -0,0 +1,45 @@ +[package] +name = "fabro-referee" +edition.workspace = true +version.workspace = true +publish = false +license.workspace = true +description = "Per-attempt, per-tier scorer — the fabro Referee plane of the AO Factory (P0 linchpin)" + +[lib] +name = "fabro_referee" +doctest = false + +[[bin]] +name = "fabro-referee" +path = "src/main.rs" + +[lints] +workspace = true + +[features] +default = [] +# Test-only: expose the in-process fake gate backend so integration +# tests can wire the runner without `ao` or `dellsrv`. NEVER enabled +# in production builds — see fabro CLAUDE.md "Test support boundaries". +test-support = [] + +[dependencies] +fabro-http.workspace = true +anyhow.workspace = true +thiserror.workspace = true +serde = { workspace = true, features = ["derive"] } +serde_json = { workspace = true } +chrono = { workspace = true, features = ["clock", "serde"] } +clap = { workspace = true, features = ["derive"] } +strum = { version = "0.27", features = ["derive"] } +tracing.workspace = true +tracing-subscriber = { workspace = true, features = ["fmt", "env-filter"] } +reqwest = { workspace = true, features = ["json", "rustls", "blocking"] } + +[dev-dependencies] +httpmock = "0.8" +base64.workspace = true +ulid.workspace = true +sha2 = "0.10" +regex = { version = "1", default-features = false } diff --git a/.salvage/fabro-74-referee/lib-and-runner.diff b/.salvage/fabro-74-referee/lib-and-runner.diff new file mode 100644 index 000000000..111257e40 --- /dev/null +++ b/.salvage/fabro-74-referee/lib-and-runner.diff @@ -0,0 +1,43 @@ +diff --git a/lib/components/fabro-referee/src/lib.rs b/lib/components/fabro-referee/src/lib.rs +index bc1482351..57adf6838 100644 +--- a/lib/components/fabro-referee/src/lib.rs ++++ b/lib/components/fabro-referee/src/lib.rs +@@ -30,6 +30,7 @@ pub mod canary; + pub mod decision_log; + pub mod emit; + pub mod gate; ++pub mod register; + pub mod runner; + pub mod types; + +diff --git a/lib/components/fabro-referee/src/runner.rs b/lib/components/fabro-referee/src/runner.rs +index 217a65c18..71d2e08c7 100644 +--- a/lib/components/fabro-referee/src/runner.rs ++++ b/lib/components/fabro-referee/src/runner.rs +@@ -26,6 +26,7 @@ use tracing::{info, warn}; + use crate::decision_log::find_decision; + use crate::emit::{append_jsonl, write_markdown_summary}; + use crate::gate::GateBackend; ++use crate::register; + use crate::types::{CURRENT_SCHEMA_VERSION, GateOutput, Route, RunRow, TaskSpec, Tier, Verdict}; + + /// What the runner emits for one task. +@@ -89,6 +90,18 @@ pub fn run( + let row = make_row(run_id, task, &route, &gate_out, backfill); + append_jsonl(sink_dir, &row) + .with_context(|| format!("emit jsonl for branch={}", route.branch))?; ++ // Optional, opt-in fan-out to the Fabro run board. The JSONL ++ // sink above is the authoritative artifact; a server outage ++ // must degrade to a logged warning, never a lost score. ++ if let Err(e) = register::register(&row, &task.base_ref) { ++ warn!( ++ run_id = %run_id, ++ task = %task.task_id, ++ branch = %route.branch, ++ err = %e, ++ "referee run registration failed; JSONL sink retained as authoritative" ++ ); ++ } + rows.push(row); + } + write_markdown_summary(sink_dir, run_id, &rows).with_context(|| "emit markdown summary")?; diff --git a/.salvage/fabro-74-referee/register.rs b/.salvage/fabro-74-referee/register.rs new file mode 100644 index 000000000..a45007cc5 --- /dev/null +++ b/.salvage/fabro-74-referee/register.rs @@ -0,0 +1,78 @@ +//! Optional external registration of referee rows in the Fabro run board. + +use anyhow::{anyhow, Context, Result}; +use serde_json::json; + +use crate::types::{RunRow, Verdict}; + +const ENABLE_ENV: &str = "FABRO_REFEREE_REGISTER_RUNS"; +const BASE_URL_ENV: &str = "FABRO_REFEREE_BASE_URL"; +const HEAD_SHA_ENV: &str = "FABRO_REFEREE_HEAD_SHA"; + +/// Register a row when explicitly enabled. Failures are returned to the caller, +/// which deliberately logs them after the authoritative JSONL write succeeds. +pub fn register(row: &RunRow, base_ref: &str) -> Result<()> { + if std::env::var(ENABLE_ENV).as_deref() != Ok("1") { + return Ok(()); + } + let base_url = std::env::var(BASE_URL_ENV) + .with_context(|| format!("{BASE_URL_ENV} must be set when {ENABLE_ENV}=1"))?; + let head_sha = std::env::var(HEAD_SHA_ENV) + .with_context(|| format!("{HEAD_SHA_ENV} must be set when {ENABLE_ENV}=1"))?; + validate_sha(base_ref, "base_ref")?; + validate_sha(&head_sha, HEAD_SHA_ENV)?; + + let verdict = match row.verdict { + Verdict::Pass => "pass", + Verdict::Fail => "fail", + }; + let dispatch_path = if row.backfill { "referee_backfill" } else { "auto_detect" }; + let kind = if row.backfill { "referee_backfill" } else { "referee" }; + let details = json!({ + "kind": kind, + "tier": row.tier, + "base_sha": base_ref, + "head_sha": head_sha, + "verdict": verdict, + "verdict_at": row.ts, + "dispatch_path": dispatch_path, + "branch": row.branch, + "synthetic": row.synthetic, + "score": row.score, + "valset_hash": row.valset_hash, + "error": serde_json::Value::Null, + "backfill_at": if row.backfill { json!(row.ts) } else { serde_json::Value::Null }, + "requester": if row.backfill { json!("fabro-referee") } else { serde_json::Value::Null }, + }); + let body = json!({ "run_id": row.run_id, "origin": { "kind": kind, "details": details } }); + let url = format!("{}/api/v1/runs/registrations", base_url.trim_end_matches('/')); + let response = reqwest::blocking::Client::builder() + .timeout(std::time::Duration::from_secs(10)) + .build()? + .post(url) + .json(&body) + .send() + .context("send referee run registration")?; + if !response.status().is_success() { + return Err(anyhow!("registration returned HTTP {}", response.status())); + } + Ok(()) +} + +fn validate_sha(value: &str, label: &str) -> Result<()> { + if value.len() != 40 || !value.bytes().all(|b| b.is_ascii_hexdigit()) { + return Err(anyhow!("{label} must be a 40-character hexadecimal commit SHA, got {value:?}")); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rejects_ref_names() { + assert!(validate_sha("HEAD", "base_ref").is_err()); + assert!(validate_sha(&"a".repeat(40), "base_ref").is_ok()); + } +} diff --git a/DELLSRV-COMPLETION-RUNBOOK.md b/DELLSRV-COMPLETION-RUNBOOK.md new file mode 100644 index 000000000..f981354d2 --- /dev/null +++ b/DELLSRV-COMPLETION-RUNBOOK.md @@ -0,0 +1,1094 @@ +# dellsrv completion runbook — operator execution, 2026-07-28 morning + +**Author:** fabro orchestrator (session fabro-71), overnight 2026-07-27→28. +**Audience:** Val, executing by hand on dellsrv. +**Status of everything below:** authored from the git/PR trail ONLY. No live +dellsrv access was available (see "Why nothing was verified live" at the end). + +> ## Read this first +> +> **Every claim about live dellsrv state in this document is UNKNOWN until Part A +> is run.** Nothing here was observed. The campaign's own commit trail contains at +> least one unproven assertion (commit `6c3ed8bac` states "golden option-2 is live +> on dellsrv" with no evidence attached), so treat the trail as a *hypothesis +> generator*, not as ground truth. +> +> Part A is read-only and answers every open question. Part B is gated on Part A. +> **Do not run any Part B step whose Part A precondition passed** — you would be +> re-doing work that is already done, and in the golden-tag case that means +> churning a live gate dependency for nothing. +> +> **🔴 There is an ACTIVE OUTAGE as of 2026-07-27 night — start at PART 0, not +> Part A.** Part 0 is self-contained (it has its own read-only diagnosis step). +> Come back to Part A once production flow is restored. +> +> One correction to how the outage was escalated: it was filed under the +> controller-durability item, but `os error 11` (EAGAIN) is resource exhaustion, +> **not** the deleted-inode problem that step B1 fixes. Part 0 explains the +> distinction. Do not expect B1 to clear the outage. + +--- + +# PART 00 — ⚡ THE EXECUTABLE LIST (operator hands, in run order) + +_Added 2026-07-28 evening. Everything here needs a host no agent in this loop can +reach. Steps 1-3 are read-only and safe to run immediately, in order. Step 4 decides. +Steps 5+ mutate and are gated on what 1-4 return._ + +**Current signature** (as of the 17:09Z poll cycle, three consecutive cycles): + +``` +POST /v1/sandboxes -> HTTP 500 +{"error":"restore_many: firecracker API PUT /snapshot/load returned 400: + {\"fault_message\":\"Load snapshot error: Failed to restore from snapshot: + Failed to build microVM from snapshot: Failed to res... <-- TRUNCATED +``` + +**Established by forensics, do not re-derive:** sandbox IDs are monotonic across the +EAGAIN→restore_many transition (`003a` … `00a7, 00a8, 00a9, 00aa, 00ab, 00ac`) with +**no reset**. An in-process counter would have reset on restart. **The controller +process has been continuously up.** This kills "controller restarted and lost its +in-memory snapshot registry." Something changed the *snapshot's restorability* while +the controller stayed up. + +--- + +### STEP 1 — Get the full fault message (read-only, ~10s) + +The truncation at `Failed to res...` is hiding the one string that names the cause. +It usually identifies the missing/mismatched piece: memory file, vmstate, kernel or +rootfs drive path, or a CPU-feature/template mismatch. + +```bash +# a) From the controller's own logs (not the poll log, which truncates): +docker logs --since 2h forkd 2>&1 | grep -iA5 'restore_many\|snapshot/load\|fault_message' | tail -40 + +# b) Or provoke one directly and read the untruncated body: +docker exec -i forkd sh <<'EOF' +TOKEN=$(cat /etc/forkd-token) +curl -sS -X POST http://127.0.0.1:8891/v1/sandboxes \ + -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ + --data '{"snapshot_tag":"zen-gate-big"}' +EOF +``` + +**This single string likely collapses steps 2-4.** If it names a specific file, go +straight to that file's forensics. + +### STEP 2 — Golden / snapshot artifact forensics (read-only, ~30s) + +Tests the three live candidates: (a) the backing golden was modified/replaced/ +truncated under a live controller, (b) the memory/vmstate file was evicted or its fd +invalidated, (c) the VM store filled up. + +```bash +# Golden artifacts + integrity state (host paths): +ls -la --time-style=full-iso /data/forkd-cow/golden/ +cat /data/forkd-cow/golden/golden.sha 2>/dev/null +cat /data/forkd-cow/golden/.golden-integrity-state 2>/dev/null +lsattr /data/forkd-cow/golden/*.ext4 2>/dev/null # is the +i immutable flag still set? + +# Snapshot artifacts — per tag, with sizes and mtimes: +ls -la --time-style=full-iso /var/lib/forkd-dellsrv/forkd-snapshots/*/ + +# THE COMPARISON THAT MATTERS: does the golden's CURRENT sha match the recorded one, +# and is its mtime NEWER than the snapshot that was built from it? +sha256sum /data/forkd-cow/golden/*.ext4 + +# Disk pressure on both stores: +df -h /data/forkd-cow /var/lib/forkd-dellsrv + +# Did the integrity service act recently? +systemctl status fabro-golden-integrity.service --no-pager | tail -20 +journalctl -u fabro-golden-integrity.service --since -48h --no-pager | tail -40 +``` + +**Decision:** golden mtime/sha newer than the snapshot, or `df` near full, or the +integrity service having fired → candidate (a)/(c) confirmed, and it explains *both* +the earlier EAGAIN (VM boots from a half-valid image, guest agent never answers) and +today's honest 400. + +### STEP 3 — Per-tag control: which snapshot is broken? (read-only, ~1min) + +Scopes the blast radius. Merged as **PR #21** on `zenprocess/fabro`, branch +`fabro-71/exec-eagain-control` — `scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh`. + +```bash +# Run BOTH. Do not pipe (piping reports tee's exit code, not the script's). +sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --tag zen-gate-big --diagnose +echo "big exit: $?" +sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --tag zen-gate-base --diagnose +echo "base exit: $?" +``` + +Read the `CONTROL-RESULT:` and `EXEC-ELAPSED:` lines. + +| big | base | meaning | +|---|---|---| +| fails | **passes** | scoped to `zen-gate-big` → re-bake only that tag (step 5) | +| fails | fails | both snapshots non-restorable → shared cause (the golden itself, or the store) | +| passes | passes | the fault is intermittent or load-dependent — do NOT re-bake; re-run under real gate load | + +### STEP 4 — Decide + +Steps 1-3 determine which of these you are in. **Do not skip to step 5 on a hunch** — +re-baking a golden that was fine wastes an hour and destroys the evidence. + +--- + +### STEP 5 — Re-bake / re-register (MUTATING, T3, maintenance window required) + +> **Open the window first.** A forkd/golden operation is disruptive; standing policy +> requires it or you page people for your own planned action: +> ```bash +> zenctl maint on 60m "golden re-bake — non-restorable snapshot, fabro gate outage" +> ``` +> Close with `zenctl maint off` when done. + +**The non-negotiable rule from the recorded lesson — this is the whole point:** +**do NOT re-baseline a golden you have not restore-booted.** Disk-clean is not +bootable. Verify in this order: + +```bash +# 1. Filesystem integrity (read-only check): +e2fsck -n /data/forkd-cow/golden/zen-gate-clean.ext4 + +# 2. REAL restore-boot canary — the step that has been missing. +# Must prove real work inside the guest, not just that it booted. +# Boot-only proof (node -v + exit 0) is EXACTLY what let a broken golden +# promote looking healthy on 2026-07-22. Assert something real: +# - node --version inside the VM +# - a real `npm ci --no-audit --no-fund` in a scratch dir +# - free disk > 2GiB inside the guest +# Only if ALL pass, continue. + +# 3. Only then re-baseline: +chattr -i /data/forkd-cow/golden/.ext4 +sha256sum /data/forkd-cow/golden/.ext4 > /data/forkd-cow/golden/golden.sha +stat /data/forkd-cow/golden/.ext4 > /data/forkd-cow/golden/.golden-integrity-state +chattr +i /data/forkd-cow/golden/.ext4 +systemctl start fabro-golden-integrity.service # expect ok=1 + +# 4. Re-register the tag(s) — a RE-REGISTER off the same rootfs, not a re-bake: +# forkd snapshot --tag --kernel --rootfs --tap \ +# --boot-wait-secs --mem-size-mib +# zen-gate-base = 1024, zen-gate-big = 4096. +# Source kernel/tap/boot-wait values from the EXISTING registration first — +# record them before you deregister anything, so rollback is exact. +``` + +**Rollback:** deregister the new tag, restore the prior registration from the values +recorded above. + +### STEP 6 — PROVE IT (nothing counts as fixed without this) + +```bash +# a) Fast precheck — trivial exec must pass on both tags: +sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --tag zen-gate-big +echo "exit: $?" + +# b) The authoritative proof — a real gate run returning PASS or a genuine FAIL, +# never another INFRA: +fabro-github-gate.sh reverdict --repo uniforme --sha \ + --test "npm ci --no-audit --no-fund && npm test" \ + --paths "src test features specs e2e public migrations scripts qa import .zp" +``` + +The `--paths` flag is load-bearing for npm repos: sparse clone materializes root only, +and without it vitest reports `No test files found` — a **false failure** that would +look like a real red verdict. + +Verify the resulting status **on GitHub** for uniforme (it is not on Forgejo — see +Part A step A6): +```bash +gh api repos/zenprocess/uniforme/commits//status \ + --jq '.state, (.statuses[] | {context, state, description})' +``` + +--- + +### DURABLE FIXES (do not skip once green — these prevent recurrence) + +**D1. Add a real restore-boot canary to `golden-validate.sh` permanently.** This is +the recorded lesson's actual prescription and belongs in the fix, not as a follow-up. +It must run real work (npm ci + free-disk assertion), not `node -v`. **Verified +2026-07-28: the equivalent hardening authored on 2026-07-22 as a zeninfra PR NEVER +MERGED** — `GATE_CANARY_DEEP` returns 0 results in a repo code search and is absent +from the on-disk `gates-mcp/ensure-base.sh`. Assume it needs authoring from scratch +unless the old unmerged branch is located first. + +**D2. forkd upstream #269 — the fail-fast capability.** #269 adds controller-side +bootability checking (`GET /v1/snapshots`) and returns **409 on `POST /v1/sandboxes` +for a non-restorable tag**, instead of attempting a doomed restore and emitting a +misleading error. The deployed gate pins **v0.5.3**, which predates #269; the +recovered binary is **v0.5.2**, older still. **Neither binary can fail fast** — both +will keep producing confusing errors for this failure class. Getting #269 into the +deployed version is the durable fix for the *diagnosability* of this whole incident. +(Upstream repo is external — operator-gated, T3.) + +**D3. Exec-hop retry asymmetry.** `~/fabro-run/forkd-shim.py` gives `sandbox-create` +extra attempts on a known transient signature but every other request shape — +including `exec` — exactly one attempt. Only worth changing if the control experiment +shows a *transient* fault; a deterministic one just fails N times instead of once. + +--- + +# PART 0 — 🔴 ACTIVE OUTAGE (escalated 2026-07-27 night) — DO THIS FIRST + +**Symptom — CONFIRMED 2026-07-28 from GitHub commit statuses** (no longer just a +relayed report; this is the verbatim `description` field the gate itself posted): + +``` +context: fabro/qa-pipeline +state: error +target_url: None +description: infrastructure: controller POST /v1/sandboxes/sb-6a686d2b-0097/exec + -> HTTP 500 {"error":"exec: read response: Resource temporarily unavaila + (truncated by GitHub at 140 chars) +``` + +Reproduce with: +`gh api repos/zenprocess/uniforme/commits//status` + +**Systematic, not a one-off — same string, three distinct sandboxes, 8+ hours:** + +| commit | time (UTC) | sandbox | +|---|---|---| +| `18257180` | 2026-07-28T00:48:57Z | `sb-6a67fa76-008a` | +| `bf8f62d` (PR #807) | 2026-07-28T08:58:17Z | `sb-6a686d2b-0097` | +| `3b72eeb` (PR #808) | 2026-07-28T09:11:56Z | `sb-6a68705c-0098` | + +Note the gate **correctly labels this `infrastructure:`** in its own description — so +the classifier is identifying it right, yet it still posts a red `error` status. That +gap is step 0.55's subject. + +**Reported blast radius:** every uniforme gate run dies, so `fabro/qa-pipeline` +never goes green, and `cfw-autodeploy` has skipped **~23 merges** on +`uniforme/preprod` over **4+ hours**. + +## ⚠️ Read this before you restart anything + +The escalation labels this "the broken forkd controller" and files it under the +controller-durability item. **Those are two different faults and the durability fix +will not resolve this one.** + +- `os error 11` is **EAGAIN** — "resource temporarily unavailable". On an exec path + this is a `fork()`/thread-spawn or VM-launch failure caused by **resource + exhaustion**: PID/task limit, `RLIMIT_NPROC`, FD limit, or memory pressure. +- The durability item (step **B1**, canonicalize the recovered v0.5.2 binary) is + about surviving a *restart* after the deleted-inode incident. It has nothing to do + with EAGAIN. **Running B1 will not clear this outage.** + +If you run B1 expecting the gate to come back and it doesn't, that is not a new +problem — it's this misattribution. Do Part 0, then B1 separately for its own reason. + +## 🔬 REVISED AGAIN 2026-07-28 (3rd pass) — both prior analyses were reading a STALE checkout + +Two rounds of "verified" facts below (mine and uniforme-781's) were drawn from a +uniforme working copy that was **345 commits behind `origin/main`** — it predates +commit `be30943` (2026-07-24, "wire the hermetic E2E lane into the fabro gate, #477") +entirely. Neither of us checked `git status` against the remote before asserting facts +from the file. Re-verified against `origin/main`; corrections below **supersede** +everything under the "PER-REPO" heading further down (kept intact underneath, struck +through in spirit not in text, because the differential reasoning in it is still +correct — only the specific command/config facts were wrong). + +**What was actually wrong:** +- uniforme's `testCmd` is not `npm test` — it's `npm ci ... && npm run qa:gate`, where + `qa:gate = npm test && npm run qa:e2e`, and `qa:e2e` boots a Cloudflare Worker + (`wrangler dev`) and launches **real Playwright Chromium** inside the guest. +- uniforme **does** have a `.zp/qa-diamond.yaml` — I said it didn't; that file is what + makes this diagnosable at all. + +**What that file actually says, verbatim comment, and it is the sharpest evidence yet:** + +```yaml +# Gate VM snapshot. The Lane-1 compiler passes this to the gate; without +# it the gate boots zen-gate-base at 1GB of RAM and the uniformetest +# suite OOMs (same class of failure that forced the serial-vitest +# workaround). zen-gate-big is the 4GB guest on which the full suite +# ran 6911/6911 green. Pin this here AND in test/qa-diamond.test.ts — +# a silent drop / typo falls back to the 1GB base with no error. +snapshot_tag: zen-gate-big +``` + +Three things this settles at once: + +1. **`zen-gate-base` is already 1024 MiB.** Golden option-2's base tag is confirmed + live by this comment (an independent source from the fabro-side commit trail) — so + Part A step A3's 512 MB trap is very likely already cleared. Still worth running A3 + to confirm `zen-gate-big` (4096 MiB) specifically exists, since that's the tag this + file actually depends on. +2. **The undersized-guest hypothesis and the EAGAIN evidence are both still right** — + the file's own author independently arrived at "OOM-class failure without the 4 GB + tag," which matches uniforme-781's `exec: Resource temporarily unavailable` finding + exactly. Chromium + a Worker + a 6911-test suite in 1 GB is more than enough to + explain EAGAIN at `fork()`/`mmap()`. +3. **The file's own author already named the exact failure mode we're chasing**: + *"a silent drop / typo falls back to the 1GB base with no error."* If the Lane-1 + compiler either doesn't read `snapshot_tag` from this file, or reads it but the + value doesn't reach the sandbox spec, uniforme silently runs on the 1 GB guest with + **no error indicating that happened** — and then EAGAINs on Chromium, indistinguishable + from the outside. + +**I cannot verify whether the compiler honors this pin — and neither can uniforme.** +`FABRO-COMPLETION-RUNBOOK.md` (the original operator mandate) asserts *"platform side +is ready: qa-diamond-compile.py reads top-level `snapshot_tag:` and the gate passes it +to `--snapshot` (merged, cal-green-proven)"* — but **that compiler is not checked into +either the fabro repo or the uniforme repo.** I grepped both trees at `origin/main`; +it isn't there. Same gap as the forkd-controller source earlier in this document: the +code that would actually answer the question lives somewhere neither of us can read. +**This makes the following the single highest-priority live check, ahead of even +Part A step A3:** + +```bash +# On dellsrv/wherever the Lane-1 compiler runs — find it first if its location +# isn't already known, then confirm it actually reads .zp/qa-diamond.yaml: +grep -rn "snapshot_tag" 2>&1 + +# Then confirm empirically, on a REAL uniforme gate invocation: which snapshot_tag +# did the actual `forkd snapshot create`/sandbox-create call receive? Compare +# against the .zp/qa-diamond.yaml pin (zen-gate-big). If the create call shows +# zen-gate-base, the pin is being silently dropped — exactly as warned. +docker exec -i forkd sh <<'EOF' +TOKEN=$(cat /etc/forkd-token) +curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8891/v1/sandboxes +EOF +# Look at the snapshot_tag field on the most recent uniforme-related sandbox. +``` + +> ### ⏱️ The timeline eliminates the "pin wasn't there yet" explanation +> +> The pin landed in `origin/main` as **`bfcc588` "ci: pin snapshot_tag to zen-gate-big +> (#664)", 2026-07-25 19:11 UTC**. The **first EAGAIN was 2026-07-25 23:44 UTC** — +> **4.5 hours later**. Every one of the 23 `failed/error` rows since (through +> 2026-07-28 09:28 UTC) occurred with `snapshot_tag: zen-gate-big` already pinned in +> the repo. +> +> So the failures are not "the pin hadn't been added yet." The pin was present and the +> gate kept EAGAINing — which means either the compiler never reads it, the value is +> dropped before sandbox-create, or `zen-gate-big` exists but cannot boot. All three +> are answered by the single check below. This is the strongest evidence in the +> document that the problem is on the **fabro/platform side**, not in uniforme's config. + +If this shows `zen-gate-base` where `zen-gate-big` was pinned: **that is the root +cause**, no rootfs work needed, no restart needed — the fix is in the compiler +(reading the field) or in whatever bridges it to the sandbox-create call, and it +directly explains why uniforme (the only repo with heavy per-VM requirements AND a +`qa-diamond.yaml` pin depending on it) is the only one failing. + +**Candidate fixes** (from uniforme-781, in the order they'd try them — this is a +fabro/platform-side judgment call, not something the uniforme side should self-serve +by editing its own gate config to make its own PRs pass): + +1. If the compiler-honors-the-pin check above comes back clean (pin IS reaching the + sandbox correctly) and EAGAIN still happens on `zen-gate-big`, the `weight: light` + label in uniforme's `.zp/project.yaml` may be under-sizing something else in the + lane-allocation path (concurrency slot, not just guest memory) — worth checking + what `weight` actually controls before assuming it's irrelevant. +2. Confirm `zen-gate-big` (4096 MiB) exists at all (Part A step A3) — if the tag is + simply missing, the pin is correct but has nothing to resolve to. +3. If both the pin and the tag check out and it *still* EAGAINs, consider splitting + the lane: keep `npm test` in the blocking gate and move `qa:e2e` (the Chromium + step) to a separate non-blocking lane, so a resource-hungry browser run cannot + dark the entire gate for a repo. This is a bigger change (affects the pass/fail + semantics uniforme relies on) — operator call, not a default. + +**In-repo doc worth reading directly**: `docs/qa/fabro-e2e-gate.md` in the uniforme +repo (added by the same `be30943` commit that wired in the E2E lane). It documents +the author's own expectations of what fabro should provide, including a "256 MB risk" +section recommending Chromium be baked into the base snapshot to avoid a per-VM +download, and an explicit fabro-owned follow-up table: + +| Follow-up (from `docs/qa/fabro-e2e-gate.md`) | Owner | +|---|---| +| Auto-discover PR heads lacking a `fabro/qa-pipeline` status and gate them | fabro | +| Stop swallowing `gh-status.sh` POST failures (`\|\| true` → real exit code) | fabro | +| Verify the driver reads `qa.testCmd` from the descriptor (or document the sync contract) | fabro | +| Bake Chromium into the base snapshot to avoid the 256 MB/per-VM download | fabro | + +That table is independent evidence (written 2026-07-24, before this outage) that the +uniforme side already flagged "does the driver actually read our descriptor?" as an +open question — which is exactly what this outage turned out to hinge on. + +**uniforme-781 deliberately did not touch `.zp/project.yaml` or `qa-diamond.yaml`** +— correctly: changing gate config to make one's own PRs pass would be the wrong actor +making that call. The weight/tier decision is explicitly left to the operator (and, if +delegated, to fabro-side code work). + +--- + +## 🔬 REVISED 2026-07-28 (superseded above by the 3rd-pass finding, kept for the +## differential reasoning, which is still correct) — it is PER-REPO, which refutes my +## first analysis + +An earlier draft of this section proposed two **host-global** hypotheses (leaked +microVMs exhausting host PIDs; golden option-2's 4 GB tag exhausting host RAM). +**Evidence from the uniforme orchestrator refutes both**, and I have corrected them +rather than leaving them to misdirect you. + +**The refuting data** (from `ao.db pr_checks`, reported by uniforme-781, all-time): + +| project | same 3-day window | result | +|---|---|---| +| `zenprocess/cal` PR #72 | 4 × `passed/success` | ✅ green | +| `zenprocess/cal` PR #56 | 4 × `failed/failure`, 1 × `passed/success` | ✅ real verdicts, both polarities | +| `zenprocess/zetronom` PR #134 | 4 × `passed/success`, 3 × `failed/failure` | ✅ real verdicts | +| `zenprocess/uniforme` | **ZERO** `passed/success` since 2026-07-21T20:59:39Z | ❌ only `error` + stuck `pending` | + +A host-global resource exhaustion would take down cal and zetronom too. It didn't. +**The gate driver can run, and can post both green and red — just not for uniforme.** +So the fault is per-repo, and any explanation that is host-wide is wrong. + +Also decisive: every uniforme `failed/error` row has **empty `details` and empty +`url`**, and on GitHub shows `state=ERROR` with `target_url=null`. An `error` with no +output means the run produced nothing at all — **the gate died before or during +setup, not at the test command.** That is a different fault from a test regression, +and it rules out "uniforme's tests broke." + +### ⛔ SUPERSEDED — corrected in the "3rd pass" section above + +Everything below this line, through the end of this subsection, was written from a +**stale uniforme checkout** (345 commits behind `origin/main`) and is factually wrong +on the specifics. Left visible rather than deleted so the reasoning trail is honest, +but **do not act on any command/config claim in this block** — use the "REVISED AGAIN +(3rd pass)" section above instead. Specifically wrong: uniforme's testCmd is not +`npm test` (it's `npm run qa:gate`, which includes a Chromium E2E lane); uniforme +**does** have a `.zp/qa-diamond.yaml`, and it already pins `snapshot_tag: zen-gate-big`. +The differential logic (uniforme is the only heavy-workload repo among the three +compared) is still correct — only the mechanism was wrong (Chromium/E2E weight, not +bare `npm ci`). + +**What the corrected picture means for the remediation ladder:** + +- **Do not restart first**, still. A restart cannot fix a mis-read config pin, and if + the compiler is honoring the pin correctly, there is nothing to restart for. +- **The likely fix is NOT Part B step B2** (registering new tags) — `zen-gate-big` + already exists per the uniforme file's own comment ("ran 6911/6911 green" on it). + The likely fix is in the **Lane-1 compiler** (find it, confirm it reads + `snapshot_tag`, confirm the value reaches the sandbox-create call) — see the live + check at the end of the 3rd-pass section above. That component is not in either + repo I could search; locating it is itself part of the fix. +- Still run step 0.1's diagnosis and Part A step A3, since a leak or an actually-missing + `zen-gate-big` tag could coexist with a compiler bug, and they're cheap to rule out. + +> **Do not restart before running step 0.1, Part A step A3, and the compiler check +> above.** A restart destroys evidence and none of the candidate causes are fixed by +> one. + +## Step 0.1 — Diagnose (read-only, ~60s) + +```bash +# a) How many sandboxes does the controller think exist? (orphan count) +docker exec -i forkd sh <<'EOF' +TOKEN=$(cat /etc/forkd-token) +curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8891/v1/sandboxes +EOF + +# b) How many Firecracker processes are actually alive? +pgrep -c firecracker || echo 0 +docker exec forkd sh -c 'pgrep -c firecracker || echo 0' + +# c) PID/task exhaustion — the direct EAGAIN cause +cat /sys/fs/cgroup/system.slice/docker-*.scope/pids.current 2>/dev/null | tail -5 +cat /sys/fs/cgroup/system.slice/docker-*.scope/pids.max 2>/dev/null | tail -5 +docker exec forkd sh -c 'cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max 2>/dev/null' +ps -eLf | wc -l # total threads on host +sysctl kernel.pid_max kernel.threads-max + +# d) Memory pressure / OOM history +free -g +docker stats --no-stream +dmesg -T 2>/dev/null | grep -iE 'oom|out of memory|cannot allocate' | tail -20 + +# e) FD limits +docker exec forkd sh -c 'ulimit -n; ls /proc/1/fd 2>/dev/null | wc -l' +``` + +**Reading the result:** + +| Observation | Hypothesis | Go to | +|---|---|---| +| Many sandboxes in (a) and/or many `firecracker` procs in (b), memory OK | **1 — leak** | 0.2 | +| `pids.current` at/near `pids.max` | **1 — leak** (exhausted task limit) | 0.2 | +| `free -g` shows little available, or `dmesg` shows OOM kills | **2 — capacity** | 0.4 | +| Few sandboxes, plenty of RAM and PIDs | Neither — controller-internal | 0.3, then escalate | + +## Step 0.2 — Reap orphans (least-disruptive remediation; try FIRST) + +**Precondition:** step 0.1 showed orphaned sandboxes or PID exhaustion. + +This is targeted and does not interrupt healthy work. + +```bash +# List, review, THEN delete. Do not blind-delete — a running gate lane is a sandbox too. +docker exec -i forkd sh <<'EOF' +TOKEN=$(cat /etc/forkd-token) +curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8891/v1/sandboxes +EOF + +# For each id you have confirmed is an orphan (not an in-flight lane): +docker exec -i forkd sh <<'EOF' +TOKEN=$(cat /etc/forkd-token) +curl -s -X DELETE -H "Authorization: Bearer $TOKEN" \ + http://127.0.0.1:8891/v1/sandboxes/ +EOF +``` + +Re-run 0.1(a)+(c). If PIDs drop and exec starts working, **stop here** and go to +step 0.5 (verification). Restarting is then unnecessary. + +## Step 0.3 — Restart (only if 0.2 was insufficient) + +> ### 🔒 Open a maintenance window FIRST +> +> A forkd/docker restart is a disruptive infra op. Standing policy requires a +> `zenctl` maintenance window before one, or you will fire `health_alert` + +> route-drift ntfy and page people for your own planned action: +> +> ```bash +> zenctl maint on 30m "forkd controller EAGAIN outage remediation" +> ``` +> +> Close it when done: `zenctl maint off`. + +Escalating ladder — **stop at the first one that works**, re-running 0.5 after each: + +```bash +# (i) Restart the forkd container. This also re-provisions the per-child netns, +# which the boot script creates at container start. +docker restart forkd +sleep 15 +docker logs --tail 50 forkd + +# (ii) If the container will not come up healthy, restart its owning unit: +sudo systemctl restart forkd-ec.service +sudo systemctl status forkd-ec.service --no-pager | tail -20 + +# (iii) Confirm the controller answers at all: +docker exec -i forkd sh <<'EOF' +TOKEN=$(cat /etc/forkd-token) +curl -s -o /dev/null -w '%{http_code}\n' \ + -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8891/v1/sandboxes +EOF +``` + +**After ANY restart, re-run Part A step A5** (per-child netns). The 2026-07-22 +three-day outage was those namespaces dying, and they are provisioned only at +container start — a restart is exactly when they can come back wrong. + +**Rollback:** there is nothing to roll back for a restart. If the container will not +start, capture `docker logs forkd` and `journalctl -u forkd-ec.service -n 100` before +trying anything further — that log is the artifact for the next person. + +## Step 0.4 — If capacity (hypothesis 2), do NOT just restart + +**Precondition:** 0.1 showed memory exhaustion or OOM kills. + +A restart clears the symptom for minutes and it returns. Options, cheapest first: + +1. **Reduce concurrency** on the gate lanes so `concurrent_lanes × mem-size-mib` + fits host RAM with headroom. +2. **Point the failing lane at `zen-gate-base` (1024 MiB) instead of `zen-gate-big` + (4096 MiB)** — a one-line `snapshot_tag:` change in the consuming repo's + `.zp/qa-diamond.yaml`, reversible. +3. Only then consider host-level capacity changes. + +Record which you chose — if `zen-gate-big` is implicated, that directly affects the +uniforme-on-`zen-gate-big` plan in the cross-project ask below, and it means golden +option-2 needs a concurrency budget attached before it is used in anger. + +## Step 0.5 — VERIFY: a real gate run on uniforme HEAD must go green + +Do not declare this resolved on a 200 from the controller. The reported failure is at +the *gate* level, so the verification has to be at the gate level too. + +**Fast precheck (~10s)** — exercises the real `per_child_netns=true` path end to end: + +```bash +sudo /usr/local/sbin/gate-health-probe.sh +echo "probe exit: $?" # 0 = create + exec + delete all succeeded +``` + +Note the README's warning: **do not pipe this** if you care about the exit code — +`probe.sh | tee log` reports `tee`'s status, not the probe's. + +**Authoritative check — a real gate run on uniforme HEAD:** + +```bash +# 1. Get the SHA you are proving green: +git -C rev-parse HEAD + +# 2. Trigger the Lane-1 QA path for that SHA (the normal gate entry point for +# uniforme — same path cfw-autodeploy waits on). + +# 3. Watch the exec that previously failed actually succeed: +docker logs -f forkd 2>&1 | grep -iE 'exec|error 11|Resource temporarily' +``` + +**Green means:** the run completes AND a `fabro/qa-pipeline` status for that SHA flips +to `success`. For **uniforme specifically, read that on GitHub** — it is not on Forgejo +(see the corrected table in Part A step A6): + +```bash +gh api repos/zenprocess/uniforme/commits//status \ + --jq '.state, (.statuses[] | {context, state, description, target_url})' +``` + +You are looking for `state: success` on context `fabro/qa-pipeline`. The **exact +before/after** is unusually clean here, so use it: the same query today returns +`state: error` with `description: 'infrastructure: controller POST +/v1/sandboxes/.../exec -> HTTP 500 ... Resource temporarily unavaila'` (truncated by +GitHub at 140 chars). That string disappearing, replaced by `success`, is your proof. + +Note the status only appears for **PR heads**, not plain commits — so verify against a +PR head SHA, not a `main` commit, or you will read "no status" as failure. + +**Then confirm the actual business symptom cleared:** `cfw-autodeploy` picks up the +backlog and the ~23 skipped `uniforme/preprod` merges drain. A green gate that +doesn't unblock autodeploy means the blockage was only partly the controller. + +## Step 0.55 — 🚨 The false-ERROR status is actively misrouting agents (fix independent of root cause) + +Reported by uniforme-781, and this is the part that causes harm *right now*, +regardless of when the root cause is fixed. + +**What happened:** a `failed/error` status with empty details causes AO to emit a +*"CI is failing on PR #N … Review the output below and push a fix"* nudge. That nudge +was delivered into the composer of **uniforme-782, a VERIFY-ONLY worker** dispatched +via `--claim-pr` to independently verify PR #807. **Had it complied, it would have +pushed commits to the very branch it was verifying** — destroying the independence of +the verification and corrupting the PR. uniforme-781 caught it and sent an override. + +The nudge also says *"review the output below"* when there **is** no output, because +an errored run produces none. + +This is our own core failure class with a new blast radius: **an infrastructure fault +rendered as a code failure, which then instructs an agent to "fix" code that was never +broken.** It can misroute agents in *any* project using `--claim-pr`, not just uniforme. + +**Two cheap mitigations, both worth doing before root cause:** + +1. **Suppress or clearly label the auto-fix nudge when `conclusion=error` AND + `details`/`url` are empty.** An infra error is not a code failure and must not tell + anyone to push a fix. *(Owner: agent-orchestrator — this is AO nudge behavior, not + the fabro gate. Needs routing to that project.)* +2. **If a run cannot start for a repo, post NO status (stay dark) rather than ERROR.** + Dark is honest and harmless; a false ERROR actively misroutes agents. + *(Owner: fabro — this is our gate poster.)* + +> **⚠️ Ask 2 contradicts documented gate behavior — resolve before implementing.** +> During the 2026-07-22 netns outage the gate **correctly suppressed** verdicts rather +> than posting wrong ones; the recorded failure mode was *"silence, not noise."* The +> gate therefore already has a dark path. Yet uniforme is now receiving posted ERRORs. +> So either a regression introduced ERROR-on-infra, or this failure takes a code path +> the suppression logic does not cover. **Find out which before changing the poster** — +> if suppression exists and regressed, the fix is restoring it, not adding a second +> mechanism. +> +> **RESOLVED 2026-07-28** — the Forgejo-vs-GitHub discrepancy is settled: uniforme's +> verdicts land on **GitHub**, it is not on Forgejo at all, and this document's earlier +> "GitHub statuses are always empty" claim was **wrong for uniforme**. Corrected in +> Part A step A6. It mattered: that claim would have told you to ignore the one surface +> carrying the actual diagnostic. +> +> ### 🐛 Separate bug, route to agent-orchestrator: `ao.db` drops the diagnostic +> +> `ao.db pr_checks.details` is **empty** while the GitHub status `description` carries +> the real error (`infrastructure: controller POST /v1/sandboxes/.../exec -> HTTP 500 +> ... Resource temporarily unavaila…`). The description is being **dropped on +> ingestion**. This is real data loss, and it is not hypothetical: it caused uniforme-781 +> to characterize these failures as "no output to inspect" when a precise diagnostic +> existed the whole time. Anyone treating `ao.db` as authoritative will make the same +> misdiagnosis. Route alongside the nudge-suppression ask. + +**Blast-radius note:** uniforme-781 is currently treating `ao.db` as authoritative, +ignoring the fabro status for merge decisions, and gating every uniforme PR on a +dispatched verify worker running `npm test` on a fresh checkout. That workaround is +holding, so this is urgent-but-not-blocking for them. Currently affected open PRs: +**#807** (fabro ERROR on head `bf8f62d`) and **#808** (stuck `in_progress`). + +The **23 stuck `in_progress`/`pending` rows** (latest 2026-07-28T09:04:08Z) are a +third symptom worth its own look: runs that never resolve at all. A gate that leaves +checks pending forever blocks merges just as effectively as a red one, and it suggests +a lane that dies without ever reporting a terminal state. + +## Step 0.6 — Check whether this outage poisoned the referee labels + +Worth 2 minutes once service is restored. An infra fault must never be recorded as a +code verdict — that is this campaign's core failure class, and 4+ hours of failures +is a lot of potential bad rows. + +```bash +# Any run rows recorded during the outage window that claim a real verdict? +ls -la ~/.ao/data/aofactory/referee/runs/ | tail -20 +grep -l 'outcome_kind' ~/.ao/data/aofactory/referee/runs/*.jsonl 2>/dev/null | tail +``` + +Rows from the outage window must be `infra` / `inconclusive`, **never** `fail`. Any +that say `fail` are contaminated labels: quarantine them before they reach GEPA or +the trainset. If the classifier recorded EAGAIN as a code failure, that is a +classifier gap worth a fix of its own — the same shape as the exit-137 gap. + +--- + +## PART A — read-only verification checklist + +Nothing in this part mutates anything. Run it top to bottom; record each verdict. +Total time ≈ 5 minutes. + +### A1. forkd-controller binary — is the deleted-inode emergency actually closed? + +Background: on 2026-07-25 the running controller was a **deleted inode** — +the on-disk binary was gone and only `/proc//exe` held it. It was recovered +by copying that back to `/usr/local/bin/forkd-controller`. The durability question +is whether the *running* process now corresponds to a real on-disk file. + +```bash +ls -l /usr/local/bin/forkd-controller +/usr/local/bin/forkd-controller --version +sha256sum /usr/local/bin/forkd-controller + +# The load-bearing check — is the RUNNING controller still a deleted inode? +pgrep -af forkd-controller || true +for p in $(pgrep -x forkd-controller 2>/dev/null); do + printf 'pid %s -> %s\n' "$p" "$(readlink -f /proc/$p/exe 2>/dev/null || echo UNREADABLE)" + ls -l /proc/$p/exe 2>/dev/null +done +``` + +- **DONE** — binary exists, `--version` reports `v0.5.2`, and `readlink + /proc//exe` resolves to `/usr/local/bin/forkd-controller` with **no + `(deleted)` suffix**. +- **NOT DONE** — `/proc//exe` shows `(deleted)`, or resolves to a path other + than `/usr/local/bin/forkd-controller`. The emergency is NOT closed; the running + process still cannot be restarted from disk. → Part B, step B1. +- **UNKNOWN / needs care** — `pgrep` finds nothing on the host. This is *expected* + if the controller runs inside the `forkd` container (see A2). Re-run the `pgrep` + and `/proc` checks inside the container: + `docker exec forkd sh -c 'pgrep -af forkd-controller; readlink -f /proc/$(pgrep -x forkd-controller | head -1)/exe'` + +> **Why `--version` alone is not sufficient:** a correct `v0.5.2` on disk tells you +> nothing about what the *running* process is executing. The 2026-07-25 failure +> mode was precisely a healthy-looking service whose binary had vanished. The +> `/proc//exe` check is the one that actually bites; do not skip it. + +### A2. Which unit supervises forkd, and does it reference the canonical binary? + +```bash +systemctl cat forkd-ec.service +systemctl is-enabled forkd-ec.service +systemctl is-active forkd-ec.service + +# What the unit actually launches — follow ExecStart to the boot script, then read it: +systemctl show -p ExecStart --value forkd-ec.service +# then, against whatever path that prints (commonly forkd-ec-boot-dellsrv.sh): +grep -n 'forkd-controller' +``` + +- **DONE** — `forkd-ec.service` is `enabled` + `active`, and the boot script it owns + references `/usr/local/bin/forkd-controller` (an absolute, on-disk path). +- **NOT DONE** — the boot script references a path under `/tmp`, a build directory, + a relative path, or a container-ephemeral location. → Part B, step B1. + +> ### ⚠️ DO NOT add a `forkd-controller.service` unit +> +> **PR #16 ("ops(dellsrv): forkd supervision unit + golden tag re-register") was +> CLOSED, not merged**, precisely because a new supervision unit would **duplicate +> `forkd-ec.service`'s ownership of the boot script** — two units racing to own one +> process. The review called this "the exact hazard the earlier review flagged." +> +> The durability fix is **canonicalizing the recovered v0.5.2 binary that +> `forkd-ec.service` already launches** (step B1), NOT a competing unit. If you find +> yourself writing a new `.service` file for the controller, stop — you are +> re-creating a rejected design. + +### A3. Golden option-2 — do BOTH tags exist at the right memory sizes? + +The gate consumes the `zen-gate-base` snapshot tag. The original was baked at +**512 MB**, which silently OOM-killed real test suites. Option 2 re-registers two +tags off the *same* existing 20 GB golden rootfs at larger memory sizes. + +```bash +# List registered snapshot tags (inside the forkd container): +docker exec forkd sh -c 'ls -la /root/.local/share/forkd/snapshots/' + +# The load-bearing check — memory.bin size PER TAG: +docker exec forkd sh -c 'for t in /root/.local/share/forkd/snapshots/*/; do + printf "%s " "$t"; stat -c %s "$t/memory.bin" 2>/dev/null || echo "(no memory.bin)"; done' +``` + +Interpret the byte size exactly: + +| memory.bin bytes | MiB | Meaning | +|---|---|---| +| **536870912** | 512 | ⛔ **The original undersized bake. Option 2 did NOT happen.** | +| 1073741824 | 1024 | ✅ `zen-gate-base` at option-2 size | +| 4294967296 | 4096 | ✅ `zen-gate-big` at option-2 size | + +- **DONE** — BOTH `zen-gate-base` (1073741824) AND `zen-gate-big` (4294967296) exist. +- **NOT DONE** — either tag is missing, or any tag still reports **536870912**. + → Part B, step B2. + +> **This is the check most likely to contradict the commit trail.** Commit +> `6c3ed8bac` asserts option-2 is already live. If you see 536870912 here, that +> assertion was wrong and every downstream conclusion built on it (including +> "uniforme can run on zen-gate-big") is void. + +### A4. gate-health-probe — is the continuous canary actually running? + +PR #17 merged the probe and its commit message records a hand-verification on +2026-07-25. That proves the **script works**; it does not prove the **timer is +installed and firing**. Those are different facts. + +```bash +systemctl is-enabled gate-health-probe.timer +systemctl list-timers gate-health-probe.timer --all +systemctl status gate-health-probe.service --no-pager | tail -20 + +# Has it actually produced output in the last day? +journalctl -t gate-health-probe --since -24h | tail -20 +journalctl -t gate-health-probe --since -24h | grep -c FORKD-GATE-ALERT || true +journalctl -t gate-health-probe --since -24h | grep -c FORKD-GATE-HEAL || true +``` + +- **DONE** — timer `enabled`, `list-timers` shows a concrete NEXT/LEFT time, and + journal shows probe entries within the last 24h. +- **NOT DONE** — `is-enabled` says `disabled`/`not-found`, or `list-timers` is + empty, or the journal has **zero** entries in 24h. → Part B, step B3. + +> **A zero ALERT count is not evidence of health.** Zero alerts and zero probe runs +> look identical in a `grep -c`. Confirm the probe *ran* (journal has entries, timer +> shows a next fire) before reading an alert count as good news. This is the exact +> shape of the 2026-07-22 three-day silent outage: the failure mode was *silence*, +> and silence reads as healthy. + +### A5. Per-child network namespaces — the 2026-07-22 outage surface + +```bash +ls -la /var/run/netns/ 2>/dev/null | grep forkd-child || echo "NO forkd-child netns" +docker exec forkd sh -c 'ls -la /var/run/netns/ 2>/dev/null' || true +``` + +- **DONE** — `forkd-child-1`, `-2`, `-3` present. +- **NOT DONE** — missing. Every gate exec will fail at infra level while appearing + silent. The netns are provisioned only by the boot script at container start → + the repair is a controlled restart of the forkd container (see PR #17's README + `--heal` path; do not improvise). + +### A6. Posting / poller — ON or OFF? + +> ### ⚠️ CORRECTED 2026-07-28 — which forge to check is PER-REPO +> +> An earlier draft of this step said "`gh api .../statuses` is always empty, never use +> it." **That is false for uniforme and it is the single most useful diagnostic we +> have.** Corrected rule: +> +> | repo | where gate verdicts land | how to read them | +> |---|---|---| +> | **uniforme** | **GitHub commit statuses** (it is **not on Forgejo at all** — the repo 404s there) | `gh api repos/zenprocess/uniforme/commits//status` | +> | **fabro** | Forgejo statuses; GitHub CI here is **Actions**, which posts **check-runs, not statuses** | `gh api .../commits//check-runs` (statuses genuinely *are* empty for fabro — a statuses-vs-check-runs artifact, not proof of darkness) | +> +> So an empty `commits/{sha}/status` means different things per repo. **Never infer +> gate health from a github.com status query alone, in either direction.** The +> reliable cross-repo surface is `ao.db pr_checks` joined to `pr`/`sessions` for +> `name='fabro/qa-pipeline'`, judged on **recency of terminal verdicts** — but see the +> `ao.db` data-loss caveat in step 0.55 before trusting its `details` column. +> +> Two more facts worth having: +> - **Statuses post only for PR heads.** Plain `main` commits and pushed branch heads +> get no status at all (verified: `a3114e1`, `942adb6`). A bare commit with no status +> is *expected*, not a dark gate. +> - **GitHub truncates the status `description` at 140 chars**, so the diagnostic it +> carries is clipped mid-sentence. For the full error, go to the controller logs. + +Check the Forgejo side and the poster process instead: + +```bash +# Is the poster present / kill-switched? (on the Mac, ~/.ao-mac/) +ls -la ~/.ao-mac/gh-status.sh* +launchctl list 2>/dev/null | grep -iE 'gh-status|poll|fabro' || echo "no launchd job loaded" + +# On dellsrv, whatever runs the poller: +systemctl list-units --all | grep -iE 'poll|gate|forkd' || true +``` + +- **CURRENT KNOWN STATE: UNKNOWN.** As of 2026-07-27 the live `~/.ao-mac/gh-status.sh` + and its `.DISABLED-until-trustworthy.bak-20260725T195158Z` backup are **byte-identical**, + which is consistent with *either* "was disabled and already restored" *or* "was + never content-disabled at all." No launchd job for it is loaded on the Mac. This + cannot be resolved from the artifacts — only by checking whether verdicts are + actually landing on Forgejo. +- **Resolve it by**: pick a recent PR that went through the gate and look for a + `fabro/qa-pipeline` status **on Forgejo**. Verdicts present and recent → posting is + ON. Nothing since 2026-07-25 → OFF. → Part B, step B4 only if OFF *and* A3+A4 both + passed. + +--- + +## PART B — mutating steps + +**Precondition discipline:** each step names the Part A check that must have +**FAILED** for the step to be needed. If that check passed, skip the step entirely. + +Order matters: B1 → B2 → B3 → B4. Do not re-enable posting (B4) before the golden +tags (B2) and the canary (B3) are confirmed good — that is how a bad verdict gets +published at scale. + +### B1. Canonicalize the recovered controller binary + +**Precondition:** A1 or A2 failed. +**Not in scope:** creating any new systemd unit — see the PR #16 warning in A2. + +The goal is that `forkd-ec.service`'s boot script launches an absolute, on-disk, +persistent binary, so a container/process restart is survivable. + +```bash +# 1. Confirm what you have before changing anything: +sha256sum /usr/local/bin/forkd-controller +/usr/local/bin/forkd-controller --version # expect v0.5.2 + +# 2. Back up before touching the boot script: +sudo cp -a .bak-$(date +%Y%m%dT%H%M%SZ) + +# 3. Edit the boot script so its controller invocation uses the absolute path +# /usr/local/bin/forkd-controller. The live invocation shape recorded during the +# 2026-07-25 recovery was: +# FORKD_TOKEN_FILE=/etc/forkd-token FORKD_BIND=0.0.0.0:8891 \ +# forkd-controller serve --bind 0.0.0.0:8891 \ +# --snapshot-root /root/.local/share/forkd/snapshots +# Change ONLY the binary reference to the absolute path. Do not restructure +# the unit or the script. + +# 4. Verify by re-running A1 and A2. Do NOT restart anything yet — a restart is +# only warranted if A1 showed a deleted inode, and it should be done in a +# zenctl maintenance window. +``` + +**Rollback:** restore the `.bak-` boot script, `systemctl daemon-reload` +if a unit file changed. + +**Note on "rebuild from source":** the original mandate asked for a source rebuild +of forkd-controller. **No forkd-controller source repo is referenced anywhere in the +fabro repo** — verified by grepping both `HEAD` and `origin/main` trees. Per your +2026-07-28 decision, the recovered v0.5.2 binary is the canonical artifact and this +step is the durability fix. If you know where that source actually lives, a real +rebuild supersedes this. + +### B2. Register the golden option-2 tags + +**Precondition:** A3 failed (a tag missing, or any tag at 536870912 bytes). + +This is a **RE-REGISTER, not a re-bake**. Both tags point at the *same* existing +20 GB golden rootfs; only the memory size differs. No rootfs rebuild, minutes of +work, reversible. + +```bash +# The registration form (run inside the forkd container; cf. cow-run.sh cmd_register): +forkd snapshot --tag \ + --kernel \ + --rootfs \ + --tap \ + --boot-wait-secs \ + --mem-size-mib + +# Option 2 = two tags off that same rootfs: +# zen-gate-base --mem-size-mib 1024 +# zen-gate-big --mem-size-mib 4096 +``` + +**Read the existing registration first** to source the kernel/rootfs/tap/boot-wait +values — do not guess them. They are recoverable from the current `zen-gate-base` +registration and from `cow-run.sh cmd_register` inside the forkd container. + +**Verify:** re-run A3. Both tags present, sizes 1073741824 and 4294967296. + +**Rollback:** deregister the new tag. The prior `zen-gate-base` registration should +be recorded (tag name + all flag values) *before* you re-register it, so it can be +restored exactly. + +> Host RAM headroom is not a concern for this change: COW warm-lanes already default +> to `FORKD_COW_MEM_MIB=12288`. Only the gate's direct `zen-gate-base` tag was baked +> small. + +### B3. Enable the gate-health canary timer + +**Precondition:** A4 failed. + +The deploy commands are already documented in +`scripts/ops/dellsrv-forkd-supervision/README.md` on `origin/main` (merged as +PR #17) — **use that file, do not improvise**. It covers install paths, the +`daemon-reload`, `enable --now`, the alert-vs-`--heal` decision, exit-code +semantics, and rollback. + +Two things from that README worth restating because they are easy to get wrong: + +- **Default is alert-only.** `--heal` is opt-in and should stay off until you have + seen the probe behave. Auto-heal that masks a recurring fault re-creates the + original blind spot. +- **Do not pipe the probe manually** if you care about its exit code — `probe.sh | + tee log.txt` gives you `tee`'s status, not the probe's. Run it via systemd, or use + `set -o pipefail`. + +**Verify:** re-run A4 — timer enabled, `list-timers` shows a next fire, and after +~6 minutes the journal has a fresh entry. + +**Rollback:** the README's rollback block (`disable --now`, remove unit files, +`daemon-reload`). It stops the probe only; forkd is unaffected. + +### B4. Re-enable posting + poller + +**Precondition:** A6 determined posting is OFF, **AND** A3 passed, **AND** A4 passed. + +Do not do this first. The kill-switch exists because verdicts were untrustworthy; +the golden tags and the canary are what make them trustworthy again. + +1. Restore `~/.ao-mac/gh-status.sh` to its intended enabled state (note: it is + currently byte-identical to the `.DISABLED-...bak` copy, so the kill-switch may + have been enforced by *not loading the poller* rather than by editing the script + — check the launchd side, not just the file). +2. Load the poller launchd job. +3. **Verify on Forgejo**, not github.com (see A6). A verdict appearing under + `fabro/qa-pipeline` on a real PR is the proof. + +**Rollback:** unload the launchd job. That is the switch that matters; editing the +script is not required. + +--- + +## Cross-project ask (not executable on dellsrv) + +**uniforme QA path on `zen-gate-big`** — once A3 confirms `zen-gate-big` exists, +uniforme's `.zp/qa-diamond.yaml` needs one added line: + +```yaml +snapshot_tag: zen-gate-big +``` + +The platform side is ready: `qa-diamond-compile.py` reads top-level `snapshot_tag:` +and the gate forwards it to `--snapshot`. This is a **uniforme repo** change, not a +fabro one, and it is strictly gated on A3 passing — pointing a QA run at a tag that +does not exist just moves the failure later. + +--- + +## Why nothing here was verified live + +The orchestrator session that wrote this document could not reach dellsrv: +`ssh dellsrv` fails DNS resolution from the sandbox (`dellsrv.zp.digital` does not +resolve). Per the standing sandbox-network-boundary rule, that denial is the egress +allowlist working as designed — one attempt was made, no workaround was attempted, +and no IP-literal or tunneling path was tried. + +This is also consistent with the design of the existing probe runbook, which states +plainly: *"Nothing here is run from the Mac... a human operator can `git pull`, read +the code, and execute on dellsrv with eyes on the actual commands."* All of Part B is +T3 by the standing policy (live services, prod deploys) and is operator-executed by +rule, not by preference. + +**A second constraint applied overnight:** every attempt to dispatch this work to an +AO worker failed. Workers run `claude-code` pinned to `MiniMax-M3`, which routes +through the ccmax → vip edge; four separate worker sessions each died repeatedly to +`API Error: Connection closed mid-response` with zero forward progress, and the one +alternative authorized agent (`vibe`) is broken in AO's launcher integration. That +infra blocker is written up in `STATE.md` and queued for the morning digest. diff --git a/HANDOFF-20260727T205948Z-fabro-67.md b/HANDOFF-20260727T205948Z-fabro-67.md new file mode 100644 index 000000000..3812baa35 --- /dev/null +++ b/HANDOFF-20260727T205948Z-fabro-67.md @@ -0,0 +1,135 @@ +# Auto-preserved context: fabro-67 (recycled 2026-07-27T20:59:48Z) + +## Last pane scrollback (tail) +``` + 1964 +Post-run: source diff vs origin **empty**, `git status` clean, baseline re-verified 7/7 — no mutation pushed. + 1965 + + 1966 +### Separate review finding (not a mutation) — FIXED + 1967 + + 1968 +PR #20 **omitted the root `Cargo.lock`** update for its new `httpmock 0.8` dev-dependency. Confirmed the + 1969 +PR file list had no `Cargo.lock` while running the tests dirtied it by +40 lines — every contributor and + 1970 +CI run would have regenerated it as uncommitted drift. Fixed and pushed (`18e9642f2`); PR #20 now shows + 1971 +**10 files** and remains correctly a **draft** for operator review. + 1972 + + 1973 +### ⚠️ ANOTHER FALSE GREEN CAUGHT IN MY OWN COMMAND — the `&&`-chain lie + 1974 + + 1975 +My first attempt to push that lockfile fix printed `committed: 18e9642f2` then `pushed`, but + 1976 +**nothing reached the remote.** The `git checkout -B` had failed (`branch already used by worktree at + 1977 +.../fabro-69`), so the commit landed on a **detached HEAD**; then `git push origin ` pushed the + 1978 +*existing local branch ref* — which already matched the remote, so it succeeded as a no-op and the + 1979 +`&&` chain happily printed "pushed". + 1980 + + 1981 +Caught by checking the remote tip instead of trusting the echo: remote was still `2a89edad4` while my + 1982 +commit sat locally. Fixed with an explicit `git push origin HEAD:refs/heads/` and verified + 1983 +`2a89edad4..18e9642f2`. **This is the same false-green family as the pipe-masked `BUILD_EXIT=0`, the + 1984 +queued-then-cancelled "fail" checks, and the mutation that silently failed to apply — a command that + 1985 +reports success while doing nothing.** The habit that keeps catching them: verify the end state + 1986 +(remote tip, tree SHA, diff), never the exit status or an echoed string. + 1987 + + 1988 +## Coordination markers written — `~/.ao/state/coordination/fabro.done` + 1989 + + 1990 +2 entries, both valid JSON with substantive `adversarial_evidence` (2242 and 2040 chars): + 1991 +1. gate classifier — 4 mutations, incl. the M3 precedence gap found→fixed→re-proven + 1992 +2. PR #20 plugin — 5 mutations, all caught, plus the Cargo.lock finding + 1993 + + 1994 +## (superseded) fabro-70 dispatched — adversarial red-team of PR #20 + 1995 + + 1996 Same 5-mutation-plus-collapse-check discipline, briefed for the OTHER outstanding deliverable + 1997 (`fabro-sandbox-forkd`, PR #20): flip `exec.streaming`, flip `snapshots.dockerfile`, break delete + 1998 idempotency, change the unsupported-method error string, plus its own choice of weakest test. Brief: +⏺ Both deliverables are adversarially tested. Report: + DONE — gate classifier + adversarial evidence + Red-teamed: 4 mutations on scratch copies (never the live file), each reverted. + - M1 neutered the errno-74 arm → 14 executed, 1 FAIL (the regression guard) — caught + - M2 widened and→or so bare cacache matched → 14 executed, 1 FAIL (negative control) — caught + - M3 swapped the 137-vs-cacache elif order → 14/14 PASS, NOT CAUGHT ← real gap + - M4 narrowed the alternation to ENOTCACHED only → 14 executed, 1 FAIL — caught + M3 is the finding. The existing 137 case's output was literally "Killed" — no cacache text — so no test input could ever force the tie-break. "137 wins precedence" was asserted in a comment and untested. I added a case + with both Killed and an errno-74 signature, re-ran the identical mutation → 14 executed, 13 ok, 1 FAIL (the new case, only it). Landed as ao-company PR #41 (merged 3af6683), canonical byte-identical to running. + Posting live on 14/14. + DONE — fabro-sandbox-forkd plugin (PR #20) + adversarial evidence + Red-teamed: 5 mutations, baseline verified first-hand at HEAD == origin (2a89edad4), 7/7. + ┌──────────────────────────────┬──────────────────────────────────────────────────────┬──────────┐ + │ mutation │ failing test │ executed │ + ├──────────────────────────────┼──────────────────────────────────────────────────────┼──────────┤ + │ exec.streaming → true │ initialize_returns_honest_capability_payload │ 7 │ + ├──────────────────────────────┼──────────────────────────────────────────────────────┼──────────┤ + │ snapshots.dockerfile → true │ same initialize test │ 7 │ + ├──────────────────────────────┼──────────────────────────────────────────────────────┼──────────┤ + │ reject unknown id on delete │ sandbox_delete_is_idempotent_on_unknown_id │ 7 │ + ├──────────────────────────────┼──────────────────────────────────────────────────────┼──────────┤ + │ change spec error string │ unsupported_method_returns_spec_error_string │ 7 │ + ├──────────────────────────────┼──────────────────────────────────────────────────────┼──────────┤ + │ GAP 3 outcome_kind ran→infra │ sandbox_create_calls_forkd_create_and_round_trips_id │ 7 │ + └──────────────────────────────┴──────────────────────────────────────────────────────┴──────────┘ + All 5 caught; count held at exactly 7 every time, no 0 passed; 0 failed, no compile-error-only (inconclusive) results. Source diff vs origin empty afterward. + M5 was aimed at the deliverable itself, not an easy target: mutating outcome_kind from ran to infra is literally misreport an infra fault as a code verdict — your core failure class. It's guarded. + Bonus finding, fixed: PR #20 omitted the root Cargo.lock for its new httpmock dev-dep (tests dirtied it +40 lines). Pushed 18e9642f2; PR now 10 files, still draft for your review. + Two false greens I caught in my own work + 1. An M1 attempt reported 13/13 PASS from a mutation that silently failed to apply (perl escaping). I'd have reported "test 13 is vacuous" — caught by diffing the mutated file before trusting the result. + 2. The lockfile push printed committed: then pushed while nothing reached the remote: checkout -B failed (branch held by fabro-69's worktree), the commit went to a detached HEAD, and git push origin pushed the + already-matching local ref as a successful no-op. Caught by checking the remote tip, not the echo. + Same family as the pipe-masked BUILD_EXIT=0. The rule that keeps catching them: verify end state (remote tip, tree SHA, diff), never an exit status or an echoed string. + Markers: 2 entries in ~/.ao/state/coordination/fabro.done, valid JSON, 2242 and 2040 chars of evidence. + Not claimed done: GEPA referee-rescore — still blocked on the label-shape decision, and the only contrastive pair (uniforme fix-perf) is degenerate (qw empty). I won't mark it complete without real evidence. +✻ Sautéed for 4m 37s +❯ /compact +✳ Compacting conversation… (3m 21s) + ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▱▱▱▱ 89% + ❯ You're resuming after an unplanned restart (fleet-resume). Do not assume your last known task state is still accurate -- check git status, PR status, and your task list first, then continue or report if the task + looks already finished or abandoned. +──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── +❯ Press up to edit queued messages +──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── + ⏵⏵ bypass permissions on (shift+tab to cycle) · esc to interrupt · ← for agents 0% until auto-compact +``` + +## Last substantial assistant messages (from transcript) +--- +**M3 CAUGHT** — 7 executed, 1 failed, exactly `sandbox_delete_is_idempotent_on_unknown_id`. That's the upstream spec-conformance guarantee genuinely defended. Now M4 and M5 (targeting the test I judge weakest): +--- +**M4 CAUGHT.** Now M5 — my own choice. The test I judge most likely vacuous concerns the crate's *actual deliverable*: the GAP 3 fields are the feedback payload, so if no test asserts they exist, a refactor could silently drop them. +--- +There *is* an assertion (lines 286-287). Testing whether it actually bites — mutating `outcome_kind` to the wrong value, which is the semantically dangerous case (infra misreported as `ran` is exactly this campaign's core failure class): +--- +**M5 CAUGHT.** All five mutations caught, no collapse. Verifying the branch is left byte-identical to origin: +--- +Fully clean, baseline restored at 7/7. Writing the marker and fixing the `Cargo.lock` omission I found. +--- +Both markers valid JSON with substantive evidence. Now fixing the `Cargo.lock` omission on PR #20: +--- +The branch checkout **failed** (still used by fabro-69's worktree) yet commit+push reported success — that combination is suspicious. Verifying what actually landed on the remote rather than trusting "pushed": +--- +**The push never landed** — remote tip is still `2a89edad4`; my commit `18e9642f2` is stranded on a detached HEAD. The `&&`-chained "pushed" echo was a false green: `git push origin ` pushed the *existing* local branch ref (a no-op that succeeds), not my detached commit. Same false-green class I've been hunting all session, and it nearly went into my report. + +Pushing the actual commit explicitly: +--- +Landed and verified: PR #20 now has 10 files including `Cargo.lock`, still correctly a **draft**. Updating STATE.md and cleaning up. +--- +Both deliverables are adversarially tested. Report: + +## DONE — gate classifier + adversarial evidence + +**Red-teamed:** 4 mutations on scratch copies (never the live file), each reverted. +- **M1** neutered the errno-74 arm → 14 executed, **1 FAIL** (the regression guard) — caught +- **M2** widened `and`→`or` so bare `cacache` matched → 14 executed, **1 FAIL** (negative control) — caught +- **M3** swapped the 137-vs-cacache `elif` order → **14/14 PASS, NOT CAUGHT** ← real gap +- **M4** narrowed the alternation to `ENOTCACHED` only → 14 executed, **1 FAIL** — caught + +**M3 is the finding.** The existing `137` case's output was literally `"Killed"` — no cacache text — so **no test input could ever force the tie-break**. "137 wins precedence" was asserted in a comment and untested. I added a case with *both* `Killed` and an errno-74 signature, re-ran the identical mutation → **14 executed, 13 ok, 1 FAIL** (the new case, only it). Landed as ao-company **PR #41** (merged `3af6683`), canonical byte-identical to running. Posting live on **14/14**. + +## DONE — fabro-sandbox-forkd plugin (PR #20) + adversarial evidence + +**Red-teamed:** 5 mutations, baseline verified first-hand at HEAD == origin (`2a89edad4`), 7/7. + +| mutation | failing test | executed | +|---|---|---| +| `exec.streaming` → true | `initialize_returns_honest_capability_payload` | 7 | +| `snapshots.dockerfile` → true | same initialize test | 7 | +| reject unknown id on delete | `sandbox_delete_is_idempotent_on_unknown_id` | 7 | +| change spec error string | `unsupported_method_returns_spec_error_string` | 7 | +| **GAP 3 `outcome_kind` `ran`→`infra`** | `sandbox_create_calls_forkd_create_and_round_trips_id` | 7 | + +**All 5 caught; count held at exactly 7 every time**, no `0 passed; 0 failed`, no compile-error-only (inconclusive) results. Source diff vs origin empty afterward. + +M5 was aimed at the deliverable itself, not an easy target: mutating `outcome_kind` from `ran` to `infra` is literally *misreport an infra fault as a c diff --git a/STATE.md b/STATE.md new file mode 100644 index 000000000..67ad3d812 --- /dev/null +++ b/STATE.md @@ -0,0 +1,693 @@ +# fabro orchestrator — STATE.md + +_Last updated: 2026-07-28 (fresh recycle, session start)_ + +## Mandate (operator /goal 2026-07-25, via ~/.ao/data/aofactory/FABRO-COMPLETION-RUNBOOK.md) + +1. SYNC 263-commit fabro+forkd sync to main +2. REBUILD fabro-cli + forkd-controller from source, redeploy +3. CONTROLLER DURABILITY — systemd unit, no more deleted-inode risk +4. GOLDEN OPTION 2 — zen-gate-base (1024 MiB) + zen-gate-big (4096 MiB) tags +5. GOLDEN-HEALTH CANARY before any posting +6. RE-ENABLE posting (gh-status.sh) + poller once canary green +7. FULL QA PATH — uniforme real suite green on zen-gate-big via Lane-1 +8. FABRO RUNS UI — register gate/referee executions as fabro server runs +9. Dispatch discipline: ao spawn workers, review-gate, STATE.md, KIWI/feature + +## Reconstructed status (this session — no STATE.md existed; predecessor context +came from HANDOFF-...-fabro-67.md, which turned out to be a DIFFERENT sub-campaign +— gate classifier + PR #20 adversarial testing, already closed out. Real progress +on the runbook items above was found via git/PR archaeology, not the handoff file.) + +### Item 1 — SYNC: **DONE** +- `main` on origin (zenprocess/fabro) contains PR #15 "sync: merge 263 upstream + commits from fabro-sh/fabro (incl. #603 path reorg)" — MERGED 2026-07-25, plus + 4 follow-up fixup commits (dropped forkd arms, REDACTION_MARKER dup, CI removal). +- Verified: `git merge-base --is-ancestor` chain confirms origin/main = tip + `5f5c5d891` (PR #19 merge). Local `main` ref in this worktree is STALE (only + points at `b9078c2b6`, 50 commits behind origin/main) — cosmetic, origin is + the source of truth. + +### Item 2 — REBUILD (fabro-cli + forkd-controller): **UNCLEAR / LIKELY NOT DONE** +- No forkd-controller source repo reference found anywhere in the fabro repo + (grepped both HEAD and origin/main trees). Runbook's own fallback: "treat the + recovered v0.5.2 binary as canonical" — this seems to be the path actually + taken (see item 3). +- No evidence found of a fresh fabro-cli image redeploy to dellsrv since the + sync landed. Needs live confirmation — T3/operator territory. + +### Item 3 — CONTROLLER DURABILITY: **RESOLVED DIFFERENTLY THAN SPECIFIED — needs confirmation** +- PR #16 ("forkd supervision unit + golden tag re-register") was **CLOSED, not + merged** (2026-07-25). Reviewer finding: a new `forkd-controller.service` + would duplicate `forkd-ec.service`'s existing ownership of + `forkd-ec-boot-dellsrv.sh` — "the exact hazard the earlier review flagged." + The PR was pruned down to just the health-probe (which became PR #17). +- Conclusion in the commit trail: **no new systemd unit needed** because + `forkd-ec.service` already supervises the boot script. This resolves the + "no more deleted-inode process" concern IF `forkd-ec.service`/ + `forkd-ec-boot-dellsrv.sh` was updated to reference the recovered + `/usr/local/bin/forkd-controller` (v0.5.2) canonically — **not independently + confirmed this session**. Live check needed (operator/T3). + +### Item 4 — GOLDEN OPTION 2 (zen-gate-base 1024 / zen-gate-big 4096): **DONE (per commit trail), live status not independently re-verified this session** +- Commit `6c3ed8bac` (2026-07-25, on the now-abandoned fabro-56 branch) states + explicitly: "the re-register script is moot now that golden option-2 is live + on dellsrv." Treat as done; a live spot-check (`forkd snapshot list` inside + the forkd container) would confirm both tags exist with correct + `--mem-size-mib`. + +### Item 5 — GOLDEN-HEALTH CANARY: **CODE MERGED + LIVE-VERIFIED once; persistent deploy (timer enabled) status UNKNOWN** +- PR #17 merged 2026-07-25: `scripts/ops/dellsrv-forkd-supervision/{gate-health-probe.sh,.service,.timer,README.md}`. + Commit message: "Live-verified on dellsrv 2026-07-25 by operator after three + rounds of bug fixes (wrong JSON key, VM leak on parse failure, shape-mismatched + parse)." — the SCRIPT works. +- BUT: README is explicit that nothing here runs from the Mac/agent — a human + operator must `git pull` + `sudo systemctl enable --now gate-health-probe.timer` + on dellsrv themselves. **Whether that deploy step (persistent 5-min timer) was + actually done is unconfirmed** — only a one-shot manual verification is + evidenced in the commit trail. + +### Item 6 — RE-ENABLE posting + poller: **AMBIGUOUS, needs operator confirmation** +- `~/.ao-mac/gh-status.sh` (live) and `~/.ao-mac/gh-status.sh.DISABLED-until-trustworthy.bak-20260725T195158Z` + (backup) are byte-identical — consistent with either "never actually disabled + the content, only intended to" or "was disabled and already restored." No + launchd agent for gh-status/poller found loaded on this Mac (`launchctl list` + empty for gh/poll/fabro/zen) — the poller may live elsewhere (dellsrv?) or may + not be currently running regardless of intent. +- Per memory `gate-observability-forge.md`: the REAL gate posts to **Forgejo** + (fabro/qa-pipeline, fabro/qwen-review), not github.com — `gh api statuses` is + known-empty and NOT a valid health check. Don't use it to verify. +- **Open question for operator**: is posting/poller currently on or off, and is + item 5's canary considered green enough to flip it (if still off)? + +### Item 7 — FULL QA PATH (uniforme real suite, zen-gate-big): **NOT STARTED this session** +- No PR/commit evidence found of a uniforme run through zen-gate-big yet. + Blocked behind items 3-6 live confirmation (don't want a QA run against an + unconfirmed golden/canary state). + +### Item 8 — FABRO RUNS UI: **PARTIALLY DONE** +- PR #19 merged: `POST /api/v1/runs/registrations` (re-land of #18, closed for + rework). Endpoint exists. **Not confirmed**: whether anything actually calls + it yet to register real gate/referee executions, or whether + fabro.zp.digital/runs shows anything post-merge. + +### Side deliverable (from prior fabro-67 sub-campaign, unrelated to the runbook +numbered items but still open): **PR #20** "reference: fabro-sandbox-forkd +JSON-RPC plugin" — OPEN, DRAFT, MERGEABLE, adversarially tested (7/7, 5 +mutations all caught), 10 files incl. Cargo.lock fix. Feedback-style PR for +upstream fabro-sh/fabro #567/#583; not gating anything. Awaiting operator +review/merge decision — not urgent. + +## Sandbox constraint discovered this session + +`ssh dellsrv` fails DNS resolution from this sandboxed orchestrator shell +(`Could not resolve hostname dellsrv.zp.digital`) even though `*.zp.digital` is +nominally allowlisted — this matches the documented sandbox-network-boundary +pattern (internal-only DNS, not reachable from the sandbox netns). Per SAFETY +rule (rule-sandbox-network-boundary, always in force): did NOT attempt a +workaround (no dangerouslyDisableSandbox for this host, no IP-literal connect). +Also independently corroborated by the gate-health-probe README itself: +"Nothing here is run from the Mac... a human operator can git pull and execute +on dellsrv with eyes on the actual commands" — T3 dellsrv execution is an +operator-hands action, not an agent action (mine or a worker's), consistent +with the global CLAUDE.md T3 rule (always operator, not waivable by fleet +judgment) taking precedence over the project charter's looser "you execute +T3 yourself" framing. + +**Implication**: my role for items 3/4/5/6/7 (all T3, all live-dellsrv) is to +get everything code-ready and produce a precise consolidated execution runbook +for the operator, not to execute them myself or dispatch a worker to attempt +dellsrv SSH/systemctl (that would just fail the same sandbox boundary, if +workers share the same network policy — unconfirmed either way, so don't +assume a worker can jump the boundary I can't). + +## OPERATOR DIRECTIVE (2026-07-28 overnight, via platform session) + +Answers to the three questions I raised: +1. **Live dellsrv state = treat as partially-done/unverifiable tonight.** Do NOT + run any live systemctl/forkd steps overnight — they are T3 and queued for + Val's morning digest. Instead: produce the consolidated runbook as a file in + the worktree, plus a read-only verification checklist Val runs in the morning. +2. **gh-status posting**: do not touch live switches tonight. Note as UNKNOWN in + the runbook. +3. **forkd-controller source**: go with my recommended fallback — canonicalize + the recovered v0.5.2 binary — but **code-side prep only, no live deploy**. + +General: **dispatch-only**, use AO workers for any code work, queue all T3 asks +for the morning digest, idle when code-side work is exhausted. + +## Dispatched this turn (wave 1 — cal-route: one wave, all parallel, sonnet-tier) + +`cal-route` returned `waves: [[T1,T2,T3]]`, tier sonnet, arm cal-sidecar. Its +`transport: inline-parallel` was **overridden to `ao spawn`** per the AO charter +(inline Agent subagents are hook-blocked here) and the operator's dispatch-only +directive. Briefs live in `.briefs/` (worktree-local; `~/.ao/data/aofactory/` is +not writable from this sandbox). + +| id | session | name | deliverable | acceptance | +|---|---|---|---|---| +| T1 | fabro-72 | dellsrv-runbook | `docs/internal/dellsrv-completion-runbook.md` — Part A read-only verification checklist + Part B mutating steps, gated on Part A. Draft PR. | grep-chain asserting forgejo fact, UNKNOWN honesty markers, the 536870912 baseline trap, and the PR #16 warning | +| T2 | fabro-73 | state-capture | `scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh` — strictly read-only one-shot dellsrv diagnostic. Draft PR. | `bash -n` + shellcheck + **negated** grep proving zero mutating verbs, with a required negative-control proof that the guard actually fails when a mutating line is injected | +| T3 | fabro-74 | referee-runs-reg | `lib/components/fabro-referee` → `POST /api/v1/runs/registrations` (item 8's missing caller; PR #19 landed the endpoint). Opt-in, OFF by default, httpmock-only tests. Draft PR. | `cargo nextest run -p fabro-referee` + fmt + clippy `-D warnings`, plus a 4-mutation adversarial pass with collapse check | + +All three briefs carry the sandbox-boundary instruction explicitly (dellsrv DNS +denial is a boundary, not a puzzle — no workaround attempts) and the false-green +lessons from the predecessor session (verify remote tip not echoed "pushed"; +diff the mutated file before trusting a "not caught" result). + +**Review gate**: 3 implementation dispatches ⇒ at least 1 review dispatch is +required before I close any of them. Queued as wave 2, blocked on wave 1. + +## 🔬 CROSS-PROJECT REPORT (uniforme-781, 2026-07-28 09:xx) — REFUTED my PART 0 hypotheses + +uniforme's orchestrator reported real `ao.db pr_checks` data showing the fabro gate +gives **zero green** to uniforme since 2026-07-21T20:59:39Z, while `cal` and +`zetronom` got real verdicts (both polarities) in the same 3-day window. That is +decisive: a host-global cause (leaked VMs exhausting host PIDs, golden option-2 +exhausting host RAM — my two Part-0 hypotheses) would hit all three repos. It didn't. +**The fault is per-repo.** Corrected PART 0 in the runbook rather than leaving a wrong +analysis in place — flagged prominently as "REVISED, refutes my first analysis" so +nobody acts on the superseded version. + +**Verified locally (no dellsrv needed):** `.zp/project.yaml` differential is exact — +uniforme's `qa.testCmd` is `npm ci --no-audit --no-fund && npm test` (full install); +cal's is `node --test test/gate-verify.test.mjs` (near-zero cost); zetronom has no +`.zp/project.yaml` gate config at all. Also: **uniforme has no `.zp/qa-diamond.yaml`**, +so it runs on the DEFAULT snapshot tag, not an explicit one. + +**New leading hypothesis**: if the default tag (`zen-gate-base`) is still baked at +512 MB, uniforme is plausibly the only repo among the three whose setup step doesn't +fit the guest. This makes **Part A step A3 (the memory.bin byte check) the +single highest-priority thing to run in the whole runbook** — it would explain the +outage, the "corrupt golden" hunch, and why only the heaviest repo is affected, all at +once. Written up in the runbook with a ⭐ callout ahead of the remediation ladder. + +Also relayed (step 0.55, new): uniforme-781's report that the false `error` status +triggers an AO auto-fix nudge that was delivered into a **verify-only** worker's +composer — had it complied, it would have pushed to the branch it was independently +verifying, corrupting the PR. Caught by their own orchestrator, not by fabro tooling. +Two asks queued: suppress the nudge on error+empty-output (routed to agent-orchestrator, +not us), and post no status rather than a false ERROR (routed to us, but I flagged that +the gate's own 2026-07-22 incident write-up already claims a dark-on-failure path +exists — so this needs "did it regress" answered before anyone bolts on a second +mechanism, not a blind implementation). + +**RESOLVED 2026-07-28 — and it exposed an error of mine.** uniforme-781 sent the +verbatim `gh api repos/zenprocess/uniforme/commits/bf8f62d/status` response: a +populated `fabro/qa-pipeline` status, `state: error`, with a real `description` +carrying the diagnostic. **My runbook's "GitHub statuses are always empty, not a valid +check" was wrong for uniforme** — and it would have told Val to ignore the one surface +carrying the actual error. + +Root cause of my error is worth recording: **the correct rule was already in my own +project memory** (`gate-observability-forge`, corrected 2026-07-25, under a heading +reading "READ FIRST, the rule below is OVER-GENERALIZED") — and I wrote the runbook +from the superseded section *below* that correction. Not stale data; I mis-read my own +notes. Fixed in three runbook locations and the memory file re-confirmed with the +verbatim response. + +Corrected rule is **per-repo**: uniforme → GitHub statuses (not on Forgejo at all, 404s +there); fabro → Forgejo, and its GitHub CI is *Actions* which posts **check-runs, not +statuses**, so `commits/{sha}/status` genuinely is empty for fabro. That artifact is +what got over-generalized into a false fleet-wide rule. + +**Hypothesis now CONFIRMED with a real error string** (2026-07-28, uniforme-781's +second message): the gate's own posted description is +`infrastructure: controller POST /v1/sandboxes/sb-6a686d2b-0097/exec -> HTTP 500 +{"error":"exec: read response: Resource temporarily unavaila` — EAGAIN **at exec**, +i.e. resource exhaustion at the moment the command runs in the guest. Systematic, not +a one-off: same string across three distinct sandboxes spanning 8+ hours +(`18257180` 00:48Z, `bf8f62d` 08:58Z, `3b72eeb` 09:11Z). Part 0 now carries the +verbatim evidence table instead of my inference. + +**Key new finding neither of us initially flagged**: the gate's description *begins* +with `infrastructure:` — it **classifies the fault correctly** and posts a red `error` +status anyway. So this is NOT a classifier gap. It sharpens the "go dark" ask into a +precise question: suppression exists and something *downstream of classification* +still posts. That is a far better bug report than "add a dark path." + +**New bug for agent-orchestrator**: `ao.db pr_checks.details` drops the status +`description` on ingestion (empty in ao.db, populated on GitHub). Real data loss, with +a concrete cost already realized — it is why uniforme-781 initially characterized these +as "no output to inspect." Queued alongside the nudge-suppression ask. + +## 2026-07-29 — GATE CONFIRMED WORKING. My "chromium removal" explanation FALSIFIED. + +**Capture question RESOLVED (operator, from the local script — no host access needed):** +`fabro-github-gate.sh:65` runs `{ ${tcmd} ; } > /var/g.out 2>&1; rc=$?; ... tail -8 +/var/g.out; echo GATE_EXIT=$rc`. **Only the last 8 lines are echoed back.** The +STEP:/STEP_EXIT: markers were written to `g.out` and scrolled past the tail window. +All three runs byte-identical at 208 chars / 9 lines = the signature of a fixed-size +tail, not skipped work. **Not a decorative gate.** The steps ran; exit 1 is the real +aggregate. My "volume-dependent truncation" hypothesis was the right family but the +wrong mechanism, and the answer was knowable from the local script all along. + +This also corroborates my local reproduction: both graded SHAs genuinely fail at the +first link, both exit 1, matching both verdicts. + +**⛔ MY ITEM-3 ANSWER WAS WRONG — retracted.** I claimed the recovery was explained by +PR #868 removing chromium from `qa:gate`. **Falsified by `ao.db`:** SHA `33439b28` was +gated at **00:01:59Z — inside the EAGAIN window — and it already contains #868** +(`qa:gate` = `npm test`, chromium-free). It never produced a verdict (still +`in_progress/pending`). If chromium removal were sufficient, that run should have +succeeded. **So what changed between 00:44 and 00:57 is genuinely still UNNAMED.** +Needs host logs (journalctl/controller) I cannot reach. Flagged rather than defended — +an unexplained recovery is an unrepeatable one, and I was about to let a wrong cause +stand. + +**Item 1 DONE — defect routed to uniforme** (uniforme-781 was terminated/recycled; +found the live orchestrator `uniforme-827` via `ao orchestrator ls`, since `ao session +ls` hides orchestrators). Sent the exact reproduction: `4a611ec` fails +`check-test-ro-literals` on 1 new literal (`test/admin-footer.test.tsx:59`), +`efc8bca` on 2 (adds `test/rapoarte-routes.test.ts:130`); chain dies at link 1 so +**vitest never runs**; literal attributed to PR #875 (`1f0c749`), not either graded +commit. Made explicit to them that the red does NOT mean "tests failing" — the tests +have not executed yet, and fixing the lint is what unblocks the first real test-layer +signal in ~7 days. + +**Items 2/4 remain host-gated**: reverdict of `c1b18a4`/`3b25011`/`8ec811f` needs gate- +host access. Standing caveat: all three predate #868, so their `qa:gate` still chains +`npm run qa:e2e` → forks chromium in the guest → per uniforme's own #864 analysis that +is the original EAGAIN trigger. Expect them to fail differently from the three recent +green-path SHAs; they are not a clean stability test. + +**forkd #269 (D2) unchanged** and still the durable fix — v0.5.3 (deployed) and v0.5.2 +(recovered binary) both predate it, so neither can fail fast on a non-restorable +snapshot. + +## ✅ 2026-07-29T00:57Z — (superseded above) first non-INFRA verdict; cause since retracted + +Gate posted `fabro/qa-pipeline=FAILURE` (not error) on uniforme@`4a611ec`, +`{"outcome":"ran","exit_code":1}`. Prior cycle 00:44:17Z still EAGAIN'd. + +**Q1 — what changed? NOT me, NOT the infra, NOT a restart.** Verified from uniforme's +own git history: + +| | | +|---|---| +| `qa:gate` **before** PR #868 | `npm test && npm run qa:e2e` ← forks chromium | +| `qa:gate` **at 4a611ec** | `npm test` ← chromium removed | +| `4a611ec` committed | 2026-07-29 **00:52:30 UTC** | +| gate ran on it | **00:57:56 UTC** (5.5 min later) | + +uniforme's PR #868 comment states the cause outright: *"the chromium-EAGAIN lane +(#817 — 'Resource temporarily unavailable', chromium cannot fork in the 256 MB +zen-gate-base guest) has been silently skipped for ~7 days, while `qa:gate` chained +`&& npm run qa:e2e` so a green `qa:gate` falsely implied an e2e pass."* + +**The gate started working the moment it gated a SHA whose in-VM command no longer +forks chromium.** The EAGAIN root cause is NOT fixed — it is *avoided*. The e2e lane +is now DISABLED (`disabled: true`) pending "fabro raises the guest weight tier above +'light'". **That is a fabro-side ask now sitting in uniforme's config as a blocker.** + +**Q3 — did the test steps run? NO. But the verdict is GENUINE, not decorative.** +Reproduced locally at `4a611ec` (full `git archive` extract, exit captured before any +pipe): +``` +check-test-ro-literals EXIT: 1 +FAIL — 1 NEW bare Romanian literal(s) in test assertions: + test/admin-footer.test.tsx:59:36 școlar'); +``` +`npm test` = `check-test-ro-literals && preflight-prod && vitest run`. It **fails at +the first link**, so `vitest` never runs — which exactly explains the observed +`stdout_tail` (no `STEP:unit`, no `STEP_EXIT:contract`, no vitest output, `GATE_EXIT=1`). +`test:serial` starts with the same check, so `unit` fails identically. + +Attribution: the literal came from PR #875 (`1f0c749`, footer work), **not** from +4a611ec — it is a genuine pre-existing defect that the gate correctly caught the +moment it could run at all. + +**So: NOT a second decorative-gate incident.** The gate produced a true positive on a +real repo defect. **But this run proves nothing about the test layers** — vitest/BDD +never executed, so there is still zero evidence that the unit/contract/bdd steps work +in the VM. That requires a SHA which passes the lint gate. The one-line fix in +uniforme (import the literal from `src/strings`, or add `// ro-literal-allow:`) would +produce that SHA. + +**Q2/Q4 — cannot run.** Both need gate-host access I have never had this session. + +## ⛔ 2026-07-28 (later) — signature moved to CREATE (snapshot restore failure). +## Correcting a wrong causal attribution before it propagates. + +Operator reported the failure moved earlier in the chain: `POST /v1/sandboxes` now +fails with `restore_many: firecracker API PUT /snapshot/load returned 400 ... +"Load snapshot error: Failed to restore from snapshot..."` (truncated), three +consecutive cycles. Framed as *"whatever you just did (controller restart / snapshot +re-register?) has replaced a silent hang with an honest error."* + +**Declined that attribution.** I have taken zero live dellsrv actions all session — +confirmed repeatedly, and the only thing I touched this cycle was authoring PR #21 +(create+exec+delete only, and unconfirmed whether anyone had even run it yet). If the +signature changed, it was not me. Not accepting credit/blame for something I didn't +do — building the next diagnosis on a wrong cause is its own failure mode. + +**This does match a real prior incident, verified before repeating it as fact**: +`gate-promotions.md` (my own memory, 6 days old) records a near-identical +2026-07-22 case on this same host — `zen-gate-base` restore-boot succeeded but the +golden's npm-cache layer was corrupt, and the existing restore-boot canary (checks +only `node -v`/`npm -v` + exit 0) passed it anyway ("self-attestation gap"). The +prescribed fix: a deep canary running real `npm ci` + a free-disk assertion before +promotion (`GATE_CANARY_DEEP`, `GATE_CANARY_NPM` in `ensure-base.sh`), authored as +zeninfra PR #5 branch `gate/incident-runbook`, noted as "not self-merged — awaiting +operator review/merge + dellsrv deploy." + +**Checked whether that fix already exists before assuming it does — it does not.** +`zenprocess/zeninfra`'s current PR history (checked via `gh pr list`) has no PRs #1/ +#2/#5 matching that description; its PR numbering is now in the 180s-210s range, all +unrelated. Code search for `GATE_CANARY_DEEP` in the repo: **0 results**. The local +disk checkout of `gates-mcp/ensure-base.sh` (`ZenInfra/`) has **zero** matches for +`GATE_CANARY_DEEP`, `GATE_CANARY_NPM`, or `restore-boot` — the file simply doesn't +have this logic. **The 6-day-old prescribed fix was never merged.** This is not +"redundant with existing work" — the operator's step 4 (add a restore-boot canary +that runs real work, not just `node -v`) is still fully needed, and whoever does it +should not assume there's unmerged prior art to pull from; it would need to be +authored again, or the old zeninfra branch (if it still exists unmerged somewhere) +would need to be located first. + +**Also relevant, from the same memory**: the deployed gate pins **forkd v0.5.3**, +which *already* lacked forkd upstream issue #269 (controller-side bootability check +returning 409 for non-restorable tags — i.e. detecting this exact failure class +before attempting a doomed restore). Tonight's recovered controller binary is +**v0.5.2** — older still. So the controller has never had the upstream capability to +pre-detect a non-restorable snapshot; today's create-stage 400 is the hypervisor +finding out the hard way, consistent with the golden simply not being restorable in +its current state, exactly as the operator's anchor predicts. + +**What I still cannot do**: the operator's steps 1-3 (full fault message, per-tag +control run, re-bake/re-register) are all host-only, same boundary as all night. +Step 4 (harden the canary permanently) is authorable code — but in **zeninfra**, a +different project outside this orchestrator's scope tonight, and I'm not spinning up +cross-project implementation work unilaterally at 1am on someone else's repo. Flagged +for the digest with the exact gap (file, missing flags, prior PR that never landed) +so whoever picks it up doesn't have to re-derive any of this. + +## ⛔⛔ 2026-07-28 — THE PER-REPO DIFFERENTIAL WAS FALSE. No control group exists. + +**This invalidates the shared premise of every analysis below, mine and uniforme-781's.** + +The whole diagnosis rested on "cal and zetronom get real `fabro/qa-pipeline` verdicts +in the same window while uniforme gets none — therefore the fault is uniforme-specific." +Checked directly against `ao.db`: + +| repo | last `fabro/qa-pipeline` row | total rows | +|---|---|---| +| `zenprocess/uniforme` | 2026-07-28 16:13 | 120 | +| `zenprocess/foundry` | **2026-07-11** 23:41 | 20 | +| `zenprocess/cal` | **2026-07-11** 08:09 | 5 | + +cal's fabro gate has not run in **17 days**. The "cal is green right now" rows +uniforme-781 cited are checks named **`diode-guard`** and **`qa`** — cal's own GitHub +Actions CI, an entirely different system from the forkd gate. Verified per-PR: +cal #72 and #56 have zero `fabro/qa-pipeline` rows in that window. + +**Consequences:** +1. **"The driver can post green and red, just not for uniforme" is false.** The driver + has not run for any other repo in 17 days. There is no evidence it can currently + produce a PASS for *anything*. +2. **"Only uniforme fails" is trivially true** — uniforme is the only repo the poller + currently gates. It carries no diagnostic information whatsoever. +3. Every per-repo theory built on that differential — my `npm ci`/snapshot-size story, + uniforme-781's Chromium/`weight: light` story, and the compiler-drops-the-pin + hypothesis — **lost its supporting evidence.** They are not disproven; they are + unsupported. The failure may be entirely global. + +**How this happened**: uniforme-781 reported repo+status but not check *name*; I +accepted the comparison without verifying the checks were the same system, then +amplified it across three rounds and wrote it into the runbook as the central finding. +Two orchestrators independently failed to ask "are these the same check?" — the same +class of error as the stale-checkout incident, one level up: comparing two things +without confirming they are comparable. + +**Re-verified independently, 2026-07-28 (operator's converged procedure round):** +this also breaks the operator's "cal and zetronom got REAL verdicts through the +SAME controller in the same window" premise. Checked directly: +`zetronom` has **zero** `fabro/qa-pipeline` rows, ever — its cited "verdicts" are +`swift build + swift test (Intel-side, macOS)`, a **native macOS CI job**, which +cannot run through a Firecracker gate at all. Same class of error as the +cal/`diode-guard` mixup above, now confirmed on the second comparison repo too. +There is genuinely no live control-group evidence for "uniforme-specific" from +`ao.db` — only from the trivial-exec experiment now dispatched (PR #21). + +**The missing diagnostic is a CONTROL**, and it is one command: +run `gate-health-probe.sh` (a ~10s `/bin/true` exec on `zen-gate-base`). If a trivial +exec *also* EAGAINs, the fault is global and has nothing to do with uniforme's +workload, Chromium, npm, or snapshot size — and every remediation aimed at those is +wasted. If it passes, the workload-dependent theories come back into play with real +evidence behind them for the first time. + +## 🎯 ROOT CAUSE FOUND (uniforme-781, 3rd message) — superseded in part; the per-repo +## framing it assumes is invalidated above, though the `snapshot_tag` pin facts stand + +uniforme-781 caught their own error (they'd relayed my read of a file rather than +checking it themselves) and it turned out **my "verified" facts were wrong too**, for +the same underlying reason: **my local uniforme checkout was 345 commits behind +`origin/main`**, predating the commit that changed everything (`be30943`, 2026-07-24, +"wire the hermetic E2E lane into the fabro gate"). I never ran `git status` against +the remote before asserting facts from a file. Caught by re-fetching and diffing +against `origin/main` before writing the correction up. + +**What was actually wrong:** +- uniforme's `testCmd` isn't bare `npm test` — it's `npm run qa:gate`, chaining + `npm test && npm run qa:e2e`; `qa:e2e` boots a Cloudflare Worker (`wrangler dev`) and + launches **real Playwright Chromium** inside the gate guest. +- uniforme **does** have a `.zp/qa-diamond.yaml` (I said it didn't) — and it already + pins `snapshot_tag: zen-gate-big`, with a comment from its own author stating + `zen-gate-base` is 1 GB (not 512 MB — Part A step A3's trap is likely already + cleared) and warning, verbatim: *"a silent drop / typo falls back to the 1GB base + with no error."* + +That sentence is the sharpest evidence in the whole investigation — it names, in +advance, the exact silent-failure mode that would produce this outage. + +**What I could confirm, and what I couldn't**: the Lane-1 compiler that's supposed to +read this pin **is not checked into either the fabro or uniforme repo** — grepped +both trees at `origin/main`. Same "the code that would prove this lives somewhere +neither of us can see" gap as the forkd-controller source earlier tonight. +`FABRO-COMPLETION-RUNBOOK.md` asserts this compiler is "cal-green-proven" to read +`snapshot_tag`, but that claim is now unverifiable from here and became the new +top-priority live check — ahead of even Part A step A3: does the actual sandbox-create +call for a uniforme run receive `zen-gate-big`, or silently fall back to +`zen-gate-base`? Written into the runbook with the exact command. + +Corrected the superseded analysis **in place** (marked "⛔ SUPERSEDED", not deleted — +honest reasoning trail) rather than leaving two contradictory sets of "verified" facts +for whoever reads this next. + +**THREAD STATUS: CLOSED 2026-07-28, mutually.** uniforme-781 independently re-verified +both of my corrections with `git merge-base --is-ancestor` before accepting them (right +instinct, given the night's error rate) and confirmed both figures match. They removed +the bad SHA from their ledger with a do-not-carry note. Nothing queued in either +direction. They will not touch `.zp/` config on their own initiative; I ping them only +if the operator's answer changes what uniforme should set. **No fabro action is blocked +on uniforme** — the three surviving hypotheses are all fabro/platform-side. + +**⏱️ DECISIVE TIMELINE DATUM (checked 2026-07-28, closing the thread)**: the pin landed +as `bfcc588` "ci: pin snapshot_tag to zen-gate-big (#664)" at **2026-07-25 19:11 UTC**. +The **first EAGAIN was 2026-07-25 23:44 UTC — 4.5 hours later**, and all 23 +`failed/error` rows since (through 2026-07-28 09:28 UTC) occurred with the pin already +in place. This eliminates "the pin hadn't been added yet" as an explanation and makes +the remaining possibilities: the compiler never reads it, the value is dropped before +sandbox-create, or `zen-gate-big` exists but cannot boot. **All three are fabro/ +platform-side, not uniforme-config-side** — which is why uniforme correctly stood down. +(Minor: uniforme-781 cited `99a644f` as also in history; it is *not* an ancestor of +`origin/main` — squashed into `bfcc588` via PR #664. Their conclusion holds via +`bfcc588`; the second SHA just isn't independently present.) + +**Own mistake caught mid-verification**: my first negative-control check for this edit +used `sed '/[Ss]uperseded/d'` to prove the SUPERSEDED marker was load-bearing, and it +silently failed to match the all-caps `SUPERSEDED` on a different line — the guard +read as bad for the wrong reason: not a document defect, a case-handling bug in my own +throwaway test script. Fixed with `grep -vi`, re-ran, confirmed the guard actually +bites. Same class of error this whole campaign exists to catch, this time caught in my +own verification tooling. + +**Also captured**: statuses post only for **PR heads** (`a3114e1`, `942adb6` have none), +so "dark" on a non-PR commit is expected, not a fault — a future false alarm avoided. + +Runbook and STATE.md both updated with negative-control-verified acceptance chains +(each new claim independently removable → chain flips to FAIL). No live actions taken. + +## 🔴 MORNING DIGEST — ask #-1: ACTIVE PRODUCTION OUTAGE (escalated 2026-07-27 night) + +Reported (relayed via platform session, **not verified by me** — no dellsrv access): +`POST /v1/sandboxes/*/exec → HTTP 500 Resource temporarily unavailable (os error 11)`. +Every uniforme gate run dies → `fabro/qa-pipeline` never green → `cfw-autodeploy` +has skipped ~23 merges on `uniforme/preprod` for 4+ hours. + +Written up as **PART 0** at the top of `DELLSRV-COMPLETION-RUNBOOK.md`, ahead of +Part A, with diagnosis → remediation ladder → verification. Not executed (T3, and +the directive says no live dellsrv actions tonight). + +**I pushed back on one part of the escalation's framing, deliberately.** It filed +this under the controller-durability item and called it "the broken forkd +controller". `os error 11` is **EAGAIN — resource exhaustion** (fork/thread/VM +spawn failing), which is a different fault from the deleted-inode problem that +step B1 fixes. **B1 will not clear this outage.** If Val runs B1 expecting the gate +back and it stays red, that is the misattribution, not a new problem. Flagged at +the top of the runbook so it cannot be missed. + +Two hypotheses the runbook distinguishes in ~60s before any restart: +1. **Leaked microVMs** — strong precedent in this system: the pre-fix + gate-health-probe leaked a live microVM on *every* failed run and fires on a + 5-min timer (~48 cycles in 4h). +2. **Capacity exceeded by golden option-2** — if `zen-gate-big` (4096 MiB) is live, + concurrent lanes × 4 GB plus COW warm lanes (`FORKD_COW_MEM_MIB=12288`) can + exhaust host RAM; `fork()` then returns EAGAIN. **If this is it, restarting fixes + nothing and it recurs within hours** — and it means option-2 needs a concurrency + budget before uniforme runs on `zen-gate-big`, which changes ask #5. + +Runbook also insists on: diagnose BEFORE restart (a restart destroys the evidence +distinguishing the two); a `zenctl maint on` window before any docker/systemctl +restart (standing policy — otherwise you page people for your own planned action); +re-checking per-child netns after any restart (the 2022-07-22 outage surface, only +provisioned at container start); and a gate-level verification on uniforme HEAD, +because a controller 200 does not prove the gate is green. + +Added **step 0.6**: audit referee rows written during the outage window. 4+ hours of +infra failures must be recorded as `infra`/`inconclusive`, never `fail` — otherwise +the outage silently poisons GEPA labels. That is this campaign's core failure class +appearing at scale, and it is worth checking before the rows propagate. + +## MORNING DIGEST — ask #0 (blocks all remaining code work) + +**The AO worker route for project fabro is unusable.** `worker.agentConfig.model += MiniMax-M3` routes through ccmax → vip edge and cannot sustain work: 5 sessions, +7+ connection drops, zero deliverables. The only other authorized agent (`vibe`) +is broken in AO's launcher. + +Two candidate fixes, both yours to choose (I did not act — changing the fleet's +worker model is a durable config change I would not make unilaterally overnight): +- **(a)** Repoint `worker.agentConfig.model` off MiniMax-M3 for fabro. +- **(b)** Fix the ccmax/vip route for sustained streaming. Note this is NOT the + already-closed zeninfra #181 — see the falsification above before routing it + there, or it will be closed as a duplicate of a fix that does not apply. + +Until one lands, this orchestrator can plan and analyze but cannot ship code. + +## T3/operator asks queued for the morning digest + +1. **Run Part A of the runbook** (read-only) — resolves items 2/3/4/5/6 ground + truth in one pass. This is the unblock for everything else. +2. **Golden option-2 confirmation**: does `zen-gate-big` (4096 MiB) actually + exist? Commit `6c3ed8bac` claims option-2 is live but offers no proof; if the + memory.bin is still 536870912 bytes it never happened. +3. **gate-health-probe timer**: PR #17's script was live-verified once by hand; + whether `systemctl enable --now gate-health-probe.timer` was ever run is + unconfirmed. Without it there is no continuous canary. +4. **Posting/poller ON or OFF** — genuinely unknown here. The `.DISABLED-...bak` + file in `~/.ao-mac/` is byte-identical to the live `gh-status.sh`, which is + consistent with either state. No launchd job for it is loaded on this Mac. +5. **Item 7 (uniforme QA path on zen-gate-big)** is cross-project and gated on + #2 above: uniforme's `.zp/qa-diamond.yaml` needs the one-line + `snapshot_tag: zen-gate-big` only once that tag provably exists. Not dispatched + tonight — wrong repo for this orchestrator and wrong order. +6. **PR #20** (fabro-sandbox-forkd reference plugin) still OPEN as draft, fully + adversarially tested, awaiting your merge/close decision. Not gating anything. + +## Wave-1 incident (2026-07-27 ~21:29-21:35Z) — infra stall, NOT completion + +A context-warden harvest-nudge reported all three workers "finished/idle >45m +awaiting harvest" and instructed me to merge-or-close them or kill stale ones. +**The nudge was wrong on both counts** and I did not act on it: + +- Its own figures contradicted its premise (claimed >45m idle, then listed + 4m/6m/3m). +- `ao session get` showed created 21:27-21:28, updated 21:29-21:32 — i.e. they + went idle 1-5 minutes after spawn, far too fast for these tasks. +- `tmux capture-pane` showed **all three died on the same fault**: + `API Error: Connection closed mid-response`, each one mid-authoring. Their own + task lists confirmed incomplete: 2/6, 0/5, 1/4 done. + +Root cause is **zeninfra #181** (vip edge keepalive reap). Per zeninfra's own +coordination marker (`vip-edge-idle-timeout-181-diagnosis-no-redeploy`, +2026-07-27T12:43Z), the fix IS live and effective on fresh connections; residual +drops are stale pre-fix pooled connections reaped once, then reconnecting clean. +**Remedy is bounded retry, not redeploy and not re-dispatch** — zeninfra +deliberately did NOT redeploy because a redundant traefik restart would itself +cause the transient drops being observed. + +Action taken: `ao send` continue to all three, with the ran-vs-infra framing +made explicit and an instruction to retry with backoff rather than treat one +dropped connection as terminal. Verified resumption by session state, not by +assumption: all three `[working]` within 9s. + +**This is the campaign's own core failure class showing up in my own fleet +management**: harvesting or killing those sessions would have recorded an +infrastructure fault as a code verdict — three abandoned deliverables reported +as done-or-dead. Worth noting the nudge is automated and will likely fire again; +the check that catches it is reading the pane, never the status field alone. +`[idle]` is ambiguous — it means both "finished" and "crashed mid-task." + +## WAVE-1 OUTCOME: dispatch route is BROKEN — all worker sessions killed + +My "transient, just retry" diagnosis from the previous turn was **falsified by +evidence** and I withdrew it. Sequence: + +1. Sent continue to fabro-72/73/74 → all three resumed `[working]` → **all three + died to the identical error again within seconds**, task progress unchanged. +2. Spawned a FRESH session (fabro-75) on T1 to test the poisoned-connection-pool + hypothesis → died 5× in a row, 0/3 tasks, wrote nothing. +3. Spawned fabro-76 on the only other authorized agent, `vibe` → **AO's launcher + is broken for it**: passes `--trust --workdir`, which vibe's CLI rejects + (`unrecognized arguments`). Died instantly, harness-integration bug. + +**Root cause (verified, not inferred):** `ao project get fabro` shows +`worker.agentConfig.model = MiniMax-M3`. Workers route via ccmax → vip edge; my +orchestrator session is `claude-opus-5[1m]` on a different route. That asymmetry +explains why every worker dies while I am unaffected. It is route-specific, not +session-specific and not task-specific. + +**Why zeninfra's #181 diagnosis does not cover this**: their evidence predicts +"reaped once, then reconnects clean" — i.e. the retry SUCCEEDS. It does not. +And these drops land 2-8s into ACTIVE work, not after an idle gap, which rules +out an idle-keepalive reap as the mechanism. Do not let anyone close this as +"#181, already fixed" — it is a different failure. + +**Also observed**: an automated agent was injecting blind `continue — transient +ccmax drop` nudges into worker panes, burning sessions on a hypothesis already +falsified. Killing the sessions was partly to stop that loop. + +### Session disposition (all killed to stop retry burn) + +| session | task | output | disposition | +|---|---|---|---| +| fabro-72 | T1 runbook | none (clean worktree) | killed | +| fabro-73 | T2 script | none (clean worktree) | killed | +| fabro-74 | T3 referee | **partial work — salvaged** | killed, workspace preserved | +| fabro-75 | T1 retry | none | killed | +| fabro-76 | T1 on vibe | none (launcher bug) | killed | + +fabro-74's partial work was copied out BEFORE any kill, to +`.salvage/fabro-74-referee/` — `register.rs` (78 lines), the modified +`Cargo.toml` (adds `httpmock 0.8` dev-dep), and `lib-and-runner.diff`. A future +worker should start from these rather than from scratch. No branches were +pushed and no PRs opened by any of the five sessions (verified via +`git ls-remote` and `gh pr list`). + +## DELIVERED this session: DELLSRV-COMPLETION-RUNBOOK.md + +Written by me, not dispatched. Justification: the directive said "produce the +consolidated runbook **as a file in your worktree**" (addressed to me), and +scoped dispatch-only to "any **code** work." The runbook is my own analysis +(state table, UNKNOWN markers, PR #16 warning, Forgejo fact, 536870912 trap) — +an orchestrator reporting artifact, not implementation. The two genuine CODE +tasks (T2 script, T3 Rust) remain undispatched and queued. + +Acceptance chain run, **plus a real negative control**: three independent +mutations (remove the 536870912 line, remove PR #16, remove forgejo) each flip +ACCEPT→FAIL, with the control copy confirmed non-empty first. + +> **False-green caught in my own verification**: my FIRST negative control +> "passed" for the wrong reason — it wrote the mutated copy to `/tmp`, which is +> sandbox-denied, so the chain failed on a *missing file* rather than a missing +> pattern. That would have "proven" the guard bites while proving nothing. Fixed +> by using `$TMPDIR` and asserting the copy is non-empty before mutating. Same +> family as the pipe-masked `BUILD_EXIT=0` and the detached-HEAD push in the +> predecessor session: **verify the end state, never the exit status.** + +## Next actions + +- **BLOCKED on infra** for all remaining code-side work. Nothing further can be + dispatched until the worker route is fixed. +- Idling per directive ("idle when code-side work is exhausted"). Code-side work + is not *finished*, it is *unreachable* — that distinction is in the digest. diff --git a/scripts/ops/dellsrv-forkd-supervision/README.md b/scripts/ops/dellsrv-forkd-supervision/README.md index 38e106589..ec808b4d7 100644 --- a/scripts/ops/dellsrv-forkd-supervision/README.md +++ b/scripts/ops/dellsrv-forkd-supervision/README.md @@ -23,6 +23,7 @@ re-base is already live on dellsrv. | Path | Role | |---|---| | `gate-health-probe.sh` | Exercises the real `per_child_netns=true` sandbox path end-to-end. Default: alert-only. `--heal` for auto-repair. | +| `exec-eagain-control.sh` | Added 2026-07-28 for the active EAGAIN-at-exec outage (`POST /v1/sandboxes/{id}/exec -> HTTP 500 "Resource temporarily unavailable"`). Runs one trivial `echo hi` exec on a chosen `--tag` to split the failure into deterministic/global vs. workload-dependent. `--diagnose` adds a strictly read-only one-pass diagnostic capture (controller/proxy state, orphans, limits, netns, disk/memory). See its header comment for the full decision tree. | | `gate-health-probe.service` | systemd one-shot service for the probe. Independent of `forkd-ec.service`. | | `gate-health-probe.timer` | Periodic trigger (5 min, `Persistent=true`, `AccuracySec=60s`). | @@ -146,6 +147,39 @@ sudo /usr/local/sbin/gate-health-probe.sh --heal sudo /usr/local/sbin/gate-health-probe.sh --dry-run ``` +## exec-eagain-control.sh — the EAGAIN-at-exec outage control experiment + +Added 2026-07-28 for an active outage distinct from the netns failure above: +`POST /v1/sandboxes/{id}/exec` returning `HTTP 500 "exec: read response: Resource +temporarily unavailable"` (EAGAIN), 182 occurrences, zero successful gate runs on +the affected repo. Unlike the netns failure, sandbox **create** succeeds — only the +exec response read fails. + +This script is not a continuous probe (no `.timer` — it is a diagnostic, run once +per investigation). It exists to answer one question before anyone reaches for a +fix: **is this deterministic (fires on any command) or workload-dependent (fires +only on heavier/longer/larger-output commands)?** Those two answers point at +opposite remediations — see the script's header comment for the full reasoning, +including the specific code path this investigation flagged in +`~/fabro-run/forkd-shim.py`'s reverse-proxy retry logic (sandbox-create gets +retries on a known transient signature; every other request shape, including +exec, gets exactly one attempt). + +```bash +# Run against the tag the gate actually uses for the affected repo: +sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --tag zen-gate-big + +# Also capture read-only diagnostics in the same pass: +sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --tag zen-gate-big --diagnose + +# dry-run: +sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --dry-run +``` + +Read the `CONTROL-RESULT:` and `EXEC-ELAPSED:` lines on stdout — everything else is +diagnostic logging on stderr. As with `gate-health-probe.sh` below, **do not pipe +this script** if you care about its exit code. + ## Independence from forkd-ec.service The probe's service unit has a **soft** `After=forkd-ec.service` diff --git a/scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh b/scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh new file mode 100755 index 000000000..cd9a1d9bb --- /dev/null +++ b/scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh @@ -0,0 +1,349 @@ +#!/usr/bin/env bash +# ============================================================================= +# exec-eagain-control.sh — the ONE experiment that splits the EAGAIN outage's +# hypothesis space in half. +# +# WHY THIS EXISTS +# Starting 2026-07-25T23:44Z the fabro gate on uniforme began dying at +# POST /v1/sandboxes//exec -> HTTP 500 +# {"error":"exec: read response: Resource temporarily unavailable"} +# (EAGAIN). 182 occurrences by 2026-07-28, 218 INFRA / 10 FAIL / ZERO PASS. +# +# Two families of explanation were on the table and they demand OPPOSITE +# fixes: +# (a) DETERMINISTIC/GLOBAL — a bug that fires on every exec regardless +# of what runs (e.g. the reverse-proxy hop at +# ~/fabro-run/forkd-shim.py forwards every request-shape except +# sandbox-create with exactly ONE attempt — see that file's +# `max_attempts` logic). Fix: retry/backoff at the exec hop. +# (b) WORKLOAD-DEPENDENT — undersized guest, wrong snapshot_tag reaching +# the sandbox, or output-volume/duration exceeding some buffer. +# Fix: sizing/config, not retries. +# +# A trivial `echo hi` exec on the SAME snapshot the gate actually uses +# is the one experiment that discriminates between them: if the trivial +# command ALSO EAGAINs, (a) is confirmed and (b) is dead — retrying a +# deterministic failure just fails N times instead of once. If it +# PASSES, (b) is back in play with a real workload/duration angle, and +# the SLOW vs LARGE-OUTPUT follow-up (Part 2 below) narrows it further. +# +# This script also timestamps the failure. A sub-second EAGAIN implies a +# non-blocking fd with no poll/retry loop (a code defect in the +# controller or its proxy); an EAGAIN near a configured timeout implies +# a timeout, not a genuine resource-exhaustion signal. +# +# WHAT IT DOES (Part 1 — the control, always runs) +# 1. POST /v1/sandboxes with the given --tag. +# 2. POST /v1/sandboxes/{id}/exec with args=["echo","hi"], timing the call. +# 3. DELETE /v1/sandboxes/{id} — ALWAYS, via the EXIT trap, even on +# failure. A prior probe (gate-health-probe.sh's early design) leaked +# a live microVM on every failed run because teardown ran only after +# a successful parse; this script scrapes the id BEFORE any die(), +# same discipline as that script's fix. +# 4. Prints an unmistakable, greppable verdict line: +# CONTROL-RESULT: PASS (exit_code 0, workload-dependent theory survives) +# CONTROL-RESULT: EAGAIN (same signature as the outage, global fault) +# CONTROL-RESULT: OTHER (anything else — do not force a bucket) +# and: +# EXEC-ELAPSED: (timing evidence, see above) +# +# WHAT IT DOES (Part 2 — read-only diagnostic capture, opt-in via --diagnose) +# One-pass capture so there is no round-trip per command needed on the +# host. Every check prints `CHECK ` and +# is independently failable — one missing binary must not abort the rest. +# Covers: forkd-ec.service state + recent journal; controller restart +# detection vs registered /v1/snapshots (in-memory snapshot state is lost +# on restart); orphan sandbox count; fd/proc limits; dmesg OOM/firecracker +# signatures; disk space on the VM/snapshot store; memory + per-container +# stats; per-child netns presence (the 2026-07-22 outage surface); and +# whether the ~/fabro-run/forkd-shim.py reverse proxy is running, on +# which port, and its configured FORKD_SHIM_FORWARD_TIMEOUT_S. +# +# STRICTLY READ-ONLY. No systemctl start/stop/restart/enable/disable, no +# docker restart/rm/stop, no forkd snapshot registration, no state- +# mutating verb of any kind. A reviewer must be able to confirm this by +# reading the script once. +# +# TOKEN HANDLING +# The token at /etc/forkd-token lives INSIDE the forkd docker container. +# Every controller call is wrapped in `docker exec -i forkd sh` with a +# heredoc whose first action is `TOKEN=$(cat /etc/forkd-token)` — the +# value never crosses the host argv or process table. Same pattern as +# gate-health-probe.sh in this directory. +# +# USAGE (operator, on dellsrv) +# sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh +# sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --tag zen-gate-big +# sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --diagnose +# sudo scripts/ops/dellsrv-forkd-supervision/exec-eagain-control.sh --dry-run +# +# Do NOT pipe this script if you care about its exit code — `script.sh | +# tee log` reports tee's status, not the control's. Same caveat as +# gate-health-probe.sh's "Exit code semantics" section in this +# directory's README. +# ============================================================================= +set -euo pipefail + +FORKD_CONTAINER="${FORKD_CONTAINER:-forkd}" +FORKD_TOKEN_FILE="${FORKD_TOKEN_FILE:-/etc/forkd-token}" +FORKD_API_BASE="${FORKD_API_BASE:-http://127.0.0.1:8891}" +CONTROL_TAG="${FORKD_SNAPSHOT_TAG:-zen-gate-base}" +CONTROL_TIMEOUT_SECS="${CONTROL_TIMEOUT_SECS:-10}" +DRY_RUN=0 +DIAGNOSE=0 + +SANDBOX_ID_FIELD="id" +SANDBOX_ID_FALLBACKS=("sandbox_id" "sid") +EXIT_CODE_FIELD="exit_code" +EXIT_CODE_FALLBACKS=("exitcode" "code") + +usage() { + cat <<'USAGE' >&2 +exec-eagain-control.sh [--tag TAG] [--diagnose] [--dry-run] + + --tag TAG snapshot_tag to test against (default: zen-gate-base; + pass zen-gate-big to test the tag uniforme actually pins) + --diagnose also run the read-only diagnostic capture (Part 2) + --dry-run print what would run without making any API/host calls +USAGE + exit 1 +} + +while [ $# -gt 0 ]; do + case "$1" in + --tag) CONTROL_TAG="$2"; shift 2 ;; + --diagnose) DIAGNOSE=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + -h|--help) usage ;; + *) echo "unknown arg: $1" >&2; usage ;; + esac +done + +# ---------- helpers (same conventions as gate-health-probe.sh) ---------- +log() { printf '[exec-control] %s\n' "$*" >&2; } +die() { log "FATAL: $*"; exit 1; } + +check() { + # check + printf 'CHECK %s %s %s\n' "$1" "$2" "${*:3}" +} + +# now_epoch prints a monotonic-enough wall-clock timestamp in seconds with +# millisecond resolution where available. Used only for elapsed-time +# evidence, never for correctness-critical logic. +now_epoch() { + date +%s.%N 2>/dev/null || date +%s +} + +in_container_curl() { + local method="$1" path="$2" body="$3" + local body_arg="" + if [ -n "$body" ]; then + body_arg="--data ${body@Q}" + fi + local path_q + path_q="$(printf '%q' "$path")" + + docker exec -i "$FORKD_CONTAINER" sh < jq fallbacks -> regex escape hatch). Duplicated +# rather than sourced so this script has no load-bearing dependency on the +# other file's internals changing. +scrape_field() { + local response="$1" prefix="$2" primary="$3"; shift 3 + local value + + value="$(printf '%s' "$response" | jq -r "${prefix}${primary} // empty" 2>/dev/null || true)" + if [ -n "$value" ] && [ "$value" != "null" ]; then + printf '%s' "$value" + return 0 + fi + + for alt in "$@"; do + value="$(printf '%s' "$response" | jq -r "${prefix}${alt} // empty" 2>/dev/null || true)" + if [ -n "$value" ] && [ "$value" != "null" ]; then + log "WARN: scraped field from fallback name '$alt'" + printf '%s' "$value" + return 0 + fi + done + + value="$(printf '%s' "$response" | grep -oE "\"${primary}\":\"[^\"]+\"" \ + | head -1 | sed -E "s/^\"${primary}\":\"([^\"]+)\"/\1/")" + if [ -n "$value" ]; then + log "WARN: scraped field via regex escape hatch (jq path failed for '$primary')" + printf '%s' "$value" + return 0 + fi + + return 1 +} + +# ---------- preflight ---------- +command -v docker >/dev/null 2>&1 || die "docker binary not found on host PATH" +command -v jq >/dev/null 2>&1 || die "jq is required to parse API responses" + +if [ "$DRY_RUN" != "1" ]; then + docker inspect "$FORKD_CONTAINER" >/dev/null 2>&1 \ + || die "docker container '$FORKD_CONTAINER' is not present" +fi + +# ============================================================================= +# PART 1 — the control experiment +# ============================================================================= +SANDBOX_ID="" +# Invoked via the EXIT trap; SC2329 is a false positive here. +# shellcheck disable=SC2329 +teardown_sandbox() { + if [ -z "$SANDBOX_ID" ]; then + log "WARN: EXIT trap teardown: SANDBOX_ID is empty — VM may have leaked; check forkd sandboxes manually and delete orphans" + return 0 + fi + log "teardown: DELETE /v1/sandboxes/$SANDBOX_ID" + if [ "$DRY_RUN" = "1" ]; then + log "DRY-RUN: would DELETE $FORKD_API_BASE/v1/sandboxes/$SANDBOX_ID" + return 0 + fi + if ! in_container_curl DELETE "/v1/sandboxes/$SANDBOX_ID" ""; then + log "WARN: teardown DELETE failed for $SANDBOX_ID (gate will reap)" + fi +} +trap 'teardown_sandbox' EXIT + +log "control: tag=$CONTROL_TAG cmd=[echo,hi] timeout_secs=$CONTROL_TIMEOUT_SECS" +create_body="$(printf '{"snapshot_tag":"%s"}' "$CONTROL_TAG")" + +if [ "$DRY_RUN" = "1" ]; then + log "DRY-RUN: would POST $FORKD_API_BASE/v1/sandboxes body=$create_body" + log "DRY-RUN: would exec echo/hi, time it, then DELETE" + [ "$DIAGNOSE" = "1" ] && log "DRY-RUN: would also run Part 2 diagnostics" + echo "CONTROL-RESULT: DRY-RUN" + exit 0 +fi + +create_response="$(in_container_curl POST "/v1/sandboxes" "$create_body")" +log "create response (raw): $create_response" + +# Scrape id BEFORE any die() — same load-bearing ordering as +# gate-health-probe.sh's fix for the 2026-07-25 leak (sb-6a64b43f-0031 +# stayed alive when the old code parsed after a check that could die() +# first). The EXIT trap must always have something to clean up. +SANDBOX_ID="$(scrape_field "$create_response" ".[0]." "$SANDBOX_ID_FIELD" "${SANDBOX_ID_FALLBACKS[@]}")" || SANDBOX_ID="" +if [ -n "$SANDBOX_ID" ]; then + log "sandbox id: $SANDBOX_ID" +else + log "WARN: could not extract sandbox id from create response — possible VM leak" + echo "CONTROL-RESULT: OTHER create_response_unparseable" + die "create response did not yield a sandbox id: $create_response" +fi + +exec_body='{"args":["echo","hi"],"timeout_secs":'"$CONTROL_TIMEOUT_SECS"'}' +log "exec: POST /v1/sandboxes/$SANDBOX_ID/exec body=$exec_body" + +start_ts="$(now_epoch)" +set +e +exec_response="$(in_container_curl POST "/v1/sandboxes/$SANDBOX_ID/exec" "$exec_body")" +exec_status=$? +set -e +end_ts="$(now_epoch)" +elapsed="$(awk -v a="$start_ts" -v b="$end_ts" 'BEGIN{printf "%.3f", (b-a)}' 2>/dev/null || echo "unknown")" + +log "exec response (raw): $exec_response" +echo "EXEC-ELAPSED: ${elapsed}" + +if [ $exec_status -ne 0 ]; then + echo "CONTROL-RESULT: OTHER in_container_curl_failed_status=$exec_status" + die "docker exec / curl transport itself failed (status=$exec_status), not a controller response" +fi + +if printf '%s' "$exec_response" | grep -qiE 'resource temporarily unavailable|os error 11|eagain|ewouldblock'; then + echo "CONTROL-RESULT: EAGAIN" + log "control EAGAIN'd on a trivial echo — fault is deterministic/global, not workload-dependent" + log "elapsed=${elapsed}s — sub-second implies a non-blocking fd with no retry loop; ~120s implies a proxy timeout (see FORKD_SHIM_FORWARD_TIMEOUT_S check under --diagnose)" + exit_after_control=1 +else + exit_code="$(scrape_field "$exec_response" "." "$EXIT_CODE_FIELD" "${EXIT_CODE_FALLBACKS[@]}")" || exit_code="" + if [ "$exit_code" = "0" ]; then + echo "CONTROL-RESULT: PASS" + log "control PASSED — the workload-dependent hypothesis family survives; try the slow/large-output follow-up next" + exit_after_control=0 + else + echo "CONTROL-RESULT: OTHER unexpected_exit_code=${exit_code:-} response=$exec_response" + exit_after_control=1 + fi +fi + +# ============================================================================= +# PART 2 — read-only diagnostic capture (opt-in: --diagnose) +# ============================================================================= +if [ "$DIAGNOSE" = "1" ]; then + log "---- Part 2: read-only diagnostics ----" + + if systemctl is-active forkd-ec.service >/dev/null 2>&1; then + check forkd-ec.service OK "$(systemctl is-active forkd-ec.service 2>&1) / $(systemctl is-enabled forkd-ec.service 2>&1)" + else + check forkd-ec.service FAIL "not active: $(systemctl is-active forkd-ec.service 2>&1 || true)" + fi + journal_tail="$(journalctl -u forkd-ec.service -n 20 --no-pager 2>&1 | tail -20 | tr '\n' '|' || echo unavailable)" + check forkd-ec.service-journal INFO "$journal_tail" + + snapshots_response="$(in_container_curl GET "/v1/snapshots" "" 2>/dev/null || echo "")" + if [ -n "$snapshots_response" ]; then + check registered-snapshots OK "$snapshots_response" + else + check registered-snapshots UNKNOWN "GET /v1/snapshots returned nothing — controller may not expose this route, or a restart lost in-memory state" + fi + + sandboxes_response="$(in_container_curl GET "/v1/sandboxes" "" 2>/dev/null || echo "")" + orphan_count="$(printf '%s' "$sandboxes_response" | jq 'length' 2>/dev/null || echo unknown)" + check orphan-sandboxes "$([ "$orphan_count" = "0" ] && echo OK || echo FAIL)" "count=$orphan_count response=$sandboxes_response" + + fd_limit="$(docker exec "$FORKD_CONTAINER" sh -c 'ulimit -n' 2>/dev/null || echo unknown)" + check fd-limit INFO "ulimit -n (in container) = $fd_limit" + + pids_current="$(docker exec "$FORKD_CONTAINER" sh -c 'cat /sys/fs/cgroup/pids.current 2>/dev/null' 2>/dev/null || echo unknown)" + pids_max="$(docker exec "$FORKD_CONTAINER" sh -c 'cat /sys/fs/cgroup/pids.max 2>/dev/null' 2>/dev/null || echo unknown)" + check pids-cgroup INFO "current=$pids_current max=$pids_max" + + ps_count="$(ps -eLf 2>/dev/null | wc -l | tr -d ' ' || echo unknown)" + check host-thread-count INFO "$ps_count" + + oom_lines="$(dmesg -T 2>/dev/null | grep -iE 'oom|out of memory|firecracker|cannot allocate' | tail -10 || true)" + if [ -n "$oom_lines" ]; then + check dmesg-oom FAIL "$(printf '%s' "$oom_lines" | tr '\n' '|')" + else + check dmesg-oom OK "no OOM/firecracker-alloc signatures in recent dmesg" + fi + + disk_line="$(df -h / 2>/dev/null | tail -1 || echo unknown)" + check disk-space INFO "$disk_line" + + mem_line="$(free -g 2>/dev/null | tail -2 | tr '\n' '|' || echo unknown)" + check memory INFO "$mem_line" + + netns_count=0 + for _ns in /var/run/netns/forkd-child-*; do + [ -e "$_ns" ] && netns_count=$((netns_count + 1)) + done + check per-child-netns "$([ "$netns_count" -ge 1 ] && echo OK || echo FAIL)" "forkd-child netns count=$netns_count" + + if pgrep -f "fabro-run/forkd-shim.py" >/dev/null 2>&1; then + shim_pid="$(pgrep -f "fabro-run/forkd-shim.py" | head -1)" + shim_env="$(tr '\0' ' ' < "/proc/$shim_pid/environ" 2>/dev/null | grep -o 'FORKD_SHIM_FORWARD_TIMEOUT_S=[0-9]*' || echo "unset (defaults to 120)")" + check forkd-shim-proxy OK "pid=$shim_pid $shim_env" + else + check forkd-shim-proxy UNKNOWN "no fabro-run/forkd-shim.py process found on host PATH via pgrep — may run under a different invocation or on a different host" + fi + + log "---- end diagnostics ----" +fi + +exit "$exit_after_control"