ops(dellsrv): control experiment for the exec EAGAIN outage

.briefs/T1-dellsrv-runbook.md

@@ -0,0 +1,102 @@
+# TASK T1 — consolidated dellsrv T3 runbook + read-only verification checklist
+
+**Docs deliverable. NO live actions.** You are writing the single consolidated T3
+execution runbook for the fabro/dellsrv completion campaign, for operator Val to run
+BY HAND in the morning.
+
+You MUST NOT ssh to dellsrv, run systemctl, run forkd, or touch any live service. The
+sandbox cannot reach `dellsrv.zp.digital` anyway — DNS denial there is the egress
+boundary working as designed. Do NOT attempt any workaround (no IP literals, no
+tunneling, no alternate hostnames). This is a pure authoring task.
+
+## Deliverable
+
+New file `docs/internal/dellsrv-completion-runbook.md` in the fabro repo. Branch off
+`origin/main`, commit, push, open a **DRAFT** PR against `zenprocess/fabro`.
+
+## Sources — read these first
+
+1. `~/.ao/data/aofactory/FABRO-COMPLETION-RUNBOOK.md` — the operator mandate (9 numbered
+   items). Authority on WHAT must happen.
+2. `scripts/ops/dellsrv-forkd-supervision/README.md` on `origin/main` — the already-merged
+   gate-health-probe runbook (PR #17). Its deploy section is the model for tone and
+   precision. Do NOT duplicate its content; REFERENCE it.
+3. `git log origin/main` plus `gh pr view 15`, `16`, `17`, `19` `--repo zenprocess/fabro`
+   — the commit/PR trail showing what already landed.
+
+## Content — exactly two parts
+
+### PART A — READ-ONLY VERIFICATION CHECKLIST (put this FIRST; it is the more important half)
+
+Numbered, copy-pasteable, strictly NON-MUTATING commands Val runs on dellsrv to establish
+ground truth. Each entry needs: (a) the exact command, (b) what output means DONE, (c) what
+output means NOT DONE. Cover at minimum:
+
+- **forkd-controller binary**: exists at `/usr/local/bin/forkd-controller`, `--version`
+  reports v0.5.2, and is NOT a deleted inode (compare against `/proc/<pid>/exe` of the
+  running controller).
+- **Which unit supervises forkd**: `systemctl cat forkd-ec.service`, `is-enabled`,
+  `is-active`, and whether `forkd-ec-boot-dellsrv.sh` references the `/usr/local/bin`
+  binary or something ephemeral.
+- **Golden option-2 tags**: list snapshots INSIDE the forkd container; confirm BOTH
+  `zen-gate-base` AND `zen-gate-big` exist; confirm each `memory.bin` size
+  (`zen-gate-base` 1024 MiB = 1073741824 bytes, `zen-gate-big` 4096 MiB = 4294967296
+  bytes). The pre-existing baseline was **536870912** bytes (512 MB) — if you see that,
+  option-2 did NOT happen.
+- **gate-health-probe**: `systemctl list-timers gate-health-probe.timer`, `is-enabled`,
+  and `journalctl -t gate-health-probe --since -24h | grep -c FORKD-GATE-ALERT`.
+- **Posting/poller state**: how to determine whether gate verdict posting is ON or OFF.
+  **CRITICAL FACT you must encode**: the fabro gate posts to the **Forgejo** forge
+  (contexts `fabro/qa-pipeline`, `fabro/qwen-review`), NOT github.com. `gh api .../statuses`
+  is ALWAYS empty and is NOT a valid health check. Say so explicitly so nobody re-derives
+  that wrong conclusion.
+
+### PART B — MUTATING STEPS, gated on Part A results
+
+Each remaining item in dependency order, each with a stated PRECONDITION (which Part A
+check must have FAILED for this step to be needed) and a ROLLBACK. Cover:
+
+- **Golden tag registration** via
+  `forkd snapshot --tag <T> --kernel <K> --rootfs <ext4> --tap <tap> --boot-wait-secs <N> --mem-size-mib <MIB>`.
+  Option 2 = `zen-gate-base` at 1024 and `zen-gate-big` at 4096, off the SAME existing
+  20 GB golden rootfs — this is a **RE-REGISTER, not a re-bake**. Rollback = deregister
+  the new tag.
+- **gate-health-probe timer enable** — commands already exist in the PR #17 README;
+  reference them, do not copy.
+- **Re-enabling posting + poller.**
+
+## Honesty requirements (non-negotiable — the whole point of the document)
+
+- Anywhere live state is UNKNOWN to us, write it as **UNKNOWN** with the Part A check that
+  resolves it. Do NOT assert live dellsrv state as fact. Everything we have is inferred
+  from commit messages, and one commit (`6c3ed8bac`) merely *claims* "golden option-2 is
+  live on dellsrv" without proof.
+- Explicitly record that **PR #16** (a `forkd-controller.service` supervision unit) was
+  CLOSED-not-merged because a new unit would DUPLICATE `forkd-ec.service`'s ownership of
+  the boot script. Any reader tempted to re-add such a unit must hit that warning. The
+  durability fix is canonicalizing the recovered v0.5.2 binary that forkd-ec already
+  launches, NOT a competing unit.
+- No invented paths or flag names. If you do not know a value (kernel path, tap name,
+  boot-wait-secs), write `<FILL FROM PART A step N>` rather than guessing.
+
+## Acceptance command (must pass; paste the real output in your report)
+
+```
+test -f docs/internal/dellsrv-completion-runbook.md \
+  && grep -qi "forgejo" docs/internal/dellsrv-completion-runbook.md \
+  && grep -q "UNKNOWN" docs/internal/dellsrv-completion-runbook.md \
+  && grep -q "536870912" docs/internal/dellsrv-completion-runbook.md \
+  && grep -qi "PR #16" docs/internal/dellsrv-completion-runbook.md \
+  && echo ACCEPT
+```
+
+## Adversarial check (also required)
+
+Pick the two most load-bearing claims in your Part A checklist and try to prove them
+WRONG from the repo/PR trail. If a check would pass even when the thing it checks is
+broken, it is a worthless check — fix it and say so. Report which checks you strengthened.
+
+## Report back
+
+File path, draft PR number, real acceptance output, adversarial findings. Do NOT mark
+done without the pasted acceptance output.

.briefs/T2-state-capture.md

@@ -0,0 +1,94 @@
+# TASK T2 — read-only dellsrv state-capture script (code-side only)
+
+**NO live actions.** You are authoring a script that the operator will run on dellsrv
+by hand. You must NOT ssh to dellsrv, run systemctl, or touch any live service. The
+sandbox cannot reach `dellsrv.zp.digital` — DNS denial there is the egress boundary
+working as designed. Do NOT attempt any workaround. Author + test locally only.
+
+## Why this exists
+
+We cannot see live dellsrv state. Four campaign items (forkd-controller durability,
+golden option-2 tags, gate-health-probe deploy, posting/poller state) are each blocked
+on ground truth we don't have. This script captures ALL of it in one paste-able run so
+the operator spends 30 seconds, not 20 minutes, and so the output is machine-comparable
+next time.
+
+## Deliverable
+
+`scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh` plus a new section in
+that directory's existing `README.md`. Branch off `origin/main`, commit, push, open a
+**DRAFT** PR against `zenprocess/fabro`.
+
+## Hard requirements
+
+- **STRICTLY READ-ONLY.** No `systemctl start|stop|restart|enable|disable`, no
+  `forkd snapshot` registration, no `docker restart`, no file writes outside a
+  `mktemp` output file, no DELETE/POST to the controller. A reviewer must be able to
+  confirm read-only-ness by reading the script once. Put a comment block at the top
+  stating this contract, and make it true.
+- **Never print secret VALUES.** The forkd token lives at `/etc/forkd-token` INSIDE the
+  `forkd` docker container. If you need it for a GET, follow the existing pattern in
+  `gate-health-probe.sh` (`docker exec -i forkd sh` with a heredoc whose first action is
+  `TOKEN=$(cat /etc/forkd-token)`) so the value never crosses the host argv or the
+  process table. Report only presence/absence, never content.
+- **Every probe must be independently failable.** Wrap each check so one missing binary
+  or one absent unit does not abort the rest — the operator needs the WHOLE picture from
+  one run. Use `|| true` guards deliberately, but never in a way that turns a real
+  failure into a silent pass; each check must print an explicit `OK` / `MISSING` /
+  `UNKNOWN` verdict line.
+- **Output must be diffable**: a stable, greppable line format, one fact per line, e.g.
+  `CHECK <name> <OK|FAIL|UNKNOWN> <detail>`. Print a summary count at the end.
+
+## Checks the script must capture
+
+1. `forkd-controller` binary: path, `--version`, sha256, and whether the RUNNING
+   controller's `/proc/<pid>/exe` is a deleted inode or resolves to the on-disk file.
+   (The 2026-07-25 emergency was exactly a deleted-inode process; this check is the
+   durability question.)
+2. `forkd-ec.service`: `systemctl cat`, `is-enabled`, `is-active`, and whether the boot
+   script it owns references `/usr/local/bin/forkd-controller` or an ephemeral path.
+3. Golden snapshot tags inside the forkd container: which tags exist, and the byte size
+   of each tag's `memory.bin`. Known baseline to flag loudly: **536870912** bytes
+   (512 MB) means golden option-2 did NOT happen. Targets are 1073741824 (1024 MiB,
+   `zen-gate-base`) and 4294967296 (4096 MiB, `zen-gate-big`).
+4. `gate-health-probe`: installed? `is-enabled`? timer scheduled? count of
+   `FORKD-GATE-ALERT` and `FORKD-GATE-HEAL` lines in the last 24h of
+   `journalctl -t gate-health-probe`.
+5. Per-child netns presence (`/var/run/netns/forkd-child-*`) — the 2026-07-22 three-day
+   silent outage was these dying.
+6. Posting/poller: whether the verdict poster is running, and where it posts.
+   **CRITICAL FACT to encode in a comment**: the gate posts to the **Forgejo** forge
+   (contexts `fabro/qa-pipeline`, `fabro/qwen-review`), NOT github.com. A `gh api
+   .../statuses` check is always empty and is NOT valid evidence of gate health. Do not
+   write a github.com-based check.
+
+## Acceptance command (must pass; paste the real output in your report)
+
+```
+bash -n scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \
+  && shellcheck -S warning scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \
+  && ! grep -nE '(systemctl (start|stop|restart|enable|disable))|(docker (restart|rm|stop))|(forkd snapshot )|(-X (POST|DELETE|PUT))' scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \
+  && grep -q 536870912 scripts/ops/dellsrv-forkd-supervision/dellsrv-state-capture.sh \
+  && echo ACCEPT
+```
+
+(The negated grep is the read-only proof — it must find NOTHING. If shellcheck is not
+installed, install it or state clearly that you could not run it; do not silently drop
+it from the acceptance chain.)
+
+## Adversarial check (required — do not skip)
+
+The acceptance command above is a **negative control**: it passes when the script
+contains no mutating verbs. Prove it actually bites. Temporarily insert a line like
+`systemctl restart forkd-ec.service` into a **scratch copy** of the script, re-run the
+acceptance chain, and confirm it FAILS. Then remove it and confirm it passes again.
+Report both outputs. A guard you never saw fail is not a guard.
+
+Also: pick the check you judge weakest (most likely to report OK while the underlying
+thing is broken) and either strengthen it or state plainly why it cannot be strengthened
+without live access.
+
+## Report back
+
+File path, draft PR number, real acceptance output, and the negative-control proof
+(the FAIL output and the restored PASS output). Do NOT mark done without both pasted.

.briefs/T3-referee-run-registration.md

@@ -0,0 +1,108 @@
+# TASK T3 — register referee scores as fabro server runs (code-side only, no live calls)
+
+Rust implementation task in `lib/components/fabro-referee`. Branch off `origin/main`,
+commit, push, open a **DRAFT** PR against `zenprocess/fabro`.
+
+## Why
+
+Campaign item 8: `fabro.zp.digital/runs` shows nothing since Jul 10 because gate execs go
+through forkd and referee runs write only the Mac JSONL sink — neither becomes a fabro
+server run. PR #19 already merged the receiving endpoint
+(`POST /api/v1/runs/registrations`, operationId `registerExternalRun`). The missing half
+is the CALLER.
+
+## Read first
+
+- `docs/public/api-reference/fabro-api.yaml` — the `/api/v1/runs/registrations` path
+  (around line 1316) and the `RunRegistrationRequest` schema (around line 11757). **This
+  spec is the source of truth for the wire contract.** Do not invent field names.
+- `lib/components/fabro-referee/src/emit.rs` — the existing sink emitter. Note its
+  idempotency design (re-running the same `(run_id, route)` overwrites rather than
+  double-appends). Your registration must respect the same replay semantics: the endpoint
+  returns 201 on create and **200 on idempotent re-registration** — treat BOTH as success.
+- `lib/components/fabro-referee/src/types.rs` — `RunRow`, `Verdict`.
+- Repo conventions in `AGENTS.md`/`CLAUDE.md`: Rust import style (types by name, functions
+  via parent module, no glob imports), the API type-ownership rules, and the test-support
+  boundary rules (`#[cfg(any(test, feature = "test-support"))]`, never in default features).
+
+## Scope — what to build
+
+Emit the registration alongside the existing sink write, **opt-in and off by default**:
+
+- Gated behind an explicit env var (e.g. `FABRO_REFEREE_REGISTER_RUNS=1` plus a base-URL
+  var). **Default OFF** — a fresh checkout must not attempt any network call.
+- Map the verdict exactly as the spec documents: `pass` → Succeeded/Completed,
+  `fail` → Failed/WorkflowError, `inconclusive` → Failed/TransientInfra.
+- Populate `origin.details` with real provenance (gate backend, route/tier, commit SHA).
+- **Truthfulness rule from the spec, enforce it**: there is no "register a placeholder"
+  path. If the source cannot produce a real verdict it MUST send `inconclusive` — never
+  fabricate `pass`. Encode this so a future refactor cannot quietly default to `pass`.
+- **Registration failure must never corrupt or block the sink write.** The JSONL sink is
+  the authoritative artifact; a server that is down must degrade to a logged warning, not
+  a lost score and not a crash.
+- The spec rejects a ref NAME where a SHA is required — validate before sending and fail
+  loudly with a clear message rather than sending garbage.
+
+Keep the diff tight: roughly a new `register.rs`, its wiring in `lib.rs`/the emit path,
+and a test file. Do not refactor unrelated referee code.
+
+## Testing — mandatory, and no live network
+
+Use `httpmock` (already a dev-dependency in this workspace — see PR #20's usage) to assert
+against a mock HTTP responder. **No test may contact a real fabro server.** Cover at
+minimum:
+
+1. Request body matches the OpenAPI `RunRegistrationRequest` shape (field names verified
+   against the YAML, not from memory).
+2. All three verdict mappings, including `inconclusive` → TransientInfra.
+3. **200 idempotent re-registration is treated as success**, not as an error.
+4. Server 5xx / connection failure → sink write still succeeds, warning logged, no panic.
+5. Feature is OFF by default: with the env var unset, no HTTP request is made at all
+   (assert the mock received zero hits).
+
+If you add shared test helpers, they go behind the `test-support` feature per the repo's
+test-support boundary rules — never exported from a production module, never in default
+features.
+
+## Acceptance command (must pass; paste the real output in your report)
+
+```
+cargo nextest run -p fabro-referee \
+  && cargo +nightly-2026-04-14 fmt --check -p fabro-referee \
+  && cargo +nightly-2026-04-14 clippy -p fabro-referee --no-deps --tests -- -D warnings \
+  && echo ACCEPT
+```
+
+Note: a full-workspace `cargo build` is hook-blocked on this Mac; crate-scoped commands
+above are the authoritative local gate. Also confirm the root `Cargo.lock` is committed if
+your change adds or changes any dependency — a prior PR in this repo (#20) shipped without
+it and every contributor regenerated it as uncommitted drift. Run `git status` after
+testing and check.
+
+## Adversarial check (required — do not skip)
+
+Run at least **4 mutations**, one at a time, each on a scratch copy, each reverted after,
+and report a table of `mutation → failing test → tests executed`:
+
+1. Change the `inconclusive` mapping to `pass` (this is the campaign's core failure class:
+   misreporting infra as a real verdict — it MUST be caught).
+2. Treat the 200 idempotent response as an error.
+3. Make a registration failure propagate and abort the sink write.
+4. Your own choice — target the test you judge WEAKEST, not an easy one.
+
+**Collapse check**: the executed test count must stay constant across every mutation. A
+mutation that yields `0 passed; 0 failed`, a reduced count, or a compile error only is
+INCONCLUSIVE, not a pass — redo it as a clean edit. Before trusting any "not caught"
+result, `diff` the mutated file to confirm the mutation actually applied (a silently
+non-applied mutation looks exactly like a vacuous test).
+
+If a mutation is NOT caught, that is the finding — add the missing test, re-run the
+identical mutation, and prove it now fails.
+
+## Report back
+
+Draft PR number, real acceptance output, the mutation table with executed counts, and
+confirmation that `git diff` vs origin is clean of mutations afterward. Verify your push
+by checking the REMOTE tip SHA, not by trusting an echoed "pushed" — a `git push origin
+<branch>` can succeed as a no-op while your commit sits on a detached HEAD. Do NOT mark
+done without pasted acceptance output.