guard: what it cannot see and what the shell can still remove; alarm: a transient failure that keeps happening

.agents/notes/implemented/2026-10-27-what-the-guard-cannot-see-and-a-failure-that-keeps-happening.md

@@ -0,0 +1,48 @@
+---
+title: What the guard cannot see, what the shell can still remove, and a failure that keeps happening
+status: implemented
+date: 2026-10-27
+scope: hookprobe
+---
+
+## Decision
+
+A second deployment running copies of these files added a deny list for its
+unattended agent and a page for failures that repeat. Run against this repo's
+own gate and failure path, three of its lessons held here too.
+
+1. **A script the guard never sees is refused, under every posture.** A pipe
+   into a shell (`curl … | sh`, `wget -qO- … | bash`, `| sudo bash`) and a
+   fetched script handed to one (`bash -c "$(curl …)"`, `sh <(curl …)`,
+   `source <(…)`, `eval "$(…)"`) were allowed in both postures: the guard read
+   `sh`, and every verb in the script stayed out of reach of every rule.
+2. **Removing or relinking a protected path from the shell is a write.**
+   `rm results/x.json`, `rm -f CLAUDE.md`, `find results -delete` and
+   `ln -sf /tmp/x .claude/skills/x` passed while `cp` and `mv` into the same
+   paths were refused — the case files the next run reads could be deleted, and
+   a skill planted whose content lives wherever the run can write.
+3. **A transient failure that keeps happening pages.** Rate limits and 5xx are
+   retried and, when they fail again, filed as one investigation's bad luck;
+   none of them is node-wide, so none paged. The third consecutive failure of
+   one kind now pages once per kind per six hours, and a run that lands says
+   so. The other deployment lost three days of rounds to a rate limit that way.
+
+Also: `aws logs start-query` (and CloudTrail Lake's) is a read in the readonly
+posture, named exactly so Athena's `start-query-execution` stays refused.
+
+## Why
+
+Each was measured, not reasoned: the commands from the other deployment's list
+went through this gate in both postures and the gaps were the ones above; the
+failure path had no count of anything but node-wide shapes.
+
+## Consequences
+
+- An installer's `curl … | sh` is refused even on a writing node; fetch it to a
+  file, read it, run what it holds. A script saved and then run is two steps and
+  still passes — the guard's stated limit, not a new one.
+- `ln -s <protected> /tmp/link` is refused too, which over-blocks a link that
+  only points at a protected file; writing through such a link was already
+  refused, and the guard errs toward over-blocking.
+- A streak lives in memory: a restart starts the count again.
+- Rollback: nothing persisted changes shape.

docs/containment.md

@@ -146,7 +146,7 @@ Every row here is a claim, and a claim is worth what its test is worth. `scripts
 | **Chained audit** — every flight-recorder line carries the hash of the one before it; `verify_chain` names the first line that stops adding up, and `/v1/selftest` walks it | A record quietly tidied after the fact. The audit is what an investigation is reconstructed from and what a compliance reader asks about, and an append-only file cannot tell a line rewritten later from one written that way. Edits, deletions and reorderings are all caught, because each line names its predecessor rather than only hashing itself | Somebody who can rewrite the audit here AND reach the pipe's ledger. Rewriting `.chain` alone is not enough since 2026-09-10: every report carries `meta.audit_head` off-box, so a rebuilt chain is locally consistent and still fails `chain_anchored` against a head another service recorded hours earlier. Two disks and two services is the bound, not proof. Lines from before the chain existed are counted, never treated as a break — an alarm that fired on all existing history would be ignored by day two | `hookprobe/gate.py` (`append_audit`, `verify_chain`), `selftest.py` |
 | **What an approved procedure may run, and with what** — `HOOKPROBE_REMEDIATION_ALLOWLIST` (full-match regexes, one per line, hot-read; no file means nothing runs), checked over the WHOLE procedure at approval and again before each step; `execution_env()` passes `PATH`/`HOME`/locale, the cloud credentials an operator mounted for the purpose (`AWS_*`, `KUBE*`, `GOOGLE_*`, `AZURE_*`) and the egress proxy variables, and nothing else; a WRITING node names what its credential may do (`HOOKPROBE_BLAST_RADIUS`, one measured line per entry as `/v1/posture` reports it) and under `enforce` a credential that can do more refuses to start, naming the excess | The one path that executes anything running a command nobody wrote down, or a half-executed 1-2-3 when the file is tightened mid-incident; a procedure reading the family's HMAC keys, the Lark app secret or the provider credential out of its own environment — until 2026-09-10 `create_subprocess_exec` passed no `env` at all; a writing node quietly growing, which under `danger-only` had nothing to be compared against for its whole life | A pattern an operator wrote too widely — the file is the decision, this only holds it, and the model proposing anything stays free. A permitted command doing something with the credentials it IS given. Anything inside the declared radius, a grant that was unwise (`iam:*` declared is `iam:*`), and a credential widened while the node runs, which is caught at the next boot and by the hourly selftest, not at the moment it changes | `hookprobe/remediation.py` (`deny_reason`, `approve`, `execute`, `execution_env`), `posture.py` (`declared_radius`, `beyond`, `verdict`) |
 | **Approval window, freshness cursor and the version pressed** — a proposal is unrunnable 24h after it was made (the span the pipe gives a card's token), and it records what was true about its condition when the steps were chosen — has it ENDED, has the investigation taken another turn; if either has moved by the time somebody presses, the procedure is retired instead of run, the button is not drawn, and the refusal returns to the chat as a report. And a press names WHICH proposal it approves: the digest of its steps, run, clock and cursor as they were shown — inside the card's signed token, or as the console listed them — and a press naming none or another version is refused with the row untouched; the approved digest goes on the row | Commands chosen from evidence that has since moved being run by a click on a week-old card; approving a fix for a condition that already cleared, or running steps lifted from a report the investigation has since superseded — both reachable at any age under a clock alone, because a clock assumes the world moves at a rate; and commands nobody saw running under an approval of others — a proposal rewritten under the same id between the reading and the press | An approval at 23 hours on evidence that went stale in one is what the cursor is for; what neither sees is anything that happened outside this pipe. Nothing here reaches out to look, and it must not — the production investigator holds no credentials for the systems it writes procedures about — so somebody fixing the target by hand without saying so in chat is invisible | `hookprobe/remediation.py` (`APPROVAL_WINDOW_SECONDS`, `stale`, `cursor`, `moved`, `content_hash`, `Changed`), `actions.py`, `events.py`, `service.py` (`report_superseded`) |
-| **Bash guard and input guard** — under `readonly`, mutating verbs of known CLIs with `aws` inverted to deny-unless-read, under `danger-only` only what no credential scope can undo; and writes to the files that steer the next run (`CLAUDE.md` and `AGENTS.md`, `.claude/`, the system prompt, `audit/`, `telemetry/`, `remediation/`, `actions/`, and the `results/` case files the prompt tells every run to read first) refused in the three shapes a write arrives in — an edit tool's path argument, the paths inside a codex `apply_patch` envelope, and a shell command that redirects, `tee`s, `sed -i`s, `cp`s or `mv`s into one of them | An over-eager model running `kubectl delete` or `aws s3 rm`; a run installing its own runbook as a persistence vector. All three write shapes were measured getting through before they were closed: `apply_patch` because the guard read argument keys and codex puts its paths inside a patch string, `AGENTS.md` because the list only knew Claude's name for the steering file, and the shell redirect on **every** runtime under **both** postures | An adversary: the rules need a binary and a verb in one pipeline segment, and separating them gets through; a redirect assembled at runtime out of two variables, or any of the endless other ways a shell can name a file. These catch the obvious forms — the over-eager model, which is the case that happens. Writes anywhere else in the workspace are allowed on purpose. **The read-only credential is what holds** | `hookprobe/guard.py`, `inputs.py`, `gate.py` (`patch_targets`, `shell_write_target`) |
+| **Bash guard and input guard** — under `readonly`, mutating verbs of known CLIs with `aws` inverted to deny-unless-read, under `danger-only` only what no credential scope can undo; and writes to the files that steer the next run (`CLAUDE.md` and `AGENTS.md`, `.claude/`, the system prompt, `audit/`, `telemetry/`, `remediation/`, `actions/`, and the `results/` case files the prompt tells every run to read first) refused in the three shapes a write arrives in — an edit tool's path argument, the paths inside a codex `apply_patch` envelope, and a shell command that redirects, `tee`s, `sed -i`s, `cp`s, `mv`s, removes (`rm`, `find -delete`) or links (`ln`) into one of them; and under every posture, a script piped or handed to a shell (`curl … \| sh`), whose verbs no rule ever reads | An over-eager model running `kubectl delete` or `aws s3 rm`, or an installer's `curl … \| sh`; a run installing its own runbook as a persistence vector, or deleting the case files the next run reads. All three write shapes were measured getting through before they were closed: `apply_patch` because the guard read argument keys and codex puts its paths inside a patch string, `AGENTS.md` because the list only knew Claude's name for the steering file, and the shell redirect on **every** runtime under **both** postures; removing and symlinking from the shell were measured getting through on 2026-10-27 | An adversary: the rules need a binary and a verb in one pipeline segment, and separating them gets through; a redirect assembled at runtime out of two variables, or any of the endless other ways a shell can name a file; a script saved to a file and then run. These catch the obvious forms — the over-eager model, which is the case that happens. Writes anywhere else in the workspace are allowed on purpose. **The read-only credential is what holds** | `hookprobe/guard.py`, `inputs.py`, `gate.py` (`patch_targets`, `shell_write_target`) |
 | **Memory shape check** — a suggested fact whose SHAPE could act waits for a human | An instruction smuggled through alert text into `CLAUDE.md` | A benign-shaped line that is simply wrong | `hookprobe/suggestions.py` |
 | **High-risk allowlist and target cooldown** — a step the report marked `risk: high` must also full-match a pattern in `HOOKPROBE_REMEDIATION_HIGH_RISK_ALLOWLIST`, deny-by-default in the same direction, both lists re-read before each step and one uncovered high step refusing the whole procedure at the click; after a procedure acts, a second one sharing its declared `target` **or** its literal command is refused for 15 minutes (`HOOKPROBE_REMEDIATION_COOLDOWN_SECONDS`, 0 disables), and one already `running` holds its target with no window at all | Arming remediation from also arming its worst half by the same gesture — `kubectl rollout restart .*` written to cover restarts has not licensed the steps the model itself calls dangerous; a fix and the rollback of that fix approved a minute apart, and two sequences interleaving their steps against one host. Held, not retired: the row stays `proposed` and the same press works after the window | **Keys the model writes.** A dangerous command the report labelled `low` never meets the strict list — the risk label is the MODEL's, so this can only ADD a requirement, and a mislabelled step is bounded by the ordinary allowlist exactly as before. A step that varies its command by a flag AND labels its target differently is not cooled; two keys because production named one target three ways while running character-identical commands, so the command half is the one that holds. Both count only what THIS node ran — somebody fixing the same box by hand is invisible, the same blind spot the freshness cursor has | `hookprobe/remediation.py` (`step_deny_reason`, `cooldown_keys`, `cooling`), `actions.py`, `service.py` (`report_cooling`) |
 | **Refused at config load** — `on_error` per stage (`pass` or `drop`, chosen per stage, never defaulted globally); an inline decider's timeout capped at 10s; no unreachable door, starved exit, or wildcard a return door can reach; a write-back channel may say `POST`, `PUT` or `PATCH` and nothing else | A stage's failure silently changing policy; an inline decider holding the sender's connection until it retries and duplicates the alert; a config change silently feeding a brain its own output; a channel configured with `DELETE`, or a `GET` whose side effect fires when a chat client fetches a link preview — the trap `/card-action` refuses to fall into | Choosing wrong inside what is legal: an explicit fail-open, a node that answers in 9s and is still the wrong place for the work, a graph that is legal and still wrong, a `POST` that does damage at the receiver — the pipe checks the verb it is asked for, never what the far side does with it | `hookrelay/processors.py` (`MAX_INLINE_TIMEOUT_SECONDS`), `config.py` (`_WRITE_METHODS`), `topology.py`, `scripts/assert_topology.py` |

hookprobe/README.md

@@ -224,7 +224,8 @@ always reach. Two things about the word *default*, before the layers:
 2. **Bash guard** — under `readonly`, a PreToolUse hook denies mutating verbs
    of kubectl, helm, docker/podman, systemctl, terraform, plus ssh/scp and
    `git push`; under `danger-only` it keeps only the estate-wide destroyers
-   listed above. It errs toward over-blocking. HTTP verbs are not policed (query APIs POST), and
+   listed above. Under both, a script piped or handed to a shell (`curl … | sh`)
+   is refused, because its verbs never reach a rule. It errs toward over-blocking. HTTP verbs are not policed (query APIs POST), and
    cloud CLIs are too many to enumerate — scope their credentials instead.
 3. **Container** — non-root, disposable, nothing precious inside, no container
    runtime and no socket (a run cannot start a container, and cannot build a

hookprobe/docs/reference.md

@@ -66,7 +66,7 @@ the next run, and `scripts/gen_reference.py --check` will say so.
 | `HOOKPROBE_PI_PROVIDER` | *(empty)* | Which provider in that configuration serves this node's model |
 | `HOOKPROBE_SKILLS` | *(empty)* | Directory of skills the engine may load |
 | `HOOKPROBE_RETURN_URL` | *(empty)* | The one door findings go back to |
-| `HOOKPROBE_ALARM_URL` | *(empty)* | Direct self-alarm for report returns that exhaust their retries: the pipe is the broken link at that moment, so this posts straight to a bot/collector URL, touching nothing on the… |
+| `HOOKPROBE_ALARM_URL` | *(empty)* | Direct self-alarm, around the pipe: report returns that exhaust their retries, a run failing in a way every run here would, the hourly selftest, and a transient failure that recurs… |
 | `HOOKPROBE_SETTING_SOURCES` | `project` | Skill layers |
 | `HOOKPROBE_CODEX_BINARY` | *(empty)* | Codex only |
 | `HOOKPROBE_CODEX_PYTHON` | *(empty)* | The interpreter the Codex adapter spawns its tool gate with |

hookprobe/hookprobe/engine.py

@@ -640,10 +640,45 @@ def transient(error: str) -> bool:
     then re-investigating was spending money on a question whose answer had
     stopped mattering. The moment to try again is the moment, not four days on.
     """
+    return transient_kind(error) is not None
+
+
+# The same transient shape, named, so a run of them can be counted as ONE thing:
+# a provider says "429" one time and "rate limit exceeded" the next.
+_TRANSIENT_KINDS: tuple[tuple[tuple[str, ...], str], ...] = (
+    (("429", "rate limit"), "rate limited"),
+    (("overloaded",), "overloaded"),
+    (
+        (
+            "500",
+            "502",
+            "503",
+            "504",
+            "520",
+            "521",
+            "522",
+            "524",
+            "internal server error",
+            "service unavailable",
+            "temporarily unavailable",
+        ),
+        "a 5xx from the provider",
+    ),
+    (("408",), "a request timeout"),
+    (("connection reset", "connection error"), "the connection dropped"),
+)
+
+
+def transient_kind(error: str) -> str | None:
+    """Which transient shape this failure is, or None when it is not one."""
     lowered = " ".join(str(error or "").split()).lower()
     if not lowered or any(marker in lowered for marker in _PERMANENT_MARKERS):
-        return False
-    return any(marker in lowered for marker in _TRANSIENT_MARKERS)
+        return None
+    if not any(marker in lowered for marker in _TRANSIENT_MARKERS):
+        return None
+    return next(
+        (kind for markers, kind in _TRANSIENT_KINDS if any(marker in lowered for marker in markers)), "transient"
+    )
 
 
 # Failures that are about the NODE, not about this investigation: whatever the

hookprobe/hookprobe/gate.py

@@ -181,9 +181,26 @@ def shell_write_target(command: str, roots: tuple[Path, ...], workdir: Path | No
 
 
 def _writes_a_named_file(words: list[str]) -> bool:
-    """Commands that write a path given as an argument rather than a redirect."""
+    """Commands that write, remove or relink a path given as an argument.
+
+    Removing and relinking count (2026-10-27). `rm results/x.json` deleted the
+    evidence the next run is told to read, and `ln -sf /tmp/x .claude/skills/x`
+    planted a skill whose content then lives wherever the run can write — both
+    passed while `cp` and `mv` into the same paths were refused.
+    """
     head = Path(words[0]).name
-    if head in ("cp", "mv", "install", "dd", "truncate", "tee"):
+    if head in ("cp", "mv", "install", "dd", "truncate", "tee", "rm", "rmdir", "unlink", "shred", "ln", "touch"):
+        return True
+    if head == "find" and (
+        "-delete" in words
+        or (
+            any(word in ("-exec", "-execdir", "-ok", "-okdir") for word in words)
+            and any(
+                Path(word).name in ("rm", "mv", "cp", "ln", "unlink", "shred", "truncate", "tee", "dd")
+                for word in words
+            )
+        )
+    ):
         return True
     # `sed` only writes with -i, and BSD's takes a separate suffix argument, so
     # the target is not at a fixed position. Every argument is a candidate.