Harden recovery, posture checks, and replay handling

.agents/notes/implemented/2026-10-27-recovery-must-not-claim-certainty.md

@@ -0,0 +1,40 @@
+---
+title: Recovery must not claim certainty
+status: implemented
+date: 2026-10-27
+scope: stack
+---
+
+## Decision
+
+Record remediation step starts and results durably, and classify interrupted
+steps as completed, unknown, or not started. Redact a step's whole output, then
+keep its last 10,000 characters (omit it past a megabyte). Reserve signed
+inbound requests with renewable SQLite leases, recording completion only after
+handling. Treat incomplete posture measurements as unverifiable, and ASK the
+instance metadata service, as the AWS CLI does, when nothing names an AWS
+credential: a role it offers is measured; no role, another vendor's service or
+nothing there is no credential. A writing node with a declared radius refuses
+an unverified measurement under enforce.
+
+Amended in review before merge: the first version assumed metadata credentials
+whenever nothing named one, which read every laptop and every probe on an
+internal network as `unverifiable` and stopped a declared writing node from
+starting; it omitted any output past 10,000 bytes, tail included; and it cut
+the pipe's explanatory comments to fit its line ceiling. The pipe's two unused
+signature methods went instead.
+
+## Why
+
+A crash between a side effect and its record cannot establish that it did not
+happen. Concurrent requests must not both enter the pipeline while the first is
+still processing, and a failed permission check cannot prove a declared radius.
+
+## Consequences
+
+- An unknown remediation step requires reconciliation before another attempt.
+- A lease can expire after a crash, allowing retries but not exactly-once external effects.
+- Patterns do not catch every secret, and proposals retain executable commands.
+- Declared writing nodes may need corrected tooling or an explicit metadata setting before starting.
+- The pipe gains a `pending_signatures` table. A rollback to an earlier pipe
+  ignores it: nothing fails to open, and a lease left in it simply expires.

docs/containment.md

@@ -19,9 +19,17 @@ a real model cannot be steered into emitting a harmful one.
 > defaults to `enforce` in the code, so the deploy composes now spell it out rather than leaving the reader to find it;
 > a `danger-only` runner records its blast radius; an identity that cannot be asked is reported *unverifiable*, never confirmed.
 > A credential whose measuring tool is not in the image is *unmeasurable* (2026-10-12): a readonly runner refuses to start
-> under `enforce`, a writing runner starts and pages, and the hourly selftest fails until the tool is back — ten days of
+> under `enforce`, as does a writing runner with a declared radius; an undeclared writing runner starts and pages, and the hourly selftest fails until the tool is back — ten days of
 > `kubectl: not found` had passed as `unverifiable` and `recorded` before this. The measurement is served on `GET /v1/posture`
 > and cited in every run's audit record.
+> An unverifiable writing runner with a declared radius also refuses under `enforce`. With no explicit AWS source, the
+> instance metadata service is asked directly whether it offers a role: one it offers is measured like any other source.
+
+> **Recovery is not exactly-once (2026-10-27).** Concurrent signed hook requests use a durable, renewable processing
+> lease; an in-flight duplicate receives 503 and can retry, while completed signatures are deduplicated. A crash after
+> an external side effect but before the completion record can still cause that effect to repeat. Remediation records
+> each step's start and result; a started step without a durable result is unknown and needs operator reconciliation.
+> Output is redacted whole before it is cut to its last 10,000 characters (omitted past a megabyte); redaction is by pattern and can miss secrets, and original proposal commands remain stored.
 
 ## What holds, and what only helps
 

hookprobe/README.md

@@ -178,11 +178,18 @@ always reach. Two things about the word *default*, before the layers:
   over the dangerous actions) and compares the answer with the declaration:
   `readonly-confirmed`, `wider-than-declared` (refuses to start under the
   default `HOOKPROBE_POSTURE_CHECK=enforce`), `unverifiable`, `unmeasurable`
-  (the measuring tool is not in the image: a readonly node refuses to start
-  under `enforce`, a writing node starts and pages), `recorded`,
+  (the measuring tool is not in the image: a readonly node or a writing node
+  with a declared radius refuses to start under `enforce`; an undeclared
+  writing node starts and pages), `recorded`,
   `no-credentials`. The verdict is at `GET /v1/posture` and at the top of
   every run's audit, so "read-only" is a measurement with a date rather than
   a sentence in this file.
+  An unverifiable writing node with a declared radius also refuses under
+  `enforce`. When nothing names an AWS credential, the instance metadata
+  service is asked directly, the way the AWS CLI asks it (no proxy, a second
+  per question): a role it offers is measured like any other source; no role,
+  another vendor's service at the address, or nothing there is no credential;
+  `AWS_EC2_METADATA_DISABLED=true` skips the question.
 - **Widening is a decision, and it is signed.** Under `danger-only` the guard
   keeps only the rules no credential scope can undo — `rm -rf`, `mkfs`/`dd`
   onto a device, fork bombs, container runtimes, `terraform destroy`,
@@ -198,6 +205,13 @@ always reach. Two things about the word *default*, before the layers:
   (`HOOKPROBE_REMEDIATION_COOLDOWN_SECONDS`, 15 minutes, 0 disables) is held
   rather than changed twice with nobody looking in between. Both are recorded
   in the run's audit with the person's action beside them.
+  Execution checkpoints distinguish completed steps, a started step with an
+  unknown outcome, and steps not started after an interruption; reconcile an
+  unknown outcome before retrying. A step's output is redacted whole and then
+  cut to its last 10,000 characters, so a secret the cut would split was seen
+  whole by the redactor; past a megabyte it is omitted instead. Redaction is by
+  pattern, not a security boundary, and the stored proposal retains its
+  original commands.
 
 1. **Credentials** — mount query-only credentials (read-only kubeconfig,
    Prometheus/Loki endpoints, viewer tokens). This is the real boundary;

hookprobe/hookprobe/posture.py

@@ -48,6 +48,8 @@
 import logging
 import os
 import time
+import urllib.error
+import urllib.request
 from collections.abc import Awaitable, Callable
 from pathlib import Path
 from typing import Any
@@ -155,7 +157,19 @@ def kubeconfig_path() -> Path | None:
 
 
 def aws_configured() -> bool:
-    if os.environ.get("AWS_ACCESS_KEY_ID"):
+    # A profile NAME alone is not a source: it names a section of a file, and
+    # the files are what the loop below looks for. An instance metadata service
+    # is not configured at all — it is asked (see metadata_offers_credentials).
+    if any(
+        os.environ.get(name)
+        for name in (
+            "AWS_ACCESS_KEY_ID",
+            "AWS_ROLE_ARN",
+            "AWS_WEB_IDENTITY_TOKEN_FILE",
+            "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI",
+            "AWS_CONTAINER_CREDENTIALS_FULL_URI",
+        )
+    ):
         return True
     for var, default in (("AWS_SHARED_CREDENTIALS_FILE", "~/.aws/credentials"), ("AWS_CONFIG_FILE", "~/.aws/config")):
         if Path(os.environ.get(var) or default).expanduser().is_file():
@@ -190,11 +204,60 @@ async def check_kube(run: Runner) -> dict[str, Any]:
     return record
 
 
+def metadata_offers_credentials(timeout: float = 1.0) -> bool:
+    """Whether an AWS instance metadata service here hands out a role's credentials.
+
+    When nothing names a credential this is the source left, and the AWS CLI
+    asks it the same two questions: a token (IMDSv2), then the role listing.
+    A role name in the answer is a credential; a 404 is an instance with no role;
+    anything else is no AWS service at all. A cloud VM of another vendor answers
+    at the same address, and an answer of any kind is not a credential — GitHub's
+    runners are Azure VMs, and counting any answer made the image refuse to start
+    there. No proxy, on purpose: the probes' egress proxy answers for addresses
+    it refuses, and the question is what this container reaches by itself.
+    """
+    base = (
+        os.environ.get("AWS_EC2_METADATA_SERVICE_ENDPOINT")
+        or (
+            "http://[fd00:ec2::254]"
+            if os.environ.get("AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE", "").lower() == "ipv6"
+            else "http://169.254.169.254"
+        )
+    ).rstrip("/")
+    opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
+    token: dict[str, str] = {}
+    put = urllib.request.Request(
+        base + "/latest/api/token", method="PUT", headers={"X-aws-ec2-metadata-token-ttl-seconds": "60"}
+    )
+    try:
+        with opener.open(put, timeout=timeout) as answer:  # nosec B310 — a fixed link-local address
+            token = {"X-aws-ec2-metadata-token": answer.read(256).decode("ascii", "replace").strip()}
+    except (OSError, ValueError):
+        pass  # IMDSv1 only, another vendor's service, or nothing: the listing decides
+    listing = urllib.request.Request(base + "/latest/meta-data/iam/security-credentials/", headers=token)
+    try:
+        with opener.open(listing, timeout=timeout) as answer:  # nosec B310
+            return bool(answer.read(1024).strip())
+    except (OSError, ValueError):  # HTTPError included: 404 is no role, anything else no AWS
+        return False
+
+
 async def check_aws(run: Runner) -> dict[str, Any]:
     if not aws_configured():
-        return {"present": False}
+        # Asked, not assumed. Assuming made every laptop and every probe on an
+        # internal network `unverifiable`, and a writing node with a declared
+        # radius refused to start over a service that cannot exist there.
+        if os.environ.get("AWS_EC2_METADATA_DISABLED", "").lower() == "true":
+            return {"present": False}
+        if not await asyncio.to_thread(metadata_offers_credentials):
+            return {"present": False}
+        # A role is offered: its credentials are measured like any other source.
     record: dict[str, Any] = {"present": True, "identity": None, "allowed": [], "errors": [], "unverifiable": False}
     code, out, err = await run(["aws", "sts", "get-caller-identity", "--output", "json"])
+    if code != 0 and "Unable to locate credentials" in (err or out):
+        # The CLI looked everywhere it looks and found nothing: an instance with
+        # no role, say. That is an absence, not a measurement that failed.
+        return {"present": False}
     if code != 0:
         record["errors"].append(f"get-caller-identity: {(err or out).strip()[:200]}")
         record["unverifiable"] = True
@@ -228,7 +291,29 @@ async def check_aws(run: Runner) -> dict[str, Any]:
         record["errors"].append(f"simulate-principal-policy: {(err or out).strip()[:200]}")
         record["unverifiable"] = True
     else:
-        record["allowed"] = aws_allowed(out)
+        try:
+            results = json.loads(out)["EvaluationResults"]
+            if not isinstance(results, list):
+                raise ValueError("invalid results")
+            record["allowed"] = sorted(
+                {
+                    str(item.get("EvalActionName"))
+                    for item in results
+                    if isinstance(item, dict) and item.get("EvalDecision") == "allowed" and item.get("EvalActionName")
+                }
+            )
+            if any(
+                not isinstance(item, dict)
+                or item.get("EvalDecision") not in {"allowed", "explicitDeny", "implicitDeny"}
+                for item in results
+            ):
+                raise ValueError("invalid decision")
+            decisions = {str(item["EvalActionName"]) for item in results if item.get("EvalDecision")}
+            if not set(DANGEROUS_AWS_ACTIONS).issubset(decisions):
+                raise ValueError("incomplete simulation")
+        except (ValueError, KeyError, TypeError, AttributeError):
+            record["errors"].append("simulate-principal-policy: invalid or incomplete results")
+            record["unverifiable"] = True
     return record
 
 
@@ -281,19 +366,21 @@ def verdict(bash_guard: str, kube: dict[str, Any], aws: dict[str, Any], declared
         # nothing to answer that with and this stays an observation, which is
         # what it was for a year. Declared, `enforce` means the same thing on
         # both postures: the credential may not exceed the declaration.
-        if declared is None:
-            return "unmeasurable" if missing else "recorded"
         if beyond(kube, aws, declared):
             return "wider-than-declared"
-        return "unmeasurable" if missing else "within-declared-radius"
-    if not kube.get("present") and not aws.get("present"):
-        return "no-credentials"
+        if missing:
+            return "unmeasurable"
+        if kube.get("errors") or aws.get("unverifiable"):
+            return "unverifiable"
+        return "recorded" if declared is None else "within-declared-radius"
     if kube.get("mutating") or aws.get("allowed"):
         return "wider-than-declared"
     if missing:
         return "unmeasurable"
     if kube.get("errors") or aws.get("unverifiable"):
         return "unverifiable"
+    if not kube.get("present") and not aws.get("present"):
+        return "no-credentials"
     return "readonly-confirmed"
 
 
@@ -395,13 +482,18 @@ async def on_startup(
         if alarm is not None:
             with contextlib.suppress(Exception):  # the page is a courtesy; the verdict below is the decision
                 await alarm(f"posture UNMEASURABLE — {text}")
-        if bash_guard == "readonly" and mode == "enforce":
+        if (bash_guard == "readonly" or declared is not None) and mode == "enforce":
             logger.error("REFUSING TO START — %s", text)
             raise PostureViolation(
                 f"posture cannot be measured as declared {bash_guard!r}: {missing} not in this image"
             )
         logger.error("%s", text)
     elif record["verdict"] == "unverifiable":
+        if alarm is not None:
+            with contextlib.suppress(Exception):
+                await alarm(f"posture UNVERIFIABLE — {line}")
+        if bash_guard != "readonly" and declared is not None and mode == "enforce":
+            raise PostureViolation(f"declared writing posture could not be verified: {line}")
         logger.warning(
             "posture could not be fully verified: %s · %s",
             line,