probe image: one tag built alike, and a credential nobody can measure is not a posture

.agents/notes/implemented/2026-10-12-a-credential-nobody-can-measure-is-not-a-posture.md

@@ -0,0 +1,54 @@
+---
+title: A credential nobody can measure is not a posture
+status: implemented
+date: 2026-10-12
+scope: stack
+---
+
+## Decision
+
+A mounted credential whose measuring tool is not in the image gets its own
+verdict, `unmeasurable`, and it is never a pass:
+
+- a `readonly` node refuses to start under `HOOKPROBE_POSTURE_CHECK=enforce`
+  and says which tool is missing;
+- a writing node (`danger-only`) starts, records the verdict, and pages on
+  `HOOKPROBE_ALARM_URL` at once;
+- the hourly selftest's posture check fails until the tool is back;
+- what WAS measured still outranks it: a violation found on one credential is
+  `wider-than-declared` whether or not the other credential's tool exists.
+
+And the shape that caused it is refused at the gate: services that share an
+image tag must build it alike (`scripts/assert_shared_images_build_alike.py`,
+in `scripts/gate.sh` and `ci-stack`). The work compose's `probe-watch` build now
+passes `KUBECTL_VERSION` exactly as `probe-plan` and `probe-work` do.
+
+## Why
+
+The work stack's three probes share `hookprobe:work`. A compose builds every
+service that has a `build:` block, and the tag holds whichever build ran last;
+`probe-watch`'s build passed no kubectl version. From 2026-09-29 the image had
+no kubectl. At every boot after that both credentialed nodes recorded
+`can-i --list: kubectl: not found` — the planner's verdict `unverifiable`, the
+work node's `recorded` with `mutating: []` — and both started. The hourly check
+asks one question, "wider than declared?", and answered held every hour. The
+stack ran this way until it was stopped on 2026-10-09; nothing paged, no gate
+went red, and the record read like a cluster with no mutating grants.
+
+`unverifiable` was built for an identity that may not ask about itself. A
+missing binary is a different silence: the far side was never asked, and the
+operator can fix it with a rebuild — which is exactly why it must be loud.
+
+## Consequences
+
+- A rebuild that drops a tool now stops a readonly node at boot instead of
+  degrading its planner quietly; a writing node keeps running but says so.
+- The posture record carries `tool_missing` on the credential it could not
+  measure, and `GET /v1/posture` and the audit cite it.
+- `assert_guards_are_tested.py` carries the mutation: turning `unmeasurable`
+  back into `recorded` must fail `tests/test_posture.py`.
+- The new gate check parses the composes by indentation and has a floor; a
+  compose written in another shape, or one that stops building something,
+  needs the floor lowered on purpose.
+- Already-built images are not rebuilt by this change; the work image needs
+  `docker compose ... build` before the next recreate.

.github/workflows/ci-stack.yml

@@ -133,6 +133,11 @@ jobs:
       # in is a comment.
       - name: the pipe and the brain are inside their stated ceilings
         run: python3 scripts/assert_weight.py
+      # Three services build one image tag, and the tag is whichever build ran
+      # last: a build arg one of them does not pass is an image the others did
+      # not mean. The work image shipped without kubectl this way (2026-09-29).
+      - name: services sharing an image tag build it alike
+        run: python3 scripts/assert_shared_images_build_alike.py
       # A live production component on the card-callback path, outside every
       # service package and therefore outside every component gate. It was
       # covered by nothing at all until it was edited and the question was asked.

deploy/docker-compose.work.yml

@@ -491,6 +491,14 @@ services:
       context: ../hookprobe
       args:
         APT_EXTRAS: ${PROBE_PLAN_APT_EXTRAS:-git awscli}
+        # ARG for ARG the planner's build: the three probes share one image tag
+        # and the tag is whichever build ran last. This block passed no kubectl
+        # version until 2026-10-12, so from 2026-09-29 the tag was this
+        # service's kubectl-less build and the planner and the work node started
+        # every day unable to measure their cluster side (posture.py says what
+        # that cost). scripts/assert_shared_images_build_alike.py keeps the
+        # three builds alike.
+        KUBECTL_VERSION: ${PROBE_PLAN_KUBECTL_VERSION:-v1.31.4}
     image: hookprobe:work
     container_name: work-probe-watch
     restart: unless-stopped
@@ -685,7 +693,11 @@ services:
     build:
       context: ../hookprobe
       args:
-        APT_EXTRAS: ${PROBE_WORK_APT_EXTRAS:-git awscli}
+        # The planner's variables, not a PROBE_WORK_ pair: the three probes share
+        # one image tag, so one set of build args is the only honest shape. A
+        # second variable with the same default would let .env fork the image
+        # and leave the tag holding whichever build ran last.
+        APT_EXTRAS: ${PROBE_PLAN_APT_EXTRAS:-git awscli}
         KUBECTL_VERSION: ${PROBE_PLAN_KUBECTL_VERSION:-v1.31.4}
     image: hookprobe:work
     container_name: work-probe-work

docs/containment.md

@@ -17,7 +17,10 @@ a real model cannot be steered into emitting a harmful one.
 > (`hookprobe/posture.py`): what `kubectl auth can-i` and `iam simulate-principal-policy` say the identity may do, against
 > the posture it declares. Declared `readonly` with credentials that can mutate refuses to start (`HOOKPROBE_POSTURE_CHECK=enforce`);
 > a `danger-only` runner records its blast radius; an identity that cannot be asked is reported *unverifiable*, never confirmed.
-> The measurement is served on `GET /v1/posture` and cited in every run's audit record.
+> A credential whose measuring tool is not in the image is *unmeasurable* (2026-10-12): a readonly runner refuses to start
+> under `enforce`, a writing runner starts and pages, and the hourly selftest fails until the tool is back — ten days of
+> `kubectl: not found` had passed as `unverifiable` and `recorded` before this. The measurement is served on `GET /v1/posture`
+> and cited in every run's audit record.
 
 ## What holds, and what only helps
 

hookprobe/README.md

@@ -83,7 +83,7 @@ caller ── POST /hooks/agent ───────────────▶
 | `GET /v1/automation` | Each class of automation, its declared ceiling, its record of proposals and the human decisions on them, and the tier the record WOULD support. Counters, never agreement rates. The page an operator reads before deciding a class has earned a step up. |
 | `POST /v1/automation/{class}/{id}/regret` | A sampling review saying an auto-applied action was wrong — the one event that resets a class's argument for its tier, and how an unattended deployment gets its human-in-the-loop back asynchronously. Operator token only; no run can label its own work. |
 | `POST /v1/runs/{key}/ruling` | Rule one run useful/useless (or clear) from the sessions page, and be told what it DID: a useless that withdraws the runbook this run distilled says so. The only ruling surface that needs no chat bridge — the card buttons were the only other one, and a deployment without a bridge has none, which is why the loops that feed on a ruling went unfed. `record_ruling` runs in-process here, so the consequence is reported synchronously rather than enqueued and forgotten. |
-| `GET /v1/posture` | what the credentials could do when this runner started, measured against the declared posture — kube rules that mutate, AWS actions that would be allowed, or `unverifiable` when the identity may not ask; the record behind "this ran read-only" (see `HOOKPROBE_POSTURE_CHECK`) |
+| `GET /v1/posture` | what the credentials could do when this runner started, measured against the declared posture — kube rules that mutate, AWS actions that would be allowed, `unverifiable` when the identity may not ask, or `unmeasurable` when the tool that would ask is not in the image; the record behind "this ran read-only" (see `HOOKPROBE_POSTURE_CHECK`) |
 | `GET /v1/work` | the board: one row per piece of WORK rather than per run, with what is blocked, what is in flight and what closed without anybody stepping in. Its `counts` also answer the two questions an on-call opens a board with — `median_answer_seconds`, how long until the work was any use, and `signals` against the items they became, which is how much noise the door folded. Both are `null` rather than `0` when nothing has answered yet, because those are different facts. Derived on every request from the run records, the proposals and the memory queue — nothing is stored twice ([work.py](hookprobe/work.py)) |
 | `POST /v1/runs/{key}/retry` | human takeover: try a failed investigation again. Continues the engine session when the failure left one — the second attempt starts from what the first gathered — and otherwise re-asks the question the run opened with. Not budget-gated, for the reason `/hooks/agent` is not |
 | `GET /v1/agent` | what this node is: name, role, runtime adapter and model, the policy it runs under (guard, budget, escalation bar, who may speak to it from chat) and its health. Settings this process resolved, answered in one place so a board can name the agent that did the work; no secret values, like `/v1/config` |
@@ -175,7 +175,9 @@ always reach. Two things about the word *default*, before the layers:
   actually do (`kubectl auth can-i --list`, `aws iam simulate-principal-policy`
   over the dangerous actions) and compares the answer with the declaration:
   `readonly-confirmed`, `wider-than-declared` (refuses to start under the
-  default `HOOKPROBE_POSTURE_CHECK=enforce`), `unverifiable`, `recorded`,
+  default `HOOKPROBE_POSTURE_CHECK=enforce`), `unverifiable`, `unmeasurable`
+  (the measuring tool is not in the image: a readonly node refuses to start
+  under `enforce`, a writing node starts and pages), `recorded`,
   `no-credentials`. The verdict is at `GET /v1/posture` and at the top of
   every run's audit, so "read-only" is a measurement with a date rather than
   a sentence in this file.

hookprobe/hookprobe/app.py

@@ -223,7 +223,11 @@ async def lifespan(_: FastAPI):
         # Before the first run: are the credentials as narrow as the posture
         # says? Under enforce a wider-than-declared runner does not come up.
         await posture.on_startup(
-            settings.workdir, settings.bash_guard, settings.posture_check, radius=settings.blast_radius
+            settings.workdir,
+            settings.bash_guard,
+            settings.posture_check,
+            radius=settings.blast_radius,
+            alarm=service.alarm,
         )
         service.recover_orphans()
         service.sweep_interrupted_remediations()

hookprobe/hookprobe/posture.py

@@ -28,11 +28,22 @@
 Unverifiable is a verdict, not an error: a read-only AWS identity often may not
 call SimulatePrincipalPolicy on itself. That is reported as such — loudly — and
 never mistaken for confirmed.
+
+Unmeasurable is the verdict for a credential whose measuring tool is not in the
+image, and it is a different silence: the far side was never asked. It happened.
+The work stack's three probes share one image tag, one of their builds passed no
+kubectl version, and from 2026-09-29 to 10-09 the planner and the work node
+started every day with `kubectl: not found` in their records — the planner
+`unverifiable`, the writer `recorded` with an empty radius — and nothing paged,
+because both verdicts start and the hourly check only asks "wider?". Now a
+readonly node refuses to start under `enforce`, a writing node starts and pages
+through the alarm channel, and the hourly selftest fails until the tool is back.
 """
 
 from __future__ import annotations
 
 import asyncio
+import contextlib
 import json
 import logging
 import os
@@ -74,7 +85,8 @@
 
 
 class PostureViolation(RuntimeError):
-    """The credentials are wider than the declared posture, and the mode is enforce."""
+    """The posture cannot be held as declared under enforce: the credentials are
+    wider than it, or could not be measured at all."""
 
 
 async def run_cli(argv: list[str]) -> tuple[int, str, str]:
@@ -157,6 +169,12 @@ async def check_kube(run: Runner) -> dict[str, Any]:
         return {"present": False}
     record: dict[str, Any] = {"present": True, "kubeconfig": str(path), "mutating": [], "errors": []}
     code, out, err = await run(["kubectl", "auth", "can-i", "--list"])
+    if code == 127:
+        # The binary is not in the image. Four more questions of it would be
+        # four more copies of this line; the verdict needs the one.
+        record["tool_missing"] = "kubectl"
+        record["errors"].append(f"can-i --list: {(err or out).strip()[:200]}")
+        return record
     if code != 0:
         record["errors"].append(f"can-i --list: {(err or out).strip()[:200]}")
     else:
@@ -180,6 +198,8 @@ async def check_aws(run: Runner) -> dict[str, Any]:
     if code != 0:
         record["errors"].append(f"get-caller-identity: {(err or out).strip()[:200]}")
         record["unverifiable"] = True
+        if code == 127:
+            record["tool_missing"] = "aws"
         return record
     try:
         arn = str(json.loads(out).get("Arn") or "")
@@ -243,20 +263,35 @@ def beyond(kube: dict[str, Any], aws: dict[str, Any], declared: set[str] | None)
     return sorted(item for item in measured if item not in declared)
 
 
+def tools_missing(kube: dict[str, Any], aws: dict[str, Any]) -> list[str]:
+    """The measuring tools a mounted credential could not be asked with."""
+    return [
+        str(record["tool_missing"]) for record in (kube, aws) if record.get("present") and record.get("tool_missing")
+    ]
+
+
 def verdict(bash_guard: str, kube: dict[str, Any], aws: dict[str, Any], declared: set[str] | None = None) -> str:
+    # A credential nobody could measure is not a posture of any kind. What WAS
+    # measured still outranks it: a violation found on one credential is a
+    # violation whether or not the other one's tool is installed.
+    missing = tools_missing(kube, aws)
     if bash_guard != "readonly":
         # A writing posture is ALLOWED to write — the question is never "can it"
         # but "is it wider than what somebody agreed to". Undeclared, there is
         # nothing to answer that with and this stays an observation, which is
         # what it was for a year. Declared, `enforce` means the same thing on
         # both postures: the credential may not exceed the declaration.
         if declared is None:
-            return "recorded"
-        return "wider-than-declared" if beyond(kube, aws, declared) else "within-declared-radius"
+            return "unmeasurable" if missing else "recorded"
+        if beyond(kube, aws, declared):
+            return "wider-than-declared"
+        return "unmeasurable" if missing else "within-declared-radius"
     if not kube.get("present") and not aws.get("present"):
         return "no-credentials"
     if kube.get("mutating") or aws.get("allowed"):
         return "wider-than-declared"
+    if missing:
+        return "unmeasurable"
     if kube.get("errors") or aws.get("unverifiable"):
         return "unverifiable"
     return "readonly-confirmed"
@@ -287,11 +322,13 @@ def summary(record: dict[str, Any]) -> str:
         parts.append(
             f"kube mutating={len(kube.get('mutating') or [])}"
             + (f" errors={len(kube['errors'])}" if kube.get("errors") else "")
+            + (f" tool-missing={kube['tool_missing']}" if kube.get("tool_missing") else "")
         )
     if aws.get("present"):
         parts.append(
             f"aws identity={aws.get('identity') or '?'} dangerous-allowed={len(aws.get('allowed') or [])}"
             + (" unverifiable" if aws.get("unverifiable") else "")
+            + (f" tool-missing={aws['tool_missing']}" if aws.get("tool_missing") else "")
         )
     return " · ".join(parts)
 
@@ -314,6 +351,7 @@ async def on_startup(
     mode: str,
     run: Runner = run_cli,
     radius: Path | None = None,
+    alarm: Callable[[str], Awaitable[bool]] | None = None,
 ) -> dict[str, Any] | None:
     """Check, record, announce — and under `enforce`, refuse to start a runner
     whose credentials are wider than it declares.
@@ -322,6 +360,11 @@ async def on_startup(
     `danger-only` runner has always started whatever it held, because "allowed
     to write" left nothing to compare against; with it, `enforce` means the same
     sentence on both postures.
+
+    `alarm` is the node's road around the pipe (`HOOKPROBE_ALARM_URL`). A
+    credential that could not be measured is said there at once, on both
+    postures: the readonly node is about to refuse and loop, and a writing node
+    is about to start with a radius nobody measured.
     """
     if mode not in MODES:
         mode = "enforce"  # a typo in the switch that decides whether to refuse must fail closed
@@ -346,6 +389,18 @@ async def on_startup(
             logger.error("REFUSING TO START — credentials wider than declared posture: %s · %s", line, detail)
             raise PostureViolation(f"credentials wider than declared posture {bash_guard!r}: {detail}")
         logger.warning("credentials wider than declared posture: %s · %s", line, detail)
+    elif record["verdict"] == "unmeasurable":
+        missing = ", ".join(tools_missing(record["kube"], record["aws"]))
+        text = f"posture cannot be measured: {missing} not in this image · {line}"
+        if alarm is not None:
+            with contextlib.suppress(Exception):  # the page is a courtesy; the verdict below is the decision
+                await alarm(f"posture UNMEASURABLE — {text}")
+        if bash_guard == "readonly" and mode == "enforce":
+            logger.error("REFUSING TO START — %s", text)
+            raise PostureViolation(
+                f"posture cannot be measured as declared {bash_guard!r}: {missing} not in this image"
+            )
+        logger.error("%s", text)
     elif record["verdict"] == "unverifiable":
         logger.warning(
             "posture could not be fully verified: %s · %s",

hookprobe/hookprobe/selftest.py

@@ -213,9 +213,15 @@ async def posture_still_holds(settings: Settings) -> dict[str, Any]:
     beyond = record.get("beyond_declared") or []
     if verdict == "wider-than-declared" and beyond:
         detail = f"{detail} · beyond: {', '.join(str(item) for item in beyond)}"[:280]
+    if verdict == "unmeasurable":
+        # The tool that would measure it is not in the image. Ten days of
+        # `kubectl: not found` passed this check hourly (2026-09-29 to 10-09)
+        # because the only question asked was "wider?".
+        missing = ", ".join(posture.tools_missing(record.get("kube") or {}, record.get("aws") or {}))
+        detail = f"{detail} · cannot measure: {missing} not in this image"[:280]
     return _check(
         "credentials are no wider than the declared posture",
-        verdict != "wider-than-declared",
+        verdict not in ("wider-than-declared", "unmeasurable"),
         detail,
         "a credential whose far side cannot be asked — `unverifiable` is a verdict, not a pass",
     )