Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion deploy/docker-compose.work.yml
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,10 @@ services:
MCPGATE_CLIENT_PLAN_TOKEN: ${PLAN_MCPGATE_TOKEN:?set PLAN_MCPGATE_TOKEN in .env}
MCPGATE_CLIENT_PLAN_TOOLS: ${PLAN_MCPGATE_TOOLS:?set PLAN_MCPGATE_TOOLS in .env}
MCPGATE_LEDGER: /data/calls.jsonl
# The proxy's refusals, beside the gate's calls: a run that tried to reach
# an address nobody listed used to be one WARNING line in a container log
# that a recreate throws away.
EGRESS_LEDGER: /data/egress.jsonl
INGRESS_FORWARD: 8088:probe-plan:8088,8089:probe-watch:8088,8090:probe-work:8088
TZ: ${WATCH_TZ:-Asia/Singapore}
ports:
Expand Down Expand Up @@ -725,7 +729,11 @@ services:
environment:
# Who may continue this node's sessions by replying in the chat thread
# (sender ids, or `*`); empty = nobody.
HOOKPROBE_FOLLOW_UP_SENDERS: ${HOOKPROBE_FOLLOW_UP_SENDERS:-}
# Its OWN variable, empty by default: the pipe already keeps a reply under
# a plan card off the write node by routing (by_work=False), and the node
# itself now refuses to continue a session from chat unless somebody is
# named HERE — a second lock on the one node that holds a write credential.
HOOKPROBE_FOLLOW_UP_SENDERS: ${WORK_FOLLOW_UP_SENDERS:-}
HOOKPROBE_RELAY_UI_URL: ${HOOKPROBE_RELAY_UI_URL:-http://127.0.0.1:8100}
# Where this node is reachable from OUTSIDE, if anywhere. Empty on every
# deployment here — the nodes bind to loopback behind the host firewall —
Expand Down
7 changes: 5 additions & 2 deletions deploy/sidecars/gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,11 @@
# How much of a call's arguments the ledger keeps. Enough to see what was asked,
# short enough that the ledger is not a copy of the chat.
ARGUMENTS_KEPT = 500
# Idle, per socket operation — see the module docstring.
UPSTREAM_TIMEOUT = 60.0
# Idle, per socket operation — see the module docstring. A knob
# (MCPGATE_UPSTREAM_TIMEOUT, seconds) so a deployment whose upstream answers in
# a second can report a silent one as failed sooner than a minute; the default
# is the minute this file always had.
UPSTREAM_TIMEOUT = float(os.environ.get("MCPGATE_UPSTREAM_TIMEOUT", "60"))
# A JSON-RPC request is a few KB and a listing of 34 tools is a few tens; this
# container has 128 MB, and a body is held about three times over while it is
# filtered. Past either bound nothing here should buffer it.
Expand Down
12 changes: 10 additions & 2 deletions deploy/sidecars/proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@
proxy on an allowlist. Everything the investigator legitimately talks to is a
name somebody wrote down; everything else is refused and, more importantly,
LOGGED. A refused connection is the signal — it is the only place that can say
"this run tried to reach somewhere nobody listed".
"this run tried to reach somewhere nobody listed". With EGRESS_LEDGER set the
refusals are also written as JSONL, the way the gate beside this writes its
calls: a container's log leaves with the container, and the refusal log is the
product, so it lives on the volume the gate's does.

Deliberately stdlib-only and small enough to read in one sitting, like the
bridge beside it. A proxy is now in the path of every call the investigator
Expand Down Expand Up @@ -64,7 +67,7 @@
import sys
import threading

from common import Server, pump
from common import Ledger, Server, pump

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
logger = logging.getLogger("egress")
Expand Down Expand Up @@ -94,6 +97,10 @@ def _rules(text: str) -> tuple[str, ...]:
PORTS = frozenset(int(p) for p in os.environ.get("EGRESS_PORTS", "80,443,8088,8100,8200,4318").split(",") if p.strip())
IDLE_SECONDS = 300
CONNECT_TIMEOUT = 20
# Refusals, one JSON line each, when a path is given; empty keeps them in the
# container log only. Best-effort like the gate's: a disk that fills must not
# turn the way out into a wall.
LEDGER = Ledger(os.environ.get("EGRESS_LEDGER", ""))
# Where a desktop container runtime maps the host machine, inside a range that
# is otherwise "this network" and that the stdlib does not call private.
HOST_MAPPED = ipaddress.ip_network("0.250.250.0/24")
Expand Down Expand Up @@ -313,6 +320,7 @@ def _refuse(self, code: int, why: str, target: str, level: int = logging.WARNING
# an address nobody wrote down, and that is worth an operator's eye
# whether it was a mistake, a dependency nobody declared, or worse.
logger.log(level, "REFUSED %s — %s", target, why)
LEDGER.write(event="egress.refused", target=target, code=code, why=why)
body = f"{why}\n".encode()
reason = {400: "Bad Request", 403: "Forbidden", 502: "Bad Gateway"}.get(code, "Forbidden")
try:
Expand Down
26 changes: 25 additions & 1 deletion deploy/sidecars/tests/test_allowlist.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,11 @@
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))


def _with(allow: str, ports: str = "80,443", internal: str = ""):
def _with(allow: str, ports: str = "80,443", internal: str = "", ledger: str = ""):
os.environ["EGRESS_ALLOW"] = allow
os.environ["EGRESS_PORTS"] = ports
os.environ["EGRESS_INTERNAL"] = internal
os.environ["EGRESS_LEDGER"] = ledger
import proxy

return importlib.reload(proxy)
Expand Down Expand Up @@ -210,6 +211,29 @@ def test_whitespace_and_stray_commas_do_not_create_a_blank_rule(rule: str) -> No
assert not p.permitted("")


def test_a_refusal_is_written_to_the_ledger_when_one_is_named(tmp_path) -> None:
"""A container's log leaves with the container; the refusal log is the product."""
import json
import socket
import threading

ledger = tmp_path / "egress.jsonl"
proxy = _with("example.com", ledger=str(ledger))
gateway = proxy.Server(("127.0.0.1", 0), proxy.Handler)
threading.Thread(target=gateway.serve_forever, kwargs={"poll_interval": 0.02}, daemon=True).start()
try:
with socket.create_connection(("127.0.0.1", gateway.server_address[1]), timeout=5) as client:
client.sendall(b"CONNECT evil.test:443 HTTP/1.1\r\nHost: evil.test\r\n\r\n")
first = client.recv(200).decode("latin-1").splitlines()[0]
finally:
gateway.shutdown()
gateway.server_close()
assert first.startswith("HTTP/1.1 403")
row = json.loads(ledger.read_text(encoding="utf-8").splitlines()[-1])
assert row["event"] == "egress.refused" and row["target"] == "evil.test:443" and row["code"] == 403
assert "allowlist" in row["why"]


def test_bytes_sent_right_behind_the_connect_reach_the_far_side() -> None:
"""A client that starts TLS without waiting for the 200 must not lose its hello.

Expand Down
2 changes: 2 additions & 0 deletions docs/containment.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ cannot reach. Its limits are in the third column; its key is not model-written.
- **Per-node credentials, and the agent's bearer reads only** — the real boundary, which lives outside this repository; inside a node the agent's own token can only `GET`.
- **Closed sets at the doors** — a payload, a verdict or a tool call picks from inside a set the config declared; free text cannot widen it. The MCP list binds the agent's TOOL calls and not its sockets, which is why the row below exists.
- **The chat's tools are on a list in another container** — a network with no route, and a list the agent's process cannot edit. The tools that write are not refused, they are unreachable; a hostile model has nowhere to send the request.
- **The one way out** — the same network with no route, and a forward proxy in another container whose list the agent's process cannot edit: an unlisted destination is unreachable, not refused. What it bounds is WHERE, not what — a listed host that accepts bodies carries whatever the model puts in them.
- **The watcher's signature is held where no agent runs** — the key is in another container and the check in front of it is against a file the agent does not write. What it bounds is which conversations can carry a signal, not whether one is honest.
- **Tool gate reachability** — bounds the EXISTENCE of the heuristics below, not their content: a node whose gate cannot answer, or was never asked, stops.
- **Budget breaker and retry caps** — arithmetic over a ledger and counters the model does not write.
Expand Down Expand Up @@ -129,6 +130,7 @@ Every row here is a claim, and a claim is worth what its test is worth. `scripts
| **Per-node credentials, and the agent's bearer reads only** — each process-external node gets its own secrets, budget and guards; inside a node the console's `HOOKPROBE_TOKEN` is blanked out of the agent's subprocess and replaced with `HOOKPROBE_AGENT_TOKEN`, which the API refuses on every method but `GET` | One node's compromise becoming every node's; an injected instruction reaching a Bash step and using the service's own write surface: `PUT /v1/memory` (no shape check, unlike the suggestion path), `PUT /v1/skills`, `/v1/system-prompt`, approving a remediation, and `POST /hooks/agent`, which spends money. The regret route *claimed* this for its whole life while the token was inherited | Anything inside a node once it is compromised; everything a `GET` exposes, which is most of a node's own state; and anything the agent can do without this API, which is the whole shell. The real boundary is the credential each node is given, and that lives outside this repository | the compose files; `hookprobe/settings.py` (`agent_token`), `app.py` (`require_token`), `gate.py` (`agent_credentials`) |
| **Closed sets at the doors** — config names the channel set a payload may pick from (the bypass lanes); `HOOKPROBE_VERDICTS` and `HOOKPROBE_MCP_TOOLS` are closed sets, and empty denies all | An injected message routing itself around a gatekeeper; an investigator's free text choosing which lane spends money; a chat message asking the agent to use a write tool on a mounted server | A wrong choice among lanes the operator already declared. **A socket.** The MCP allowlist is enforced where the agent asks for a TOOL and says nothing about a request the agent makes itself: measured on 2026-09-30, one plain HTTP POST from inside the watcher reached the chat client's own MCP port, with no credential, and called a tool this list does not name. The row below is the answer; this one alone was advice | `hookrelay/config.py`, `routing.py`; `hookprobe/reports.py` (`verdict`), `engine.py` (`mcp_deny_reason`) |
| **The chat's tools are on a list in another container** — the probes sit on a docker network with `internal: true`, so they have no route to the host or the internet at all; `mcp-gate` holds the only reach to the chat client and forwards a `tools/call` only when the tool's name is on the calling node's list, recording both halves. Several MCP servers can share its one port as routes: each keeps its own credential, which never reaches another, and a node's list names tools per route | An agent that goes around its own MCP allowlist with an HTTP request, which is what the row above could not stop. The writes are not refused, they are unreachable: the watcher sees 8 read tools where the server offers 34, 20 of them writes | Anything a PERMITTED read does: an argument is not checked, and what a read returns still goes to the model provider. The server-to-client stream is declined rather than relayed (405), so a chat server that pushed notifications would not reach the agent through this. A client token lives in the agent's own MCP config by construction — it admits exactly the tools that agent already had, which is the point of the list being here rather than there | `deploy/sidecars/gate.py`, `deploy/docker-compose.work.yml` (`probe_net`) |
| **The one way out** — the probes' only route to anything is the egress proxy in the doors container, on an allowlist of names: a bare name matches exactly, a leading dot admits a tree, and two shapes (`@aws-api`, `@aws-s3`) admit AWS service APIs and buckets without admitting the hostnames anyone can obtain under that suffix. A listed name is resolved and refused if any answer is a loopback, link-local, private, reserved or host-mapped address; compose peers (`EGRESS_INTERNAL`) must resolve inside the compose network; the proxy connects to the address it vetted and never looks the name up twice; only the listed ports open; every refusal is logged and, with `EGRESS_LEDGER` set, written to a ledger that outlives the container | A credential read out of a tool's own environment and posted to an address nobody listed — measured before this existed, a Bash step inside the investigator could read the provider key and reach anywhere; a listed name that resolves to the host or the LAN; a name used as a DNS exfiltration channel, since the list is checked before any lookup; a tunnel to 22 or 5432 | A listed host that accepts attacker-chosen bodies: the model gateway takes POST bodies and anyone can create an S3 bucket, so the shapes narrow the door without closing it; what a permitted destination returns still reaches the model; on a deployment without the internal network a client that ignores the proxy variables goes direct | `deploy/sidecars/proxy.py` (`permitted`, `decide`, `resolve`, `connect`), `deploy/docker-compose.work.yml` (`probe_net`, `EGRESS_ALLOW`, `EGRESS_INTERNAL`) |
| **The watcher's signature is held where no agent runs** — `watch-signer` holds `WATCH_INGEST_SECRET`; the watcher posts its signal unsigned to it, and it signs only a signal naming a conversation the scanner offered THIS round, with a level and kind from a closed set, text cut to a length, and a ceiling per round | A round steered by a colleague's message manufacturing a request from a conversation nobody scanned — at `high`, as a `task`, which buys a paid planner run and a card the operator reads as real. Until 2026-09-30 the secret sat in a file the agent could read, so the door's only check answered yes to whatever the round chose to say | A round that read a real conversation and describes it dishonestly: the offer bounds WHICH conversations can carry a signal, not whether the signal is true. Nothing here is a judgement about content, and a deployment with no prescan states no offer, which is passed through rather than read as an offer of nothing. The producer half of an origin is a closed set only where `WATCH_SIGNER_PRODUCERS` names one (the work compose does: the chat tool, Jira, the scanner); the scanner's own `scanner-notes` subject is admitted without an offer at `low`/`note`, so a round can always post one cheap note; and the offer is this round's or the round before it, so a run that outlasts a tick is checked against what it was handed, one round wide. Its ledger is also the record of what was reported (conversation and offered cursor); `scripts/assert_watch_signed.py` reads it against the pipe's ledger once per tick, and a signal in the pipe with no signer row is the one thing it pages about | `deploy/sidecars/signer.py`, `scripts/post_watch_signal.py` |
| **Tool gate reachability** — before its first turn a node requires its gate to refuse a call no posture permits, and after every turn it requires the flight recorder to show the gate was actually asked; failing either stops the node | A node running with no gate at all while reporting one. Not hypothetical: the first live run of a spawned-gate adapter had no gate, because the hook command could not import hookprobe — codex logged it and carried on, and a `kubectl delete` ran to completion on a node whose /v1/agent said `bash_guard: readonly`. The second half catches the other shape of the same failure: a gate that answers perfectly and is never consulted, which is what driving codex through its app-server does | Anything that can write into the run's own volume. Codex trusts hooks only after a person approves them and `exec` has nobody to ask, so the adapter passes `--dangerously-bypass-hook-trust` and vets the source by writing it: whoever else can put a file in that directory installs a hook that runs as the probe | `hookprobe/gate.py` (`verify`), `engine_codex.py`, `engine_pi.py` |
| **Budget breaker and retry caps** — window spend ≥ ceiling refuses new autonomous runs; one resume per run after a restart and one automatic retry per turn after a transient provider error, both under the breaker | An unattended loop spending without limit; a crash loop becoming a spend loop — a process that dies on every boot buys one continuation, not one per restart forever | Operator-driven doors, which are never gated; and the first duplicate charge: each cap allows exactly one more attempt, which is the point — what it bounds is the loop, not the pair | `hookprobe/service.py` (`_MAX_RESUMES`, `_MAX_AUTO_RETRIES`) |
Expand Down
10 changes: 10 additions & 0 deletions docs/deployments.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,16 @@ offered that round, with a level and kind from a closed set, text cut to a
length and a ceiling per round. The refusals are in
`work-data/watch-signer/signals.jsonl`.

**The pipe's ledger is a bind mount, and only the container opens it.**
`work-data/hookrelay/hookrelay.db` runs in WAL (`hookrelay/store.py`) on a
directory the desktop container runtime shares from the host, and WAL's
shared-memory file is not coherent across that boundary: a sibling deployment
on this machine records it as a silent-corruption hazard and runs its databases
in rollback mode. Nothing on the host opens this one while the pipe runs, and
that is the rule rather than a habit. Read it with the container stopped and
the file opened immutably (`?immutable=1`); the integrity check of 2026-10-19
was clean at 446 events.

**What a review of those two days found, fixed on 2026-10-05.** The timer's
contract-violation signal had signed from the file the watcher's copy was
deleted from, so every violation since the redeploy failed to post with one
Expand Down
Loading