Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 45 additions & 9 deletions .claude/commands/provision.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,23 +206,59 @@ If chosen in the interview — and **after** halting the loop, so an in-flight d
If chosen in the interview — this gives the human's **admin** account zero-inbound-exposure SSH
(Cloudflare Tunnel + Access + short-lived certs); it is **never** a path into the agent user, whose
no-`authorized_keys` invariant from Phase 2 stands:
- Walk the runbook top to bottom; on native Linux its WSL2-specific gotchas (the `loopback0` ufw rule,
`localhostForwarding`) drop out. Almost everything is human-terminal or Cloudflare-dashboard work
(sshd config, `cloudflared` install + tunnel, the Access app and SSH CA) — print the blocks, then verify.
- Interplay with Phase 7's egress allowlist: `cloudflared` runs as its own system user, not the agent
user, so the UID-matched nftables chain does not (and must not) allowlist anything for it.
- **Read the runbook's top banner before starting.** Its steps 4–5 are superseded: Cloudflare retired
the per-application SSH CA, so `cloudflared access ssh-gen` cannot mint a certificate and the
documented client config cannot work. Steps 1–3 (hardened loopback-only sshd, dedicated tunnel,
Access application) are still correct.
- **Set expectations with the human up front**, because the successor changes the deal: Access for
Infrastructure requires the **WARP client on every device** in Traffic + DNS mode, uses a private
network route instead of the public hostname, and offers **no browser-rendered terminal**. That is a
device-level posture change, not a config detail — get a deliberate yes before building toward it,
and offer stopping with the server side complete as a legitimate outcome. Everything server-side
(CA, sshd, tunnel) carries forward whenever they resume.
- Prefer Access for Infrastructure over the self-managed-keys path: the latter needs WARP *too* while
reintroducing long-lived `authorized_keys`, reversing the property the design exists for.
- **Verify the CA fingerprint matches the account gateway CA** (`ssh-keygen -lf`, compared against
`GET /accounts/$ID/access/gateway_ca`). On a live run the dashboard-pasted key differed from the
signing CA, and a cert signed by an untrusted CA fails exactly like a client-side fault.
- `cloudflared` should run as a **system** unit under its own unprivileged user, not a `--user` unit:
the admin account often has `Linger=no`, and remote access must not depend on a login session —
it is the recovery path for a box you cannot otherwise reach. Use `Type=simple`; sd_notify support
varies by build, and a wrong `Type` presents as a unit stuck in `activating`.
- Interplay with Phase 7's egress fence: `cloudflared` runs as its own system user, not the agent
user, so the UID-matched nftables chain does not (and must not) apply to it.
- Watch for a **pre-existing DNS record** on the chosen hostname — often an orphan from the
decommissioned box, since proxied records hide their target from `dig`. Check the dashboard, then
`--overwrite-dns`. List the account's other tunnels too (`cloudflared tunnel list`): each is a
standing ingress path, and stale ones are the same hygiene issue as unrotated tokens.
- Verify: `sshd -T` shows loopback-only `ListenAddress`, `PasswordAuthentication no`,
`PermitRootLogin no`, and `AllowUsers` limited to the admin (+ email-local-part alias); the
`cloudflared` tunnel unit is active; the human confirms a real login from another device lands as
the admin account; `~<agent_user>/.ssh/authorized_keys` still does not exist.
`cloudflared` tunnel unit is active with registered edge connections; `ss -tln` shows **no**
non-loopback listeners; the human confirms a real login from another device lands as the admin
account; and `/etc/ssh/principals/` plus `~<agent_user>/.ssh` confirm the agent gained no SSH path.

## Phase 9 — final verification (`9-verify`)

Walk HARDENING.md's **"Checklist deltas"** for the worked example plus:
- One loop tick landed in `.claude/state/loop-ticks.jsonl` (or the census explains why not).
- A test notification arrived through the `notify` seam.
- The old machine: its loop/daemons decommissioned and its tokens rotated (Phase 3).
- A test notification arrived through the `notify` seam — **triggered, not assumed**. Force a real
blocked-egress event and confirm the human's phone actually buzzed.
- `ss -tln` shows no non-loopback listeners.
- The old machine: its loop/daemons decommissioned and its tokens rotated (Phase 3); any stale
Cloudflare tunnels on the account deleted.
- If remote SSH was set up (Phase 8): the runbook's §6 "Verify the security posture" block passes.
- **`claude-rc-<slug>.service` is a known trap.** `arm-loop.sh` passes `--spawn` to avoid the
first-run question, but the server still prompts `Enable Remote Control? (y/n)` inside its tmux
session — so the unit reports `active` while sitting on a prompt, doing nothing. `systemctl` cannot
see this; check with `tmux capture-pane -p -t rc-<slug> | tail`. Either answer it (and accept that
anyone with the human's Anthropic account can drive the agent) or
`systemctl --user disable --now claude-rc-<slug>.service`.

**Before signing off, state the unattended posture plainly.** If the human is leaving the box alone,
walk through the combination they are actually leaving running: the loop's `stop_after_days` horizon,
whether remote access exists, and what the only feedback channel is. A loop armed indefinitely on a
box with no remote shell and one notification channel is a decision worth naming out loud — recommend
a short horizon (`--stop-after-days 14`) so it fails safe, and let them overrule it.

## Report

Expand Down
2 changes: 1 addition & 1 deletion .claude/scripts/notify.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
# - positional args $1/$2/$3 = severity/title/body-line
# - env vars NOTIFY_SEVERITY / NOTIFY_TITLE / NOTIFY_BODY = the same three
# Example commands (all empty by default; pick ONE in your adapter):
# ntfy: "curl -s -d \"$NOTIFY_BODY\" -H \"Title: $NOTIFY_TITLE\" -H \"Priority: $NOTIFY_SEVERITY\" ntfy.sh/<your-topic>"
# ntfy: "curl -s -d \"$NOTIFY_BODY\" -H \"Title: $NOTIFY_TITLE\" -H \"Priority: $NOTIFY_SEVERITY\" https://ntfy.sh/<your-topic>"
# notify-send: "notify-send \"$NOTIFY_TITLE\" \"$NOTIFY_BODY\""
# webhook curl: "curl -s -X POST -H 'Content-Type: application/json' -d \"{\\\"severity\\\":\\\"$NOTIFY_SEVERITY\\\",\\\"title\\\":\\\"$NOTIFY_TITLE\\\",\\\"body\\\":\\\"$NOTIFY_BODY\\\"}\" https://example.invalid/hook"
#
Expand Down
40 changes: 36 additions & 4 deletions docs/HARDENING.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,10 @@ sudo gpasswd -d "$USER" docker # if you don't need Docker in this distro

### Worked example: dedicated Linux server (maximum containment)

> **Working files:** [`examples/dedicated-server/`](../examples/dedicated-server/) carries the
> nftables fence, egress alarm, auditd rules, divergence tripwire and systemd drop-in from a real
> run of this section on bare-metal Ubuntu 26.04, with the reasoning behind each choice.

> **Guided:** run **`/orchestrator:provision`** on the new box to be walked through this section
> interactively — an interview, phase-by-phase checkpoints with verification, and resumable progress in
> `.claude/state/provision-progress.json`. This section stays the source of truth; the command executes it.
Expand Down Expand Up @@ -465,13 +469,32 @@ loop needs — the Anthropic API, GitHub, your notifier — and **log the drops*
# /etc/nftables.d/recode-agent.nft — adjust the resolver/set mechanics to taste;
# domain-based filtering needs a resolving frontend (e.g. a local proxy) or periodic set refresh.
table inet recode_agent {
set allowed_v4 { type ipv4_addr; }
set allowed_v6 { type ipv6_addr; }

chain output {
type filter hook output priority 0;
meta skuid != "recode-agent" accept
type filter hook output priority 0; policy accept;
# JUMP on a positive UID match — do NOT write `meta skuid != <uid> accept`.
# Kernel-generated packets (ICMPv6 MLD/ND, DHCP renewal) have NO owning
# socket, so `skuid != N` never matches them and they fall through to the
# drop below — silently filtering the host's own network stack. The symptom
# arrives days later as a lease that won't renew or IPv6 quietly degrading,
# with nothing to connect it back to this file.
meta skuid 1001 jump agent_out # numeric uid; `id -u <agent>`
}

chain agent_out {
oifname "lo" accept
ct state established,related accept
# Link-local multicast is local discovery noise: deny WITHOUT the log
# prefix, or mDNS/MLD will page you forever and the alarm stops being read.
ip daddr 224.0.0.0/4 drop
ip6 daddr ff00::/8 drop
udp dport 53 accept # DNS (or pin to a local resolver)
tcp dport 443 ip daddr @allowed_v4 accept # populate from api.anthropic.com, github.com, ntfy.sh
counter log prefix "recode-agent-egress-drop " drop
tcp dport 53 accept
ip daddr @allowed_v4 tcp dport 443 accept # refreshed from DNS on a timer
ip6 daddr @allowed_v6 tcp dport 443 accept
counter log prefix "recode-agent-egress-drop " level warn drop
}
}
```
Expand All @@ -496,6 +519,15 @@ Three things the sketch above glosses over, learned the hard way:
would rather not run a resolver-refresh loop, the alternative is coarse — permit DNS and TCP 443 to
*any* destination — but understand that this stops odd ports and non-HTTPS exfil only, and gives you
no meaningful exfiltration alarm, which is most of the value.
- **Resolve the hosts the agent actually contacts, not the ones you assume.** `statsig.anthropic.com`
(in an earlier version of this list) has **no A record**; Claude Code's feature-flag traffic goes to
`api.statsig.com` / `statsigapi.net` / `events.statsigapi.net` / `featureassets.org`, on Google
Cloud. Leaving them out is defensible — the loop works without telemetry — but it pages you forever,
and an alarm that cries wolf gets ignored. Decide deliberately; don't discover it as noise.
- **Alerts must be readable and de-duplicated.** A raw kernel log line (`IN= OUT= SRC= DST= LEN=…`)
is unreadable on a phone. Parse `DST`/`DPT`/`PROTO`, reverse-resolve the address, and suppress
repeats per destination (15 minutes works) — otherwise one blocked endpoint produces dozens of
identical pushes and the channel becomes noise.

**6. Detection.** Single-purpose boxes make auditing cheap:
- `auditd` watches on `.claude/scripts/`, `.claude/self/`, `.env`, and the unit files — any write
Expand Down
59 changes: 57 additions & 2 deletions docs/REMOTE_SSH_RUNBOOK.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,50 @@
# Remote SSH into a WSL2 box — Cloudflare Tunnel + Access runbook

> ## ⚠️ Steps 4–5 are SUPERSEDED — read this first (verified 2026-08)
>
> Cloudflare has **retired the per-application SSH CA** this runbook depends on. On a current
> account, *Access → Service auth → SSH → Generate certificate* is **disabled**, and
> `cloudflared access ssh-gen` fails with `Bad request, please create CA for application` — the
> client never obtains a certificate, so nothing on the server side can fix it. Confirmed with the
> account CA correctly trusted (see below); the failure is upstream of `sshd`.
>
> **Steps 1–3 remain correct and worth doing** — loopback-only hardened `sshd`, the dedicated
> outbound-only tunnel, and the Access application in front of the hostname.
>
> The successor is **Access for Infrastructure**, which is a different topology:
> - The Cloudflare One Client (**WARP**) is required on *every* client device, in Traffic + DNS mode.
> - Connectivity uses a **private network route** (Networking → Routes → Tunnel CIDR), not the public
> hostname ingress in step 2.
> - You register a **target** (Access controls → Targets: hostname + IP + virtual network) and create
> an **Infrastructure application** (protocol SSH, port 22) whose policy lists the exact UNIX
> usernames each person may log in as.
> - Clients then use plain `ssh user@<target-ip>` — no `ProxyCommand`, no `ssh-gen`. `scp`/`sftp`/
> `rsync` work too, which the browser terminal never allowed.
> - **No browser-rendered SSH terminal** is offered for infrastructure apps.
>
> The account-wide CA public key comes from *Access controls → Service credentials → SSH →
> Add a certificate → Generate SSH CA*, or over the API:
>
> ```bash
> curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/access/gateway_ca" \
> --request POST --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" | jq -r .result.public_key
> ```
>
> **Verify the CA you install is the one Cloudflare signs with.** On a live run the key pasted from
> the dashboard did *not* match the account gateway CA (`SHA256:UdAX7Pben…` vs `SHA256:K3EDSsU0olUb…`).
> A cert signed by an untrusted CA fails identically to a client-side problem and will send you
> chasing the wrong layer. `TrustedUserCAKeys` accepts multiple CAs, one per line — trust both and
> compare with `ssh-keygen -lf /etc/ssh/cloudflare_access_ca.pub`, which must list every fingerprint.
>
> There is also a **self-managed-keys** path (Tunnel → private route → plain `authorized_keys`). It
> still requires WARP, and it reintroduces long-lived keys — reversing the "nothing long-lived to
> steal" property that motivates layer 4 below. Prefer Access for Infrastructure if you are installing
> WARP anyway.
>
> **If sshd listens on `127.0.0.1` only** (as step 1 configures), register the target as `127.0.0.1`
> so `cloudflared` — running on the same box — dials its own loopback and the property survives. The
> LAN-IP fallback needs a second `ListenAddress` and makes sshd reachable from your LAN.

> Written for WSL2 Ubuntu, but applies almost verbatim to any Linux box — the WSL2-specific
> gotchas (the `loopback0` ufw rule, the `localhostForwarding` note) simply drop out on native
> Linux. Every gotcha below was hit for real on a live deployment.
Expand Down Expand Up @@ -139,8 +184,11 @@ Zero Trust dashboard → **Access → Applications → Add → Self-hosted**:

## 4. *(dashboard + terminal)* Short-lived certificates

Zero Trust dashboard → **Access → Service auth → SSH**: select the `ssh.example.com` app and
**Generate certificate**. Copy the public key it shows, then back on the box:
> **SUPERSEDED** — *Service auth → SSH → Generate certificate* is disabled on current accounts. Get
> the **account-wide** CA from *Access controls → Service credentials → SSH*, or the `gateway_ca` API
> call in the banner at the top of this file. The `sudo tee` step below is still exactly right; only
> where the key comes from has changed. Verify with `ssh-keygen -lf` that the fingerprint matches the
> account CA before assuming a client-side fault.

```bash
sudo tee /etc/ssh/cloudflare_access_ca.pub > /dev/null <<'EOF'
Expand All @@ -155,6 +203,13 @@ systemctl --user enable --now cloudflared-ssh.service

## 5. Clients

> **SUPERSEDED** — everything in this section depends on `cloudflared access ssh-gen`, which no
> longer mints certificates (see the banner). Kept for the historical record and because the
> `Match exec` reasoning is still the correct shape *if* you ever have a working short-lived-cert
> source. On a current account, clients use WARP + Access for Infrastructure and connect with plain
> `ssh user@<target-ip>`, with allowed usernames set in the application policy rather than in
> `AuthorizedPrincipalsFile`.

**Laptop** (install `cloudflared` there first). GOTCHA: a bare
`ProxyCommand cloudflared access ssh` only proxies the TCP stream — it never fetches the
short-lived cert, and since sshd trusts ONLY the Cloudflare CA the result is
Expand Down
Loading
Loading