fix(coganchor): keep an agent's own runtime local, and read a command as a command

.gitignore

@@ -223,3 +223,6 @@ __marimo__/
 
 .humanize/
 *.trace.json
+
+# The concurrency rig writes its workspaces, mock target and results here.
+bench/*/lab/

bench/coganchor-concurrency/README.md

@@ -0,0 +1,79 @@
+# How many anchored agents fit on one machine
+
+A rig for the one question humanize deliberately refuses to answer for you. [Many turns at
+once](../../docs/features/concurrency.md) says it plainly:
+
+> **How wide it runs is a question about the machine**, not about this library, so nothing
+> caps it.
+
+This measures the machine. It runs real `hmz`, with real coganchor interception, against a
+mock controlled end, and climbs the concurrency ladder until something stops behaving.
+
+## Table of Contents
+
+- [What is real and what stands in](#what-is-real-and-what-stands-in)
+- [Install](#install)
+- [Usage](#usage)
+- [What a rung means](#what-a-rung-means)
+- [Results](#results)
+
+## What is real and what stands in
+
+| | |
+| --- | --- |
+| **Real** | `hmz` itself: the backends, the argv they build, the anchor, the ptrace supervisor, the mirror, the protocol, and the agent CLIs — `claude`, `codex`, `grok`, `kimi`, `dsh` — as installed. |
+| **Stood in for** | The target's *data*: `hmz anchor serve` really serves, over a real TCP channel, but the workspace it serves is synthetic. |
+| **Stood in for** | The model provider: one local server answering the Anthropic Messages, OpenAI Responses and OpenAI Chat shapes with a scripted turn. |
+
+Both stand-ins run **outside** the constrained cgroup, deliberately. In production the target
+is another machine and the model is somebody else's API, so neither belongs in the budget
+being sized. `ramp.py` records the target's own CPU alongside the measurement, so a run can
+say whether the stand-in was anywhere near its own limit — if it was, the ceiling found is a
+fact about the rig rather than about the machine.
+
+Every backend is given the **same** scripted turn — three shell commands, each reading a
+seeded file and appending to another — so the numbers compare backends rather than prompts.
+A turn counts only if the agent said the sentence the script ends on *and* its work is on the
+target, which is what distinguishes a turn that ran from one that merely reported.
+
+## Install
+
+Needs `uv`, the five agent CLIs on `PATH`, and `sudo` for the cgroup:
+
+```sh
+npm install --global @xai-official/grok @moonshot-ai/kimi-code @deepseek-ai/dsh
+```
+
+## Usage
+
+```sh
+bash backdrop.sh start                 # stand-ins up, outside the cgroup
+bash ladder.sh claude                  # climb until two rungs in a row misbehave
+RUNGS="256 320 384" APPEND=1 bash ladder.sh codex
+bash refine.sh codex 208 224 240       # narrow a ceiling, keeping stderr
+python3 summarise.py                   # the table
+bash backdrop.sh stop
+```
+
+`run_one.sh <backend> <n>` is one rung on its own. The cgroup is the whole of the constraint:
+
+```sh
+sudo systemd-run --scope -p AllowedCPUs=0-7,112-119 -p MemoryMax=64G -p MemorySwapMax=0 ...
+```
+
+`AllowedCPUs` names eight physical cores **and their hyperthread siblings** — sixteen logical
+CPUs, which is what a 16-vCPU machine is, and what `nproc` reports inside the scope. Naming
+`0-15` instead would quietly hand the benchmark sixteen *physical* cores.
+
+## What a rung means
+
+One rung is N agents in one `hmz` process, one session each, one turn each, all going at
+once — the fan-out shape the concurrency guide describes, aimed at an anchor. Each agent gets
+a workspace of its own, mirrored from a copy of its own on the mock target.
+
+A rung is **normal** when every agent finished, said the sentence, and left its work on the
+target. The ladder stops after two consecutive rungs that are not.
+
+## Results
+
+See [RESULTS.md](RESULTS.md).

bench/coganchor-concurrency/RESULTS.md

@@ -0,0 +1,109 @@
+# How many anchored agents fit in 16 CPUs and 64 GiB
+
+Measured 2026-08-29 on an AMD EPYC 7B13, confined to eight physical cores and their
+hyperthread siblings (sixteen logical CPUs, which is what `nproc` reports inside the scope)
+and 64 GiB with no swap. Real `hmz`, real coganchor, real agent CLIs; the target's data and
+the model provider stood in for, both outside the cgroup. See [README](README.md) for what
+that means exactly.
+
+Every backend ran the same turn: three shell commands, each reading a seeded file on the
+target and appending to another. A turn counts only if the agent said the sentence the script
+ends on **and** its work is on the target.
+
+## The short answer
+
+There are three different numbers, and conflating them is how this question gets answered
+wrongly.
+
+| | claude | codex | grok | kimi | dsh |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| **Runs at full speed** (p95 within 2× of one agent alone) | **24** | **24** | **16** | **16** | **16** |
+| **Best throughput** (turns/minute peaks here) | **48** | **128** | **48** | **64** | **32** |
+| — turns/min there | 336 | 583 | 274 | 116 | 444 |
+| **Still all-correct** (everything lands, just slowly) | **640** | **256** | **>832** | **192** | **>832** |
+| What stops it there | 64 GiB | its own app-server | not found | 64 GiB | not found |
+
+**Before any of that, on a stock machine, everything stops at about 320** — see
+[The first ceiling is descriptors](#the-first-ceiling-is-descriptors).
+
+So: if you want turns to run at the speed they run at alone, **16–24 anchored agents** is the
+honest number for a 16-CPU box. If you want the most work done per hour and do not care that
+each turn takes longer, **32–128**. Past that you are only lengthening the queue: claude's
+throughput peaks at 336 turns/min with 48 agents and has fallen to 222 with 640.
+
+## Cost per agent
+
+| | claude | codex | grok | kimi | dsh |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| memory | 102 MiB | 27 MiB | 44 MiB | **323 MiB** | 68 MiB |
+| processes | 23 | **45** | **60** | 13 | 16 |
+| CPU per turn | 2.0 s | 0.9 s | 2.8 s | 3.6 s | 1.1 s |
+| one turn, alone | 2.8 s | 1.3 s | 2.4 s | 4.5 s | 1.1 s |
+
+Memory is what decides kimi and claude; nothing else got near 64 GiB. Kimi's 323 MiB is its
+`kimi web` daemon, which humanize starts one of per agent.
+
+## The first ceiling is descriptors
+
+**hmz holds about three file descriptors per concurrent anchored agent.** A stock login has
+a soft `RLIMIT_NOFILE` of 1024, so the first wall anybody meets is at roughly **320 agents**,
+whatever the backend and however much RAM is free. It arrives as `[Errno 24] Too many open
+files` from dsh, and as a bare exit from codex, which is the same thing seen from further
+away.
+
+With the soft limit raised to the hard one, the same rung goes from 329/384 to **384/384**.
+Everything above is measured with it raised.
+
+```sh
+ulimit -n 1048576     # or LimitNOFILE in the unit that runs hmz
+```
+
+## Where each one actually stops
+
+- **claude — 640, on memory.** 640 agents used 63.74 of the 64 GiB and every turn still
+  landed. That is the last rung measured; there is 0.26 GiB of headroom left at it, so the
+  next one up was not attempted rather than shown to fail.
+- **kimi — 192, on memory.** 192 used 60.5 GiB; 208 pinned the cgroup at exactly 64.00 GiB
+  and the OOM killer took daemons out, which the agents saw as `Remote end closed connection
+  without response`. 42 of 208 failed.
+- **codex — 256, on codex.** Not memory (6.9 GiB), not descriptors, not the target: at 320 it
+  loses 13 agents to `app server stopped mid-turn`, at 512 it loses 96, and the ones that die
+  die at ~22 s while the survivors take ~45 s. The codex app-server gives up when the machine
+  is oversubscribed. It is also the one backend sensitive to being run in a tight loop: 256
+  passes cleanly on its own and loses one agent when it follows a 192-agent rung immediately,
+  because several thousand processes from the previous rung are still going away.
+- **grok and dsh — no ceiling found.** Both did 832 of 832 with nothing failing, at 36 GiB
+  and 55 GiB. 832 was the rig's slot count, not the machine's limit. Turns take a long time
+  there — 9 minutes for grok, 10 for dsh — but they all land.
+
+## Was the stand-in the bottleneck?
+
+No. The mock controlled end never exceeded **3%** of the 208 CPUs it had to itself, and was
+under 1% for most runs. The measured cgroup sat at 80–87% throughout. Every ceiling above is
+a fact about the 16-CPU machine or about the agent, not about the rig.
+
+## Three things this found in coganchor
+
+None of codex, kimi or grok could take an anchored turn at all before these. Each is a
+separate way for the agent's own runtime to end up on the wrong machine, and each looks like
+a backend problem until you look.
+
+1. **Argv entries were read with `PATH_MAX` as the ceiling.** An argv entry is not a path;
+   the kernel allows `MAX_ARG_STRLEN`. Codex prefixes every `bash -lc` with a preamble longer
+   than 4 KiB, so what reached the target was the first 4096 bytes of the command — which
+   parses, runs, and means something else. Codex reported every tool call as successful while
+   nothing happened on the target.
+2. **`#!/usr/bin/env node` sent the agent to the target.** `env` runs here and then searches
+   `PATH` for the interpreter, one `execve` per directory. The first candidate names a path
+   that does not exist here, which is not the agent's own by name — so it was run on the
+   target, where the name resolves. Kimi's entire agent process ran on the target, read the
+   target's `HOME`, and could not find the account it was signed in with. Codex escaped this
+   only because its runtime is listed by hand.
+3. **A binary in the agent's own state directory was kept local as a path but not as a
+   program.** grok installs its native binary under `~/.grok/bin` and re-execs it; that exec
+   went to the target, and grok reported `Not signed in`.
+
+## Raw data
+
+`data/ladder-*.jsonl` and `data/refine-codex.jsonl` hold one line per rung.
+`data/stock-nofile/` holds the first climb, before the descriptor limit was raised.

bench/coganchor-concurrency/backdrop.sh

@@ -0,0 +1,127 @@
+#!/bin/bash
+# Everything the measurement needs that is NOT part of what is being measured:
+#
+#   * the stand-in model provider, which in production is a vendor's API
+#   * `hmz anchor serve`, which in production is the target machine
+#
+# Both are started outside the constrained cgroup, deliberately: the question is how many
+# anchored agents fit in 16 CPUs and 64 GiB, not how many agents plus the machine they are
+# working on fit there.
+set -u
+
+TMP=$(cd "$(dirname "$0")" && pwd)
+LAB=${LAB:-$TMP/lab}
+ROOT=$(cd "$TMP/../.." && pwd)
+MODEL_PORT=${MODEL_PORT:-18081}
+TARGET_PORT=${TARGET_PORT:-18090}
+SLOTS=${SLOTS:-192}
+# Where the agent CLIs are. `command -v` only finds them if this shell already has them on
+# PATH, which it need not; $AGENT_BIN says so outright.
+AB=${AGENT_BIN:-}
+if [ -z "$AB" ]; then
+  _kimi=$(command -v kimi || true)
+  [ -n "$_kimi" ] && AB=$(dirname "$_kimi")
+fi
+AB=${AB:-$HOME/.local/agents/bin}
+[ -x "$AB/kimi" ] || { echo "no kimi under $AB; set AGENT_BIN"; exit 1; }
+
+stop() {
+  for pid in $(pgrep -f "standin_model[.]py" || true); do kill "$pid" 2>/dev/null || true; done
+  for pid in $(pgrep -f "anchor serve --listen" || true); do kill "$pid" 2>/dev/null || true; done
+  sleep 1
+}
+
+case "${1:-start}" in
+stop) stop; echo "backdrop down"; exit 0 ;;
+esac
+
+stop
+mkdir -p "$LAB/ws" "$LAB/tgt" "$LAB/homes" "$LAB/logs" "$LAB/runs"
+
+# ---------------------------------------------------------------- the stand-in provider
+STANDIN_LOG="$LAB/logs/standin.log" nohup python3 \
+  $TMP/standin_model.py "$MODEL_PORT" \
+  > "$LAB/logs/standin.out" 2>&1 &
+sleep 1
+curl -sS "http://127.0.0.1:$MODEL_PORT/v1/models" > /dev/null || { echo "standin failed"; exit 1; }
+
+# ---------------------------------------------------------------- the mock controlled end
+# One listener serves every session; its export table is fixed at start, so a slot for each
+# concurrency level this rig will ever reach is declared up front.  The directories behind
+# them are wiped and reseeded per run by ramp.py.
+EXPORTS=()
+for i in $(seq 0 $((SLOTS - 1))); do
+  EXPORTS+=(--export "$LAB/ws/$i:$LAB/tgt/$i")
+done
+cd "$ROOT"
+# Several listeners rather than one.  A listener is a Python process holding a thread per
+# session, and one of those would eventually be the thing that runs out -- which would make
+# the ceiling a fact about the stand-in rather than about the machine running hmz.
+LISTENERS=${LISTENERS:-4}
+for n in $(seq 0 $((LISTENERS - 1))); do
+  port=$((TARGET_PORT + n))
+  nohup ./.venv/bin/python -m hmz anchor serve --listen "127.0.0.1:$port" "${EXPORTS[@]}" \
+    > "$LAB/logs/target-$port.out" 2>&1 &
+done
+sleep 8
+for n in $(seq 0 $((LISTENERS - 1))); do
+  port=$((TARGET_PORT + n))
+  grep -q "listening" "$LAB/logs/target-$port.out" || {
+    echo "target listener on $port failed:"; cat "$LAB/logs/target-$port.out"; exit 1; }
+done
+
+# ---------------------------------------------------------------- one HOME per backend
+# Shared across that backend's concurrent agents, which is how humanize runs on a real
+# machine: many agents, one user, one state directory each CLI keeps its sessions in.
+BASE="http://127.0.0.1:$MODEL_PORT"
+
+for backend in claude codex grok kimi dsh; do
+  H="$LAB/homes/$backend"
+  rm -rf "$H"; mkdir -p "$H"
+done
+
+# codex: a provider in its own home, so humanize's `codex app-server` finds it.
+mkdir -p "$LAB/homes/codex/.codex"
+cat > "$LAB/homes/codex/.codex/config.toml" <<TOML
+model = "standin-1"
+model_provider = "standin"
+approval_policy = "never"
+sandbox_mode = "danger-full-access"
+
+[model_providers.standin]
+name = "standin"
+base_url = "$BASE/v1"
+wire_api = "responses"
+env_key = "OPENAI_API_KEY"
+TOML
+
+# grok: the same, plus a symlink to the already-unpacked binary so no session pays to
+# decompress a 166 MB payload into a cold home.
+mkdir -p "$LAB/homes/grok/.grok/bin"
+ln -sf /home/ubuntu/.grok/bin/grok-1.0.13 "$LAB/homes/grok/.grok/bin/grok"
+# And a `grok` on PATH that *is* the binary rather than the npm trampoline.  Anchored, the
+# trampoline's exec of the real binary is routed to the target -- it is not on the list of
+# programs coganchor keeps here as the agent's own runtime, the way Codex's are -- and grok
+# then reports "Not signed in".  A tarball install looks exactly like this, so measuring
+# through it measures grok rather than the trampoline.
+mkdir -p "$LAB/bin"
+ln -sf /home/ubuntu/.grok/bin/grok-1.0.13 "$LAB/bin/grok"
+cat > "$LAB/homes/grok/.grok/config.toml" <<TOML
+[cli]
+installer = "npm"
+
+[endpoints]
+xai_api_base_url = "$BASE/v1"
+cli_chat_proxy_base_url = "$BASE/v1"
+TOML
+
+# kimi: imported through its own non-interactive provider command, which is the only way in.
+env -i HOME="$LAB/homes/kimi" PATH=/usr/bin:/bin:/usr/local/bin \
+  "$AB/kimi" provider add --api-key standin-key "$BASE/api.json" > "$LAB/logs/kimi-provider.log" 2>&1
+printf 'default_model = "standin/standin-1"\n' > "$LAB/homes/kimi/.kimi-code/head.toml"
+cat "$LAB/homes/kimi/.kimi-code/head.toml" "$LAB/homes/kimi/.kimi-code/config.toml" \
+  > "$LAB/homes/kimi/.kimi-code/config.new"
+mv "$LAB/homes/kimi/.kimi-code/config.new" "$LAB/homes/kimi/.kimi-code/config.toml"
+grep -q "standin" "$LAB/homes/kimi/.kimi-code/config.toml" || { echo "kimi provider import failed"; exit 1; }
+
+echo "backdrop up: model on $MODEL_PORT, $LISTENERS mock targets from $TARGET_PORT, $SLOTS slots"