Local: a stage stopped by a signal writes its spend line once more

docs/LOCAL.md

@@ -19,7 +19,7 @@ Factory's worker and reviewer, run on your own machine instead of a GitHub runne
 - The harness holds no GitHub token at all: what it needs from GitHub is in `$FACTORY_IN`. Factory's own calls use one fine-grained token for this one repository, never your `gh` login; the token reaches each stage on a pipe it reads once, never its environment or a file, and Factory's own git gets it by per-process env config. No sandbox can read `/proc` (Claude Code and `srt`; Codex runs in its own pid namespace). Copilot's sandbox runs each command in a pid namespace of its own too, whose pid 1 is the bubblewrap Copilot starts with its whole environment, the token included (#100); on Linux Factory's own `bwrap`, first on Copilot's PATH, starts the real one with an empty environment, and bubblewrap sets each command's environment itself. Copilot also writes each command's network rules into its own `TMPDIR` before it loads them, and opens that directory to every command, so a command could rewrite the next one's rules and reach past Copilot's proxy to `srt`'s (whose list adds Copilot's API hosts). On Linux Copilot therefore runs with a `TMPDIR` of its own, in the stage's usage directory, made afresh before every start and denied to its commands, and writes no logs (Copilot opens its log directory to every command whatever its policy says); Factory's `bwrap` gives each command the stage's temp directory as its `TMPDIR`. Copilot's sandbox binds only the system's own directories (`/usr/bin`, `/usr/lib`, `/etc` and the like) into each command, so a node your version manager keeps in your home (nvm, mise, asdf, Volta) would be out of its commands' reach (#111). The stage's node is the first `node` on your PATH, by its real path (where that is a version manager's shim, the node the shim chose, which runs Factory). Its install prefix (`bin`'s parent, so npm comes too) is read-only in each command's sandbox, with its `etc` (a global npmrc) denied, and Factory's `bwrap` puts its `bin` first on each command's PATH, after Factory's git. Nothing else of your home is bound. The prefix is never your home or a directory above it, nor in or above a credential store, this repository's `.factory` directory or the stages' temp directories, where a sandbox could swap it for a link. The same holds for the repository's package manager, and for Claude Code too, whose sandbox likewise shows nothing of your home but its working and added directories (#137): a repository whose `packageManager` (else its lockfile) names pnpm or yarn gets the one your PATH resolves to (corepack's shim, its own install, or, behind a version manager's shim, the version `packageManager` pins in that manager's installs: mise's and asdf's `installs/pnpm/<version>`, Volta's `tools/image/pnpm/<version>`; Factory never runs the manager, which would read the clone's own configuration), linked from a `toolbin` directory in the stage's usage directory, which no sandbox may write, and first on each command's PATH (Copilot through Factory's `bwrap`, Claude Code through `CLAUDE_ENV_FILE`). Only the npm package it belongs to (corepack, pnpm) is readable, read-only, else the executable alone; corepack's downloads live in the Objective's own `COREPACK_HOME`. Codex's sandbox reads the whole system but the credential stores, so its commands find both as you do. Before every Copilot start a probe checks all of this: a Copilot session in the outer sandbox, with the stage's policy and no credential, against a model Factory serves on that sandbox's own loopback (no network, no bill), runs one command. That command must find a random marker Copilot holds in no environment it can read, Copilot's network rules nowhere it can list, Copilot's `TMPDIR` read-only and no Copilot log it can read, nothing it writes in your home, the probe's Copilot home or Copilot's release left there, the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused; the stage's node must be the `node` it finds, and it, the `npm` beside it and the repository's pnpm or yarn must run (else the stage Blocks with `the stage's toolchain does not run in a command`); and it must end with its pass line. A broken rule Blocks the stage (`sandbox`) before any command runs; a probe that leaves no result is tried once more, then Blocks. It takes a second or two. Claude Code on Linux gets the same probe before every start (it has no command that runs one command in its sandbox either): a Claude Code session launched as the run is, with a throwaway config directory in the usage directory, a stand-in key and no credential of yours, against a stand-in for the Anthropic API on its outer sandbox's own loopback (no network, no bill), runs one command. That command must find and run the stage's node and the repository's pnpm or yarn, find a random marker Claude Code holds in no environment it can read, leave nothing it writes in your home or the usage directory, find the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused, and end with its pass line; a broken rule Blocks the stage (`sandbox`) with the rule. `doctor.mjs --local` runs it too (`claude sandbox: start`, around a throwaway clone with your checkout's `package.json`), and names what local mode resolves: `node: v24.20.0 → /usr/bin/node`, `package manager: pnpm 12.18.0 (corepack) → …/corepack/dist/pnpm.js`; a pinned package manager it cannot resolve is a FAIL.
 - Each Objective gets its own clone under `<your clone>/.factory/<issue>/`, which is git-excluded. `serve` removes it once the issue is closed; a Blocked Objective keeps its directory so a re-label picks up where it stood. Its installed dependencies (ignored files) carry over between stages and runs, but nothing unpushed does: each stage starts on the branch exactly as GitHub has it, and removes the untracked files, uncommitted changes and stopped rebase an earlier stage or run left there. It also removes the git lock files (`.git/*.lock`, `.git/refs/**/*.lock`) a run interrupted by a reboot or a killed `serve` left, and logs each: a lock older than the machine's last boot always goes; a newer one goes only where `/proc` shows no process with its working directory or an open file in the clone (Linux; on macOS only the boot rule applies). Otherwise the lock stays and the log names it.
 - GitHub holds all state. A claim comment with a heartbeat marks an Objective a local loop is running, so the Action and other loops leave it alone.
-- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered.
+- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. A stage stopped by a signal (Ctrl-C, SIGTERM or SIGHUP, or its loop stopping) writes that reading once more as it stops, after its harness is killed, when it is more than the line shows, waiting at most 5 s for GitHub. The reading is what the harness has reported so far: Claude Code reports each model response and Copilot each usage checkpoint, so their passes killed before the next write still count; Codex reports usage only when a turn ends, so a Codex pass killed before its turn ends adds nothing for that pass. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. The stopping write records them when it landed or may have (a 5xx answer, or no answer in 5 s, which includes a connection still being opened then), so it is never counted twice; one that may have landed but did not leaves that part uncounted. When it certainly did not land (the connection refused or its host not found at once, or a refusal), nothing is recorded and the resuming stage counts that part. Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered.
 
 ## Platforms
 
@@ -63,7 +63,7 @@ Every command takes `--repo OWNER/NAME` and `--dir CLONE` too. `--help` or `-h`
 - **Hand-off:** the harness writes nothing to GitHub. It reads what Factory rendered into `$FACTORY_IN` and leaves its plan, PR text, replies, note and exit record in `$FACTORY_OUT`, both under the stage's temp directory; Factory reads them inside the same sandbox, then pushes and posts. `deliver` and `post-review` do the same for a session's own work, with the local token, else your `gh` login. A session reads the issue only through `factory issue`, and `deliver` and `post-review` check its author and last editor again, as every stage does: an issue a person who cannot push wrote or last edited is refused (deliver Blocks it, nothing pushed).
 - **Routing:** label an Objective `factory:local` and the local loop takes it, while the Action leaves it alone. `--all` (or `FACTORY_LOCAL_ALL=1`) takes every Objective. The loop's sweep (stale Running, Done) touches only the Objectives it takes, and a Running one a local loop claimed; before its Done pass removes a closed Objective's state labels under your login, it posts a `done` marker naming them, which the scorer reads as Factory's.
 - **Questions and Blocks:** an Objective whose stage ends on this machine with a Question or a Block waits for you. No `serve` tick takes it again while its stage runs, or after it ends on a GitHub read that does not show the stop yet; it starts again only after a trusted answer, the same as on the Action (re-applying its label, or marking a Question's draft PR ready). If the stop never reached GitHub (a fresh read of the issue shows it Queued), it runs again, as on the Action, and the tick line names every Objective it holds.
-- **Stopping and pausing:** stop `serve`, `run` or `review` with Ctrl-C, SIGTERM or SIGHUP (a closed terminal, a parent session ending) and its Objectives pause: they stay Running with their claims. The `factory` front end passes each signal on; the loop ends every stage process, and each stage kills its harness's whole process group before it exits, since the harness runs in a group of its own that no terminal signal reaches. A second signal a second or more after the first exits at once. The same pause happens when the machine sleeps or shuts down. The next `serve` on the same machine resumes them first, before reviews and new work, and `factory run <issue>` takes one up the same way: an open PR waiting for review goes to review; Running work starts a fresh work pass in the Objective's clone, on the branch as GitHub has it (a killed pass's session and unpushed commits are not carried over). Other machines and the Action leave a paused Objective alone; the dispatcher Blocks one only after a day away, and re-applying its label brings it back.
+- **Stopping and pausing:** stop `serve`, `run` or `review` with Ctrl-C, SIGTERM or SIGHUP (a closed terminal, a parent session ending) and its Objectives pause: they stay Running with their claims. The `factory` front end passes each signal on; the loop ends every stage process, and each stage kills its harness's whole process group and writes its spend so far (at most 5 s) before it exits, since the harness runs in a group of its own that no terminal signal reaches. A second signal a second or more after the first exits at once. The same pause happens when the machine sleeps or shuts down. The next `serve` on the same machine resumes them first, before reviews and new work, and `factory run <issue>` takes one up the same way: an open PR waiting for review goes to review; Running work starts a fresh work pass in the Objective's clone, on the branch as GitHub has it (a killed pass's session and unpushed commits are not carried over). Other machines and the Action leave a paused Objective alone; the dispatcher Blocks one only after a day away, and re-applying its label brings it back.
 - **What a start finds left over:** each stage names its own process and its harness's (pid, start time, the harness's run tag) in its usage directory while they run. Before a stage starts, and before `run`, a resume or the cleanup of a closed Objective, a harness that a dead stage of the same Objective left running (its loop killed with SIGKILL, say) is killed. Factory signals it only when the process is still the one recorded, by its start time, and its group still carries the run's tag; never by matching command lines. A stage still running on this machine is not run twice: `run` refuses and `serve` skips that Objective, naming the pid to stop. When the machine cannot tell whether the recorded pid is still that stage (its start time unreadable), it refuses too and names the record file: delete it once you have checked that pid is no Factory worker.
 - **Keeping `serve` running after the terminal closes:** `nohup` does not help, because Node resets an ignored SIGHUP when it starts. Start it in a session of its own instead: on Linux `setsid -f factory serve >serve.log 2>&1 </dev/null`; on macOS, which has no `setsid`, `factory serve >serve.log 2>&1 </dev/null &!` in zsh (`&` then `disown` in bash); anywhere, inside `tmux` or `screen`. Stop it with `kill <pid>`, which pauses its Objectives as above. On WSL2, Windows may shut the distribution down after its idle timeout once no WSL terminal is attached, and a detached `serve` does not prevent that: keep one WSL window open (with `serve` in `tmux` there, say) or raise the idle timeout in `.wslconfig`. Nothing restarts `serve` after a reboot; start it again.
 - **Claims and deadlines:**

plugin/scripts/lib/harness.mjs

@@ -446,18 +446,55 @@ export const SESSION_TOTALS_FILE = "session-totals.json";
  * counted (SESSION_TOTALS_FILE).
  */
 export async function stageSpendSoFar() {
-  let usd = STAGE_SPEND.done;
-  let estimated = STAGE_SPEND.estimated;
-  const totals = Object.fromEntries(SESSION_TOTALS);
+  // What has finished is taken before any meter is read: a pass that ends meanwhile is never counted twice.
+  const base = stageSpendBase();
+  const reads = [];
   for (const [m, { session }] of STAGE_SPEND.open) {
     const read = await m.usd().catch(() => 0);
-    usd += read;
     const est = read > 0 && !!m.estimated?.();
-    if (est) estimated = true;
-    if (!session) continue;
     // The running pass's session as this figure counts it; none known (or an estimate) leaves the session out, so the
     // next stage counts the pass again rather than not at all.
-    const counted = est ? null : await m.session?.().catch(() => null);
+    const counted = session && !est ? await m.session?.().catch(() => null) : null;
+    reads.push({ session, read, est, counted });
+  }
+  return stageSpendOf(base, reads);
+}
+
+/**
+ * stageSpendSoFar read at once, with no await: each running pass's meter from its stream alone (live-spend.mjs `now`).
+ * For a process that is stopping (session.mjs progressReporter), where nothing asynchronous runs any more.
+ */
+export function stageSpendNow() {
+  const base = stageSpendBase();
+  const reads = [];
+  for (const [m, { session }] of STAGE_SPEND.open) {
+    let r = null;
+    try {
+      r = m.now?.() ?? null;
+    } catch {
+      // read as nothing
+    }
+    const read = Math.max(0, Number(r?.usd) || 0);
+    const est = read > 0 && !!m.estimated?.();
+    reads.push({ session, read, est, counted: session && !est ? (r?.session ?? null) : null });
+  }
+  return stageSpendOf(base, reads);
+}
+
+const stageSpendBase = () => ({ done: STAGE_SPEND.done, estimated: STAGE_SPEND.estimated, totals: Object.fromEntries(SESSION_TOTALS) });
+
+/**
+ * The stage's spend: what had finished (`base`) and its running passes' readings ({session, read, est, counted}), with
+ * its commit (stageSpendSoFar).
+ */
+function stageSpendOf(base, reads) {
+  let usd = base.done;
+  let estimated = base.estimated;
+  const totals = base.totals;
+  for (const { session, read, est, counted } of reads) {
+    usd += read;
+    if (est) estimated = true;
+    if (!session) continue;
     if (counted) totals[session] = counted;
     else delete totals[session];
   }