From 30b928e61c20314e17ccd2d8f241aafc47da2a58 Mon Sep 17 00:00:00 2001 From: Kirk Marple Date: Fri, 13 Nov 2026 17:08:26 -0800 Subject: [PATCH 1/3] Local: a stage stopped by a signal writes its spend line once more A stage killed before its next progress-comment write after a pass had spent left that pass at the last line's figure ($0 for a first pass). On SIGTERM, SIGINT or SIGHUP the stage now kills its harness's group, reads the stage's spend at once (stageSpendNow, from each running pass's stream) and, when that is more than the comment shows, PATCHes the comment it last wrote, waiting at most 5 s in a worker thread so no other stage code runs. The session totals that reading counted are recorded as for any interim write, so a resuming stage never counts them twice. Fixes #178 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CS53Gu2D3bE8mKgFN92XP5 --- docs/LOCAL.md | 4 +-- plugin/scripts/lib/harness.mjs | 51 ++++++++++++++++++++++---- plugin/scripts/lib/live-spend.mjs | 53 ++++++++++++++++----------- plugin/scripts/lib/proc.mjs | 29 ++++++++++++--- plugin/scripts/lib/rest-sync.mjs | 41 +++++++++++++++++++++ plugin/scripts/lib/session.mjs | 59 +++++++++++++++++++++++++------ plugin/test/stage-spend.test.mjs | 33 +++++++++++++---- 7 files changed, 221 insertions(+), 49 deletions(-) create mode 100644 plugin/scripts/lib/rest-sync.mjs diff --git a/docs/LOCAL.md b/docs/LOCAL.md index 1c209bb..160e757 100644 --- a/docs/LOCAL.md +++ b/docs/LOCAL.md @@ -19,7 +19,7 @@ Factory's worker and reviewer, run on your own machine instead of a GitHub runne - The harness holds no GitHub token at all: what it needs from GitHub is in `$FACTORY_IN`. Factory's own calls use one fine-grained token for this one repository, never your `gh` login; the token reaches each stage on a pipe it reads once, never its environment or a file, and Factory's own git gets it by per-process env config. No sandbox can read `/proc` (Claude Code and `srt`; Codex runs in its own pid namespace). Copilot's sandbox runs each command in a pid namespace of its own too, whose pid 1 is the bubblewrap Copilot starts with its whole environment, the token included (#100); on Linux Factory's own `bwrap`, first on Copilot's PATH, starts the real one with an empty environment, and bubblewrap sets each command's environment itself. Copilot also writes each command's network rules into its own `TMPDIR` before it loads them, and opens that directory to every command, so a command could rewrite the next one's rules and reach past Copilot's proxy to `srt`'s (whose list adds Copilot's API hosts). On Linux Copilot therefore runs with a `TMPDIR` of its own, in the stage's usage directory, made afresh before every start and denied to its commands, and writes no logs (Copilot opens its log directory to every command whatever its policy says); Factory's `bwrap` gives each command the stage's temp directory as its `TMPDIR`. Copilot's sandbox binds only the system's own directories (`/usr/bin`, `/usr/lib`, `/etc` and the like) into each command, so a node your version manager keeps in your home (nvm, mise, asdf, Volta) would be out of its commands' reach (#111). The stage's node is the first `node` on your PATH, by its real path (where that is a version manager's shim, the node the shim chose, which runs Factory). Its install prefix (`bin`'s parent, so npm comes too) is read-only in each command's sandbox, with its `etc` (a global npmrc) denied, and Factory's `bwrap` puts its `bin` first on each command's PATH, after Factory's git. Nothing else of your home is bound. The prefix is never your home or a directory above it, nor in or above a credential store, this repository's `.factory` directory or the stages' temp directories, where a sandbox could swap it for a link. The same holds for the repository's package manager, and for Claude Code too, whose sandbox likewise shows nothing of your home but its working and added directories (#137): a repository whose `packageManager` (else its lockfile) names pnpm or yarn gets the one your PATH resolves to (corepack's shim, its own install, or, behind a version manager's shim, the version `packageManager` pins in that manager's installs: mise's and asdf's `installs/pnpm/`, Volta's `tools/image/pnpm/`; Factory never runs the manager, which would read the clone's own configuration), linked from a `toolbin` directory in the stage's usage directory, which no sandbox may write, and first on each command's PATH (Copilot through Factory's `bwrap`, Claude Code through `CLAUDE_ENV_FILE`). Only the npm package it belongs to (corepack, pnpm) is readable, read-only, else the executable alone; corepack's downloads live in the Objective's own `COREPACK_HOME`. Codex's sandbox reads the whole system but the credential stores, so its commands find both as you do. Before every Copilot start a probe checks all of this: a Copilot session in the outer sandbox, with the stage's policy and no credential, against a model Factory serves on that sandbox's own loopback (no network, no bill), runs one command. That command must find a random marker Copilot holds in no environment it can read, Copilot's network rules nowhere it can list, Copilot's `TMPDIR` read-only and no Copilot log it can read, nothing it writes in your home, the probe's Copilot home or Copilot's release left there, the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused; the stage's node must be the `node` it finds, and it, the `npm` beside it and the repository's pnpm or yarn must run (else the stage Blocks with `the stage's toolchain does not run in a command`); and it must end with its pass line. A broken rule Blocks the stage (`sandbox`) before any command runs; a probe that leaves no result is tried once more, then Blocks. It takes a second or two. Claude Code on Linux gets the same probe before every start (it has no command that runs one command in its sandbox either): a Claude Code session launched as the run is, with a throwaway config directory in the usage directory, a stand-in key and no credential of yours, against a stand-in for the Anthropic API on its outer sandbox's own loopback (no network, no bill), runs one command. That command must find and run the stage's node and the repository's pnpm or yarn, find a random marker Claude Code holds in no environment it can read, leave nothing it writes in your home or the usage directory, find the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused, and end with its pass line; a broken rule Blocks the stage (`sandbox`) with the rule. `doctor.mjs --local` runs it too (`claude sandbox: start`, around a throwaway clone with your checkout's `package.json`), and names what local mode resolves: `node: v24.20.0 → /usr/bin/node`, `package manager: pnpm 12.18.0 (corepack) → …/corepack/dist/pnpm.js`; a pinned package manager it cannot resolve is a FAIL. - Each Objective gets its own clone under `/.factory//`, which is git-excluded. `serve` removes it once the issue is closed; a Blocked Objective keeps its directory so a re-label picks up where it stood. Its installed dependencies (ignored files) carry over between stages and runs, but nothing unpushed does: each stage starts on the branch exactly as GitHub has it, and removes the untracked files, uncommitted changes and stopped rebase an earlier stage or run left there. It also removes the git lock files (`.git/*.lock`, `.git/refs/**/*.lock`) a run interrupted by a reboot or a killed `serve` left, and logs each: a lock older than the machine's last boot always goes; a newer one goes only where `/proc` shows no process with its working directory or an open file in the clone (Linux; on macOS only the boot rule applies). Otherwise the lock stays and the log names it. - GitHub holds all state. A claim comment with a heartbeat marks an Objective a local loop is running, so the Action and other loops leave it alone. -- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered. +- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. A stage stopped by a signal (Ctrl-C, SIGTERM or SIGHUP, or its loop stopping) writes that reading once more as it stops, after its harness is killed, when it is more than the line shows: a pass killed before the next write still counts. It waits at most 5 s for GitHub. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. The stopping write records them unless GitHub refuses it, so one that lands after its 5 s still counts once (one that never lands leaves that part uncounted). Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered. ## Platforms @@ -63,7 +63,7 @@ Every command takes `--repo OWNER/NAME` and `--dir CLONE` too. `--help` or `-h` - **Hand-off:** the harness writes nothing to GitHub. It reads what Factory rendered into `$FACTORY_IN` and leaves its plan, PR text, replies, note and exit record in `$FACTORY_OUT`, both under the stage's temp directory; Factory reads them inside the same sandbox, then pushes and posts. `deliver` and `post-review` do the same for a session's own work, with the local token, else your `gh` login. A session reads the issue only through `factory issue`, and `deliver` and `post-review` check its author and last editor again, as every stage does: an issue a person who cannot push wrote or last edited is refused (deliver Blocks it, nothing pushed). - **Routing:** label an Objective `factory:local` and the local loop takes it, while the Action leaves it alone. `--all` (or `FACTORY_LOCAL_ALL=1`) takes every Objective. The loop's sweep (stale Running, Done) touches only the Objectives it takes, and a Running one a local loop claimed; before its Done pass removes a closed Objective's state labels under your login, it posts a `done` marker naming them, which the scorer reads as Factory's. - **Questions and Blocks:** an Objective whose stage ends on this machine with a Question or a Block waits for you. No `serve` tick takes it again while its stage runs, or after it ends on a GitHub read that does not show the stop yet; it starts again only after a trusted answer, the same as on the Action (re-applying its label, or marking a Question's draft PR ready). If the stop never reached GitHub (a fresh read of the issue shows it Queued), it runs again, as on the Action, and the tick line names every Objective it holds. -- **Stopping and pausing:** stop `serve`, `run` or `review` with Ctrl-C, SIGTERM or SIGHUP (a closed terminal, a parent session ending) and its Objectives pause: they stay Running with their claims. The `factory` front end passes each signal on; the loop ends every stage process, and each stage kills its harness's whole process group before it exits, since the harness runs in a group of its own that no terminal signal reaches. A second signal a second or more after the first exits at once. The same pause happens when the machine sleeps or shuts down. The next `serve` on the same machine resumes them first, before reviews and new work, and `factory run ` takes one up the same way: an open PR waiting for review goes to review; Running work starts a fresh work pass in the Objective's clone, on the branch as GitHub has it (a killed pass's session and unpushed commits are not carried over). Other machines and the Action leave a paused Objective alone; the dispatcher Blocks one only after a day away, and re-applying its label brings it back. +- **Stopping and pausing:** stop `serve`, `run` or `review` with Ctrl-C, SIGTERM or SIGHUP (a closed terminal, a parent session ending) and its Objectives pause: they stay Running with their claims. The `factory` front end passes each signal on; the loop ends every stage process, and each stage kills its harness's whole process group and writes its spend so far (at most 5 s) before it exits, since the harness runs in a group of its own that no terminal signal reaches. A second signal a second or more after the first exits at once. The same pause happens when the machine sleeps or shuts down. The next `serve` on the same machine resumes them first, before reviews and new work, and `factory run ` takes one up the same way: an open PR waiting for review goes to review; Running work starts a fresh work pass in the Objective's clone, on the branch as GitHub has it (a killed pass's session and unpushed commits are not carried over). Other machines and the Action leave a paused Objective alone; the dispatcher Blocks one only after a day away, and re-applying its label brings it back. - **What a start finds left over:** each stage names its own process and its harness's (pid, start time, the harness's run tag) in its usage directory while they run. Before a stage starts, and before `run`, a resume or the cleanup of a closed Objective, a harness that a dead stage of the same Objective left running (its loop killed with SIGKILL, say) is killed. Factory signals it only when the process is still the one recorded, by its start time, and its group still carries the run's tag; never by matching command lines. A stage still running on this machine is not run twice: `run` refuses and `serve` skips that Objective, naming the pid to stop. When the machine cannot tell whether the recorded pid is still that stage (its start time unreadable), it refuses too and names the record file: delete it once you have checked that pid is no Factory worker. - **Keeping `serve` running after the terminal closes:** `nohup` does not help, because Node resets an ignored SIGHUP when it starts. Start it in a session of its own instead: on Linux `setsid -f factory serve >serve.log 2>&1 serve.log 2>&1 `, which pauses its Objectives as above. On WSL2, Windows may shut the distribution down after its idle timeout once no WSL terminal is attached, and a detached `serve` does not prevent that: keep one WSL window open (with `serve` in `tmux` there, say) or raise the idle timeout in `.wslconfig`. Nothing restarts `serve` after a reboot; start it again. - **Claims and deadlines:** diff --git a/plugin/scripts/lib/harness.mjs b/plugin/scripts/lib/harness.mjs index 87e17f5..6c03e02 100644 --- a/plugin/scripts/lib/harness.mjs +++ b/plugin/scripts/lib/harness.mjs @@ -446,18 +446,55 @@ export const SESSION_TOTALS_FILE = "session-totals.json"; * counted (SESSION_TOTALS_FILE). */ export async function stageSpendSoFar() { - let usd = STAGE_SPEND.done; - let estimated = STAGE_SPEND.estimated; - const totals = Object.fromEntries(SESSION_TOTALS); + // What has finished is taken before any meter is read: a pass that ends meanwhile is never counted twice. + const base = stageSpendBase(); + const reads = []; for (const [m, { session }] of STAGE_SPEND.open) { const read = await m.usd().catch(() => 0); - usd += read; const est = read > 0 && !!m.estimated?.(); - if (est) estimated = true; - if (!session) continue; // The running pass's session as this figure counts it; none known (or an estimate) leaves the session out, so the // next stage counts the pass again rather than not at all. - const counted = est ? null : await m.session?.().catch(() => null); + const counted = session && !est ? await m.session?.().catch(() => null) : null; + reads.push({ session, read, est, counted }); + } + return stageSpendOf(base, reads); +} + +/** + * stageSpendSoFar read at once, with no await: each running pass's meter from its stream alone (live-spend.mjs `now`). + * For a process that is stopping (session.mjs progressReporter), where nothing asynchronous runs any more. + */ +export function stageSpendNow() { + const base = stageSpendBase(); + const reads = []; + for (const [m, { session }] of STAGE_SPEND.open) { + let r = null; + try { + r = m.now?.() ?? null; + } catch { + // read as nothing + } + const read = Math.max(0, Number(r?.usd) || 0); + const est = read > 0 && !!m.estimated?.(); + reads.push({ session, read, est, counted: session && !est ? (r?.session ?? null) : null }); + } + return stageSpendOf(base, reads); +} + +const stageSpendBase = () => ({ done: STAGE_SPEND.done, estimated: STAGE_SPEND.estimated, totals: Object.fromEntries(SESSION_TOTALS) }); + +/** + * The stage's spend: what had finished (`base`) and its running passes' readings ({session, read, est, counted}), with + * its commit (stageSpendSoFar). + */ +function stageSpendOf(base, reads) { + let usd = base.done; + let estimated = base.estimated; + const totals = base.totals; + for (const { session, read, est, counted } of reads) { + usd += read; + if (est) estimated = true; + if (!session) continue; if (counted) totals[session] = counted; else delete totals[session]; } diff --git a/plugin/scripts/lib/live-spend.mjs b/plugin/scripts/lib/live-spend.mjs index b681a1b..cccec99 100644 --- a/plugin/scripts/lib/live-spend.mjs +++ b/plugin/scripts/lib/live-spend.mjs @@ -113,36 +113,30 @@ export function liveSpend(harness, model = null, { prior = null, creditUsd = COP codexLast = { input: n(e.usage.input_tokens), output: n(e.usage.output_tokens), cached: n(e.usage.cached_input_tokens), written: n(e.usage.cache_write_input_tokens) }; } else if (harness === "copilot" && e.type === "session.usage_checkpoint") copilotNano = Math.max(copilotNano, n(e.data?.totalNanoAiu)); }; - // Codex: the thread's running total as read now (its stream's, or its rollout's when larger) and its share since `prior`. - const codexRead = async () => { + // Codex: the thread's running total as read now (its stream's, or `rolloutText`'s when larger) and its share since `prior`. + const codexFrom = (rolloutText) => { const p = prior?.tokens || {}; const since = (t) => ({ input: Math.max(0, n(t.input) - n(p.input)), output: Math.max(0, n(t.output) - n(p.output)), cached: Math.max(0, n(t.cached) - n(p.cached)), written: Math.max(0, n(t.written) - n(p.written)) }); let best = { total: codexLast ?? {}, usd: codexUsd(codex.model, since(codex)) }; - if (rollout && thread) { - // A read that fails throws: the relay refuses the call rather than read the spend as $0. - const total = rolloutTotal(await rollout(thread), thread); - if (total) { - const fromRollout = codexUsd(codex.model, since(total)); - best = fromRollout > best.usd ? { total, usd: fromRollout } : { total: best.total, usd: best.usd }; - } + const total = rolloutText === null ? null : rolloutTotal(rolloutText, thread); + if (total) { + const fromRollout = codexUsd(codex.model, since(total)); + best = fromRollout > best.usd ? { total, usd: fromRollout } : { total: best.total, usd: best.usd }; } return best; }; - // Async: Codex's rollout is read when asked (once per relayed call). - const usd = async () => { + // A read that fails throws: the relay refuses the call rather than read the spend as $0. + const codexRead = async () => codexFrom(rollout && thread ? await rollout(thread) : null); + const usdOf = (r) => { if (harness === "claude") return [...claude.values()].reduce((s, m) => s + claudeUsageUsd(m.usage, m.model), 0); - if (harness === "codex") return (await codexRead()).usd; + if (harness === "codex") return r.usd; if (harness === "copilot") return Math.max(0, (copilotNano / 1e9) * creditUsd - n(prior?.usd)); return 0; }; - /** - * The resumed session's running total as this meter counts it now (`prior` plus the reading, in sinceSession's shape), - * for the next stage to start from (harness.mjs stageSpendSoFar); null for a fresh session, which no later stage resumes. - */ - const session = async () => { + // The resumed session's running total from a reading (`r`: Codex's, `read`: the pass's dollars). + const sessionOf = (r, read) => { if (typeof prior?.usd !== "number") return null; if (harness === "codex") { - const r = await codexRead(); const pt = prior.tokens || {}; const most = (k) => Math.max(n(pt[k]), n(r.total[k])); const tokens = { input: most("input"), output: most("output"), cached: most("cached"), written: most("written") }; @@ -150,14 +144,33 @@ export function liveSpend(harness, model = null, { prior = null, creditUsd = COP const share = codexUsd(codex.model, { input: tokens.input - n(pt.input), output: tokens.output - n(pt.output), cached: tokens.cached - n(pt.cached), written: tokens.written - n(pt.written) }); return { ...prior, usd: prior.usd + Math.min(r.usd, share), tokens }; } - const read = await usd(); if (harness === "copilot") return { ...prior, usd: prior.usd + read, aiu: Math.max(n(prior.aiu), copilotNano / 1e9) }; return { ...prior, usd: prior.usd + read }; }; + // Async: Codex's rollout is read when asked (once per relayed call). + const usd = async () => usdOf(harness === "codex" ? await codexRead() : null); + /** + * The resumed session's running total as this meter counts it now (`prior` plus the reading, in sinceSession's shape), + * for the next stage to start from (harness.mjs stageSpendSoFar); null for a fresh session, which no later stage resumes. + */ + const session = async () => { + if (typeof prior?.usd !== "number") return null; + const r = harness === "codex" ? await codexRead() : null; + return sessionOf(r, harness === "codex" ? r.usd : await usd()); + }; + /** + * `{usd, session}` read at once, from the stream alone (no rollout read), both from the one reading: what a stopping + * process can still read (harness.mjs stageSpendNow). For Codex with a rollout reader, at most what usd() reads. + */ + const now = () => { + const r = harness === "codex" ? codexFrom(null) : null; + const read = usdOf(r); + return { usd: read, session: sessionOf(r, read) }; + }; const estimated = () => { if (harness === "claude") return [...claude.values()].some((m) => claudeRate(m.model) === CLAUDE_PRICES.upperBound); if (harness === "codex") return priceCodex(codex.model, { input: 1, output: 1, cached: 0, written: 0 }) === null; return false; }; - return { line, usd, estimated, session }; + return { line, usd, estimated, session, now }; } diff --git a/plugin/scripts/lib/proc.mjs b/plugin/scripts/lib/proc.mjs index 361c954..52ca74f 100644 --- a/plugin/scripts/lib/proc.mjs +++ b/plugin/scripts/lib/proc.mjs @@ -465,6 +465,18 @@ export const GROUP_RECORD = /^harness-group-\d+\.json$/; /** This process's live `group` runs: leader pid → {tag, record}. */ const GROUPS = new Map(); let trapped = false; +/** What runs once the groups are killed on a stop signal, before the exit (atStop). */ +const AT_STOP = new Set(); + +/** + * Run `fn(sig)` when a stop signal ends a process with a live or past `group` run: after its groups are killed, before + * it exits, synchronously (stopGroups: no other code of the process runs meanwhile), so `fn` must bound its own time. + * A throw is ignored. Returns a function that removes it. + */ +export function atStop(fn) { + AT_STOP.add(fn); + return () => AT_STOP.delete(fn); +} /** `sysctl -n kern.boottime` ("{ sec = 1731450000, usec = 12345 } Thu Nov 12 ...") as its seconds, a string; else null. */ export const boottimeSeconds = (text) => /\bsec = (\d+)/.exec(String(text || ""))?.[1] ?? null; @@ -628,10 +640,11 @@ function untrackGroup(pid) { } /** - * On SIGTERM, SIGHUP or SIGINT: every live group run's process group and tagged processes are killed, then the process - * exits as the signal would have ended it (128 + its number), unless another handler of that signal is installed and - * decides when. All at once, with no await between: no code of the stage runs (a next pass, a post) once it is told - * to stop, and nothing of the harness is left when it exits. A killed pass's session is not carried over anyway. + * On SIGTERM, SIGHUP or SIGINT: every live group run's process group and tagged processes are killed, then what atStop + * registered runs, then the process exits as the signal would have ended it (128 + its number), unless another handler + * of that signal is installed and decides when. All at once, with no await between: no code of the stage runs (a next + * pass, a post) once it is told to stop, and nothing of the harness is left when it exits. A killed pass's session is + * not carried over anyway. */ function stopGroups(sig) { for (const [pid, g] of [...GROUPS]) { @@ -643,6 +656,14 @@ function stopGroups(sig) { killTagged(g.tag); untrackGroup(pid); } + for (const fn of [...AT_STOP]) { + AT_STOP.delete(fn); + try { + fn(sig); + } catch { + // the exit goes ahead + } + } if (process.listenerCount(sig) <= 1) process.exit(128 + (osConstants.signals[sig] || 15)); } diff --git a/plugin/scripts/lib/rest-sync.mjs b/plugin/scripts/lib/rest-sync.mjs new file mode 100644 index 0000000..5ecd130 --- /dev/null +++ b/plugin/scripts/lib/rest-sync.mjs @@ -0,0 +1,41 @@ +// One GitHub REST call made synchronously, for a process that is stopping (proc.mjs atStop): the call runs in a worker +// thread while this thread waits on it, at most `ms`, so no other code of the process runs meanwhile. +import { MessageChannel, Worker, isMainThread, receiveMessageOnPort, workerData } from "node:worker_threads"; + +/** + * github.mjs `rest(path, {method, body, token})`, waited on for at most `ms` (the call's own retries end then too). + * Returns {ok: true, data} or {ok: false, error, status}: `status` is GitHub's answer to a refused call, 0 when there was + * none (no answer in time, the network). Never throws. The token stays in memory: the thread gets it as data. + */ +export function restSync(path, { method = "GET", body, token }, { ms = 5_000 } = {}) { + const done = new Int32Array(new SharedArrayBuffer(4)); + const { port1, port2 } = new MessageChannel(); + let worker; + try { + // No node options of this process (`--input-type=module -e` would keep the thread from loading this file). + worker = new Worker(new URL(import.meta.url), { workerData: { restSync: true, path, method, body, token, deadlineAt: Date.now() + ms, done, port: port2 }, transferList: [port2], execArgv: [], stdout: true, stderr: true }); + } catch (e) { + return { ok: false, error: String(e?.message || e).slice(0, 200), status: 0 }; + } + worker.unref(); + const waited = Atomics.wait(done, 0, 0, ms); + const got = receiveMessageOnPort(port1)?.message ?? null; + void worker.terminate(); + port1.close(); + if (got) return got; + return { ok: false, error: waited === "timed-out" ? `no answer in ${ms} ms` : "the call ended with no answer", status: 0 }; +} + +if (!isMainThread && workerData?.restSync) { + const { path, method, body, token, deadlineAt, done, port } = workerData; + let answer; + try { + const { rest } = await import("./github.mjs"); + answer = { ok: true, data: await rest(path, { method, body, token, deadlineAt, noRateWait: true }) }; + } catch (e) { + answer = { ok: false, error: String(e?.message || e).slice(0, 200), status: Number(e?.status) || 0 }; + } + port.postMessage(answer); + Atomics.store(done, 0, 1); + Atomics.notify(done, 0); +} diff --git a/plugin/scripts/lib/session.mjs b/plugin/scripts/lib/session.mjs index 1524f95..6eed781 100644 --- a/plugin/scripts/lib/session.mjs +++ b/plugin/scripts/lib/session.mjs @@ -7,7 +7,7 @@ import { homedir, tmpdir } from "node:os"; import { join } from "node:path"; import { backend, spendInProgress } from "./backend.mjs"; import { runValidation } from "./checkout.mjs"; -import { SESSION_TOTALS_FILE, copilotCreditUsd, finalText, runHarness, seedSessionTotal, stageSpendSoFar } from "./harness.mjs"; +import { SESSION_TOTALS_FILE, copilotCreditUsd, finalText, runHarness, seedSessionTotal, stageSpendNow, stageSpendSoFar } from "./harness.mjs"; import { addLabels, awaitChecks, @@ -50,7 +50,8 @@ import { } from "./github.mjs"; import { log, redactSecrets, secretForms } from "./inputs.mjs"; import { EXIT_PROMPT, activityLine, blockMarker, answerDuring, bodyFindingId, bodyFindings, conflictsAt, decisionsReadMarker, decisionsReadOf, decisionsSection, harnessActivity, isPlaceholder, knownWriter, maintainerDecisions, pinnedMaintainer, marker, mergeChecked, mergeRead, parseExitRecord, questionCheckPrompt, refusalReason, renderTaskCard, taskCardValues, unreadDecisions } from "./logic.mjs"; -import { agentUser, exec, git } from "./proc.mjs"; +import { agentUser, atStop, exec, git } from "./proc.mjs"; +import { restSync } from "./rest-sync.mjs"; import { DIFF_FILE, checkFile, isSplitPart, partsRecord, prFooter, putFiles, sanitize } from "./handoff.mjs"; import { TokenKeeper } from "./token-keeper.mjs"; import { noticeFromParent } from "./app-token.mjs"; @@ -763,16 +764,23 @@ export const progressRunId = (env = process.env) => /** The progress comment's structured tail: the stage's harness, then its spend, decisions read, triage and plan-time asks. */ export const progressTail = ({ harness = null, model = "", spend = null, spendParts = null, read = null, triage = null, asked = null } = {}) => `${harness ? `\n${harnessMarker(harness, model)}` : ""}${spend === null ? "" : `\n${spendMarker(spend, spendParts)}`}${read ? `\n${decisionsReadMarker(read)}` : ""}${triage ? `\n${triageMarker(triage)}` : ""}${asked ? `\n${planAskedMarker(asked)}` : ""}`; +/** How long a stage stopped by a signal waits on its last spend line (progressReporter `soFarNow`) before it exits. */ +export const STOP_WRITE_MS = 5_000; + /** * `soFar`: the stage's spend so far (harness.mjs stageSpendSoFar), written as a spend line with every write until * progress.spend records the final figure, so a stage that dies (killed, the machine restarted) still leaves the spend * Factory last knew: marked `usdLowerBound`, or `usdUpperBound` when part of it is priced at the table's dearest rates. * Null when the stage records no spend in its progress comment (backend.mjs spendInProgress), or for a `--here` run. + * `soFarNow`: the same read at once (harness.mjs stageSpendNow). A stage stopped by a signal (proc.mjs atStop: its + * harness's group already killed) writes it into the comment one last time, within STOP_WRITE_MS, when it is more than + * the comment's line shows, so a pass killed before the next status write is not left at the last line's figure. */ -export function progressReporter(repo, issue, { token }, { stage, runId = progressRunId(), harness = null, model = "", soFar = spendInProgress() ? stageSpendSoFar : null }) { +export function progressReporter(repo, issue, { token }, { stage, runId = progressRunId(), harness = null, model = "", soFar = spendInProgress() ? stageSpendSoFar : null, soFarNow = soFar === stageSpendSoFar ? stageSpendNow : null }) { const lines = []; const links = []; const started = Date.now(); + const key = marker(issue, "progress", `${stage}:${runId}`); let status = ""; let spend = null; let spendParts = null; @@ -781,20 +789,50 @@ export function progressReporter(repo, issue, { token }, { stage, runId = progre let asked = null; let lastWrite = 0; let writing = null; + // The comment as the last write that landed left it: its id and the spend its line shows. + let commentId = null; + let shown = 0; + let stopHook = null; const clock = (t) => `${new Date(t).toISOString().slice(11, 16)}Z`; const runUrl = process.env.GITHUB_RUN_ID && process.env.GITHUB_SERVER_URL ? `${process.env.GITHUB_SERVER_URL}/${repo.full}/actions/runs/${process.env.GITHUB_RUN_ID}` : null; - const write = async () => { + // Not final: a lower bound, or, with a reading at the price table's dearest rates, an estimate marked as such. + const interimOf = (live) => ({ spend: live.usd, spendParts: live.estimated ? { usdUpperBound: true } : { usdLowerBound: true } }); + const bodyOf = (interim) => { const head = status ? `**${status}**\n\n` : ""; const where = [runUrl ? `[run ${runId}](${runUrl})` : `run ${runId}`, ...links].join(" · "); + const tail = progressTail({ harness, model, ...interim, read, triage, asked }); + return `${head}Factory ${stage} ${where}: started ${clock(started)}, ${Math.round((Date.now() - started) / 60000)} min at the last line.\n${lines.join("\n")}${tail}`; + }; + /** + * Stopped by a signal, the harness already killed: the live reading once more, if it is more than the line shows. Only + * a refusal GitHub answered leaves the session totals unrecorded; a write that may have landed (no answer in time) + * records them, so the next stage never counts again what this line may show (it may count less: a lower bound). + */ + const stopWrite = () => { + if (spend !== null || commentId === null) return; + let live = null; + try { + live = soFarNow(); + } catch { + return; + } + if (!Number.isFinite(live?.usd) || !(live.usd > shown)) return; + const r = restSync(`repos/${repo.full}/issues/comments/${commentId}`, { method: "PATCH", body: { body: `${key}\n${bodyOf(interimOf(live))}` }, token }, { ms: STOP_WRITE_MS }); + if (!r.ok) log(`progress: the stopped stage's spend line ($${live.usd}) may not have landed: ${r.error}`); + if (r.ok || !(r.status >= 400 && r.status < 500)) live.commit?.(); + }; + const write = async () => { const live = spend === null && soFar ? await soFar().catch(() => null) : null; - // Not final: a lower bound, or, with a reading at the price table's dearest rates, an estimate marked as such. const counted = spend === null && Number.isFinite(live?.usd); - const interim = counted ? { spend: live.usd, spendParts: live.estimated ? { usdUpperBound: true } : { usdLowerBound: true } } : { spend, spendParts }; - const tail = progressTail({ harness, model, ...interim, read, triage, asked }); - const body = `${head}Factory ${stage} ${where}: started ${clock(started)}, ${Math.round((Date.now() - started) / 60000)} min at the last line.\n${lines.join("\n")}${tail}`; + const interim = counted ? interimOf(live) : { spend, spendParts }; + const body = bodyOf(interim); lastWrite = Date.now(); - const wrote = await upsertComment(repo, issue, marker(issue, "progress", `${stage}:${runId}`), body, { token }).then( - () => true, + const wrote = await upsertComment(repo, issue, key, body, { token }).then( + (c) => { + if (Number.isInteger(c?.id)) commentId = c.id; + shown = Number(interim.spend) || 0; + return true; + }, (e) => { log(`progress: ${e.message.slice(0, 120)}`); return false; @@ -802,6 +840,7 @@ export function progressReporter(repo, issue, { token }, { stage, runId = progre ); // Only a line that landed says what was counted: the session totals it counted are recorded after it, never before. if (wrote && counted) live.commit?.(); + if (wrote && soFarNow && commentId !== null) stopHook ??= atStop(stopWrite); }; const progress = async (phase) => { // A phase can quote the harness (an exit reason, the tool it ran last): sanitised like all harness text. diff --git a/plugin/test/stage-spend.test.mjs b/plugin/test/stage-spend.test.mjs index 48413b0..8bbf73a 100644 --- a/plugin/test/stage-spend.test.mjs +++ b/plugin/test/stage-spend.test.mjs @@ -69,8 +69,8 @@ const LIB = (f) => pathToFileURL(join(SCRIPTS, "lib", f)).href; /** * A fake Codex: each call adds one turn to its thread's running total (kept in FAKE_STATE), as real Codex reports it. - * FAKE_SPLIT=1: the turn is reported half done, the process lingers, then its other half lands in the thread (a pass - * that runs on after its stage was killed) and it writes `/done`. + * FAKE_SPLIT=1: the turn is reported half done, the process lingers (FAKE_LINGER ms, 1500 by default), then its other + * half lands in the thread (a pass that runs on after its stage was killed) and it writes `/done`. */ function fakeCodex(bin) { writeFileSync(join(bin, "codex"), `#!${process.execPath} @@ -84,7 +84,7 @@ add(split ? 0.5 : 1); const id = resume || "01a1c6ba-16b2-7a13-b2da-47bf51b4e724"; console.log(JSON.stringify({ type: "thread.started", thread_id: id })); console.log(JSON.stringify({ type: "turn.completed", usage: { input_tokens: t.input, cached_input_tokens: t.cached, cache_write_input_tokens: 0, output_tokens: t.output, reasoning_output_tokens: 0 } })); -if (split) setTimeout(() => { add(0.5); fs.writeFileSync(process.env.FAKE_STATE + "/done", ""); }, 1500); +if (split) setTimeout(() => { add(0.5); fs.writeFileSync(process.env.FAKE_STATE + "/done", ""); }, Number(process.env.FAKE_LINGER) || 1500); `, { mode: 0o755 }); } @@ -137,6 +137,8 @@ test("a stage killed after a real pass leaves that pass's dollars in its default * - after: a phase line after the pass, then killed; * - window: killed as the pass ends, before any write shows it; * - mid: a phase line while the pass runs, then killed; the pass runs on and finishes in the session; + * - term: SIGTERM while the pass runs (as `factory serve` stops a stage), with no write since the pass began; the pass + * runs in its own process group, as locally, and is killed with the stage; * - failed: a phase line that lands, a second pass, a phase line that fails to land, then killed. * Returns every stage's last spend line. */ @@ -165,11 +167,17 @@ async function stages(plan) { const resume = (await restoreTranscript({ checkout: "/c", workDir: join(root, "ws"), harness: "codex" })).id; delete process.env.FACTORY_LOCAL; await progress("harness"); - const pass = () => runHarness({ harness: "codex", model: "gpt-6.1-sol", prompt: "p", maxUsd: 0, resume, cwd: root, env: { PATH: process.env.PATH, FAKE_STATE: root, FAKE_SPLIT: how === "mid" ? "1" : "0" }, timeoutMs: 60_000, eventsFile: join(usageDir, "events.jsonl"), label: "work-1" }); + const local = how === "term" ? { FACTORY_LOCAL_SANDBOX: JSON.stringify({ addDirs: [], clone: null }), FAKE_LINGER: "60000" } : {}; + const pass = () => runHarness({ harness: "codex", model: "gpt-6.1-sol", prompt: "p", maxUsd: 0, resume, cwd: root, env: { PATH: process.env.PATH, FAKE_STATE: root, FAKE_SPLIT: how === "mid" || how === "term" ? "1" : "0", ...local }, timeoutMs: 60_000, eventsFile: join(usageDir, "events.jsonl"), label: "work-1" }); const run = pass(); - if (how === "mid") { + if (how === "mid" || how === "term") { const until = Date.now() + 10_000; while (!((await stageSpendSoFar()).usd > 0) && Date.now() < until) await new Promise((r) => setTimeout(r, 20)); + if (how === "term") { + process.stdout.write("spent\\n"); + await run; + process.exit(1); + } await progress("validation"); process.exit(137); } @@ -198,8 +206,14 @@ async function stages(plan) { try { for (const [i, how] of plan.entries()) { const s = stageProcess(script(`s${i}`, how), env); - await s.exited; + if (how === "term") { + const until = Date.now() + 15_000; + while (!/spent/.test(s.out()) && s.child.exitCode === null && Date.now() < until) await new Promise((r) => setTimeout(r, 20)); + s.child.kill("SIGTERM"); + } + const code = await s.exited; assert.ok(!s.err(), s.err()); + if (how === "term") assert.equal(code, 143, "the stage exits as SIGTERM ends it"); // A killed stage's pass runs on: the next stage starts once it has finished. const until = Date.now() + 15_000; if (how === "mid") while (!existsSync(join(root, "done")) && Date.now() < until) await new Promise((r) => setTimeout(r, 50)); @@ -222,6 +236,7 @@ for (const [how, killed, own, passes] of [ ["window", "as its pass ends, before a write shows it", 0, 3], ["mid", "in the middle of a pass that then runs on", PASS / 2, 3], ["failed", "after a second pass whose write failed to land", PASS, 4], + ["term", "by SIGTERM in the middle of a pass, before a write shows it", PASS / 2, 2.5], ]) { test(`a stage killed ${killed}, then resumed: the stages' lines sum to the spend, each pass counted once`, async () => { const lines = await stages(["end", how, "end"]); @@ -233,6 +248,12 @@ for (const [how, killed, own, passes] of [ }); } +test("a stage stopped by SIGTERM before any status write shows its first pass leaves that pass's reading, a lower bound", async () => { + // A fresh session that no later stage resumes: this line is all that records the pass. + const [line] = await stages(["term"]); + assert.ok(near(line.usd, PASS / 2) && line.usd > 0 && line.usdLowerBound === true, JSON.stringify(line)); +}); + test("a final write waits for a status write in flight, and its spend line replaces the interim one", async () => { const { progressReporter } = await import("../scripts/lib/session.mjs"); const { repoOf } = await import("../scripts/lib/github.mjs"); From 6d6b036eb626023ed1fbcd917cc2da89a2f72846 Mon Sep 17 00:00:00 2001 From: Kirk Marple Date: Fri, 13 Nov 2026 17:20:47 -0800 Subject: [PATCH 2/3] Stop write: getter token, redaction here, commit only when it may have landed - restSync reads a TokenKeeper getter on this thread (a function cannot go to the worker), and redacts this process's held secrets from the body here (the worker's github.mjs holds none). - One try, and a `sent` field from structure: false when the request never left (the thread or token failed first, github.mjs `unsent` connect codes) or GitHub refused it (4xx); true for a 5xx, no answer in time or a connection lost after sending. The stop write records the session totals only when it landed or may have. - Tests use the getter shape; new cases for a stop write never sent and one answered 502, and restSync's token, redaction and `sent`. - LOCAL.md: Codex reports usage at a turn's end, so a Codex pass killed in its first turn still reads $0. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CS53Gu2D3bE8mKgFN92XP5 --- docs/LOCAL.md | 2 +- plugin/scripts/lib/rest-sync.mjs | 51 ++++++++++++----- plugin/scripts/lib/session.mjs | 11 ++-- plugin/test/stage-spend.test.mjs | 97 ++++++++++++++++++++++++++++---- 4 files changed, 130 insertions(+), 31 deletions(-) diff --git a/docs/LOCAL.md b/docs/LOCAL.md index 160e757..f09cfb4 100644 --- a/docs/LOCAL.md +++ b/docs/LOCAL.md @@ -19,7 +19,7 @@ Factory's worker and reviewer, run on your own machine instead of a GitHub runne - The harness holds no GitHub token at all: what it needs from GitHub is in `$FACTORY_IN`. Factory's own calls use one fine-grained token for this one repository, never your `gh` login; the token reaches each stage on a pipe it reads once, never its environment or a file, and Factory's own git gets it by per-process env config. No sandbox can read `/proc` (Claude Code and `srt`; Codex runs in its own pid namespace). Copilot's sandbox runs each command in a pid namespace of its own too, whose pid 1 is the bubblewrap Copilot starts with its whole environment, the token included (#100); on Linux Factory's own `bwrap`, first on Copilot's PATH, starts the real one with an empty environment, and bubblewrap sets each command's environment itself. Copilot also writes each command's network rules into its own `TMPDIR` before it loads them, and opens that directory to every command, so a command could rewrite the next one's rules and reach past Copilot's proxy to `srt`'s (whose list adds Copilot's API hosts). On Linux Copilot therefore runs with a `TMPDIR` of its own, in the stage's usage directory, made afresh before every start and denied to its commands, and writes no logs (Copilot opens its log directory to every command whatever its policy says); Factory's `bwrap` gives each command the stage's temp directory as its `TMPDIR`. Copilot's sandbox binds only the system's own directories (`/usr/bin`, `/usr/lib`, `/etc` and the like) into each command, so a node your version manager keeps in your home (nvm, mise, asdf, Volta) would be out of its commands' reach (#111). The stage's node is the first `node` on your PATH, by its real path (where that is a version manager's shim, the node the shim chose, which runs Factory). Its install prefix (`bin`'s parent, so npm comes too) is read-only in each command's sandbox, with its `etc` (a global npmrc) denied, and Factory's `bwrap` puts its `bin` first on each command's PATH, after Factory's git. Nothing else of your home is bound. The prefix is never your home or a directory above it, nor in or above a credential store, this repository's `.factory` directory or the stages' temp directories, where a sandbox could swap it for a link. The same holds for the repository's package manager, and for Claude Code too, whose sandbox likewise shows nothing of your home but its working and added directories (#137): a repository whose `packageManager` (else its lockfile) names pnpm or yarn gets the one your PATH resolves to (corepack's shim, its own install, or, behind a version manager's shim, the version `packageManager` pins in that manager's installs: mise's and asdf's `installs/pnpm/`, Volta's `tools/image/pnpm/`; Factory never runs the manager, which would read the clone's own configuration), linked from a `toolbin` directory in the stage's usage directory, which no sandbox may write, and first on each command's PATH (Copilot through Factory's `bwrap`, Claude Code through `CLAUDE_ENV_FILE`). Only the npm package it belongs to (corepack, pnpm) is readable, read-only, else the executable alone; corepack's downloads live in the Objective's own `COREPACK_HOME`. Codex's sandbox reads the whole system but the credential stores, so its commands find both as you do. Before every Copilot start a probe checks all of this: a Copilot session in the outer sandbox, with the stage's policy and no credential, against a model Factory serves on that sandbox's own loopback (no network, no bill), runs one command. That command must find a random marker Copilot holds in no environment it can read, Copilot's network rules nowhere it can list, Copilot's `TMPDIR` read-only and no Copilot log it can read, nothing it writes in your home, the probe's Copilot home or Copilot's release left there, the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused; the stage's node must be the `node` it finds, and it, the `npm` beside it and the repository's pnpm or yarn must run (else the stage Blocks with `the stage's toolchain does not run in a command`); and it must end with its pass line. A broken rule Blocks the stage (`sandbox`) before any command runs; a probe that leaves no result is tried once more, then Blocks. It takes a second or two. Claude Code on Linux gets the same probe before every start (it has no command that runs one command in its sandbox either): a Claude Code session launched as the run is, with a throwaway config directory in the usage directory, a stand-in key and no credential of yours, against a stand-in for the Anthropic API on its outer sandbox's own loopback (no network, no bill), runs one command. That command must find and run the stage's node and the repository's pnpm or yarn, find a random marker Claude Code holds in no environment it can read, leave nothing it writes in your home or the usage directory, find the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused, and end with its pass line; a broken rule Blocks the stage (`sandbox`) with the rule. `doctor.mjs --local` runs it too (`claude sandbox: start`, around a throwaway clone with your checkout's `package.json`), and names what local mode resolves: `node: v24.20.0 → /usr/bin/node`, `package manager: pnpm 12.18.0 (corepack) → …/corepack/dist/pnpm.js`; a pinned package manager it cannot resolve is a FAIL. - Each Objective gets its own clone under `/.factory//`, which is git-excluded. `serve` removes it once the issue is closed; a Blocked Objective keeps its directory so a re-label picks up where it stood. Its installed dependencies (ignored files) carry over between stages and runs, but nothing unpushed does: each stage starts on the branch exactly as GitHub has it, and removes the untracked files, uncommitted changes and stopped rebase an earlier stage or run left there. It also removes the git lock files (`.git/*.lock`, `.git/refs/**/*.lock`) a run interrupted by a reboot or a killed `serve` left, and logs each: a lock older than the machine's last boot always goes; a newer one goes only where `/proc` shows no process with its working directory or an open file in the clone (Linux; on macOS only the boot rule applies). Otherwise the lock stays and the log names it. - GitHub holds all state. A claim comment with a heartbeat marks an Objective a local loop is running, so the Action and other loops leave it alone. -- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. A stage stopped by a signal (Ctrl-C, SIGTERM or SIGHUP, or its loop stopping) writes that reading once more as it stops, after its harness is killed, when it is more than the line shows: a pass killed before the next write still counts. It waits at most 5 s for GitHub. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. The stopping write records them unless GitHub refuses it, so one that lands after its 5 s still counts once (one that never lands leaves that part uncounted). Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered. +- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. A stage stopped by a signal (Ctrl-C, SIGTERM or SIGHUP, or its loop stopping) writes that reading once more as it stops, after its harness is killed, when it is more than the line shows, waiting at most 5 s for GitHub. The reading is what the harness has reported so far: Claude Code reports each model response and Copilot each usage checkpoint, so their passes killed before the next write still count; Codex reports usage only when a turn ends, so a Codex pass killed in its first turn still reads $0. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. The stopping write records them when it landed or may have (a 5xx answer, no answer in 5 s), so it is never counted twice; one that may have landed but did not leaves that part uncounted. When it certainly did not land (GitHub unreachable, or a refusal), nothing is recorded and the resuming stage counts that part. Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered. ## Platforms diff --git a/plugin/scripts/lib/rest-sync.mjs b/plugin/scripts/lib/rest-sync.mjs index 5ecd130..66bc896 100644 --- a/plugin/scripts/lib/rest-sync.mjs +++ b/plugin/scripts/lib/rest-sync.mjs @@ -1,41 +1,62 @@ // One GitHub REST call made synchronously, for a process that is stopping (proc.mjs atStop): the call runs in a worker // thread while this thread waits on it, at most `ms`, so no other code of the process runs meanwhile. import { MessageChannel, Worker, isMainThread, receiveMessageOnPort, workerData } from "node:worker_threads"; +import { redactSecrets } from "./inputs.mjs"; + +// The worker's progress, in its shared slot: nothing sent yet, the request on its way, an answer posted. +const IDLE = 0; +const SENDING = 1; +const DONE = 2; /** - * github.mjs `rest(path, {method, body, token})`, waited on for at most `ms` (the call's own retries end then too). - * Returns {ok: true, data} or {ok: false, error, status}: `status` is GitHub's answer to a refused call, 0 when there was - * none (no answer in time, the network). Never throws. The token stays in memory: the thread gets it as data. + * github.mjs `rest(path, {method, body, token})`, one try (no retry), waited on for at most `ms`. `token`: a string or + * a getter (TokenKeeper's), read here. `body` has this process's held secrets redacted here (inputs.mjs holdSecrets): + * the thread's own module state holds none. Never throws. Returns {ok: true, data}, or {ok: false, error, sent}: `sent` + * is false only when the request certainly never reached GitHub (the thread or token failed before it, a connect-level + * failure: github.mjs `unsent`) or GitHub answered with a refusal (4xx); true when it may have landed (a 5xx, no answer + * before `ms`, a connection lost after sending). */ export function restSync(path, { method = "GET", body, token }, { ms = 5_000 } = {}) { - const done = new Int32Array(new SharedArrayBuffer(4)); + const state = new Int32Array(new SharedArrayBuffer(4)); const { port1, port2 } = new MessageChannel(); let worker; try { + const t = typeof token === "function" ? token() : token; + const data = { restSync: true, path, method, body: body === undefined ? undefined : redactSecrets(body), token: t || undefined, state, port: port2 }; // No node options of this process (`--input-type=module -e` would keep the thread from loading this file). - worker = new Worker(new URL(import.meta.url), { workerData: { restSync: true, path, method, body, token, deadlineAt: Date.now() + ms, done, port: port2 }, transferList: [port2], execArgv: [], stdout: true, stderr: true }); + worker = new Worker(new URL(import.meta.url), { workerData: data, transferList: [port2], execArgv: [], stdout: true, stderr: true }); } catch (e) { - return { ok: false, error: String(e?.message || e).slice(0, 200), status: 0 }; + port1.close(); + return { ok: false, error: String(e?.message || e).slice(0, 200), sent: false }; } worker.unref(); - const waited = Atomics.wait(done, 0, 0, ms); - const got = receiveMessageOnPort(port1)?.message ?? null; + const end = Date.now() + ms; + for (let v = Atomics.load(state, 0); v !== DONE && Date.now() < end; v = Atomics.load(state, 0)) Atomics.wait(state, 0, v, end - Date.now()); + const last = Atomics.load(state, 0); + const got = last === DONE ? (receiveMessageOnPort(port1)?.message ?? null) : null; void worker.terminate(); port1.close(); if (got) return got; - return { ok: false, error: waited === "timed-out" ? `no answer in ${ms} ms` : "the call ended with no answer", status: 0 }; + return { ok: false, error: last === DONE ? "the call ended with no answer" : `no answer in ${ms} ms`, sent: last !== IDLE }; } if (!isMainThread && workerData?.restSync) { - const { path, method, body, token, deadlineAt, done, port } = workerData; + const { path, method, body, token, state, port } = workerData; let answer; + let github = null; try { - const { rest } = await import("./github.mjs"); - answer = { ok: true, data: await rest(path, { method, body, token, deadlineAt, noRateWait: true }) }; + github = await import("./github.mjs"); + Atomics.store(state, 0, SENDING); + Atomics.notify(state, 0); + // One try: a deadline that is already here leaves rest no wait to retry in, so the answer says what this request did. + answer = { ok: true, data: await github.rest(path, { method, body, token, deadlineAt: Date.now(), noRateWait: true }) }; } catch (e) { - answer = { ok: false, error: String(e?.message || e).slice(0, 200), status: Number(e?.status) || 0 }; + const status = Number(e?.status) || 0; + const refused = status > 0 && status < 500; + const sent = Atomics.load(state, 0) === SENDING && !(github && e instanceof github.GitHubError && (e.unsent === true || refused)); + answer = { ok: false, error: String(e?.message || e).slice(0, 200), sent }; } port.postMessage(answer); - Atomics.store(done, 0, 1); - Atomics.notify(done, 0); + Atomics.store(state, 0, DONE); + Atomics.notify(state, 0); } diff --git a/plugin/scripts/lib/session.mjs b/plugin/scripts/lib/session.mjs index 6eed781..1aa9b14 100644 --- a/plugin/scripts/lib/session.mjs +++ b/plugin/scripts/lib/session.mjs @@ -804,9 +804,10 @@ export function progressReporter(repo, issue, { token }, { stage, runId = progre return `${head}Factory ${stage} ${where}: started ${clock(started)}, ${Math.round((Date.now() - started) / 60000)} min at the last line.\n${lines.join("\n")}${tail}`; }; /** - * Stopped by a signal, the harness already killed: the live reading once more, if it is more than the line shows. Only - * a refusal GitHub answered leaves the session totals unrecorded; a write that may have landed (no answer in time) - * records them, so the next stage never counts again what this line may show (it may count less: a lower bound). + * Stopped by a signal, the harness already killed: the live reading once more, if it is more than the line shows. The + * session totals it counted are recorded when the write landed or may have (rest-sync.mjs `sent`: a 5xx, no answer in + * time), so the next stage never counts again what this line may show (it may count less: a lower bound); not when it + * certainly did not land (never sent, or refused), so the next stage counts what this line could not. */ const stopWrite = () => { if (spend !== null || commentId === null) return; @@ -818,8 +819,8 @@ export function progressReporter(repo, issue, { token }, { stage, runId = progre } if (!Number.isFinite(live?.usd) || !(live.usd > shown)) return; const r = restSync(`repos/${repo.full}/issues/comments/${commentId}`, { method: "PATCH", body: { body: `${key}\n${bodyOf(interimOf(live))}` }, token }, { ms: STOP_WRITE_MS }); - if (!r.ok) log(`progress: the stopped stage's spend line ($${live.usd}) may not have landed: ${r.error}`); - if (r.ok || !(r.status >= 400 && r.status < 500)) live.commit?.(); + if (!r.ok) log(`progress: the stopped stage's spend line ($${live.usd}) ${r.sent ? "may not have landed" : "was not written"}: ${r.error}`); + if (r.ok || r.sent) live.commit?.(); }; const write = async () => { const live = spend === null && soFar ? await soFar().catch(() => null) : null; diff --git a/plugin/test/stage-spend.test.mjs b/plugin/test/stage-spend.test.mjs index 8bbf73a..9950824 100644 --- a/plugin/test/stage-spend.test.mjs +++ b/plugin/test/stage-spend.test.mjs @@ -110,7 +110,7 @@ test("a stage killed after a real pass leaves that pass's dollars in its default import { runHarness } from ${JSON.stringify(LIB("harness.mjs"))}; import { repoOf } from ${JSON.stringify(LIB("github.mjs"))}; process.env.FACTORY_LOCAL = "1"; - const progress = progressReporter(repoOf("owner/repo"), 7, { token: "t" }, { stage: "work", runId: "1", harness: "codex" }); + const progress = progressReporter(repoOf("owner/repo"), 7, { token: () => "t" }, { stage: "work", runId: "1", harness: "codex" }); delete process.env.FACTORY_LOCAL; const rec = await runHarness({ harness: "codex", model: "gpt-6.1-sol", prompt: "p", maxUsd: 0, cwd: process.env.FAKE_STATE, env: { PATH: process.env.PATH, FAKE_STATE: process.env.FAKE_STATE }, timeoutMs: 60_000, label: "work-1" }); await progress("validation"); @@ -139,6 +139,8 @@ test("a stage killed after a real pass leaves that pass's dollars in its default * - mid: a phase line while the pass runs, then killed; the pass runs on and finishes in the session; * - term: SIGTERM while the pass runs (as `factory serve` stops a stage), with no write since the pass began; the pass * runs in its own process group, as locally, and is killed with the stage; + * - term-unsent: as term, but GitHub is unreachable for the stop's write (nothing sent); + * - term-5xx: as term, but GitHub answers the stop's write with a 502 (it may have landed); * - failed: a phase line that lands, a second pass, a phase line that fails to land, then killed. * Returns every stage's last spend line. */ @@ -149,7 +151,12 @@ async function stages(plan) { mkdirSync(join(root, "bin")); mkdirSync(ws); fakeCodex(join(root, "bin")); - const env = { GITHUB_API_URL: gh.url, FAKE_STATE: root, CODEX_HOME: join(root, "codex"), PATH: `${join(root, "bin")}:${process.env.PATH}` }; + // An address nothing listens on: a request to it is refused before anything is sent. + const dead = createServer(); + await new Promise((r) => dead.listen(0, "127.0.0.1", r)); + const deadUrl = `http://127.0.0.1:${dead.address().port}`; + await new Promise((r) => dead.close(r)); + const env = { GITHUB_API_URL: gh.url, FAKE_DEAD: deadUrl, FAKE_STATE: root, CODEX_HOME: join(root, "codex"), PATH: `${join(root, "bin")}:${process.env.PATH}` }; const script = (name, how) => ` import { existsSync, mkdirSync, writeFileSync } from "node:fs"; import { join } from "node:path"; @@ -163,17 +170,21 @@ async function stages(plan) { const usageDir = join(root, "ws", "usage-work-" + ${JSON.stringify(name)}); mkdirSync(usageDir, { recursive: true }); process.env.FACTORY_LOCAL = "1"; - const progress = progressReporter(repoOf("owner/repo"), 7, { token: "t" }, { stage: "work", runId: ${JSON.stringify(name)}, harness: "codex" }); + // The token as a stage holds it: TokenKeeper's getter (worker.mjs, review.mjs). + const progress = progressReporter(repoOf("owner/repo"), 7, { token: () => "t" }, { stage: "work", runId: ${JSON.stringify(name)}, harness: "codex" }); const resume = (await restoreTranscript({ checkout: "/c", workDir: join(root, "ws"), harness: "codex" })).id; delete process.env.FACTORY_LOCAL; await progress("harness"); - const local = how === "term" ? { FACTORY_LOCAL_SANDBOX: JSON.stringify({ addDirs: [], clone: null }), FAKE_LINGER: "60000" } : {}; - const pass = () => runHarness({ harness: "codex", model: "gpt-6.1-sol", prompt: "p", maxUsd: 0, resume, cwd: root, env: { PATH: process.env.PATH, FAKE_STATE: root, FAKE_SPLIT: how === "mid" || how === "term" ? "1" : "0", ...local }, timeoutMs: 60_000, eventsFile: join(usageDir, "events.jsonl"), label: "work-1" }); + const local = how.startsWith("term") ? { FACTORY_LOCAL_SANDBOX: JSON.stringify({ addDirs: [], clone: null }), FAKE_LINGER: "60000" } : {}; + const pass = () => runHarness({ harness: "codex", model: "gpt-6.1-sol", prompt: "p", maxUsd: 0, resume, cwd: root, env: { PATH: process.env.PATH, FAKE_STATE: root, FAKE_SPLIT: how === "mid" || how.startsWith("term") ? "1" : "0", ...local }, timeoutMs: 60_000, eventsFile: join(usageDir, "events.jsonl"), label: "work-1" }); const run = pass(); - if (how === "mid" || how === "term") { + if (how === "mid" || how.startsWith("term")) { const until = Date.now() + 10_000; while (!((await stageSpendSoFar()).usd > 0) && Date.now() < until) await new Promise((r) => setTimeout(r, 20)); - if (how === "term") { + if (how.startsWith("term")) { + // The status line the stop's write carries (no write now: the last was moments ago), or where it goes. + if (how === "term-5xx") progress.live("FAIL-THIS", { everyMs: 3_600_000 }); + if (how === "term-unsent") process.env.GITHUB_API_URL = process.env.FAKE_DEAD; process.stdout.write("spent\\n"); await run; process.exit(1); @@ -206,14 +217,14 @@ async function stages(plan) { try { for (const [i, how] of plan.entries()) { const s = stageProcess(script(`s${i}`, how), env); - if (how === "term") { + if (how.startsWith("term")) { const until = Date.now() + 15_000; while (!/spent/.test(s.out()) && s.child.exitCode === null && Date.now() < until) await new Promise((r) => setTimeout(r, 20)); s.child.kill("SIGTERM"); } const code = await s.exited; assert.ok(!s.err(), s.err()); - if (how === "term") assert.equal(code, 143, "the stage exits as SIGTERM ends it"); + if (how.startsWith("term")) assert.equal(code, 143, "the stage exits as SIGTERM ends it"); // A killed stage's pass runs on: the next stage starts once it has finished. const until = Date.now() + 15_000; if (how === "mid") while (!existsSync(join(root, "done")) && Date.now() < until) await new Promise((r) => setTimeout(r, 50)); @@ -237,8 +248,13 @@ for (const [how, killed, own, passes] of [ ["mid", "in the middle of a pass that then runs on", PASS / 2, 3], ["failed", "after a second pass whose write failed to land", PASS, 4], ["term", "by SIGTERM in the middle of a pass, before a write shows it", PASS / 2, 2.5], + // Its stop write never sent: no session totals recorded, so the resuming stage counts the killed half pass. + ["term-unsent", "by SIGTERM in the middle of a pass, its stop write never sent", 0, 2.5], + // Its stop write answered 502, so it may have landed: the totals are recorded, and the half pass is never counted twice + // (here it did not land, so it is not counted at all: a lower bound). + ["term-5xx", "by SIGTERM in the middle of a pass, its stop write answered 502", 0, 2], ]) { - test(`a stage killed ${killed}, then resumed: the stages' lines sum to the spend, each pass counted once`, async () => { + test(`a stage killed ${killed}, then resumed: the stages' lines sum to ${passes} passes' spend, none counted twice`, async () => { const lines = await stages(["end", how, "end"]); const sum = lines.reduce((s, l) => s + l.usd, 0); assert.ok(near(sum, passes * PASS), `${JSON.stringify(lines)} sum to ${sum}, not ${passes * PASS}`); @@ -254,6 +270,67 @@ test("a stage stopped by SIGTERM before any status write shows its first pass le assert.ok(near(line.usd, PASS / 2) && line.usd > 0 && line.usdLowerBound === true, JSON.stringify(line)); }); +test("restSync: a getter's token, the held secrets redacted here, and whether the request may have landed", async () => { + const seen = []; + const server = createServer((req, res) => { + let b = ""; + req.on("data", (c) => (b += c)); + req.on("end", () => { + seen.push({ url: req.url, auth: req.headers.authorization, body: b }); + if (req.url === "/hang") return; + res.setHeader("content-type", "application/json"); + if (req.url === "/422" || req.url === "/502") { + res.statusCode = Number(req.url.slice(1)); + return res.end('{"message":"no"}'); + } + res.end('{"id":5}'); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const base = `http://127.0.0.1:${server.address().port}`; + const dead = createServer(); + await new Promise((r) => dead.listen(0, "127.0.0.1", r)); + const deadUrl = `http://127.0.0.1:${dead.address().port}`; + await new Promise((r) => dead.close(r)); + const script = ` + import { restSync } from ${JSON.stringify(LIB("rest-sync.mjs"))}; + import { holdSecrets } from ${JSON.stringify(LIB("inputs.mjs"))}; + holdSecrets(["WORKERSECRETVALUE123"]); + const call = (url, token = () => "tok-1") => { + const r = restSync(url, { method: "PATCH", body: { body: "status: echo WORKERSECRETVALUE123" }, token }, { ms: 1500 }); + return r.ok ? { ok: true, data: r.data } : { ok: false, sent: r.sent }; + }; + console.log(JSON.stringify({ + landed: call(${JSON.stringify(base)} + "/ok"), + refused: call(${JSON.stringify(base)} + "/422"), + failed: call(${JSON.stringify(base)} + "/502"), + silent: call(${JSON.stringify(base)} + "/hang"), + unreachable: call(${JSON.stringify(deadUrl)} + "/x"), + noToken: call(${JSON.stringify(base)} + "/never", () => { throw new Error("no token"); }), + })); + `; + const s = stageProcess(script, {}); + const code = await s.exited; + server.closeAllConnections(); + server.close(); + assert.equal(code, 0, s.err()); + assert.deepEqual(JSON.parse(s.out().trim().split("\n").at(-1)), { + landed: { ok: true, data: { id: 5 } }, + refused: { ok: false, sent: false }, + failed: { ok: false, sent: true }, + silent: { ok: false, sent: true }, + unreachable: { ok: false, sent: false }, + noToken: { ok: false, sent: false }, + }); + // One try each (no retry), none for the token that failed; the getter's token sent, the held secret never. + assert.deepEqual(seen.map((x) => x.url), ["/ok", "/422", "/502", "/hang"]); + assert.equal(seen[0].auth, "Bearer tok-1"); + for (const x of seen) { + assert.doesNotMatch(x.body, /WORKERSECRETVALUE123/); + assert.match(x.body, /echo \*\*\*/); + } +}); + test("a final write waits for a status write in flight, and its spend line replaces the interim one", async () => { const { progressReporter } = await import("../scripts/lib/session.mjs"); const { repoOf } = await import("../scripts/lib/github.mjs"); From 276d802ef6768c329bebbed9927ab4a65d1bf1ac Mon Sep 17 00:00:00 2001 From: Kirk Marple Date: Fri, 13 Nov 2026 17:24:45 -0800 Subject: [PATCH 3/3] LOCAL.md: the stop write's landed and not-landed cases as the code decides them Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CS53Gu2D3bE8mKgFN92XP5 --- docs/LOCAL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/LOCAL.md b/docs/LOCAL.md index f09cfb4..f3f5dee 100644 --- a/docs/LOCAL.md +++ b/docs/LOCAL.md @@ -19,7 +19,7 @@ Factory's worker and reviewer, run on your own machine instead of a GitHub runne - The harness holds no GitHub token at all: what it needs from GitHub is in `$FACTORY_IN`. Factory's own calls use one fine-grained token for this one repository, never your `gh` login; the token reaches each stage on a pipe it reads once, never its environment or a file, and Factory's own git gets it by per-process env config. No sandbox can read `/proc` (Claude Code and `srt`; Codex runs in its own pid namespace). Copilot's sandbox runs each command in a pid namespace of its own too, whose pid 1 is the bubblewrap Copilot starts with its whole environment, the token included (#100); on Linux Factory's own `bwrap`, first on Copilot's PATH, starts the real one with an empty environment, and bubblewrap sets each command's environment itself. Copilot also writes each command's network rules into its own `TMPDIR` before it loads them, and opens that directory to every command, so a command could rewrite the next one's rules and reach past Copilot's proxy to `srt`'s (whose list adds Copilot's API hosts). On Linux Copilot therefore runs with a `TMPDIR` of its own, in the stage's usage directory, made afresh before every start and denied to its commands, and writes no logs (Copilot opens its log directory to every command whatever its policy says); Factory's `bwrap` gives each command the stage's temp directory as its `TMPDIR`. Copilot's sandbox binds only the system's own directories (`/usr/bin`, `/usr/lib`, `/etc` and the like) into each command, so a node your version manager keeps in your home (nvm, mise, asdf, Volta) would be out of its commands' reach (#111). The stage's node is the first `node` on your PATH, by its real path (where that is a version manager's shim, the node the shim chose, which runs Factory). Its install prefix (`bin`'s parent, so npm comes too) is read-only in each command's sandbox, with its `etc` (a global npmrc) denied, and Factory's `bwrap` puts its `bin` first on each command's PATH, after Factory's git. Nothing else of your home is bound. The prefix is never your home or a directory above it, nor in or above a credential store, this repository's `.factory` directory or the stages' temp directories, where a sandbox could swap it for a link. The same holds for the repository's package manager, and for Claude Code too, whose sandbox likewise shows nothing of your home but its working and added directories (#137): a repository whose `packageManager` (else its lockfile) names pnpm or yarn gets the one your PATH resolves to (corepack's shim, its own install, or, behind a version manager's shim, the version `packageManager` pins in that manager's installs: mise's and asdf's `installs/pnpm/`, Volta's `tools/image/pnpm/`; Factory never runs the manager, which would read the clone's own configuration), linked from a `toolbin` directory in the stage's usage directory, which no sandbox may write, and first on each command's PATH (Copilot through Factory's `bwrap`, Claude Code through `CLAUDE_ENV_FILE`). Only the npm package it belongs to (corepack, pnpm) is readable, read-only, else the executable alone; corepack's downloads live in the Objective's own `COREPACK_HOME`. Codex's sandbox reads the whole system but the credential stores, so its commands find both as you do. Before every Copilot start a probe checks all of this: a Copilot session in the outer sandbox, with the stage's policy and no credential, against a model Factory serves on that sandbox's own loopback (no network, no bill), runs one command. That command must find a random marker Copilot holds in no environment it can read, Copilot's network rules nowhere it can list, Copilot's `TMPDIR` read-only and no Copilot log it can read, nothing it writes in your home, the probe's Copilot home or Copilot's release left there, the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused; the stage's node must be the `node` it finds, and it, the `npm` beside it and the repository's pnpm or yarn must run (else the stage Blocks with `the stage's toolchain does not run in a command`); and it must end with its pass line. A broken rule Blocks the stage (`sandbox`) before any command runs; a probe that leaves no result is tried once more, then Blocks. It takes a second or two. Claude Code on Linux gets the same probe before every start (it has no command that runs one command in its sandbox either): a Claude Code session launched as the run is, with a throwaway config directory in the usage directory, a stand-in key and no credential of yours, against a stand-in for the Anthropic API on its outer sandbox's own loopback (no network, no bill), runs one command. That command must find and run the stage's node and the repository's pnpm or yarn, find a random marker Claude Code holds in no environment it can read, leave nothing it writes in your home or the usage directory, find the clone's `.git/config` read-only, the stage's temp directory writable and a host off the list refused, and end with its pass line; a broken rule Blocks the stage (`sandbox`) with the rule. `doctor.mjs --local` runs it too (`claude sandbox: start`, around a throwaway clone with your checkout's `package.json`), and names what local mode resolves: `node: v24.20.0 → /usr/bin/node`, `package manager: pnpm 12.18.0 (corepack) → …/corepack/dist/pnpm.js`; a pinned package manager it cannot resolve is a FAIL. - Each Objective gets its own clone under `/.factory//`, which is git-excluded. `serve` removes it once the issue is closed; a Blocked Objective keeps its directory so a re-label picks up where it stood. Its installed dependencies (ignored files) carry over between stages and runs, but nothing unpushed does: each stage starts on the branch exactly as GitHub has it, and removes the untracked files, uncommitted changes and stopped rebase an earlier stage or run left there. It also removes the git lock files (`.git/*.lock`, `.git/refs/**/*.lock`) a run interrupted by a reboot or a killed `serve` left, and logs each: a lock older than the machine's last boot always goes; a newer one goes only where `/proc` shows no process with its working directory or an open file in the clone (Linux; on macOS only the boot rule applies). Otherwise the lock stays and the log names it. - GitHub holds all state. A claim comment with a heartbeat marks an Objective a local loop is running, so the Action and other loops leave it alone. -- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. A stage stopped by a signal (Ctrl-C, SIGTERM or SIGHUP, or its loop stopping) writes that reading once more as it stops, after its harness is killed, when it is more than the line shows, waiting at most 5 s for GitHub. The reading is what the harness has reported so far: Claude Code reports each model response and Copilot each usage checkpoint, so their passes killed before the next write still count; Codex reports usage only when a turn ends, so a Codex pass killed in its first turn still reads $0. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. The stopping write records them when it landed or may have (a 5xx answer, no answer in 5 s), so it is never counted twice; one that may have landed but did not leaves that part uncounted. When it certainly did not land (GitHub unreachable, or a refusal), nothing is recorded and the resuming stage counts that part. Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered. +- Each local run writes its dollars into its progress comment, so the `daily_usd` cap counts local and Actions spend together, on any machine. Every stage writes one: $0 when it ran no harness pass. While a stage runs, each write of its comment carries the spend Factory has read so far, marked `usdLowerBound` (`usdUpperBound` when part of it is priced at a price table's dearest rates, for a model with no row), so a stage that is killed or loses its machine still counts what it spent. A stage stopped by a signal (Ctrl-C, SIGTERM or SIGHUP, or its loop stopping) writes that reading once more as it stops, after its harness is killed, when it is more than the line shows, waiting at most 5 s for GitHub. The reading is what the harness has reported so far: Claude Code reports each model response and Copilot each usage checkpoint, so their passes killed before the next write still count; Codex reports usage only when a turn ends, so a Codex pass killed before its turn ends adds nothing for that pass. After each such write lands, the stage records in its usage directory (`session-totals.json`) the session totals that line counted; a stage that resumes the live session after a killed one starts from them, so what the killed stage's line counted is not counted again and what it missed is counted by the resuming stage. The stopping write records them when it landed or may have (a 5xx answer, or no answer in 5 s, which includes a connection still being opened then), so it is never counted twice; one that may have landed but did not leaves that part uncounted. When it certainly did not land (the connection refused or its host not found at once, or a refusal), nothing is recorded and the resuming stage counts that part. Two exceptions: a reading priced at a table's dearest rates is not recorded, so the resuming stage counts that pass again (an over-count, never an under-count); and a stage killed after its result.json but before its last write keeps its last interim line, and the stage after it starts from that result.json. A `--here` run writes no spend line; it stays unmetered. ## Platforms