Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .changeset/cli-contract-busy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
"@managers/server": minor
---

`managers busy`, `GET /api/busy`, a `service stop` that refuses when Managers is mid-turn, and a SIGTERM that finishes or records the work under way

Wave F2 of the CLI-contract rollout (edspencer/cli-contract SPEC 0.2, §4), ported from Paddock 0.81.

**New: `managers busy [--json]`.** Asks the running instance whether it is in the
middle of something a restart would lose. It exits **0** not busy, **75** busy,
**3** when nothing answers, and **1** when something answered but not as
Managers' busy check. 75 is `EX_TEMPFAIL`, which Watchtower's
`lifecycle.pre-update` hook reads as "skip this one".

**`service stop` and `service restart` now run that check first and refuse with
exit 75 unless `--force` is given.** A script that stops Managers unconditionally
needs `--force`, or needs to handle 75: that is the behaviour change in this
release. `service uninstall` and `service start` don't check. A forced stop
prints the message it would have refused with, prefixed `Overriding:`.
**`--force` anywhere else is now a usage error (exit 2)** rather than being
silently ignored. When the busy work is chat turns, the refusal says to run the
command from a terminal rather than from inside a Managers chat.

**New: `GET /api/busy`**, answering `{ busy, version, counts }` with four counts:
`chat` (chat turns running, triggers and schedules included), `background`
(background tasks a live session still holds after its turn), `job` (herdctl
scheduler jobs) and **`run`** (Managers trigger runs in flight: from the moment a
fire writes its `running` record until that record is finished and committed,
which is wider than the turn at both ends). No authentication, because it
carries counts and nothing else; in-memory state only.

**New: `<data-dir>/runtime.json`**, written 0600 once the server is listening and
removed when it stops, holding `{ pid, port, url, version, startedAt }`.

**SIGTERM now finishes or records the work under way, instead of closing at
once.** If chat turns or trigger runs are in flight, a stop waits up to 8 seconds
for them to end by themselves; whatever is left is recorded — each trigger run's
record is finished as `failed`, `error: interrupted by shutdown` (through its own
completion hook, so a consolidation still gets its `#reflection` episode and the
record is committed), and the log names what was cut short. An idle instance
stops at once. Signals are idempotent and the whole stop is bounded at 15
seconds, inside launchd's 20-second kill budget. Under Docker, whose default
`docker stop` timeout is 10 seconds, give the container a `stop_grace_period` of
at least 20 seconds to let the wait run.
9 changes: 5 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,11 @@ jobs:
# The CLI contract, checked from the outside by @edspencer/cli-conformance: `--version` is
# bare, `--help` exits 0, an unknown flag exits 2 with the usage after the message,
# `service status` exits 4 with nothing installed and its --json matches the shared schema,
# `serve`'s first stdout line is `Managers <version>: <url>`, and /api/health answers with
# that same version. `cli-contract.json` at the root says which checks apply and why the
# two that don't (`busy`, wave C1) don't — a NEWER runner's new check fails until this repo
# says whether it applies, which is the point.
# `busy` exits 3 with nothing running and its --json matches the schema too, `serve`'s first
# stdout line is `Managers <version>: <url>`, and /api/health answers with that same version.
# `cli-contract.json` at the root says which checks apply; since wave C1 every one of the
# nine does, with no skips — and a NEWER runner's new check fails until this repo says
# whether IT applies, which is the point.
#
# `--dir dist-npm`, not the repo root: Managers' publishable package is SYNTHESIZED from two
# private workspaces by scripts/make-npm-package.mjs (see RELEASING.md), so `npm pack` at the
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,11 @@ runs it in the background from login (a launchd LaunchAgent on macOS, a
`systemd --user` unit on Linux), with `uninstall`, `start`, `stop`, `restart`,
`status [--json]` and `logs [-f] [-n <lines>]`. `service status` exits **0** when
it is up and answering, **3** when installed but not, and **4** when there is no
unit; a bad command, flag or value exits **2**. The CLI follows the shared
unit; a bad command, flag or value exits **2**. `managers busy` exits **75** while
a chat turn or trigger run is in flight (0 when idle, 3 when nothing is running),
and `service stop`/`restart` refuse with 75 unless given `--force`. A SIGTERM
waits briefly for in-flight work, then records what it cut short; under Docker,
give the container a `stop_grace_period` of 20 s or more. The CLI follows the shared
[CLI contract](https://github.com/edspencer/cli-contract) (`cli-contract.json`
says which of its checks apply).

Expand Down
8 changes: 2 additions & 6 deletions cli-contract.json
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,8 @@
"unknown-flag": true,
"service-status": true,
"service-status-json": true,
"busy": {
"skip": "no `managers busy` until wave F2 of the CLI-contract rollout"
},
"busy-json": {
"skip": "no `managers busy` until wave F2 of the CLI-contract rollout"
},
"busy": true,
"busy-json": true,
"serve-banner": true,
"serve-health": true
}
Expand Down
9 changes: 8 additions & 1 deletion packages/server/src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ import { installHerdctlBridgeLoopbackBind } from "./herdctl-bridge-bind.js";
import { renderIndexHtml } from "./brand.js";
import { makeChatHandler } from "./ws.js";
import type { SessionHub } from "./session-hub.js";
import type { BackgroundRegistry } from "./background-live.js";
import { SweepService } from "./sweep.js";
import { ArchiveStore } from "./archive.js";
import { StarStore } from "./star.js";
Expand Down Expand Up @@ -107,6 +108,12 @@ export interface BuiltApp {
* same way, via the route context).
*/
hub: SessionHub;
/**
* The live background-work registry (#604), exposed for the same reason `hub`
* is — it is WS-layer state a non-socket caller has to read. `GET /api/busy`
* reaches it through the route context; a test reaches it here.
*/
background: BackgroundRegistry;
/** Tear down the fleet + close the server (no process.exit, for tests). */
close: () => Promise<void>;
}
Expand Down Expand Up @@ -655,5 +662,5 @@ export async function buildApp(opts: BuildAppOptions = {}): Promise<BuiltApp> {
// `hub` rides along like every other built piece: it is the WS layer's turn
// bookkeeping, and the only place a caller can observe or annotate a turn
// without a live socket (see SessionHub.noteCancel).
return { app, cfg, projects, herdctl, git, githubAuth, sweep, archive, star, readState, unread, parentDetach, runProvenance, queuedMessage, transcriber, events, triggers, managers, autocommit, dataSync, hub: chatHandler.managementOpsContext.hub, close };
return { app, cfg, projects, herdctl, git, githubAuth, sweep, archive, star, readState, unread, parentDetach, runProvenance, queuedMessage, transcriber, events, triggers, managers, autocommit, dataSync, hub: chatHandler.managementOpsContext.hub, background: chatHandler.managementOpsContext.background, close };
}
20 changes: 16 additions & 4 deletions packages/server/src/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,10 @@
* paddock holds no key material, only the JWKS URL, and
* rejects missing/invalid/expired tokens with 401.
*
* Paddock's health route (`/api/health`) is always exempt so the proxy and
* monitoring can poll a server that is otherwise locked down. Only registered
* routes may be exempted — see HEALTH_PATHS below for why (issue #569).
* Managers' health route (`/api/health`) is always exempt so the proxy and
* monitoring can poll a server that is otherwise locked down, as is the CLI
* contract's `/api/busy` — which carries counts and nothing else. Only
* registered routes may be exempted — see HEALTH_PATHS below for why (#569).
*
* Registered in app.ts BEFORE the routes so the `onRequest` hook guards every
* REST + WS request. The decorator + hook are added directly to the root app
Expand Down Expand Up @@ -78,7 +79,18 @@ declare module "fastify" {
* to a real route returning JSON, so the set cannot drift from the router again.
*/
export const HEALTH_PATHS: ReadonlySet<string> = new Set<string>([
"/api/health", // paddock's own health route (routes/meta.ts)
"/api/health", // Managers' own health route (routes/meta.ts)
// The CLI contract's busy check (SPEC §4, §8). Exempt for the same reason
// health is, and safe for a reason health does not need: the answer is four
// integers and the version, with no chat title, session id or project name in
// it — SPEC §10 keeps `reasons` out until there is a way for a caller to prove
// it is local. `managers service stop`, `managers update` and Watchtower's
// pre-update hook all ask it with no credential to offer, and a 401 would be
// read by `cli/busy.ts` as exit 1 rather than as "not busy" — so gating it
// would fail every stop on an authenticated instance.
//
// If this answer ever grows a label, this line has to be revisited with it.
"/api/busy", // routes/meta.ts
]);

/**
Expand Down
24 changes: 24 additions & 0 deletions packages/server/src/background-live.ts
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,30 @@ export class BackgroundRegistry {
return !!e && e.tasks.size > 0;
}

/**
* How many live background tasks there are, across the sessions `keep`
* accepts.
*
* The `/api/busy` check (CLI contract SPEC §4) passes herdctl's SDK liveness
* as `keep`, for the reason the per-process caveat in the module header gives:
* nothing resets this set when a one-shot `claude` exits, so on
* `driveMode: batch` a finished turn's tasks sit here indefinitely. Filtering
* on a live session is what stops a stale entry reporting a batch instance
* busy for good — and it costs nothing in `session` mode, where the reaper
* holds a session open precisely BECAUSE it has background work.
*
* Counted rather than listed so the busy check allocates nothing per task
* (§4: in-memory, well under a second). {@link snapshot} is the one that
* builds arrays, and it is for the connect-time frame.
*/
liveTaskCount(keep: (sessionId: string) => boolean = () => true): number {
let n = 0;
for (const [sessionId, e] of this.bySession) {
if (e.tasks.size > 0 && keep(sessionId)) n += e.tasks.size;
}
return n;
}

/**
* Forget a session's set. Called when the session's process restarts or its
* stream ends — see the per-process caveat in the module header.
Expand Down
133 changes: 133 additions & 0 deletions packages/server/src/busy.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
/**
* What `GET /api/busy` answers (CLI contract SPEC §4, ROADMAP "Busy, per tool").
*
* ## What "busy" means here
*
* Only work **a restart would lose or cut short** — the kind someone would be
* annoyed to find killed. For Managers that is four things: Paddock's three, which
* the ROADMAP row names, and Managers' own trigger runs:
*
* - **`chat`** — a chat turn running. This is the one that matters: a turn is a
* live `claude` subprocess writing a transcript, and `herdctl.stop()` ends it
* with `waitForJobs: false`. A fired trigger or a cron schedule is NOT a
* fourth kind, because every one of them runs as a first-class chat on the
* hub (`ws-triggers.ts` — one `startAgentTurn` for every trigger type), so it
* is already counted here.
* - **`background`** — background work a live SDK session is still holding
* after its turn ended (#604): a `run_in_background` shell, a sub-agent, a
* `Monitor`. These outlive the `Turn` record, so the hub cannot see them.
* - **`job`** — herdctl's scheduler running a job. Managers routes a keeper's
* schedule fire back onto the hub, so in practice this double-counts with
* `chat` for the window where both are true; a count that is occasionally
* high by one is the right failure direction for a check whose answer is
* "wait a moment".
* - **`run`** — a Managers trigger run in flight (`ManagersState.liveRuns`):
* from the moment a fire writes its `status: running` record until its
* completion hook has finished that record and committed it. That window is
* wider than the chat turn at both ends — the briefing is built before the
* turn starts, and the record, the `#reflection` episode and the commit are
* written after it ends — and a restart inside it leaves a run failed with
* "interrupted by restart". The overlap with `chat` while the turn runs is
* the same deliberate high-by-one as `job`.
*
* ## What is deliberately NOT busy
*
* - **Idle live sessions.** A session the reaper is holding open with no turn
* and no background task resumes after a restart; that is the whole point of
* a resumable session.
* - **The post-turn sweeper.** It is re-runnable from its watermark, so losing
* one is a re-sweep rather than lost work. The ROADMAP row leaves it out too.
* - **Open browsers.** A page reconnects to `/ws` and replays.
*
* ## Cheap, and in-memory only
*
* §4 requires this to read in-memory state, never walk the disk, and answer in
* well under a second — it is called by Watchtower's pre-update hook and by
* every `service stop`. So each source here is a `Map.size` or a filtered
* iteration over one, and {@link collectBusy} is synchronous.
*
* That rules out the obvious source for `job`. `fleetStatus().counts.runningJobs`
* is what the ROADMAP names, and `FleetStatus` is built by `getAgentInfo()`,
* which **reads the fleet state file off disk** on every call (herdctl's
* `StatusQueries.readFleetStateSnapshot`) — the same re-parse that was 61% of
* this server's CPU once. `counts.runningJobs` reduces to
* `Σ scheduler.getRunningJobCount(agent)`, and the scheduler also publishes
* `getTotalRunningJobCount()` over the same in-memory map, so Managers asks the
* scheduler directly for an identical number with no file read.
*
* ## No labels, ids or titles
*
* Counts only, for every caller, until SPEC §10's first question (how a CLI
* proves it is local) is designed. That is why this route needs no auth: there
* is nothing in the answer to protect. A chat's TITLE would be a different
* matter, and is exactly what `reasons` will carry once there is a way to know
* who is asking — so nothing here may grow a string field by accident.
*/

/** The non-zero kinds the CLI's message names: `Managers is busy: 2 chats are running.` */
export interface BusyCounts {
/** Chat turns running — a human turn, a fired trigger, a cron fire. */
chat: number;
/** Background tasks live sessions are still holding after their turn (#604). */
background: number;
/** herdctl scheduler jobs in flight. */
job: number;
/** Managers trigger runs in flight (a `running` run record this process owns). */
run: number;
}

/** The `/api/busy` body (SPEC §4). */
export interface BusyBody {
busy: boolean;
version: string;
counts: BusyCounts;
}

/**
* The in-memory readings, injected so the computation is unit-testable
* without a fleet, a hub or a socket. Each must be cheap enough to call on every
* request — see the module header.
*/
export interface BusySources {
/** Sessions with a running turn: `hub.runningSessions().length`. */
runningChats: () => number;
/** Live background tasks, counted only in sessions whose SDK session is live. */
backgroundTasks: () => number;
/** Scheduler jobs running right now. */
runningJobs: () => number;
/** Trigger runs in flight in this process: `managers.liveRuns.size`. Absent → 0. */
runningRuns?: () => number;
}

/**
* Read the counts and decide busy.
*
* `busy` is `counts` having any non-zero member rather than a separately
* computed predicate, so the sentence the CLI prints can never disagree with the
* boolean it printed it for. A source that throws counts as zero: a busy check
* that 500s is worse than one that under-reports, because a 500 is read by
* `busy-client.ts` as "something answered, but not as Managers' busy check" —
* exit 1 — which fails a `service stop` outright instead of letting it proceed.
*/
export function collectBusy(sources: BusySources, version: string): BusyBody {
const counts: BusyCounts = {
chat: safeCount(sources.runningChats),
background: safeCount(sources.backgroundTasks),
job: safeCount(sources.runningJobs),
run: safeCount(sources.runningRuns ?? (() => 0)),
};
return {
busy: counts.chat > 0 || counts.background > 0 || counts.job > 0 || counts.run > 0,
version,
counts,
};
}

function safeCount(read: () => number): number {
try {
const n = read();
return Number.isInteger(n) && n > 0 ? n : 0;
} catch {
return 0;
}
}
Loading
Loading