feat(runtime): drain in-flight runs on shutdown instead of dropping them

[RatulMaharaj]

Stacked on #185. Fifth in the series.

Refs #176, deliberately not closing it. See scope below.

The problem

SIGTERM meant exit. Docker waits 10 seconds before killing the container, and that window was spent doing nothing: an in-flight run died mid-tool-call, its side effects already committed to the world.

#182 made the evidence survive. This makes the run survive.

What changed

Draining is a state, not a slower stop. RunScheduler.drain() stops accepting work and resolves once everything already accepted has finished. submit() then throws DrainingError, which the service turns into a refusal the sender actually sees, audited like a queue-full refusal. Events already queued still run, because they were accepted.

limits.drain_timeout, default 8 seconds, under docker stop's 10s grace so the normal path completes rather than being killed partway. Past it the agent stops anyway.

Runs that outlast the drain are recorded, not lost. They keep their open row, and the crash sweep from #182 marks them error_crashed at next start. That pairing is the point: draining alone would turn "died mid-run" into "died mid-run slightly later", and only the row makes it visible.

/healthz reports draining and in_flight. Both states are ok: true and only one should be sent more work, so a deployment can watch a drain rather than infer it from failures.

A bug this turned up

The first version had drain() call stop() unconditionally, which closes the store. A run that outlasted the timeout then reached closeRun and got Error: database is not open - a slow run turned into a failed write by the shutdown meant to protect it. A test caught it.

A timed-out drain now shuts down everything except the store and leaves the file to the process exit. WAL makes an unclosed database safe to reopen, and a run that finishes in the last moments still records itself.

Testing

496 pass, deno task ok clean. Five new:

• scheduler: accepted work finishes while new submits are refused, on both lane and laneless paths; an idle scheduler drains immediately; a drain waits for work queued behind a lane rather than only what is running
• service: the in-flight run completes, a late event is refused with an audit row, and the drain reports zero remaining; a run that outlasts the deadline leaves an open row and the drain reports it rather than hanging

Scope

Points 1, 2 and 5 of the issue. Not done, and why:

• Telling the triggers to stop reading (point 3). A draining agent still holds the Discord gateway and still accepts webhooks at the transport, then refuses them at the scheduler. The refusal is correct and audited; refusing at the source is better and touches every trigger.
• A goodbye frame on tty (point 4). The one case where silence is actively bad, and it needs the close-frame work in meet: end a meeting with a durable record the agent can recall #173.
• Durable delivery. The issue's own ordering note says A crash mid-run loses the run's transcript and audit trail #60 and reliability: add a durable event inbox, idempotency, and delivery retry #92 first. A crash mid-run loses the run's transcript and audit trail #60 is fix(runtime): persist a run as it happens, not after it returns #182 in this stack; reliability: add a durable event inbox, idempotency, and delivery retry #92 is not built. Until it is, a drained run's reply can still be lost between finishing and sending - the loss moves from mid-run to mid-send rather than disappearing. Worth knowing before this reads as "shutdown is solved".

Decisions from the issue

• What a drain does to the queue. Accepted events run; new ones are refused. The issue notes this becomes wrong if reliability: add a durable event inbox, idempotency, and delivery retry #92 makes the queue durable, at which point the refusal should become a hand-off. The refusal is honest today and the change is localized to one method.
• Whether a run can be too long to drain. Yes, and it is left to the timeout. Distinguishing "finish this run" from "close this session" needs the session concept from meet: end a meeting with a durable record the agent can recall #173.