Repository navigation
fix(kv-unified): close the predecessor's open receipt flight before a new activation submits - #154
Conversation
… new activation submits The kv-unified wire receipt fires per provider attempt, before the adapter call, and only that attempt's usage event settles it (driveStream's usage handler). An attempt that dies without usage — transport error, idle timeout, a framework cancel for a budget restart or endTurn — left its flight open until the old driveStream's `finally` drained it, and every successor path (error-policy retry from inside the failed stream's event loop, budget restart, the next wake after an abort) starts the new stream first. The successor's first receipt then hit the ledger's single-flight guard and the recovery itself failed: [inference-failed] agent=devops consecutive=1: kv-unified submission devops:46:1789544019150:4 is still in flight (2026-09-16 07:33Z) Agent now keeps the live receipt queue and, at the start of every activation, fails whatever the previous one left open before any new receipt can begin. The per-stream `finally` drain stays as the backstop; whichever runs first empties the shared queue. Also: inference-log records below the blob threshold are persisted through their JSON view. The compiled request carries the receipt hook (a function), which Chronicle rejected with "JS functions cannot be represented as a serde_json::Value" — thrown from the failure path it was logging. Test: framework-level repro (real kv-unified strategy, mock membrane whose first attempt dies after the receipt) and an Agent-level ordering test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Anarchid
left a comment
There was a problem hiding this comment.
🟢 CLEAR
Reviewer: Codex (GPT-5.6 Sol)
Reviewed head: caec770447c164f3873d303e548f43f8dcbccffe
No material findings.
The predecessor/successor ownership is coherent: the agent-level queue remains shared with the original stream's teardown closures, the new activation drains the predecessor before installing its own queue, and the old finally therefore becomes an idempotent no-op. The inference-log adjustment also stores the same JSON view used for thresholding, so the receipt callback function can no longer escape into Chronicle on the small-record path. The focused framework reproduction covers the critical ordering: the failed submission is closed before the successor's first provider pull, while the successor remains available for usage acceptance.
Tooling results
git diff --check 8b612f5e28ecdda4b89f6138043e7c45edf53263...caec770447c164f3873d303e548f43f8dcbccffe— passed.npx tsc --noEmit— passed with TypeScript 5.9.3.node --import tsx --test test/kv-unified-open-flight-retry.test.ts test/kv-unified-wiring.test.ts— 2/2 test files passed.node --import tsx --test test/framework.test.ts— passed.node --import tsx --test test/state-scaling.test.ts test/state-scaling-e2e.test.ts— 2/2 test files passed.npm run build— passed, including the postbuild executable-bit step.npm test— the local compiled runner repeated the repository's known stall after ten passing files and emitted no failure; it was stopped after the bounded wait. The five exact-head GitHub checks (four Node/platform build-and-test jobs plus changelog validation) are green.
Verdict: no blocker or non-blocking follow-up identified in the changed paths. Review confidence is high for the receipt-flight ordering and JSON persistence fix; the only limitation is the pre-existing local full-suite runner stall described above.
— Reviewed by GPT-5.6 Sol via OpenAI Codex.
Symptom
Devops agent (conhost d1ce7d2 = AF 0.13.0 + CM 0.8.0 + membrane 0.5.85,
foldingStrategy: kv-unified, gpt-6-astra), 2026-09-16 07:33:47Z:First failure in its streak; no provider error journaled before it. The receipt
:4was created at 07:33:39Z, a fresh compile ran at 07:33:47Z and threw. The turn was lost; the agent resumed on the next heartbeat.Mechanism
The kv-unified wire receipt fires once per provider attempt, inside membrane's
streamOnce, before the adapter call. AF's hook binds it withstrategy.beginKvUnifiedSubmission, whose ledger is single-flight. Only that attempt'susageevent accepts it (driveStream's usage handler), or the old driveStream'sfinallyfails it.An attempt that dies without a usage event — transport error, idle timeout, a framework cancel for a budget restart or endTurn — leaves the flight open until that
finally. Every path that starts a successor stream runs before it:await startAgentStream()from inside the failed stream's event loop;abortedstream resets the agent and the next wake starts immediately (and prints nothing to the journal — onlyinference:failedhas a console printer, which is why the box shows no earlier failure).The successor's first receipt then hits the ledger's guard and the recovery itself fails.
Agent.startStreamWithInjectionsrequiresidle, so the successor is always a new stream id and the error names the predecessor's pending id — exactly the shape above.Fix
Agentkeeps the live receipt queue and, at the start of every activation, fails whatever the previous activation left open before any new receipt can begin (failOpenKvSubmissions). The per-streamfinallydrain stays as the backstop; whichever runs first empties the shared queue, and failing an already-settled id is a ledger no-op.logInference: records below the blob threshold are persisted through their JSON view. The compiled request carries the receipt hook (a function), which Chronicle rejected withJS functions cannot be represented as a serde_json::Value— thrown from the failure path it was logging. Production requests take the blob path, so this bit only small requests (tests), but it masked the repro on first run.Companion: anima-research/context-manager#101 makes the ledger tolerate the same race on its side (defence in depth; either fix alone clears the incident).
Tests
test/kv-unified-open-flight-retry.test.ts: framework-level repro with the real CM 0.8.0 kv-unified strategy and a mock membrane whose first attempt dies after the receipt. Onmainit reproduces the production error verbatim; with the fix the retry completes the turn and the dead flight is failed before the successor's first pull.test/kv-unified-wiring.test.ts: Agent-level ordering test (activation 2 fails activation 1's flight before its own receipt begins; the predecessor's late drain finds nothing; the successor's flight stays live for its usage event).🤖 Generated with Claude Code
https://claude.ai/code/session_01AhZDu1aVjg2wmu9kceQcxa