Skip to content

fix(kv-unified): close the predecessor's open receipt flight before a new activation submits - #154

Merged
Anarchid merged 1 commit into
anima-research:mainfrom
Anarchid:fix/kv-unified-open-flight-on-retry
Sep 17, 2026
Merged

Anarchid merged 1 commit into
anima-research:mainfrom
Anarchid:fix/kv-unified-open-flight-on-retry

Conversation

@Anarchid

@Anarchid Anarchid commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Symptom

Devops agent (conhost d1ce7d2 = AF 0.13.0 + CM 0.8.0 + membrane 0.5.85, foldingStrategy: kv-unified, gpt-6-astra), 2026-09-16 07:33:47Z:

[inference-failed] agent=devops consecutive=1: kv-unified submission devops:46:1789544019150:4 is still in flight

First failure in its streak; no provider error journaled before it. The receipt :4 was created at 07:33:39Z, a fresh compile ran at 07:33:47Z and threw. The turn was lost; the agent resumed on the next heartbeat.

Mechanism

The kv-unified wire receipt fires once per provider attempt, inside membrane's streamOnce, before the adapter call. AF's hook binds it with strategy.beginKvUnifiedSubmission, whose ledger is single-flight. Only that attempt's usage event accepts it (driveStream's usage handler), or the old driveStream's finally fails it.

An attempt that dies without a usage event — transport error, idle timeout, a framework cancel for a budget restart or endTurn — leaves the flight open until that finally. Every path that starts a successor stream runs before it:

  • the error-policy retry is await startAgentStream() from inside the failed stream's event loop;
  • a budget/physical-window restart cancels the live stream and requeues, and the restart deliberately does not wait for the old teardown;
  • an aborted stream resets the agent and the next wake starts immediately (and prints nothing to the journal — only inference:failed has a console printer, which is why the box shows no earlier failure).

The successor's first receipt then hits the ledger's guard and the recovery itself fails. Agent.startStreamWithInjections requires idle, so the successor is always a new stream id and the error names the predecessor's pending id — exactly the shape above.

Fix

  • Agent keeps the live receipt queue and, at the start of every activation, fails whatever the previous activation left open before any new receipt can begin (failOpenKvSubmissions). The per-stream finally drain stays as the backstop; whichever runs first empties the shared queue, and failing an already-settled id is a ledger no-op.
  • logInference: records below the blob threshold are persisted through their JSON view. The compiled request carries the receipt hook (a function), which Chronicle rejected with JS functions cannot be represented as a serde_json::Value — thrown from the failure path it was logging. Production requests take the blob path, so this bit only small requests (tests), but it masked the repro on first run.

Companion: anima-research/context-manager#101 makes the ledger tolerate the same race on its side (defence in depth; either fix alone clears the incident).

Tests

  • test/kv-unified-open-flight-retry.test.ts: framework-level repro with the real CM 0.8.0 kv-unified strategy and a mock membrane whose first attempt dies after the receipt. On main it reproduces the production error verbatim; with the fix the retry completes the turn and the dead flight is failed before the successor's first pull.
  • test/kv-unified-wiring.test.ts: Agent-level ordering test (activation 2 fails activation 1's flight before its own receipt begins; the predecessor's late drain finds nothing; the successor's flight stays live for its usage event).
  • Full suite: 790/790.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AhZDu1aVjg2wmu9kceQcxa

… new activation submits

The kv-unified wire receipt fires per provider attempt, before the adapter
call, and only that attempt's usage event settles it (driveStream's usage
handler). An attempt that dies without usage — transport error, idle
timeout, a framework cancel for a budget restart or endTurn — left its
flight open until the old driveStream's `finally` drained it, and every
successor path (error-policy retry from inside the failed stream's event
loop, budget restart, the next wake after an abort) starts the new stream
first. The successor's first receipt then hit the ledger's single-flight
guard and the recovery itself failed:

  [inference-failed] agent=devops consecutive=1: kv-unified submission
  devops:46:1789544019150:4 is still in flight   (2026-09-16 07:33Z)

Agent now keeps the live receipt queue and, at the start of every
activation, fails whatever the previous one left open before any new receipt
can begin. The per-stream `finally` drain stays as the backstop; whichever
runs first empties the shared queue.

Also: inference-log records below the blob threshold are persisted through
their JSON view. The compiled request carries the receipt hook (a function),
which Chronicle rejected with "JS functions cannot be represented as a
serde_json::Value" — thrown from the failure path it was logging.

Test: framework-level repro (real kv-unified strategy, mock membrane whose
first attempt dies after the receipt) and an Agent-level ordering test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@Anarchid Anarchid left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 CLEAR

Reviewer: Codex (GPT-5.6 Sol)

Reviewed head: caec770447c164f3873d303e548f43f8dcbccffe

No material findings.

The predecessor/successor ownership is coherent: the agent-level queue remains shared with the original stream's teardown closures, the new activation drains the predecessor before installing its own queue, and the old finally therefore becomes an idempotent no-op. The inference-log adjustment also stores the same JSON view used for thresholding, so the receipt callback function can no longer escape into Chronicle on the small-record path. The focused framework reproduction covers the critical ordering: the failed submission is closed before the successor's first provider pull, while the successor remains available for usage acceptance.

Tooling results

  • git diff --check 8b612f5e28ecdda4b89f6138043e7c45edf53263...caec770447c164f3873d303e548f43f8dcbccffe — passed.
  • npx tsc --noEmit — passed with TypeScript 5.9.3.
  • node --import tsx --test test/kv-unified-open-flight-retry.test.ts test/kv-unified-wiring.test.ts — 2/2 test files passed.
  • node --import tsx --test test/framework.test.ts — passed.
  • node --import tsx --test test/state-scaling.test.ts test/state-scaling-e2e.test.ts — 2/2 test files passed.
  • npm run build — passed, including the postbuild executable-bit step.
  • npm test — the local compiled runner repeated the repository's known stall after ten passing files and emitted no failure; it was stopped after the bounded wait. The five exact-head GitHub checks (four Node/platform build-and-test jobs plus changelog validation) are green.

Verdict: no blocker or non-blocking follow-up identified in the changed paths. Review confidence is high for the receipt-flight ordering and JSON persistence fix; the only limitation is the pre-existing local full-suite runner stall described above.

— Reviewed by GPT-5.6 Sol via OpenAI Codex.

@Anarchid
Anarchid merged commit 1bd3ab9 into anima-research:main Sep 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant