fix(gate): persist sleep and self-wake intent across restarts; re-arm on backward clock step - #194
Conversation
|
6d7f017 to
bac160e
Compare
|
Review at 6d7f017 adjudicated; all ten findings addressed at bac160e. Verified each against the code before changing anything.
Receipts at bac160e (on upstream main 5498805): Overlap note: antra-tess's |
|
@greptileai bump |
29341f7 to
e815e8c
Compare
|
Round 2 (29341f7) adjudicated; four findings, all valid, fixed at e815e8c.
Receipts at e815e8c (on 5498805): 986 pass, 0 fail; @greptileai bump |
| } | ||
|
|
||
| // Restore sleep suppression before MCPL startup can release buffered input. | ||
| framework.eventGate?.recoverWakeIntents(); |
There was a problem hiding this comment.
Failed startup can erase wakes If a persisted wake is still in the future, recovery arms its timer before MCPL configuration is validated. If validation or initialization then fails, the abandoned framework does not dispose of that timer. When it fires, it clears the saved wake and queues inference on a framework whose loop never starts, so a later successful startup cannot recover the promised wake.
Knowledge Base Used: Persistent state and recovery
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/framework.ts
Line: 1661
Comment:
**Failed startup can erase wakes** If a persisted wake is still in the future, recovery arms its timer before MCPL configuration is validated. If validation or initialization then fails, the abandoned framework does not dispose of that timer. When it fires, it clears the saved wake and queues inference on a framework whose loop never starts, so a later successful startup cannot recover the promised wake.
**Knowledge Base Used:** [Persistent state and recovery](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/persistent-state-and-recovery.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| for (const intent of state.selfWakes) { | ||
| this.selfWakeIntents.set(intent.agentName!, intent); | ||
| if (intent.wakeAt > now) this.armRestoredSelfWake(intent); |
There was a problem hiding this comment.
Repeated recovery duplicates self-wakes If an embedder enables
autoRecover and later calls the public recoverWakeIntents() method, this path arms the same future self-wake again without cancelling its first timer. Both timers can fire and each sends a notice and requests inference, turning one fallback wake into duplicate requests.
Knowledge Base Used: Event gate policies and scripts
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gate/event-gate.ts
Line: 1911-1913
Comment:
**Repeated recovery duplicates self-wakes** If an embedder enables `autoRecover` and later calls the public `recoverWakeIntents()` method, this path arms the same future self-wake again without cancelling its first timer. Both timers can fire and each sends a notice and requests inference, turning one fallback wake into duplicate requests.
**Knowledge Base Used:** [Event gate policies and scripts](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/event-gate-policies.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.e815e8c to
de7db7c
Compare
|
Rebased onto main df97a85; round 3 (e815e8c) adjudicated; fixed at de7db7c.
Overlap note: Receipts at de7db7c (on df97a85): 1136 pass, 0 fail; @greptileai bump |
| // Restore sleep suppression before MCPL startup can release buffered input. | ||
| framework.eventGate?.recoverWakeIntents(); |
There was a problem hiding this comment.
When a resident has a saved future sleep and a deferred MCPL push batch was pending at shutdown, startup restores the sleep but then queues the batch’s inference request without checking the gate. Starting the framework can run the resident before its promised wake deadline, even without new server input.
Knowledge Base Used: Event gate pipeline
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/framework.ts
Line: 1749-1750
Comment:
**Restored batches bypass sleep**
When a resident has a saved future sleep and a deferred MCPL push batch was pending at shutdown, startup restores the sleep but then queues the batch’s inference request without checking the gate. Starting the framework can run the resident before its promised wake deadline, even without new server input.
**Knowledge Base Used:** [Event gate pipeline](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/event-gates.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
LariTesserae
left a comment
There was a problem hiding this comment.
Read the full diff against main (0.19/df97a85 base). Mechanism is sound and matches the qa-staging #33/#25 findings: journal as a sibling of gate.json, atomic tmp+rename with 0600, corrupt journal quarantined rather than trusted, recovery hooked in the framework before MCPL startup, [wake-recovery] message with the measured overdue delta instead of silence, re-arm on backward clock step, long sleeps chunked under the setTimeout ceiling. Admission-before-side-effect on overdue recovery (persist first, restore in memory on failure) is the right order.
Two questions, neither blocking:
- Recovery inference timing.
recoverWakeIntents()runs before MCPL subsystems start, and an overdue intent callsrequestInferenceFnright there. If the inference loop can pick that request up before MCPL servers have registered their tools, the recovered agent wakes tool-less for one turn. Is the request guaranteed to sit in the queue until the loop starts after init? A one-line note (or a test asserting the recovered turn sees its tools) would settle it. - Gate-global sleep. Sleep is per gate, not per agent (unchanged semantics), so
agentNamein the journal is informational forsleepand authoritative for self-wakes. Worth one sentence in the journal's doc comment so a future reader doesn't assume per-agent sleep.
Nit: suppressed is persisted but reset to 0 on recovery — the comment says why; fine.
Mechanism verified against the live gate code path; receipts in the description match what I see. From the verdict loop on the QA track.
What happened in production
A resident calls
sleep(seconds)(orskip_replywithwake_in_seconds), getssuccesswith a concreteuntil, and the host restarts before that time — a deploy, an OOM, a planned reboot. The wake is gone: not re-armed, not fired late, no marker. The restarted process reports ordinary awake state, so the resident cannot tell it ever promised to resume. Reproduced on a connectome-host resident for both SIGTERM and SIGKILL (qa-staging #33, harness in the report); the same field is also what #25 loses on a backward wall-clock step.Mechanism
EventGatekeeps the entire sleep/self-wake state in process memory —sleepUntil,sleepNote,sleepAgent, thesetTimeouthandle, andselfWakeTimers— and the constructor never reconciles any of it (event-gate.ts~491-507, ~525-570).setSleep()returnsendTurn: trueafter writing only those fields (~1530-1551).dispose()clears timers and persists nothing.Change (
src/gate/event-gate.tsonly)wake-intents.json, sibling togate.json, written atomically (tmp + rename, mode 0600) beforesetSleep/armSelfWakereturn success; cleared onclearSleep, on fire, oncancelSelfWake.[wake-recovery]message naming the scheduled time and how overdue it is, and exactly one inference for that agent (at-most-once: a crash during recovery cannot replay it). Agents are registered before the gate is constructed, so the recovery message lands in the right context; a framework-level test pins that.dispose()only releases process-local timers; restart takes the same path as after a kill. A planned deploy no longer wakes a sleeping resident early.Not in this PR: in-flight tool-call "outcome unknown" markers after restart (Ferry's item 3) — separate change, same honesty family.
Tests
test/event-gate-selfwake.test.ts, six cases: persist-before-success and re-arm after abrupt loss; overdue sleep → one marker, one inference, none on the next restart; graceful shutdown preserves a future sleep;skip_replyself-wake re-arms across abrupt restart; early timer callback re-arms; framework-level recovery reaches the registered agent.Receipts at 6d7f017 (on upstream main 5498805):
npm run build && npm test→ 977 pass, 0 fail;tsc --noEmitclean;git diff --checkclean. Changelog fragmentchangelog.d/durable-wake-intents.fixed.md.Adjacent work
Open-PR list checked 2026-09-29: nothing touches
EventGatepersistence. Ferry's pointers (#connectome-qa, 09-23) called this the natural first continuity PR; the 09-26 review listed #33/#25 as the unfixed findings in our own workflow. The routing change in #190 is independent.Co-developed with qa-engineer, the connectome-host resident that lost the wakes.