Skip to content

fix(gate): persist sleep and self-wake intent across restarts; re-arm on backward clock step - #194

Open
rufalupa666-netizen wants to merge 2 commits into
anima-research:mainfrom
rufalupa666-netizen:fix/durable-sleep-intent
Open

rufalupa666-netizen wants to merge 2 commits into
anima-research:mainfrom
rufalupa666-netizen:fix/durable-sleep-intent

Conversation

@rufalupa666-netizen

Copy link
Copy Markdown

What happened in production

A resident calls sleep(seconds) (or skip_reply with wake_in_seconds), gets success with a concrete until, and the host restarts before that time — a deploy, an OOM, a planned reboot. The wake is gone: not re-armed, not fired late, no marker. The restarted process reports ordinary awake state, so the resident cannot tell it ever promised to resume. Reproduced on a connectome-host resident for both SIGTERM and SIGKILL (qa-staging #33, harness in the report); the same field is also what #25 loses on a backward wall-clock step.

Mechanism

EventGate keeps the entire sleep/self-wake state in process memory — sleepUntil, sleepNote, sleepAgent, the setTimeout handle, and selfWakeTimers — and the constructor never reconciles any of it (event-gate.ts ~491-507, ~525-570). setSleep() returns endTurn: true after writing only those fields (~1530-1551). dispose() clears timers and persists nothing.

Change (src/gate/event-gate.ts only)

  • Durable intent journal wake-intents.json, sibling to gate.json, written atomically (tmp + rename, mode 0600) before setSleep / armSelfWake return success; cleared on clearSleep, on fire, on cancelSelfWake.
  • Startup reconciliation in the constructor: deadline still in the future → re-arm for the remainder; deadline passed → clear the entry first, then one [wake-recovery] message naming the scheduled time and how overdue it is, and exactly one inference for that agent (at-most-once: a crash during recovery cannot replay it). Agents are registered before the gate is constructed, so the recovery message lands in the right context; a framework-level test pins that.
  • Graceful shutdown preserves intent. dispose() only releases process-local timers; restart takes the same path as after a kill. A planned deploy no longer wakes a sleeping resident early.
  • Backward clock step (Triumvirate fixes #25): a sleep timer that fires before its wall-clock deadline re-arms for the remaining wall time instead of consuming the only callback.

Not in this PR: in-flight tool-call "outcome unknown" markers after restart (Ferry's item 3) — separate change, same honesty family.

Tests

test/event-gate-selfwake.test.ts, six cases: persist-before-success and re-arm after abrupt loss; overdue sleep → one marker, one inference, none on the next restart; graceful shutdown preserves a future sleep; skip_reply self-wake re-arms across abrupt restart; early timer callback re-arms; framework-level recovery reaches the registered agent.

Receipts at 6d7f017 (on upstream main 5498805): npm run build && npm test → 977 pass, 0 fail; tsc --noEmit clean; git diff --check clean. Changelog fragment changelog.d/durable-wake-intents.fixed.md.

Adjacent work

Open-PR list checked 2026-09-29: nothing touches EventGate persistence. Ferry's pointers (#connectome-qa, 09-23) called this the natural first continuity PR; the 09-26 review listed #33/#25 as the unfixed findings in our own workflow. The routing change in #190 is independent.

Co-developed with qa-engineer, the connectome-host resident that lost the wakes.

@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 2/5 Tier: apex

[High risk] Adds persistent wake-intent recovery across process restarts.

The PR is not yet safe to merge because restored MCPL work can wake a sleeping resident early, and two earlier wake-recovery failures remain.

Findings

  1. P1 Restored batches bypass sleep ▶
  2. P1 Repeated recovery duplicates self-wakes ▶
Fix with agent prompt
### Issue 1
src/framework.ts:1749-1750
When a resident has a saved future sleep and a deferred MCPL push batch was pending at shutdown, startup restores the sleep but then queues the batch’s inference request without checking the gate. Starting the framework can run the resident before its promised wake deadline, even without new server input.

### Issue 2
src/gate/event-gate.ts:1911-1913
If an embedder enables `autoRecover` and later calls the public `recoverWakeIntents()` method, this path arms the same future self-wake again without cancelling its first timer. Both timers can fire and each sends a notice and requests inference, turning one fallback wake into duplicate requests.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR journals sleep and self-wake intent, restores future deadlines or reconciles overdue ones on startup, and re-arms sleep timers after backward clock steps.

  • The rebased MCPL batch-recovery path can start a turn despite a restored future sleep.

Reviews (4) · Last reviewed commit: "fix(gate): unref sleep and self-wake tim..."

Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts
Comment thread src/gate/event-gate.ts Outdated
@rufalupa666-netizen

Copy link
Copy Markdown
Author

Review at 6d7f017 adjudicated; all ten findings addressed at bac160e. Verified each against the code before changing anything.

  1. Shared journal path — valid. Journal is now derived from the gate's own config path (gate.json → gate.wake-intents.json); two gates in one directory no longer share a file.
  2. Recovery erased before initialization — valid, and the most important one. Recovery no longer runs in the constructor. EventGate.recoverWakeIntents() is called once at the end of a successful AgentFramework.create(); a due entry is removed from the journal only after its inference has been enqueued. Test: create() failing after gate construction leaves the intent in the journal for the next start.
  3. Write failure strands the resident — valid. The timer is armed before the journal write; the write is wrapped and logged, never leaving sleep set without a timer. Same on the expiry paths.
  4. Anonymous sleep not persisted — valid. Persisted with no agent name; on recovery it fans out to every registered agent, exactly as a normal expiry does.
  5. Unreadable journal treated as empty — valid. ENOENT is the only "empty" case; any read/parse failure quarantines the file as <name>.corrupt-<timestamp> and logs it, so the intents stay available for diagnosis instead of being overwritten.
  6. Startup write on empty state — valid. Nothing is written when there is nothing to change; a read-only gate directory starts as before.
  7. Long sleeps spin timers — valid. Each timer interval is clamped to 2^31−1 ms and re-armed in chunks until the wall-clock deadline.
  8. Malformed entries prevent startup — valid. Entries are validated (agent name string-or-absent, numeric armedAt/wakeAt, string source); invalid ones are skipped and logged.
  9. Suppressed count — not persisted per event, deliberately: a write per suppressed event is the wrong cost. getSleepState().suppressed is now documented as counting the current process only and resets after recovery.
  10. Recovery drops the note — valid. The saved note is included in the [wake-recovery] marker and in the inference reason.

Receipts at bac160e (on upstream main 5498805): npm run build && npm test → 983 pass, 0 fail; tsc --noEmit clean; git diff --check clean; targeted wake suites 20/20.

Overlap note: antra-tess's feat/mcpl-event-coalescing (today) also touches event-gate.ts fields and dispose(); we'll rebase on whichever lands first.

@rufalupa666-netizen

Copy link
Copy Markdown
Author

@greptileai bump

Comment thread src/framework.ts Outdated
Comment thread src/gate/event-gate.ts Outdated
Comment thread src/gate/event-gate.ts
Comment thread src/gate/event-gate.ts Outdated
@rufalupa666-netizen

Copy link
Copy Markdown
Author

Round 2 (29341f7) adjudicated; four findings, all valid, fixed at e815e8c.

  1. Startup can wake sleeping residents — valid. recoverWakeIntents() now runs in create() before MCPL servers are initialised and their buffered input released, so a restored sleep is in place when input is evaluated. Test added.
  2. Overdue wakes can replay — valid; we over-corrected in round 1. Order is now: remove the due entry (durable) → marker → inference. If the journal write fails, that entry is skipped and logged and stays for the next start — at-most-once preserved, at the cost of a delayed wake rather than a duplicate. Test added.
  3. Standalone gates skip restoration — valid. EventGate gains autoRecover (default false) for embedders that construct a gate outside AgentFramework.create(); recoverWakeIntents() documented.
  4. Wake timers cannot keep processes alive — valid, and it contradicts the unref() suggested on EventGate.dispose() does not clear sleepTimer #192. Reverted: dispose() now clears every timer (which was EventGate.dispose() does not clear sleepTimer #192's actual complaint), so unref() is unnecessary and would let a standalone process exit before a promised wake. Test keeps asserting dispose() leaves no live timer.

Receipts at e815e8c (on 5498805): 986 pass, 0 fail; tsc --noEmit clean; git diff --check clean.

@greptileai bump

Comment thread src/framework.ts
}

// Restore sleep suppression before MCPL startup can release buffered input.
framework.eventGate?.recoverWakeIntents();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Failed startup can erase wakes If a persisted wake is still in the future, recovery arms its timer before MCPL configuration is validated. If validation or initialization then fails, the abandoned framework does not dispose of that timer. When it fires, it clears the saved wake and queues inference on a framework whose loop never starts, so a later successful startup cannot recover the promised wake.

Knowledge Base Used: Persistent state and recovery

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/framework.ts
Line: 1661

Comment:
**Failed startup can erase wakes** If a persisted wake is still in the future, recovery arms its timer before MCPL configuration is validated. If validation or initialization then fails, the abandoned framework does not dispose of that timer. When it fires, it clears the saved wake and queues inference on a framework whose loop never starts, so a later successful startup cannot recover the promised wake.

**Knowledge Base Used:** [Persistent state and recovery](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/persistent-state-and-recovery.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread src/gate/event-gate.ts
Comment on lines +1911 to +1913
for (const intent of state.selfWakes) {
this.selfWakeIntents.set(intent.agentName!, intent);
if (intent.wakeAt > now) this.armRestoredSelfWake(intent);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Repeated recovery duplicates self-wakes If an embedder enables autoRecover and later calls the public recoverWakeIntents() method, this path arms the same future self-wake again without cancelling its first timer. Both timers can fire and each sends a notice and requests inference, turning one fallback wake into duplicate requests.

Knowledge Base Used: Event gate policies and scripts

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gate/event-gate.ts
Line: 1911-1913

Comment:
**Repeated recovery duplicates self-wakes** If an embedder enables `autoRecover` and later calls the public `recoverWakeIntents()` method, this path arms the same future self-wake again without cancelling its first timer. Both timers can fire and each sends a notice and requests inference, turning one fallback wake into duplicate requests.

**Knowledge Base Used:** [Event gate policies and scripts](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/event-gate-policies.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@rufalupa666-netizen

Copy link
Copy Markdown
Author

Rebased onto main df97a85; round 3 (e815e8c) adjudicated; fixed at de7db7c.

  1. Failed startup can erase wakes — valid. Recovery now runs after all configuration/MCPL validation and before buffered input is released; if create() still fails after recovery, the failure path disposes the gate (clears timers, keeps the journal). Test: create() fails after recovery → journal intact, no live timer.
  2. Repeated recovery duplicates self-wakes — valid. recoverWakeIntents() is idempotent per intent: an already-armed intent is not armed again. Test with autoRecover + explicit call.
  3. Wake timers cannot keep processes alive — stale: it cites event-gate.ts:undefined-1795; no unref() remains in event-gate.ts at e815e8c or de7db7c (reverted in round 2, noted on EventGate.dispose() does not clear sleepTimer #192).

Overlap note: fix/batched-wake-addressed-locus (antra-tess) adds an EventGate option resolveRouteChannel; this PR adds autoRecover to the same options interface — trivial textual overlap, no semantic one.

Receipts at de7db7c (on df97a85): 1136 pass, 0 fail; tsc --noEmit clean; git diff --check clean.

@greptileai bump

Comment thread src/framework.ts
Comment on lines +1749 to +1750
// Restore sleep suppression before MCPL startup can release buffered input.
framework.eventGate?.recoverWakeIntents();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Restored batches bypass sleep

When a resident has a saved future sleep and a deferred MCPL push batch was pending at shutdown, startup restores the sleep but then queues the batch’s inference request without checking the gate. Starting the framework can run the resident before its promised wake deadline, even without new server input.

Knowledge Base Used: Event gate pipeline

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/framework.ts
Line: 1749-1750

Comment:
**Restored batches bypass sleep**

When a resident has a saved future sleep and a deferred MCPL push batch was pending at shutdown, startup restores the sleep but then queues the batch’s inference request without checking the gate. Starting the framework can run the resident before its promised wake deadline, even without new server input.

**Knowledge Base Used:** [Event gate pipeline](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/event-gates.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@LariTesserae LariTesserae left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff against main (0.19/df97a85 base). Mechanism is sound and matches the qa-staging #33/#25 findings: journal as a sibling of gate.json, atomic tmp+rename with 0600, corrupt journal quarantined rather than trusted, recovery hooked in the framework before MCPL startup, [wake-recovery] message with the measured overdue delta instead of silence, re-arm on backward clock step, long sleeps chunked under the setTimeout ceiling. Admission-before-side-effect on overdue recovery (persist first, restore in memory on failure) is the right order.

Two questions, neither blocking:

  1. Recovery inference timing. recoverWakeIntents() runs before MCPL subsystems start, and an overdue intent calls requestInferenceFn right there. If the inference loop can pick that request up before MCPL servers have registered their tools, the recovered agent wakes tool-less for one turn. Is the request guaranteed to sit in the queue until the loop starts after init? A one-line note (or a test asserting the recovered turn sees its tools) would settle it.
  2. Gate-global sleep. Sleep is per gate, not per agent (unchanged semantics), so agentName in the journal is informational for sleep and authoritative for self-wakes. Worth one sentence in the journal's doc comment so a future reader doesn't assume per-agent sleep.

Nit: suppressed is persisted but reset to 0 on recovery — the comment says why; fine.

Mechanism verified against the live gate code path; receipts in the description match what I see. From the verdict loop on the QA track.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants