Skip to content

feat(recovery): abandoned session detection and backfill for abruptly terminated sessions #737

Open

Description

@Songraf

Problem or Use Case

When a session ends abruptly — power loss, kill -9, OOM kill, Docker container restart without graceful shutdown, laptop sleep, SSH disconnect — the SessionEnd hook never fires. As a result:

  1. Session stays status="active" forever in the store
  2. No graph extraction, compression, or consolidation runs for that session's observations
  3. The stale-sweep (mem::evict) eventually deletes the entire session with reason: "stale_session_without_summary" — all observations are permanently lost
  4. The session's endedAt is never set, so there's no way to distinguish "still active" from "abandoned"

This is a known design issue (#308) with extensive discussion but no complete solution shipped as of v0.9.24. PR #387 added partial protection (extract before evict) but doesn't address the root cause.

Specific Use Case: Hermes Agent in Docker

Hermes Agent runs in a Docker container. When the host machine shuts down, the container is stopped via docker stop (SIGTERM) or killed abruptly. The Hermes gateway catches SIGTERM and calls on_session_end, but:

  • If SIGKILL is used, or power is lost, no cleanup happens
  • Even with SIGTERM, async hook work can be killed before completion (anthropics/claude-code#415777)
  • Historical JSONL transcripts exist on disk (Hermes saves them automatically) but cannot currently be re-imported after the fact because import-jsonl only supports Claude Code format (separate issue)

What We Want

Part A: Abandoned Session Detection and Recovery

A backend process (or CLI command) that:

  1. Detects abandoned sessions: finds sessions with status="active" and updatedAt older than a configurable threshold (default: 4 hours)
  2. Runs the full session-end pipeline on them:
    • Triggers event::session::stopped (graph extraction, slot reflection)
    • Runs the consolidation pipeline (compress, extract lessons, build crystals)
    • Sets status="completed" and endedAt timestamp
  3. Logs the action to the audit trail

CLI interface:

# On-demand: find and recover abandoned sessions
agentmemory reap --threshold=4h

# Background daemon mode (configurable interval)
agentmemory reap --daemon --interval=3600000

Configuration:

AGENTMEMORY_REAP_THRESHOLD_MS=14400000    # 4 hours, default
AGENTMEMORY_REAP_INTERVAL_MS=3600000      # 1 hour, 0 = disabled

Part B: Post-Restart Session Reconciliation

When agentmemory daemon starts up (especially after container restart):

  1. Scan for sessions with status="active" that have updatedAt older than threshold
  2. For each, run the session-end pipeline as in Part A
  3. Log recovered sessions to audit trail

Part C: Viewer and Diagnostics

  • In the AgentMemory viewer (port 3113), show a banner: "N abandoned sessions detected" with a "Reap now" button
  • In doctor output, warn about abandoned sessions: ⚠ 3 sessions abandoned (>4h since last activity)
  • In memory_diagnose, include abandoned-sessions check with actionable output

Scope

Medium-Large. Parts A and B are independently shipable. Part C depends on A/B.

Existing Work

Activity

  1. rohitg00 commented on Oct 8, 2026

    @rohitg00
    Owner

    @Songraf v0.9.30 adds the detection half: an hourly sweep marks sessions abandoned after AGENTMEMORY_SESSION_SWEEP_STALE_HOURS without deleting anything. Backfill of abandoned sessions is not included. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions