Skip to content

Verify: lock gaps in delete, relate and RMW paths #464

Closed

Description

@possiblyneal

Part of #448

Question

Do delete paths skip the mem::remember lock, does mem::relate lock the wrong key, do session and graph read-modify-writes race, and does the circuit breaker let every caller through when half-open? Each needs a concurrent reproduction. Upstream: rohitg00#1728, rohitg00#1729, rohitg00#1421, rohitg00#1732.

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    Verdict #464: lock gaps (worktree tmp/wt/verify-464, branch verify/464-locks, base fdc3d45)

    Tests: test/memory-write-locks.test.ts (8), test/circuit-breaker.test.ts (+4 half-open cases, 2 fail on base),
    test/resilient-half-open.test.ts (1). On base: 11 fail / 13 pass across the 3 files, identical on 3 runs (11/11 of the
    new race and half-open tests fail 3/3). With the fix 24/24 pass. npm run typecheck: 0 errors. Related suites
    (relations, governance, auto-forget, evict, retention, remember*, graph-eviction, resilient/provider) 97+153 pass.
    Fix patch of src/: tmp/verify/464/fix.patch. No build needed (no hook change).

    rohitg00#1728 delete paths skip the mem:remember lock: APPLY

    Holds here, and wider than claimed. mem::forget (remember.ts) also takes no lock in this fork, so it is a fifth
    unlocked path beside mem::governance-delete, mem::governance-bulk, mem::evict (2 sites), mem::auto-forget and
    mem::retention-evict. Repro: mem::remember reinforcing X is paused at its kv.set (it read X first), the delete runs
    to completion, remember's set resurrects X. Tests " is not undone by a remember that read the Memory first":
    6 tests, fail 3/3 on base (X returned strength 6 after being reported deleted).
    Fix: MEMORY_WRITE_LOCK ("mem:remember", src/state/memory-utils.ts; not in keyed-mutex.ts because 13 tests vi.mock
    that module without it) taken per memory around get+delete in each path. Not around the bulk loops.
    Operator reach: ~nil. Live copy holds 42 Memories; deletes of Memories are rare operator or TTL actions.
    Class: not class 2. Locking only; no Recall/Injection/storage content change.

    rohitg00#1729 relate locks the id pair: APPLY

    Holds. relations.ts keyed the lock mem:<a>:<b>; in this fork mem::evolve takes NO lock at all (upstream's
    description assumed it did), so relate x evolve and evolve x remember/delete also interleave.
    Tests: "two relates sharing a Memory keep both relatedIds" (relatedIds [mem_c] only) and "a relate that read a Memory
    before evolve does not mark the superseded version latest" (isLatest true): fail 3/3 base.
    Fix: relate and evolve both run under MEMORY_WRITE_LOCK (option 1 of the issue). The read is inside the lock, so no
    separate re-read patch is needed. relations.ts diff is mostly re-indentation (git diff -w).
    Operator reach: nil. mem:relations has 0 rows in the live copy: relate and evolve are not used.
    Class: not class 2.

    rohitg00#1421 session and graph RMW locks: REJECT (already covered here)

    Graph: every graph writer already holds GRAPH_WRITE_LOCK ("mem:graph:write", graph.ts:845): persistGraphDelta
    (graph.ts:852), cascade.ts:33, temporal-graph.ts:256, mesh.ts:377/547/552, graph-eviction.ts:239,
    merge-file-nodes.ts:273, startup-maintenance.ts:196. Graph extraction is off by default and its rows deleted (#420).
    Sessions: api::session::start runs under obs:<id> and uses field ops (api.ts:770-795); observe's implicit create is
    inside obs: (observe.ts:471); replay import is under obs: with kv.update (replay.ts:412-426); the other writers use
    path-level kv.update. The one unlocked whole-row kv.set(KV.sessions) that resets observationCount,
    event::session::started (events.ts:74), is unreachable: nothing publishes agentmemory.session.started
    (sdk.ts:168 "Nothing in the daemon publishes to ..."). migrate.ts:153 and export-import.ts:403 are offline whole-row
    imports. Retry if: the Engine gains a publisher for agentmemory.session.* topics, or a second process writes sessions.
    Not separately tested; code citation only.

    rohitg00#1732 breaker lets every caller through half-open: APPLY

    Holds: isAllowed returns true for every caller once half-open, and ResilientProvider.call checks it before the
    concurrency slot, so a burst all reach a provider that is down. Tests: breaker "lets one caller probe ..." and
    "releases a probe that never reports back ..." fail on base; ResilientProvider test (8 concurrent calls after the
    recovery window, provider down) makes 8 inner calls on base, 1 with the fix.
    Fix: one probe in half-open (probeStartedAt), re-admitted after recoveryTimeoutMs if it never reports; success and
    failure clear it. Addition beyond the issue: a probe that ends "provider busy" or "content filter" records neither,
    so ResilientProvider calls releaseProbe() there, otherwise a busy provider would pin the breaker for 30 s.
    Operator reach: only after an LLM outage; breaker state is not in the store, unmeasured.
    Class: not class 2 (changes LLM call fan-out during an outage, not Recall content).

    Bar

    • typecheck 0 errors; npm test 3,129 passed / 1 skipped; eval:gate and eval:gate:rules pass; eval:suite fast vs base-fdc3d459: every metric Δ 0.000 (15 of 15).
    • Branch verify/464-locks, uncommitted, worktree tmp/wt/verify-464.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions