Skip to content

feat(graph): compact bloated graph provenance once at boot - #1455

Merged
rohitg00 merged 17 commits into
mainfrom
feat/graph-compact-on-boot
Oct 1, 2026
Merged

rohitg00 merged 17 commits into
mainfrom
feat/graph-compact-on-boot

Conversation

@rohitg00

@rohitg00 rohitg00 commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Stacked on #1454. Base is contrib/1445-graph-provenance; retarget this PR to main after #1454 merges.

Why

Stores bloated by the unbounded sourceObservationIds bug (#1124 and its duplicates) crash-loop or time out on graph search. #1454 caps new writes at 32 and adds POST /agentmemory/graph/compact, but most affected users will never call it. The release should heal them on its own.

How it works

  • After Ready and after the viewer starts, a timer (3 s, unref'd) starts runGraphCompactOnBoot in the background. Boot, search and the viewer never wait on it.
  • It calls the existing compactGraphProvenance in sorted slices of 100 for nodes, edges and the new history scope (superseded temporal edges in mem:graph:edge-history), with a 200 ms pause between slices, then one snapshot call. Each record takes graph:persist once inside compactGraphProvenance; the boot runner holds no lock, so nothing nests the keyed mutex.
  • Progress lives in mem:config under graph-compact-on-boot:v1: phase, offset, phase total, counters, and the cap. It is saved after every slice. On restart it resumes from the saved offset. Once it is done it never runs again. A marker written for a different cap or version runs it once more.
  • A failing slice (or a failing progress write) is retried 3 times with backoff, then the run stops as failed, logs one warning with the manual command, and keeps the worker running. The next start resumes from the saved offset.
  • Visibility: concise boot log lines (start or resume, one per scope, done with totals), graphCompaction on /agentmemory/status (JSON and HTML) and a "Provenance compaction" row on the viewer Health page: pending, running with <scope> x of y, done with counts, failed with the reason and the curl command, or off. Running adds an info problem; failed adds a warning.
  • An audit entry (graph_compact, function mem::graph-compact-on-boot) is written when ids were removed.
  • POST /agentmemory/graph/compact accepts scope: "history", and the unsliced call now covers it too.

Setting

AGENTMEMORY_GRAPH_COMPACT_ON_BOOT (default on). Set it to false to skip the pass. Read through isGraphCompactOnBootEnabled() in src/config.ts; skills:gen picked it up.

Verification

Gate: build ok, skills:gen no drift beyond the new variable, skills:check ok, npx vitest run --exclude test/integration.test.ts 2241 passed (15 new in test/graph-compact-boot.test.ts), tsc 28 errors (limit 30).

New tests cover a full pass over nodes, edges, history and snapshot; a clean store finishing with no record writes; resume after the process dies mid-run (records before the saved offset are not read again); idempotency; the completion marker skipping later runs; a cap change re-running it; a store that lists in a different order on every call; bounded retries then failure reporting and a later resume; the runner not blocking its caller and running once at a time; boot wiring after the viewer and never awaited; the opt-out; and the status problems and HTML row.

Live, isolated instance (REST 4611, viewer 4613, engine 49911, fake embeddings on 4699). Seeded 2,000 nodes, 3,000 edges and 200 history edges at 120 ids each, every 50th row at 5,000 ids, plus a snapshot with 5 hot nodes:

File store Redis 8 (:6393)
Before: rows over 32 5,200 / 5,200 5,200 / 5,200
Before: graph/query 410,883 B 410,883 B
Killed with SIGKILL at edges 1000 of 3000 edges 1000 of 3000
Resumed at edges 500 (last durable write) edges 1000
After: rows over 32, max ids 0, 32 (nodes, edges, history, snapshot) 0, 32
After: graph/query 3,507 B 3,507 B
Ids removed 1,043,200 1,043,200
During the run: smart-search, viewer, graph/query all 200, worst 187 ms / 7 ms / 33 ms all 200, worst 214 ms / 7 ms / 66 ms
Third boot skipped, no log line, status done skipped, status done

Opt-out boot reported off in status. The viewer Health row rendered "done, 5,201 records trimmed, 1,043,200 ids removed" with no console errors. On the file store the engine had not flushed the last few slices when it was killed, so the run resumed a little earlier and re-trimmed those rows; that is harmless because the pass is idempotent, and the record count in the summary counts them twice.

Not covered

Rows reachable by neither index (written by import, mesh sync or snapshot restore before #1454) stay as they are, the same as for the manual endpoint.

david-waltermire and others added 14 commits September 29, 2026 21:52
Each merge in persistGraphDelta added every observation id in the batch
to every node and edge it touched, with no limit. Hot nodes grew without
bound. On one store, 98% of the graph's bytes were these ids: 426 per
node and 362 per edge on average. Nodes took 323 MB and edges 398 MB.
Every kv.list traversal of the graph carried them.

- A merge now adds the incoming record's own sourceObservationIds, not
  the whole batch. Heuristic extraction knows the exact source, so that
  attribution is now kept.
- Each record keeps its newest 32 ids (MAX_GRAPH_SOURCE_OBSERVATIONS).
  A re-seen id moves to the newest end. New records are trimmed on first
  write. Retrieval returns at most 20 results, so 32 covers it.
- The temporal-graph node merge uses the same rule.
- New mem::graph-compact and POST /agentmemory/graph/compact trim records
  written before the cap. They walk the name and edge-key indexes and
  read and write one key at a time, so they never list the nodes or
  edges scope. An optional scope, offset and limit runs one slice per
  call, so each call finishes between worker reconnects. Re-running a
  slice is safe.
- Compaction takes the graph:persist lock per record, so a trim cannot
  overwrite a merge that lands during it.

persistGraphDelta no longer takes the batch id list.

Measured on a live store: compaction trimmed 18,324 nodes and 32,654
edges and removed 18.9 million ids in 45 s with no retries. Nodes went
from 323 to 30 MB and edges from 398 to 42 MB.

Fixes #1374

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Review found two defects in POST /agentmemory/graph/compact.

Invalid input returned HTTP 200. mem::graph-compact reports a bad scope,
offset or limit as { success: false }, so the route passed that through
with 200. The route now rejects an unknown scope, a negative or
non-integer offset, and a non-positive or non-integer limit with 400,
before invoking compaction. It accepts JSON numbers only, so "5" is
rejected. Other success:false results still return 200, because that
envelope also carries operational failures.

Every failure was reported as "graph disabled". The catch returned
graphDisabledResponse(), a 503 with GRAPH_EXTRACTION_ENABLED advice, but
compaction is registered whether or not graph extraction is enabled. A
timeout, such as an unsliced run on a large graph, now returns 504 and
says to run in slices. Any other failure returns 500.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
mem::graph-compact catches a storage error and returns
{ success: false, error } instead of throwing. sdk.trigger therefore
resolves, and POST /agentmemory/graph/compact answered 200.

The route validates its input first, so a success:false result reaching
this point is an operational failure. It now returns 500 with the generic
"Graph compaction failed" message. The detailed error stays in the
server-side log, which mem::graph-compact already writes.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The explanations above boundSources and compactGraphProvenance are now
doc comments, word for word. mergeNode, mergeEdge and the two compaction
inner helpers get one-line doc comments, and so do the test helpers. This
answers the docstring-coverage pre-merge check. No behavior changes.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Each comment names the functions its registrar exposes.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The temporal parser gives every edge the whole batch's observation ids,
and the edge write stored them without the cap that nodes already had. A
batch of more than 32 observations wrote an edge over the cap.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The repo keeps code free of comments; the behaviour is covered by tests
and the endpoint is documented in the README and REST reference.
Sliced compaction paged over kv.list(index) by offset across calls. The
Redis backend returns list entries in a different order on each call, so
a full walk could end with nextOffset null while records were never
visited. Sort the ids before slicing; ids carry a base36 timestamp, so
records inserted during a walk sort to the end.

mem::graph-compact now records a graph_compact audit entry with the
counts whenever it removes ids, and its registration sits above the
graph-reset notes it was splitting.
mem::import, mem::mesh-receive / mesh pull and mem::snapshot-restore
wrote graph nodes and edges as received. They do not write the name or
edge-key index, so compaction cannot reach those rows, and a pre-fix
export or peer brought the bloat straight back. Each path now keeps the
newest 32 source observation ids, the same bound extraction uses.
snapshot-restore passes parsed JSON records typed as
{ id } & Record<string, unknown>, which tsc rejected against the
narrower generic bound.
Add the endpoint to the key endpoints table and describe when to run it,
the scope/offset/limit slicing loop, and the final scope "snapshot" call
a sliced run needs.
Stores hit by the unbounded sourceObservationIds bug (#1124) crash-loop
or time out on graph search, and most of them will never call
POST /agentmemory/graph/compact. After the worker is ready and the
viewer is up, a background pass now walks nodes, edges and the temporal
edge history in sorted slices of 100 with a 200 ms pause, then trims the
cached snapshot. It reuses compactGraphProvenance, so each record takes
graph:persist once and nothing nests the lock.

Progress and the completion marker live in mem:config under
graph-compact-on-boot:v1 with the cap, so a restart resumes from the
saved offset, a finished store never runs it again, and a future cap
change runs it once more. A failing slice is retried three times, then
reported in the boot log, /agentmemory/status and the viewer Health
page with the manual command; the worker keeps running.

AGENTMEMORY_GRAPH_COMPACT_ON_BOOT=false turns it off. The manual
endpoint gains a history scope for superseded temporal edges.
@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agentmemory Ready Ready Preview Oct 1, 2026 11:09am UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Each slice is now counted with a dry run first. When the slice has
records to trim, the plan (scanned, trimmed, ids removed, next offset)
is saved in the progress record before any record is written, then the
slice is applied and the plan is folded into the totals in the same
write that advances the offset. A restart that lost the last progress
write finds the saved plan at its offset and reuses those counts, so a
re-processed slice never adds to the totals twice and never drops the
records it already trimmed. The snapshot step follows the same rule.

compactGraphProvenance gains a dryRun option that counts without
writing. Status, viewer and boot-log fields are unchanged.

The new test captures the store before every progress write of a full
run, restarts from each of those states, and checks that the final
scanned, trimmed and ids-removed totals equal a clean single run.
Base automatically changed from contrib/1445-graph-provenance to main October 1, 2026 11:08
@rohitg00
rohitg00 merged commit 976ad3e into main Oct 1, 2026
10 checks passed
@rohitg00
rohitg00 deleted the feat/graph-compact-on-boot branch October 1, 2026 11:10

This branch was successfully deployed

1 active deployment
Preview — 5df56906 Deployed Oct 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants