Repository navigation
feat(graph): compact bloated graph provenance once at boot - #1455
Merged
Merged
Conversation
Each merge in persistGraphDelta added every observation id in the batch to every node and edge it touched, with no limit. Hot nodes grew without bound. On one store, 98% of the graph's bytes were these ids: 426 per node and 362 per edge on average. Nodes took 323 MB and edges 398 MB. Every kv.list traversal of the graph carried them. - A merge now adds the incoming record's own sourceObservationIds, not the whole batch. Heuristic extraction knows the exact source, so that attribution is now kept. - Each record keeps its newest 32 ids (MAX_GRAPH_SOURCE_OBSERVATIONS). A re-seen id moves to the newest end. New records are trimmed on first write. Retrieval returns at most 20 results, so 32 covers it. - The temporal-graph node merge uses the same rule. - New mem::graph-compact and POST /agentmemory/graph/compact trim records written before the cap. They walk the name and edge-key indexes and read and write one key at a time, so they never list the nodes or edges scope. An optional scope, offset and limit runs one slice per call, so each call finishes between worker reconnects. Re-running a slice is safe. - Compaction takes the graph:persist lock per record, so a trim cannot overwrite a merge that lands during it. persistGraphDelta no longer takes the batch id list. Measured on a live store: compaction trimmed 18,324 nodes and 32,654 edges and removed 18.9 million ids in 45 s with no retries. Nodes went from 323 to 30 MB and edges from 398 to 42 MB. Fixes #1374 Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Review found two defects in POST /agentmemory/graph/compact.
Invalid input returned HTTP 200. mem::graph-compact reports a bad scope,
offset or limit as { success: false }, so the route passed that through
with 200. The route now rejects an unknown scope, a negative or
non-integer offset, and a non-positive or non-integer limit with 400,
before invoking compaction. It accepts JSON numbers only, so "5" is
rejected. Other success:false results still return 200, because that
envelope also carries operational failures.
Every failure was reported as "graph disabled". The catch returned
graphDisabledResponse(), a 503 with GRAPH_EXTRACTION_ENABLED advice, but
compaction is registered whether or not graph extraction is enabled. A
timeout, such as an unsliced run on a large graph, now returns 504 and
says to run in slices. Any other failure returns 500.
Signed-off-by: David Waltermire <davewaltermire@gmail.com>
mem::graph-compact catches a storage error and returns
{ success: false, error } instead of throwing. sdk.trigger therefore
resolves, and POST /agentmemory/graph/compact answered 200.
The route validates its input first, so a success:false result reaching
this point is an operational failure. It now returns 500 with the generic
"Graph compaction failed" message. The detailed error stays in the
server-side log, which mem::graph-compact already writes.
Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The explanations above boundSources and compactGraphProvenance are now doc comments, word for word. mergeNode, mergeEdge and the two compaction inner helpers get one-line doc comments, and so do the test helpers. This answers the docstring-coverage pre-merge check. No behavior changes. Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Each comment names the functions its registrar exposes. Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The temporal parser gives every edge the whole batch's observation ids, and the edge write stored them without the cap that nodes already had. A batch of more than 32 observations wrote an edge over the cap. Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The repo keeps code free of comments; the behaviour is covered by tests and the endpoint is documented in the README and REST reference.
Sliced compaction paged over kv.list(index) by offset across calls. The Redis backend returns list entries in a different order on each call, so a full walk could end with nextOffset null while records were never visited. Sort the ids before slicing; ids carry a base36 timestamp, so records inserted during a walk sort to the end. mem::graph-compact now records a graph_compact audit entry with the counts whenever it removes ids, and its registration sits above the graph-reset notes it was splitting.
mem::import, mem::mesh-receive / mesh pull and mem::snapshot-restore wrote graph nodes and edges as received. They do not write the name or edge-key index, so compaction cannot reach those rows, and a pre-fix export or peer brought the bloat straight back. Each path now keeps the newest 32 source observation ids, the same bound extraction uses.
snapshot-restore passes parsed JSON records typed as
{ id } & Record<string, unknown>, which tsc rejected against the
narrower generic bound.
Add the endpoint to the key endpoints table and describe when to run it, the scope/offset/limit slicing loop, and the final scope "snapshot" call a sliced run needs.
Stores hit by the unbounded sourceObservationIds bug (#1124) crash-loop or time out on graph search, and most of them will never call POST /agentmemory/graph/compact. After the worker is ready and the viewer is up, a background pass now walks nodes, edges and the temporal edge history in sorted slices of 100 with a 200 ms pause, then trims the cached snapshot. It reuses compactGraphProvenance, so each record takes graph:persist once and nothing nests the lock. Progress and the completion marker live in mem:config under graph-compact-on-boot:v1 with the cap, so a restart resumes from the saved offset, a finished store never runs it again, and a future cap change runs it once more. A failing slice is retried three times, then reported in the boot log, /agentmemory/status and the viewer Health page with the manual command; the worker keeps running. AGENTMEMORY_GRAPH_COMPACT_ON_BOOT=false turns it off. The manual endpoint gains a history scope for superseded temporal edges.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Each slice is now counted with a dry run first. When the slice has records to trim, the plan (scanned, trimmed, ids removed, next offset) is saved in the progress record before any record is written, then the slice is applied and the plan is folded into the totals in the same write that advances the offset. A restart that lost the last progress write finds the saved plan at its offset and reuses those counts, so a re-processed slice never adds to the totals twice and never drops the records it already trimmed. The snapshot step follows the same rule. compactGraphProvenance gains a dryRun option that counts without writing. Status, viewer and boot-log fields are unchanged. The new test captures the store before every progress write of a full run, restarts from each of those states, and checks that the final scanned, trimmed and ids-removed totals equal a clean single run.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1454. Base is
contrib/1445-graph-provenance; retarget this PR tomainafter #1454 merges.Why
Stores bloated by the unbounded
sourceObservationIdsbug (#1124 and its duplicates) crash-loop or time out on graph search. #1454 caps new writes at 32 and addsPOST /agentmemory/graph/compact, but most affected users will never call it. The release should heal them on its own.How it works
Readyand after the viewer starts, a timer (3 s, unref'd) startsrunGraphCompactOnBootin the background. Boot, search and the viewer never wait on it.compactGraphProvenancein sorted slices of 100 fornodes,edgesand the newhistoryscope (superseded temporal edges inmem:graph:edge-history), with a 200 ms pause between slices, then onesnapshotcall. Each record takesgraph:persistonce insidecompactGraphProvenance; the boot runner holds no lock, so nothing nests the keyed mutex.mem:configundergraph-compact-on-boot:v1: phase, offset, phase total, counters, and the cap. It is saved after every slice. On restart it resumes from the saved offset. Once it isdoneit never runs again. A marker written for a different cap or version runs it once more.failed, logs one warning with the manual command, and keeps the worker running. The next start resumes from the saved offset.graphCompactionon/agentmemory/status(JSON and HTML) and a "Provenance compaction" row on the viewer Health page: pending, running with<scope> x of y, done with counts, failed with the reason and the curl command, or off. Running adds an info problem; failed adds a warning.graph_compact, functionmem::graph-compact-on-boot) is written when ids were removed.POST /agentmemory/graph/compactacceptsscope: "history", and the unsliced call now covers it too.Setting
AGENTMEMORY_GRAPH_COMPACT_ON_BOOT(default on). Set it tofalseto skip the pass. Read throughisGraphCompactOnBootEnabled()insrc/config.ts;skills:genpicked it up.Verification
Gate: build ok,
skills:genno drift beyond the new variable,skills:checkok,npx vitest run --exclude test/integration.test.ts2241 passed (15 new intest/graph-compact-boot.test.ts), tsc 28 errors (limit 30).New tests cover a full pass over nodes, edges, history and snapshot; a clean store finishing with no record writes; resume after the process dies mid-run (records before the saved offset are not read again); idempotency; the completion marker skipping later runs; a cap change re-running it; a store that lists in a different order on every call; bounded retries then failure reporting and a later resume; the runner not blocking its caller and running once at a time; boot wiring after the viewer and never awaited; the opt-out; and the status problems and HTML row.
Live, isolated instance (REST 4611, viewer 4613, engine 49911, fake embeddings on 4699). Seeded 2,000 nodes, 3,000 edges and 200 history edges at 120 ids each, every 50th row at 5,000 ids, plus a snapshot with 5 hot nodes:
graph/querygraph/queryOpt-out boot reported
offin status. The viewer Health row rendered "done, 5,201 records trimmed, 1,043,200 ids removed" with no console errors. On the file store the engine had not flushed the last few slices when it was killed, so the run resumed a little earlier and re-trimmed those rows; that is harmless because the pass is idempotent, and the record count in the summary counts them twice.Not covered
Rows reachable by neither index (written by import, mesh sync or snapshot restore before #1454) stay as they are, the same as for the manual endpoint.