Repository navigation
fix(graph): bound and attribute node and edge provenance (carries #1445) - #1454
Conversation
Each merge in persistGraphDelta added every observation id in the batch to every node and edge it touched, with no limit. Hot nodes grew without bound. On one store, 98% of the graph's bytes were these ids: 426 per node and 362 per edge on average. Nodes took 323 MB and edges 398 MB. Every kv.list traversal of the graph carried them. - A merge now adds the incoming record's own sourceObservationIds, not the whole batch. Heuristic extraction knows the exact source, so that attribution is now kept. - Each record keeps its newest 32 ids (MAX_GRAPH_SOURCE_OBSERVATIONS). A re-seen id moves to the newest end. New records are trimmed on first write. Retrieval returns at most 20 results, so 32 covers it. - The temporal-graph node merge uses the same rule. - New mem::graph-compact and POST /agentmemory/graph/compact trim records written before the cap. They walk the name and edge-key indexes and read and write one key at a time, so they never list the nodes or edges scope. An optional scope, offset and limit runs one slice per call, so each call finishes between worker reconnects. Re-running a slice is safe. - Compaction takes the graph:persist lock per record, so a trim cannot overwrite a merge that lands during it. persistGraphDelta no longer takes the batch id list. Measured on a live store: compaction trimmed 18,324 nodes and 32,654 edges and removed 18.9 million ids in 45 s with no retries. Nodes went from 323 to 30 MB and edges from 398 to 42 MB. Fixes #1374 Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Review found two defects in POST /agentmemory/graph/compact.
Invalid input returned HTTP 200. mem::graph-compact reports a bad scope,
offset or limit as { success: false }, so the route passed that through
with 200. The route now rejects an unknown scope, a negative or
non-integer offset, and a non-positive or non-integer limit with 400,
before invoking compaction. It accepts JSON numbers only, so "5" is
rejected. Other success:false results still return 200, because that
envelope also carries operational failures.
Every failure was reported as "graph disabled". The catch returned
graphDisabledResponse(), a 503 with GRAPH_EXTRACTION_ENABLED advice, but
compaction is registered whether or not graph extraction is enabled. A
timeout, such as an unsliced run on a large graph, now returns 504 and
says to run in slices. Any other failure returns 500.
Signed-off-by: David Waltermire <davewaltermire@gmail.com>
mem::graph-compact catches a storage error and returns
{ success: false, error } instead of throwing. sdk.trigger therefore
resolves, and POST /agentmemory/graph/compact answered 200.
The route validates its input first, so a success:false result reaching
this point is an operational failure. It now returns 500 with the generic
"Graph compaction failed" message. The detailed error stays in the
server-side log, which mem::graph-compact already writes.
Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The explanations above boundSources and compactGraphProvenance are now doc comments, word for word. mergeNode, mergeEdge and the two compaction inner helpers get one-line doc comments, and so do the test helpers. This answers the docstring-coverage pre-merge check. No behavior changes. Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Each comment names the functions its registrar exposes. Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The temporal parser gives every edge the whole batch's observation ids, and the edge write stored them without the cap that nodes already had. A batch of more than 32 observations wrote an edge over the cap. Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The repo keeps code free of comments; the behaviour is covered by tests and the endpoint is documented in the README and REST reference.
Sliced compaction paged over kv.list(index) by offset across calls. The Redis backend returns list entries in a different order on each call, so a full walk could end with nextOffset null while records were never visited. Sort the ids before slicing; ids carry a base36 timestamp, so records inserted during a walk sort to the end. mem::graph-compact now records a graph_compact audit entry with the counts whenever it removes ids, and its registration sits above the graph-reset notes it was splitting.
mem::import, mem::mesh-receive / mesh pull and mem::snapshot-restore wrote graph nodes and edges as received. They do not write the name or edge-key index, so compaction cannot reach those rows, and a pre-fix export or peer brought the bloat straight back. Each path now keeps the newest 32 source observation ids, the same bound extraction uses.
snapshot-restore passes parsed JSON records typed as
{ id } & Record<string, unknown>, which tsc rejected against the
narrower generic bound.
Add the endpoint to the key endpoints table and describe when to run it, the scope/offset/limit slicing loop, and the final scope "snapshot" call a sliced run needs.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. 🧰 Additional context used📚 Code guidelines (1)No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (18)
💤 Files with no reviewable changes (1)
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review. 📝 WalkthroughWalkthroughThe change caps graph provenance at 32 observation IDs and adds an operation to compact existing node, edge, and cached-snapshot provenance. An authenticated REST endpoint exposes compaction, and audit records, tests, and API documentation cover the new behavior. ChangesGraph provenance management
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant Client
participant GraphCompactAPI as api::graph-compact
participant GraphCompactFunction as mem::graph-compact
participant Compaction as compactGraphProvenance
participant StateKV
participant Audit as safeAudit
Client->>GraphCompactAPI: POST with scope and page options
GraphCompactAPI->>GraphCompactFunction: Invoke with validated options
GraphCompactFunction->>Compaction: Compact selected provenance
Compaction->>StateKV: Read and update graph records or snapshot
Compaction-->>GraphCompactFunction: Return scan and trim counts
GraphCompactFunction->>Audit: Record audit entry when IDs were removed
GraphCompactFunction-->>GraphCompactAPI: Return success or failure result
GraphCompactAPI-->>Client: Return HTTP response
Merge Risk: ⚪ Minimal · up to This change limits graph provenance to the newest 32 observation IDs and adds an authenticated compaction endpoint for stores that already hold oversized records. No concrete merge-blocking risk was found. Existing bloated stores still need a manual compaction run after deploy. Security Architecture ReviewSecurity architecture risk: 🟡 Moderate · up to The provenance cap reduces uncontrolled record growth. However, compaction requests are not reliably workload-bounded, and some provenance removals can occur without an audit entry. Authentication follows the existing administrative pattern rather than introducing a stronger boundary. Retained concerns
Security review detailsSecurity Blast Radius
Security Findings and Attack Paths
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 3.70% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 27 functions across 13 files. (4 skipped: 4 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Carries #1445 by @david-waltermire into main with the review follow-ups, so it can land for the release.
Merge this with "Create a merge commit" (not squash): #1445's commits are included unchanged, so GitHub then marks #1445 as merged and the authorship stays with @david-waltermire.
From #1445
Graph nodes and edges kept every observation id that ever touched them, so
sourceObservationIdsgrew without bound. On aged stores this made graph rows huge and crash-looped the worker during search (#1124, #825, #1153, #1168, #1339, #1360, #1374, #1378, #1443).POST /agentmemory/graph/compacttrims rows that are already bloated, in slices (scope,offset,limit, loop untilnextOffsetis null, then a final{"scope":"snapshot"}call).Follow-ups added here
mem::graph-compactwrites agraph_compactaudit entry when it removes ids, with a viewer label.graph_compactandsession_sweep).Verified
npm run build,npm run skills:check: pass, noskills:gendriftnpx vitest run --exclude test/integration.test.ts: 2226 passed, 1 skippedtsc: 28 errors (baseline 30)Stores that are already bloated need one compaction run; a follow-up PR adds a one-time background compaction at boot.
Summary by CodeRabbit
New Features
Improvements