Skip to content

fix(graph): bound and attribute node and edge provenance (carries #1445) - #1454

Merged
rohitg00 merged 13 commits into
mainfrom
contrib/1445-graph-provenance
Oct 1, 2026
Merged

rohitg00 merged 13 commits into
mainfrom
contrib/1445-graph-provenance

Conversation

@rohitg00

@rohitg00 rohitg00 commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Carries #1445 by @david-waltermire into main with the review follow-ups, so it can land for the release.

Merge this with "Create a merge commit" (not squash): #1445's commits are included unchanged, so GitHub then marks #1445 as merged and the authorship stays with @david-waltermire.

From #1445

Graph nodes and edges kept every observation id that ever touched them, so sourceObservationIds grew without bound. On aged stores this made graph rows huge and crash-looped the worker during search (#1124, #825, #1153, #1168, #1339, #1360, #1374, #1378, #1443).

  • Each node and edge records only its own observation ids, capped at 32, including temporal graph edges.
  • POST /agentmemory/graph/compact trims rows that are already bloated, in slices (scope, offset, limit, loop until nextOffset is null, then a final {"scope":"snapshot"} call).

Follow-ups added here

  • Compaction sorts the ids before slicing. The Redis backend returns lists in a different order on every call, so an unsorted sliced walk skipped records. A test with a shuffling store fails without the sort.
  • Provenance is also capped on import, mesh sync and snapshot restore, which wrote rows compaction could not reach. Each path has a test.
  • mem::graph-compact writes a graph_compact audit entry when it removes ids, with a viewer label.
  • README documents the endpoint and the slicing loop.
  • Removed the comments the PR added.
  • Merged current main (the audit operation list and viewer labels now include both graph_compact and session_sweep).

Verified

  • Live on Redis and on the file store: 300 nodes and 600 edges seeded with 120 ids each, a sliced walk with limit 20 left 0 rows over 32, and a second run removed nothing.
  • Earlier live check: a store bloated on main to 30,005 ids per hot row (8.3 MB graph query) compacted to 32 ids and 173 KB, search kept working, the viewer node detail still shows provenance, and 3 concurrent extractors saw no deadlock.
  • npm run build, npm run skills:check: pass, no skills:gen drift
  • npx vitest run --exclude test/integration.test.ts: 2226 passed, 1 skipped
  • tsc: 28 errors (baseline 30)

Stores that are already bloated need one compaction run; a follow-up PR adds a one-time background compaction at boot.

Summary by CodeRabbit

  • New Features

    • Added an API endpoint to compact graph provenance for nodes, edges, or snapshots, with optional pagination for large stores.
    • Compaction reports scanned and trimmed records, and records an audit entry when observation IDs are removed.
  • Improvements

    • Graph provenance is now limited to the 32 most recent observation IDs across imports, merges, and snapshot restoration.
    • Updated API documentation to describe compaction and pagination.

david-waltermire and others added 13 commits September 29, 2026 21:52
Each merge in persistGraphDelta added every observation id in the batch
to every node and edge it touched, with no limit. Hot nodes grew without
bound. On one store, 98% of the graph's bytes were these ids: 426 per
node and 362 per edge on average. Nodes took 323 MB and edges 398 MB.
Every kv.list traversal of the graph carried them.

- A merge now adds the incoming record's own sourceObservationIds, not
  the whole batch. Heuristic extraction knows the exact source, so that
  attribution is now kept.
- Each record keeps its newest 32 ids (MAX_GRAPH_SOURCE_OBSERVATIONS).
  A re-seen id moves to the newest end. New records are trimmed on first
  write. Retrieval returns at most 20 results, so 32 covers it.
- The temporal-graph node merge uses the same rule.
- New mem::graph-compact and POST /agentmemory/graph/compact trim records
  written before the cap. They walk the name and edge-key indexes and
  read and write one key at a time, so they never list the nodes or
  edges scope. An optional scope, offset and limit runs one slice per
  call, so each call finishes between worker reconnects. Re-running a
  slice is safe.
- Compaction takes the graph:persist lock per record, so a trim cannot
  overwrite a merge that lands during it.

persistGraphDelta no longer takes the batch id list.

Measured on a live store: compaction trimmed 18,324 nodes and 32,654
edges and removed 18.9 million ids in 45 s with no retries. Nodes went
from 323 to 30 MB and edges from 398 to 42 MB.

Fixes #1374

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Review found two defects in POST /agentmemory/graph/compact.

Invalid input returned HTTP 200. mem::graph-compact reports a bad scope,
offset or limit as { success: false }, so the route passed that through
with 200. The route now rejects an unknown scope, a negative or
non-integer offset, and a non-positive or non-integer limit with 400,
before invoking compaction. It accepts JSON numbers only, so "5" is
rejected. Other success:false results still return 200, because that
envelope also carries operational failures.

Every failure was reported as "graph disabled". The catch returned
graphDisabledResponse(), a 503 with GRAPH_EXTRACTION_ENABLED advice, but
compaction is registered whether or not graph extraction is enabled. A
timeout, such as an unsliced run on a large graph, now returns 504 and
says to run in slices. Any other failure returns 500.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
mem::graph-compact catches a storage error and returns
{ success: false, error } instead of throwing. sdk.trigger therefore
resolves, and POST /agentmemory/graph/compact answered 200.

The route validates its input first, so a success:false result reaching
this point is an operational failure. It now returns 500 with the generic
"Graph compaction failed" message. The detailed error stays in the
server-side log, which mem::graph-compact already writes.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The explanations above boundSources and compactGraphProvenance are now
doc comments, word for word. mergeNode, mergeEdge and the two compaction
inner helpers get one-line doc comments, and so do the test helpers. This
answers the docstring-coverage pre-merge check. No behavior changes.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
Each comment names the functions its registrar exposes.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The temporal parser gives every edge the whole batch's observation ids,
and the edge write stored them without the cap that nodes already had. A
batch of more than 32 observations wrote an edge over the cap.

Signed-off-by: David Waltermire <davewaltermire@gmail.com>
The repo keeps code free of comments; the behaviour is covered by tests
and the endpoint is documented in the README and REST reference.
Sliced compaction paged over kv.list(index) by offset across calls. The
Redis backend returns list entries in a different order on each call, so
a full walk could end with nextOffset null while records were never
visited. Sort the ids before slicing; ids carry a base36 timestamp, so
records inserted during a walk sort to the end.

mem::graph-compact now records a graph_compact audit entry with the
counts whenever it removes ids, and its registration sits above the
graph-reset notes it was splitting.
mem::import, mem::mesh-receive / mesh pull and mem::snapshot-restore
wrote graph nodes and edges as received. They do not write the name or
edge-key index, so compaction cannot reach those rows, and a pre-fix
export or peer brought the bloat straight back. Each path now keeps the
newest 32 source observation ids, the same bound extraction uses.
snapshot-restore passes parsed JSON records typed as
{ id } & Record<string, unknown>, which tsc rejected against the
narrower generic bound.
Add the endpoint to the key endpoints table and describe when to run it,
the scope/offset/limit slicing loop, and the final scope "snapshot" call
a sliced run needs.
@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agentmemory Ready Ready Preview Oct 1, 2026 8:55am UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used
📚 Code guidelines (1)
AGENTS.md — auto-discovered

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: c477962a-a962-477e-a992-5e88b03c4260

📥 Commits

Reviewing files that changed from the base of the PR and between e950d58 and 1deedde.

📒 Files selected for processing (18)
  • AGENTS.md
  • README.md
  • plugin/skills/agentmemory-rest-api/REFERENCE.md
  • src/functions/export-import.ts
  • src/functions/graph-import.ts
  • src/functions/graph.ts
  • src/functions/mesh.ts
  • src/functions/snapshot.ts
  • src/functions/temporal-graph.ts
  • src/index.ts
  • src/triggers/api.ts
  • src/types.ts
  • src/viewer/index.html
  • test/export-import.test.ts
  • test/graph-compact.test.ts
  • test/graph-provenance-cap.test.ts
  • test/mesh.test.ts
  • test/snapshot.test.ts
💤 Files with no reviewable changes (1)
  • src/functions/graph-import.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The change caps graph provenance at 32 observation IDs and adds an operation to compact existing node, edge, and cached-snapshot provenance. An authenticated REST endpoint exposes compaction, and audit records, tests, and API documentation cover the new behavior.

Changes

Graph provenance management

Layer / File(s) Summary
Bound provenance on graph writes
src/functions/graph.ts, src/functions/graph-import.ts, src/functions/export-import.ts, src/functions/mesh.ts, src/functions/snapshot.ts, src/functions/temporal-graph.ts, test/graph-provenance-cap.test.ts, test/export-import.test.ts, test/mesh.test.ts, test/snapshot.test.ts
Graph merges and ingestion paths retain at most the newest 32 source observation IDs. Tests cover merge, import, mesh receive, temporal extraction, and snapshot restore.
Compaction operation
src/functions/graph.ts, test/graph-compact.test.ts
The compaction function trims selected records or cached snapshot entries, supports scoped pagination, and reports scan and removal counts. Tests cover compaction, sliced walks, and audit behavior.
REST endpoint and operation reference
src/triggers/api.ts, src/types.ts, src/viewer/index.html, README.md, plugin/skills/agentmemory-rest-api/REFERENCE.md, AGENTS.md, src/index.ts, test/graph-compact.test.ts
The endpoint validates request parameters and returns HTTP responses for success, failure, and timeout results. Audit operation text, endpoint counts, API references, and endpoint tests are updated.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant GraphCompactAPI as api::graph-compact
  participant GraphCompactFunction as mem::graph-compact
  participant Compaction as compactGraphProvenance
  participant StateKV
  participant Audit as safeAudit
  Client->>GraphCompactAPI: POST with scope and page options
  GraphCompactAPI->>GraphCompactFunction: Invoke with validated options
  GraphCompactFunction->>Compaction: Compact selected provenance
  Compaction->>StateKV: Read and update graph records or snapshot
  Compaction-->>GraphCompactFunction: Return scan and trim counts
  GraphCompactFunction->>Audit: Record audit entry when IDs were removed
  GraphCompactFunction-->>GraphCompactAPI: Return success or failure result
  GraphCompactAPI-->>Client: Return HTTP response
Loading

Merge Risk: ⚪ Minimal · up to 1deed

This change limits graph provenance to the newest 32 observation IDs and adds an authenticated compaction endpoint for stores that already hold oversized records. No concrete merge-blocking risk was found. Existing bloated stores still need a manual compaction run after deploy.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 1deed

The provenance cap reduces uncontrolled record growth. However, compaction requests are not reliably workload-bounded, and some provenance removals can occur without an audit entry. Authentication follows the existing administrative pattern rather than introducing a stronger boundary.

Retained concerns

  • Medium · security · inferred: The new maintenance entrypoint does not enforce bounded work: omitted limits process every indexed record, positive limits have no ceiling, and every finite page still loads and sorts the entire index. A reachable caller can request expensive store-wide work; operators recovering large stores can encounter the same containment problem. HTTP 504 mapping alone does not prove that processing stops. This is a new workload path, not a demonstrated expansion of administrative authority.
  • Low · security · observed: Compaction mutation and audit recording do not cover every terminal state. Snapshot-only trimming writes changed provenance without incrementing idsRemoved, so it produces no graph_compact audit entry. A later write failure can also leave earlier record removals committed while bypassing audit; retrying those already-bounded records cannot recover their removal counts. This weakens visibility into the newly introduced provenance-removal operation.
Security review details

Security Blast Radius

  • observed — A compaction invocation can select all indexed nodes and edges plus cached-snapshot provenance in the supplied store. There is no tenant or asset selector. Its direct writes trim provenance fields rather than deleting underlying observation records or changing graph endpoints.

Security Findings and Attack Paths

  • inferred — A caller able to reach the route and satisfy its configured authentication can repeatedly submit empty-body or large-limit requests to initiate store-wide work. A finite limit still permits whole-index enumeration. Availability impact depends on store size and runtime behavior; this does not establish privilege escalation, and the existing administrative routes already share the same authentication boundary.
  • observed — The new audit path omits snapshot-only removals and mutations completed before a subsequent compaction failure. Successful node or edge removal normally produces an operation-level entry, but audit failure is swallowed and no HTTP caller identity is forwarded.

Trust Boundaries and Controls

  • observed — The HTTP-to-runtime transition forwards only scope, offset, and limit. Both layers validate parameters, and HTTP failures use generic messages instead of exposing internal compaction errors. These controls restrict input shape and error disclosure, not maximum work or store-wide authority.

Resilience and Maintainability Implications

  • observed — The retention cap limits provenance accumulation on subsequent writes, and per-record idempotency supports repeating compaction after interruption. These safeguards do not make the operation transactional: completed writes survive a later failure, while audit executes only after successful completion.

Hardening Proposals

  • proposed — Consider an enforced finite default, maximum page size, and execution budget, together with storage-level cursor pagination so a small page does not enumerate the entire index. Define timeout and cancellation behavior explicitly.
  • proposed — Include snapshot removals in mutation accounting and preserve durable progress or audit information for partial completion. Distinguish operation completion from audit failure, and attribute requests where a trustworthy caller identity is available.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 3.70% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 27 functions across 13 files. (4 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: bounding and attributing graph node and edge provenance. The issue reference is additional but does not reduce clarity.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 3.70% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 27 functions across 13 files. (4 skipped: 4 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch was successfully deployed

1 active deployment
Preview — 1deedde1 Deployed Oct 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants