Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Hook scripts in `src/hooks/` are standalone Node.js scripts (no iii-sdk import).
## Current Stats (v0.9.29)

- 54 MCP tools (8 visible by default, `AGENTMEMORY_TOOLS=all` for all)
- 134 REST endpoints
- 135 REST endpoints
- 6 MCP resources, 3 MCP prompts
- 12 hooks, 17 skills
- 260+ iii functions
Expand Down
16 changes: 15 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1671,7 +1671,7 @@ Create `~/.agentmemory/.env`:

<h2 id="api"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/tags/light/section-api.svg"><img src="assets/tags/section-api.svg" alt="API" height="32" /></picture></h2>

134 endpoints on port `3111`. The REST API binds to `127.0.0.1` by default. Protected endpoints require `Authorization: Bearer <secret>` when `AGENTMEMORY_SECRET` is set, and mesh sync endpoints require `AGENTMEMORY_SECRET` on both peers.
135 endpoints on port `3111`. The REST API binds to `127.0.0.1` by default. Protected endpoints require `Authorization: Bearer <secret>` when `AGENTMEMORY_SECRET` is set, and mesh sync endpoints require `AGENTMEMORY_SECRET` on both peers.

<details>
<summary>Key endpoints</summary>
Expand All @@ -1691,13 +1691,27 @@ Create `~/.agentmemory/.env`:
| `GET` | `/agentmemory/export` | Export all data |
| `POST` | `/agentmemory/import` | Import from JSON |
| `POST` | `/agentmemory/graph/query` | Knowledge graph query |
| `POST` | `/agentmemory/graph/compact` | Trim oversized graph provenance |
| `POST` | `/agentmemory/team/share` | Share with team |
| `GET` | `/agentmemory/audit` | Audit trail |

Full endpoint list: [`src/triggers/api.ts`](src/triggers/api.ts)

</details>

**Compacting graph provenance.** Each knowledge graph node and edge keeps the ids of the newest 32 observations it came from. Stores written before that cap can hold thousands of ids per hot node, which makes graph search and the viewer slow or drops the worker. Run `POST /agentmemory/graph/compact` once after upgrading if graph queries are slow or time out. It trims every node, edge and the cached snapshot to the cap, never lists the full graph, and is safe to re-run. When it trims ids it writes a `graph_compact` audit entry.

```bash
curl -X POST http://localhost:3111/agentmemory/graph/compact -H "Content-Type: application/json" -d '{}'
```

On a large store, or when the call returns 504, run it in slices. Send `scope` (`nodes` or `edges`), `offset` and `limit`, then call again with the returned `nextOffset` until it is `null`. Do this for `nodes` and for `edges`, and finish with one `{"scope":"snapshot"}` call, because a sliced run does not touch the cached snapshot.

```bash
curl -X POST http://localhost:3111/agentmemory/graph/compact -H "Content-Type: application/json" -d '{"scope":"nodes","offset":0,"limit":200}'
curl -X POST http://localhost:3111/agentmemory/graph/compact -H "Content-Type: application/json" -d '{"scope":"snapshot"}'
```

---

<h2 id="development"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/tags/light/section-development.svg"><img src="assets/tags/section-development.svg" alt="Development" height="32" /></picture></h2>
Expand Down
3 changes: 2 additions & 1 deletion plugin/skills/agentmemory-rest-api/REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Generated from `src/triggers/api.ts`. Do not edit the block below by hand; run `
<!-- AUTOGEN:rest START - generated by scripts/skills/generate.ts, do not edit by hand -->
The REST API is the primary surface. All paths are under `http://localhost:3111` (override with `--port`). When `AGENTMEMORY_SECRET` is set, send `Authorization: Bearer $AGENTMEMORY_SECRET`; localhost is otherwise open.

134 registered endpoints:
135 registered endpoints:

| Method | Path |
| --- | --- |
Expand Down Expand Up @@ -55,6 +55,7 @@ The REST API is the primary surface. All paths are under `http://localhost:3111`
| POST | `/agentmemory/governance/bulk-delete` |
| DELETE | `/agentmemory/governance/memories` |
| POST | `/agentmemory/graph/build` |
| POST | `/agentmemory/graph/compact` |
| POST | `/agentmemory/graph/extract` |
| POST | `/agentmemory/graph/import-graphify` |
| GET | `/agentmemory/graph/node` |
Expand Down
5 changes: 3 additions & 2 deletions src/functions/export-import.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ import { indexRecords } from "./search.js";
import { resetLessonIndex } from "./lessons.js";
import { logger } from "../logger.js";
import { budgetImportedObservationSources } from "./observation-source-budget.js";
import { boundRecordSources } from "./graph.js";

// Bounded-concurrency chunk size for the import delete/write loops. A
// "replace" or "merge" of a large export (up to MAX_TOTAL_OBSERVATIONS,
Expand Down Expand Up @@ -516,7 +517,7 @@ export function registerExportImportFunction(sdk: IIIClient, kv: StateKV): void
const existing = await kv.get(KV.graphNodes, node.id).catch(() => null);
if (existing) { stats.skipped++; return; }
}
await kv.set(KV.graphNodes, node.id, node);
await kv.set(KV.graphNodes, node.id, boundRecordSources(node));
}),
);
}
Expand All @@ -527,7 +528,7 @@ export function registerExportImportFunction(sdk: IIIClient, kv: StateKV): void
const existing = await kv.get(KV.graphEdges, edge.id).catch(() => null);
if (existing) { stats.skipped++; return; }
}
await kv.set(KV.graphEdges, edge.id, edge);
await kv.set(KV.graphEdges, edge.id, boundRecordSources(edge));
}),
);
}
Expand Down
1 change: 0 additions & 1 deletion src/functions/graph-import.ts
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,6 @@ export function registerGraphImportFunction(sdk: IIIClient, kv: StateKV): void {
kv,
parsed.nodes,
parsed.edges,
[],
);

await recordAudit(kv, "import", "mem::graph::import-graphify", [], {
Expand Down
190 changes: 170 additions & 20 deletions src/functions/graph.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import {
buildGraphExtractionPrompt,
} from "../prompts/graph-extraction.js";
import { isGraphExtractionEnabled } from "../config.js";
import { recordAudit } from "./audit.js";
import { recordAudit, safeAudit } from "./audit.js";
import { logger } from "../logger.js";

// #753: keep the response payload below the iii state channel ceiling.
Expand Down Expand Up @@ -301,35 +301,46 @@ function snapshotPushEdgeIfBothInTop(
}
}

export const MAX_GRAPH_SOURCE_OBSERVATIONS = 32;

export function boundSources(existing: string[], incoming: string[]): string[] {
const merged = new Set(existing);
for (const id of incoming) {
merged.delete(id);
merged.add(id);
}
return [...merged].slice(-MAX_GRAPH_SOURCE_OBSERVATIONS);
}

export function boundRecordSources<R extends object>(record: R): R {
const sources = (record as { sourceObservationIds?: unknown } | null)?.sourceObservationIds;
if (!Array.isArray(sources) || sources.length <= MAX_GRAPH_SOURCE_OBSERVATIONS) return record;
return { ...record, sourceObservationIds: boundSources([], sources as string[]) };
}

function mergeNode(
existing: GraphNode,
incoming: GraphNode,
obsIds: string[],
capturedAt: string,
): GraphNode {
return {
...existing,
sourceObservationIds: [
...new Set([
...existing.sourceObservationIds,
...incoming.sourceObservationIds,
...obsIds,
]),
],
sourceObservationIds: boundSources(
existing.sourceObservationIds ?? [],
incoming.sourceObservationIds ?? [],
),
properties: { ...existing.properties, ...incoming.properties },
updatedAt: capturedAt,
};
}

function mergeEdge(
existing: GraphEdge,
obsIds: string[],
): GraphEdge {
function mergeEdge(existing: GraphEdge, incoming: GraphEdge): GraphEdge {
return {
...existing,
sourceObservationIds: [
...new Set([...existing.sourceObservationIds, ...obsIds]),
],
sourceObservationIds: boundSources(
existing.sourceObservationIds ?? [],
incoming.sourceObservationIds ?? [],
),
};
}

Expand Down Expand Up @@ -580,7 +591,6 @@ export async function persistGraphDelta(
kv: StateKV,
nodes: GraphNode[],
edges: GraphEdge[],
obsIds: string[],
): Promise<{ newNodeCount: number; newEdgeCount: number }> {
return withKeyedLock("graph:persist", async () => {
const snap = (await readSnapshotStrict(kv)) ?? emptySnapshot();
Expand Down Expand Up @@ -621,7 +631,7 @@ export async function persistGraphDelta(

if (existing) {
idRemap.set(node.id, existing.id);
const merged = mergeNode(existing, node, obsIds, capturedAt);
const merged = mergeNode(existing, node, capturedAt);
await kv.set(KV.graphNodes, existing.id, merged);
// Update topNodes entry if present so a stale clone isn't
// returned from the snapshot fast path.
Expand All @@ -631,6 +641,7 @@ export async function persistGraphDelta(
snapMutated = true;
}
} else {
node.sourceObservationIds = boundSources([], node.sourceObservationIds ?? []);
await kv.set(KV.graphNodes, node.id, node);
await kv.set(KV.graphNameIndex, indexKey, node.id);
await kv.set(KV.graphNodeDegree, node.id, 0);
Expand Down Expand Up @@ -671,7 +682,7 @@ export async function persistGraphDelta(
}

if (existing) {
const merged = mergeEdge(existing, obsIds);
const merged = mergeEdge(existing, edge);
await kv.set(KV.graphEdges, existing.id, merged);
// Replace cached topEdges entry too if present.
const topIdx = snap.topEdges.findIndex((e) => e.id === existing!.id);
Expand All @@ -680,6 +691,7 @@ export async function persistGraphDelta(
snapMutated = true;
}
} else {
edge.sourceObservationIds = boundSources([], edge.sourceObservationIds ?? []);
await kv.set(KV.graphEdges, edge.id, edge);
await kv.set(KV.graphEdgeKey, eKey, edge.id);
snap.stats.totalEdges += 1;
Expand Down Expand Up @@ -709,6 +721,120 @@ export async function persistGraphDelta(
});
}

export interface GraphCompactResult {
nodesScanned: number;
nodesTrimmed: number;
edgesScanned: number;
edgesTrimmed: number;
idsRemoved: number;
snapshotTrimmed: boolean;
total?: number;
nextOffset: number | null;
}

export type GraphCompactScope = "nodes" | "edges" | "snapshot";

export interface GraphCompactOptions {
scope?: GraphCompactScope;
offset?: number;
limit?: number;
}

const COMPACT_SCOPES: readonly GraphCompactScope[] = ["nodes", "edges", "snapshot"];

export async function compactGraphProvenance(
kv: StateKV,
opts: GraphCompactOptions = {},
): Promise<GraphCompactResult> {
const { scope } = opts;
if (scope !== undefined && !COMPACT_SCOPES.includes(scope)) {
throw new Error(`unknown compact scope: ${String(scope)}`);
}
const offset = opts.offset ?? 0;
const limit = opts.limit ?? Number.POSITIVE_INFINITY;
if (!Number.isInteger(offset) || offset < 0) {
throw new Error(`compact offset must be a non-negative integer: ${String(opts.offset)}`);
}
if (!(limit === Number.POSITIVE_INFINITY || (Number.isInteger(limit) && limit > 0))) {
throw new Error(`compact limit must be a positive integer: ${String(opts.limit)}`);
}

const result: GraphCompactResult = {
nodesScanned: 0,
nodesTrimmed: 0,
edgesScanned: 0,
edgesTrimmed: 0,
idsRemoved: 0,
snapshotTrimmed: false,
nextOffset: null,
};

const trimScope = async <R extends { sourceObservationIds: string[] }>(
indexScope: string,
recordScope: string,
): Promise<{ scanned: number; trimmed: number }> => {
const allIds = [...new Set(await kv.list<string>(indexScope))]
.filter((id): id is string => typeof id === "string")
.sort();
const end = Math.min(allIds.length, offset + limit);
result.total = allIds.length;
result.nextOffset = end < allIds.length ? end : null;
let scanned = 0;
let trimmed = 0;
for (const id of allIds.slice(offset, end)) {
await withKeyedLock("graph:persist", async () => {
const record = await kv.get<R>(recordScope, id);
if (!record) return;
scanned += 1;
const sources = record.sourceObservationIds ?? [];
if (sources.length <= MAX_GRAPH_SOURCE_OBSERVATIONS) return;
const bounded = boundSources([], sources);
result.idsRemoved += sources.length - bounded.length;
await kv.set(recordScope, id, { ...record, sourceObservationIds: bounded });
trimmed += 1;
});
}
return { scanned, trimmed };
};

const trimSnapshot = () =>
withKeyedLock("graph:persist", async () => {
const snap = await readSnapshot(kv);
if (!snap) return;
const trimList = <R extends { sourceObservationIds: string[] }>(list: R[]) =>
list.map((r) => {
const sources = r.sourceObservationIds ?? [];
if (sources.length <= MAX_GRAPH_SOURCE_OBSERVATIONS) return r;
result.snapshotTrimmed = true;
return { ...r, sourceObservationIds: boundSources([], sources) };
});
const topNodes = trimList(snap.topNodes);
const topEdges = trimList(snap.topEdges);
if (result.snapshotTrimmed) {
await kv.set(KV.graphSnapshot, SNAPSHOT_KEY, { ...snap, topNodes, topEdges });
}
});

if (scope === undefined || scope === "nodes") {
const n = await trimScope<GraphNode>(KV.graphNameIndex, KV.graphNodes);
result.nodesScanned = n.scanned;
result.nodesTrimmed = n.trimmed;
}
if (scope === undefined || scope === "edges") {
const e = await trimScope<GraphEdge>(KV.graphEdgeKey, KV.graphEdges);
result.edgesScanned = e.scanned;
result.edgesTrimmed = e.trimmed;
}
if (scope === undefined || scope === "snapshot") {
await trimSnapshot();
}
if (scope === undefined) {
delete result.total;
result.nextOffset = null;
}
return result;
}

export function registerGraphFunction(
sdk: IIIClient,
kv: StateKV,
Expand Down Expand Up @@ -772,7 +898,6 @@ export function registerGraphFunction(
kv,
nodes,
edges,
obsIds,
);

await recordAudit(kv, "observe", "mem::graph-extract", obsIds, {
Expand Down Expand Up @@ -1135,6 +1260,31 @@ export function registerGraphFunction(
}
});

sdk.registerFunction("mem::graph-compact", async (data?: GraphCompactOptions) => {
const started = Date.now();
try {
const result = await compactGraphProvenance(kv, data ?? {});
const tookMs = Date.now() - started;
logger.info("Graph provenance compacted", { ...result, tookMs });
if (result.idsRemoved > 0) {
await safeAudit(kv, "graph_compact", "mem::graph-compact", [], {
scope: data?.scope ?? "all",
offset: data?.offset,
limit: data?.limit,
nodesTrimmed: result.nodesTrimmed,
edgesTrimmed: result.edgesTrimmed,
idsRemoved: result.idsRemoved,
snapshotTrimmed: result.snapshotTrimmed,
});
}
return { success: true, ...result, tookMs };
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
logger.error("Graph provenance compaction failed", { error: msg });
return { success: false, error: msg };
}
});

// #814 v2 + #825: clean-restart escape hatch for corpora of any
// size, including the legacy 75K+ case that crashes kv.list.
//
Expand Down
Loading
Loading