Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
6715feb
fix(graph): bound and attribute node and edge provenance
david-waltermire Sep 25, 2026
82b9e48
fix(api): validate graph/compact input and report its real failures
david-waltermire Sep 30, 2026
b00eaed
fix(api): return 500 when graph compaction reports a failure
david-waltermire Sep 30, 2026
93143ed
docs: add doc comments to the provenance and compaction helpers
david-waltermire Oct 1, 2026
477dd0d
docs: add doc comments to the graph function registrars
david-waltermire Oct 1, 2026
a909039
fix(graph): bound temporal edge provenance before it is written
david-waltermire Oct 1, 2026
364dfe7
Merge remote-tracking branch 'origin/main' into fix-1445
rohitg00 Oct 1, 2026
05138d3
chore(graph): drop code comments added with provenance compaction
rohitg00 Oct 1, 2026
d84e5da
fix(graph): walk compaction slices in a stable order and audit the run
rohitg00 Oct 1, 2026
fdfbf5d
fix(graph): bound provenance on import, mesh sync and snapshot restore
rohitg00 Oct 1, 2026
683ce83
fix(graph): accept loosely typed records in boundRecordSources
rohitg00 Oct 1, 2026
ebe12f9
docs: explain POST /agentmemory/graph/compact
rohitg00 Oct 1, 2026
1deedde
chore: merge main after the session sweep and capture filter landed
rohitg00 Oct 1, 2026
71891b6
feat(graph): compact bloated graph provenance once at boot
rohitg00 Oct 1, 2026
97ad9e2
fix(graph): keep boot compaction totals exact across crashes
rohitg00 Oct 1, 2026
bbab447
docs(env): document the boot-time graph compaction setting
rohitg00 Oct 1, 2026
5df5690
chore: merge main after the graph provenance PR landed
rohitg00 Oct 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@
# BM25_WEIGHT=0.4 # Hybrid search weight for BM25 leg
# VECTOR_WEIGHT=0.6 # Hybrid search weight for vector leg
# AGENTMEMORY_GRAPH_WEIGHT=0.2 # Graph traversal bonus on smart-search ranking
# AGENTMEMORY_GRAPH_COMPACT_ON_BOOT=false # Skip the one-time background trim of oversized graph provenance at startup
# TOKEN_BUDGET=2000 # Max tokens injected via mem::context per session
# MAX_OBS_PER_SESSION=500 # Per-session observation cap before consolidation kicks in
# SUMMARIZE_CHUNK_SIZE=400 # When mem::summarize sees a session larger than this, it chunks observations and map-reduces (chunk-summarize → reduce-merge) to stay within the LLM's context window. Default 400 ≈ 50k tokens per chunk at ~110 tok/obs. Native sessions are capped by MAX_OBS_PER_SESSION; chunking primarily matters for bulk-imported jsonl sessions, which bypass that cap.
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1699,13 +1699,15 @@ Full endpoint list: [`src/triggers/api.ts`](src/triggers/api.ts)

</details>

**Compacting graph provenance.** Each knowledge graph node and edge keeps the ids of the newest 32 observations it came from. Stores written before that cap can hold thousands of ids per hot node, which makes graph search and the viewer slow or drops the worker. Run `POST /agentmemory/graph/compact` once after upgrading if graph queries are slow or time out. It trims every node, edge and the cached snapshot to the cap, never lists the full graph, and is safe to re-run. When it trims ids it writes a `graph_compact` audit entry.
**Compacting graph provenance.** Each knowledge graph node and edge keeps the ids of the newest 32 observations it came from. Stores written before that cap can hold thousands of ids per hot node, which makes graph search and the viewer slow or drops the worker. agentmemory fixes this by itself: on the first start after upgrading it trims every node, edge, superseded edge (the temporal graph history) and the cached snapshot to the cap in the background, in small slices with a pause between them, so search, capture and the viewer keep working. It saves its progress, resumes after a restart and never runs again once it has finished. `/agentmemory/status` and the viewer Health page show it as pending, running (with the current scope and position), done or failed. Set `AGENTMEMORY_GRAPH_COMPACT_ON_BOOT=false` to turn it off.

To run it by hand, call `POST /agentmemory/graph/compact`. It walks the name and edge-key indexes instead of listing every node and edge, and is safe to re-run. When it trims ids it writes a `graph_compact` audit entry.

```bash
curl -X POST http://localhost:3111/agentmemory/graph/compact -H "Content-Type: application/json" -d '{}'
```

On a large store, or when the call returns 504, run it in slices. Send `scope` (`nodes` or `edges`), `offset` and `limit`, then call again with the returned `nextOffset` until it is `null`. Do this for `nodes` and for `edges`, and finish with one `{"scope":"snapshot"}` call, because a sliced run does not touch the cached snapshot.
On a large store, or when the call returns 504, run it in slices. Send `scope` (`nodes`, `edges` or `history`), `offset` and `limit`, then call again with the returned `nextOffset` until it is `null`. Do this for `nodes`, `edges` and `history`, and finish with one `{"scope":"snapshot"}` call, because a sliced run does not touch the cached snapshot.

```bash
curl -X POST http://localhost:3111/agentmemory/graph/compact -H "Content-Type: application/json" -d '{"scope":"nodes","offset":0,"limit":200}'
Expand Down
3 changes: 2 additions & 1 deletion plugin/skills/agentmemory-config/REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Generated by scanning `src/` for `AGENTMEMORY_*` usage. Do not edit the block below by hand; run `npm run skills:gen` after adding or removing a variable. Internal markers ending in two underscores are excluded.

<!-- AUTOGEN:env START - generated by scripts/skills/generate.ts, do not edit by hand -->
Configuration is read from the environment and from `~/.agentmemory/.env` (no `export` prefix). 58 recognized variables:
Configuration is read from the environment and from `~/.agentmemory/.env` (no `export` prefix). 59 recognized variables:

- `AGENTMEMORY_AGENT_SCOPE`
- `AGENTMEMORY_ALLOW_AGENT_SDK`
Expand All @@ -27,6 +27,7 @@ Configuration is read from the environment and from `~/.agentmemory/.env` (no `e
- `AGENTMEMORY_EXPORT_ROOT`
- `AGENTMEMORY_FOLLOWUP_WINDOW_SECONDS`
- `AGENTMEMORY_FORCE_PROXY`
- `AGENTMEMORY_GRAPH_COMPACT_ON_BOOT`
- `AGENTMEMORY_GRAPH_WEIGHT`
- `AGENTMEMORY_III_CONFIG`
- `AGENTMEMORY_III_VERSION`
Expand Down
4 changes: 4 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,10 @@ export function getSessionSweepStaleHours(): number {
return parsed > 0 ? parsed : 24;
}

export function isGraphCompactOnBootEnabled(): boolean {
return getMergedEnv()["AGENTMEMORY_GRAPH_COMPACT_ON_BOOT"] !== "false";
}

export function isDropStaleIndexEnabled(): boolean {
return getMergedEnv()["AGENTMEMORY_DROP_STALE_INDEX"] === "true";
}
Expand Down
316 changes: 316 additions & 0 deletions src/functions/graph-compact-boot.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,316 @@
import type { StateKV } from "../state/kv.js";
import { KV } from "../state/schema.js";
import {
MAX_GRAPH_SOURCE_OBSERVATIONS,
compactGraphProvenance,
type GraphCompactResult,
type GraphCompactScope,
} from "./graph.js";
import { safeAudit } from "./audit.js";

export const GRAPH_COMPACT_BOOT_VERSION = 1;
export const GRAPH_COMPACT_BOOT_KEY = `graph-compact-on-boot:v${GRAPH_COMPACT_BOOT_VERSION}`;
export const GRAPH_COMPACT_BOOT_FUNCTION_ID = "mem::graph-compact-on-boot";
export const GRAPH_COMPACT_BOOT_DELAY_MS = 3000;

const SLICED_SCOPES = ["nodes", "edges", "history"] as const;
type SlicedScope = (typeof SLICED_SCOPES)[number];
type BootPhase = SlicedScope | "snapshot";

export interface GraphCompactBootProgress {
version: number;
cap: number;
status: "running" | "done" | "failed";
phase: BootPhase;
offset: number;
phaseTotal: number | null;
scanned: number;
trimmed: number;
idsRemoved: number;
pending?: GraphCompactBootSlicePlan;
startedAt: string;
updatedAt: string;
completedAt?: string;
error?: string;
}

export interface GraphCompactBootSlicePlan {
phase: BootPhase;
offset: number;
scanned: number;
trimmed: number;
idsRemoved: number;
nextOffset: number | null;
total: number | null;
}

export type GraphCompactBootState = "off" | "pending" | "running" | "done" | "failed";

export interface GraphCompactBootStatus {
state: GraphCompactBootState;
phase?: BootPhase;
processed?: number;
total?: number | null;
scanned?: number;
trimmed?: number;
idsRemoved?: number;
startedAt?: string;
completedAt?: string;
error?: string;
}

export interface GraphCompactBootOptions {
sliceSize?: number;
pauseMs?: number;
maxAttempts?: number;
retryDelayMs?: number;
log?: (msg: string) => void;
warn?: (msg: string) => void;
sleep?: (ms: number) => Promise<void>;
now?: () => Date;
}

const DEFAULT_SLICE_SIZE = 100;
const DEFAULT_PAUSE_MS = 200;
const DEFAULT_MAX_ATTEMPTS = 3;
const DEFAULT_RETRY_DELAY_MS = 2000;

let current: GraphCompactBootStatus = { state: "pending" };
let running: Promise<GraphCompactBootStatus> | null = null;

export function getGraphCompactBootStatus(): GraphCompactBootStatus {
return { ...current };
}

export function setGraphCompactBootDisabled(): void {
current = { state: "off" };
}

export function resetGraphCompactBootStatus(): void {
current = { state: "pending" };
running = null;
}

function statusFrom(p: GraphCompactBootProgress): GraphCompactBootStatus {
return {
state: p.status,
phase: p.phase,
processed: p.offset,
total: p.phaseTotal,
scanned: p.scanned,
trimmed: p.trimmed,
idsRemoved: p.idsRemoved,
startedAt: p.startedAt,
completedAt: p.completedAt,
error: p.error,
};
}

function isResumable(p: GraphCompactBootProgress | null): p is GraphCompactBootProgress {
return (
!!p &&
p.version === GRAPH_COMPACT_BOOT_VERSION &&
p.cap === MAX_GRAPH_SOURCE_OBSERVATIONS &&
(p.phase === "snapshot" || (SLICED_SCOPES as readonly string[]).includes(p.phase)) &&
Number.isInteger(p.offset) &&
p.offset >= 0
);
}

function storedPlan(p: GraphCompactBootProgress, phase: BootPhase): GraphCompactBootSlicePlan | null {
const plan = p.pending;
if (!plan || plan.phase !== phase || plan.offset !== p.offset) return null;
const counts = [plan.scanned, plan.trimmed, plan.idsRemoved];
return counts.every((n) => Number.isInteger(n) && n >= 0) ? plan : null;
}

function scannedOf(scope: SlicedScope, r: GraphCompactResult): [number, number] {
if (scope === "nodes") return [r.nodesScanned, r.nodesTrimmed];
if (scope === "edges") return [r.edgesScanned, r.edgesTrimmed];
return [r.historyScanned, r.historyTrimmed];
}

export function describeGraphCompactBoot(s: GraphCompactBootStatus): string {
if (s.state === "off") return "off (AGENTMEMORY_GRAPH_COMPACT_ON_BOOT=false)";
if (s.state === "pending") return "pending";
const where =
s.phase === "snapshot" || s.total === null || s.total === undefined
? `${s.phase ?? "nodes"}`
: `${s.phase} ${s.processed ?? 0} of ${s.total}`;
if (s.state === "running") return `running, ${where}`;
if (s.state === "failed") return `failed at ${where}: ${s.error ?? "unknown error"}`;
return `done, ${s.trimmed ?? 0} records trimmed, ${s.idsRemoved ?? 0} ids removed`;
}

export function runGraphCompactOnBoot(
kv: StateKV,
opts: GraphCompactBootOptions = {},
): Promise<GraphCompactBootStatus> {
if (running) return running;
running = runOnce(kv, opts).finally(() => {
running = null;
});
return running;
}

async function runOnce(kv: StateKV, opts: GraphCompactBootOptions): Promise<GraphCompactBootStatus> {
const sliceSize = opts.sliceSize ?? DEFAULT_SLICE_SIZE;
const pauseMs = opts.pauseMs ?? DEFAULT_PAUSE_MS;
const maxAttempts = Math.max(1, opts.maxAttempts ?? DEFAULT_MAX_ATTEMPTS);
const retryDelayMs = opts.retryDelayMs ?? DEFAULT_RETRY_DELAY_MS;
const log = opts.log ?? (() => {});
const warn = opts.warn ?? log;
const sleep = opts.sleep ?? ((ms: number) => new Promise<void>((r) => setTimeout(r, ms)));
const now = opts.now ?? (() => new Date());

const attempt = async <T>(fn: () => Promise<T>): Promise<T> => {
let lastErr: unknown;
for (let i = 1; i <= maxAttempts; i++) {
try {
return await fn();
} catch (err) {
lastErr = err;
if (i < maxAttempts) await sleep(retryDelayMs * i);
}
}
throw lastErr;
};

let progress: GraphCompactBootProgress | null = null;
const save = async () => {
if (!progress) return;
progress.updatedAt = now().toISOString();
current = statusFrom(progress);
await attempt(() => kv.set(KV.config, GRAPH_COMPACT_BOOT_KEY, progress));
};

try {
const stored = await attempt(() => kv.get<GraphCompactBootProgress>(KV.config, GRAPH_COMPACT_BOOT_KEY));
if (isResumable(stored) && stored.status === "done") {
current = statusFrom(stored);
return getGraphCompactBootStatus();
}
const startedAt = now().toISOString();
if (isResumable(stored)) {
progress = { ...stored, status: "running", error: undefined };
log(`Graph provenance compaction: resuming in the background at ${progress.phase} ${progress.offset}`);
} else {
progress = {
version: GRAPH_COMPACT_BOOT_VERSION,
cap: MAX_GRAPH_SOURCE_OBSERVATIONS,
status: "running",
phase: "nodes",
offset: 0,
phaseTotal: null,
scanned: 0,
trimmed: 0,
idsRemoved: 0,
startedAt,
updatedAt: startedAt,
};
log(`Graph provenance compaction: started in the background (cap ${MAX_GRAPH_SOURCE_OBSERVATIONS} ids per record)`);
}
await save();

const p = progress;
for (const scope of SLICED_SCOPES) {
if (p.phase === "snapshot") break;
if (SLICED_SCOPES.indexOf(p.phase) > SLICED_SCOPES.indexOf(scope)) continue;
p.phase = scope;
let phaseTrimmed = 0;
let phaseScanned = 0;
for (;;) {
const slice = { scope: scope as GraphCompactScope, offset: p.offset, limit: sliceSize };
let plan = storedPlan(p, scope);
if (!plan) {
const r = await attempt(() => compactGraphProvenance(kv, { ...slice, dryRun: true }));
const [scanned, trimmed] = scannedOf(scope, r);
plan = {
phase: scope,
offset: p.offset,
scanned,
trimmed,
idsRemoved: r.idsRemoved,
nextOffset: r.nextOffset,
total: r.total ?? null,
};
if (plan.trimmed > 0) {
p.pending = plan;
await save();
}
}
if (plan.trimmed > 0) await attempt(() => compactGraphProvenance(kv, slice));
phaseScanned += plan.scanned;
phaseTrimmed += plan.trimmed;
p.scanned += plan.scanned;
p.trimmed += plan.trimmed;
p.idsRemoved += plan.idsRemoved;
p.pending = undefined;
p.phaseTotal = plan.total;
p.offset = plan.nextOffset ?? p.phaseTotal ?? p.offset;
await save();
if (plan.nextOffset === null) break;
if (pauseMs > 0) await sleep(pauseMs);
}
log(`Graph provenance compaction: ${scope} done (${phaseScanned} checked, ${phaseTrimmed} trimmed)`);
const next = SLICED_SCOPES[SLICED_SCOPES.indexOf(scope) + 1];
p.phase = next ?? "snapshot";
p.offset = 0;
p.phaseTotal = null;
await save();
}

let snapPlan = storedPlan(p, "snapshot");
if (!snapPlan) {
const dry = await attempt(() => compactGraphProvenance(kv, { scope: "snapshot", dryRun: true }));
snapPlan = {
phase: "snapshot",
offset: p.offset,
scanned: 0,
trimmed: dry.snapshotTrimmed ? 1 : 0,
idsRemoved: 0,
nextOffset: null,
total: null,
};
if (snapPlan.trimmed > 0) {
p.pending = snapPlan;
await save();
}
}
if (snapPlan.trimmed > 0) await attempt(() => compactGraphProvenance(kv, { scope: "snapshot" }));
const snapshotTrimmed = snapPlan.trimmed > 0;
p.trimmed += snapPlan.trimmed;
p.pending = undefined;
p.status = "done";
p.completedAt = now().toISOString();
await save();
if (p.idsRemoved > 0) {
await safeAudit(kv, "graph_compact", GRAPH_COMPACT_BOOT_FUNCTION_ID, [], {
trigger: "boot",
recordsTrimmed: p.trimmed,
idsRemoved: p.idsRemoved,
snapshotTrimmed,
});
}
log(
`Graph provenance compaction: done (${p.scanned} records checked, ${p.trimmed} trimmed, ${p.idsRemoved} ids removed)`,
);
return getGraphCompactBootStatus();
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
if (progress) {
progress.status = "failed";
progress.error = msg;
current = statusFrom(progress);
try {
await kv.set(KV.config, GRAPH_COMPACT_BOOT_KEY, { ...progress, updatedAt: now().toISOString() });
} catch {}
} else {
current = { state: "failed", error: msg };
}
warn(
`Graph provenance compaction failed (${describeGraphCompactBoot(current)}). It resumes on the next start; to run it now, POST /agentmemory/graph/compact.`,
);
return getGraphCompactBootStatus();
}
}
Loading
Loading