Repository navigation
perf(graph): extract only observations captured since the last extract #1268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
da57e96
253b542
781e5d1
22ce830
32ed3e5
ae7b834
0a21a06
c3ed775
a00adeb
d457d7a
23a8cf6
b1f8d66
5bc5f77
8901b5f
62f593c
f84dae6
43eae53
5c08308
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| import { TriggerAction, type IIIClient } from "iii-sdk"; | ||
| import type { CompressedObservation, Memory, Session } from "../types.js"; | ||
| import { KV } from "../state/schema.js"; | ||
| import { KV, fingerprintId } from "../state/schema.js"; | ||
| import { StateKV } from "../state/kv.js"; | ||
| import { emitLiveEvent } from "../viewer/live-events.js"; | ||
| import { isReflectEnabled } from "../functions/slots.js"; | ||
|
|
@@ -10,14 +10,24 @@ import { | |
| getConsolidationCooldownMs, | ||
| isAgentScopeIsolated, | ||
| isConsolidationEnabled, | ||
| isGraphExtractionEnabled, | ||
| } from "../config.js"; | ||
| import { getGraphGeneration } from "../state/graph-generation.js"; | ||
| import { logger } from "../logger.js"; | ||
| import { noteMemoryChange, noteSessionChange } from "../state/viewer-counts.js"; | ||
|
|
||
| // Global marker recording when corpus consolidation last ran, used to debounce | ||
| // the per-turn session-stop fan-out. | ||
| const CONSOLIDATION_MARKER_KEY = "consolidation:lastRun"; | ||
|
|
||
| const observationFingerprint = (obs: CompressedObservation[], context: string): string => | ||
| fingerprintId("gx", `${context}|${obs.map((o) => o.id).sort().join(",")}`); | ||
|
|
||
| async function graphExtractContext(kv: StateKV): Promise<string> { | ||
| const generation = await getGraphGeneration(kv).catch(() => null); | ||
| return `${generation ?? "legacy"}|${isGraphExtractionEnabled() ? "llm" : "heuristic"}`; | ||
| } | ||
|
|
||
| async function consolidationDueUnserialized(kv: StateKV): Promise<boolean> { | ||
| const cooldownMs = getConsolidationCooldownMs(); | ||
| if (cooldownMs <= 0) return true; // debounce disabled | ||
|
|
@@ -70,7 +80,44 @@ export function registerEventTriggers(sdk: IIIClient, kv: StateKV): void { | |
| ); | ||
| const compressed = observations.filter((o) => o.title); | ||
| if (compressed.length > 0) { | ||
| fireVoid("mem::graph-extract", { observations: compressed }); | ||
| const session = await kv | ||
| .get<Session>(KV.sessions, data.sessionId) | ||
| .catch(() => null); | ||
| const at = session?.graphExtractedAt; | ||
| const mark = session?.graphExtractedDigest; | ||
| const context = await graphExtractContext(kv); | ||
| let batch = compressed; | ||
| if (typeof at === "string") { | ||
| const seen = compressed.filter((o) => o.timestamp <= at); | ||
| if (observationFingerprint(seen, context) === mark) { | ||
|
Comment on lines
+88
to
+92
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| batch = compressed.filter((o) => o.timestamp > at); | ||
|
rohitg00 marked this conversation as resolved.
Resolved
rohitg00 marked this conversation as resolved.
Resolved
|
||
| } else { | ||
| logger.info("graph-extract watermark stale, re-extracting session", { | ||
| sessionId: data.sessionId, | ||
| atOrBelow: seen.length, | ||
| total: compressed.length, | ||
| }); | ||
| } | ||
| } | ||
| if (batch.length > 0) { | ||
| const newest = batch.reduce( | ||
| (max, o) => (o.timestamp > max ? o.timestamp : max), | ||
| "", | ||
| ); | ||
| await sdk.trigger({ | ||
| function_id: "mem::graph-extract", | ||
| payload: { observations: batch }, | ||
| action: TriggerAction.Void(), | ||
|
Comment on lines
+107
to
+110
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| }); | ||
| if (session) await kv.update(KV.sessions, data.sessionId, [ | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| { type: "set", path: "graphExtractedAt", value: newest }, | ||
| { | ||
| type: "set", | ||
| path: "graphExtractedDigest", | ||
| value: observationFingerprint(compressed, context), | ||
| }, | ||
| ]); | ||
| } | ||
| } | ||
| } catch (err) { | ||
| logger.warn("graph-extract trigger failed", { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
rg -n -C 3 'getGraphGeneration|graphControl|generation|graph/build' src/state/graph-generation.ts src/functions/graph.ts src/triggers/events.tsRepository: rohitg00/agentmemory
Length of output: 9337
Script executed:
Repository: rohitg00/agentmemory
Length of output: 13674
Script executed:
Repository: rohitg00/agentmemory
Length of output: 4844
Remove the silent
legacyfallback so a failed generation lookup skips the stop.graphExtractContextconverts a failed generation lookup intolegacy. Before a reset, a session digest can be stored withlegacy. After a reset, a failed lookup on a stop with no newer observations can match that stored digest and skip extraction. The reset does not extract session observations or clear the watermark, so the new graph stays missing that session's data until a later successful stop. Propagating the error instead lets the existingcatchlog it and skip the watermark write, so the next stop retries.Suggested fix
Committable suggestion