Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# ======================================================================# agentmemory configuration
# ======================================================================#
# =============================================================================
# agentmemory configuration
# =============================================================================
Comment thread
guibeira marked this conversation as resolved.
Resolved
#

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Commits retain prohibited attribution trailers

Both 0a056c7 and eb131ee contain Co-authored-by: iii harness (claude-opus-5-5). The repository prohibits AI attribution headers in commits.

Learn more

The repository's contribution rules prohibit attribution headers naming Claude in commit messages. Both new commit messages include a Co-authored-by trailer naming claude-opus-5-5; this is separate from their missing DCO sign-offs.

Example: The commit restoring this header ends with Co-authored-by: iii harness (claude-opus-5-5) <noreply@iii.dev>.

Recommended fix: Remove the prohibited trailers while amending the two commits.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

648883a and f608db8 no longer carry the attribution trailer. The follow-up fixup commit 0ec2a9b still does; the fixup commits will be squashed into f608db8 and the trailer removed before merge. Leaving this open until then.

# Copy this file to `~/.agentmemory/.env` (or to your project root if you
# prefer scoped config) and uncomment the lines you want to override.
#
Expand Down
2 changes: 1 addition & 1 deletion docs/upgrades/iii-0.26.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Use the same `--instance`, `--port` and `--data-dir` flags you used to start it.
- Observers use `capture` only. Background observers receive the `Observed` payload and read the event from its `input` or `output`; HTTP route handlers still receive the `HttpRequest`.
- Failure audit observers capture `["input", "error"]` with an `error` filter for thrown errors, and `["input", "output"]` with an `output.success == false` filter for soft failures.
- When the state or http worker has not registered within 30 seconds of boot, `runtime.bindings` in `/agentmemory/status` reports that provider as `pending` instead of falling back. Its observers attach as soon as the worker connects.
- A pipeline step runs as a direct call while its own observer is not active (pending, rejected or suspended), so compression and post-session work keep running. `runtime.bindings.reasons` lists the steps that use direct calls.
- A compression or post-session step runs as a direct call while its own observer is not active (pending, rejected or suspended), so that work keeps running. `runtime.bindings.reasons` lists the steps that use direct calls. Delete cascades, relation reactions, failure audits and post-remember graph extraction have no direct-call path: `runtime.bindings.reasons` lists them separately, and their work waits until their observers are active.
- The compression sweep retries a failed compression at most 5 times and sends pending observations before failed ones.

## Custom deployments
Expand Down
2 changes: 1 addition & 1 deletion src/functions/status.ts
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,7 @@ export function evaluateStatus(input: StatusInputs): StatusReport {
problems.push({
level: "error",
code: "observer-rejected",
message: `${failedObservers} observer${failedObservers === 1 ? " was" : "s were"} rejected by the engine. Rejected route or state observers leave those endpoints or reactions inactive; rejected pipeline observers fall back to direct calls.`,
message: `${failedObservers} observer${failedObservers === 1 ? " was" : "s were"} rejected by the engine. Rejected route or state observers leave those endpoints or reactions inactive. Rejected compression and post-session observers fall back to direct calls; rejected delete-cascade, relation-reaction, failure-audit and post-remember graph observers stop that work until restart.`,
fix: "Check runtime.bindings.reasons in /agentmemory/status for the conflicting route, remove the conflict, then restart agentmemory.",
});
}
Expand Down
51 changes: 39 additions & 12 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ import { bootLog, bootWarn } from "./logger.js";
import { runtimeMetadataPath } from "./runtime-paths.js";
import { getBindings } from "./runtime/bindings.js";
import { waitForStateWorker, onStateWorkerReturn } from "./runtime/state-worker.js";
import { singleFlight, shouldRearmIndex, rearmIndexRecovery, backfillFailedOutright } from "./runtime/recovery.js";
import { singleFlight, shouldRearmIndex, rearmIndexRecovery, backfillFailedOutright, createRetryBackoff } from "./runtime/recovery.js";
import { checkDuplicateWorkers } from "./runtime/duplicates.js";
import { registerPipelineObservers } from "./runtime/pipeline-observers.js";
import { registerPostSessionFunctions } from "./functions/post-session.js";
Expand Down Expand Up @@ -489,10 +489,11 @@ async function main() {
registerPipelineObservers(sdk);

const bindings = getBindings(sdk);
let sweepTimer: ReturnType<typeof setInterval> | null = null;
if (isAutoCompressEnabled()) {
const sweepMs = compressionSweepMs();
if (sweepMs > 0) {
const sweepTimer = setInterval(() => { void runCompressionSweep(); }, sweepMs);
sweepTimer = setInterval(() => { void runCompressionSweep(); }, sweepMs);
sweepTimer.unref();
}
}
Expand Down Expand Up @@ -559,18 +560,22 @@ async function main() {
);
} else {
console.error(
`[agentmemory] Refusing to start: persisted vector index has ` +
`[agentmemory] Not loading the persisted vector index: it has ` +
`${mismatches.length} of ${loaded.vector.size} vectors with the ` +
`wrong dimension. Active provider (${embeddingProvider?.name}) ` +
`declares ${activeDim}; dimensions seen on disk: ${distinct}. ` +
`First mismatched obsIds: ${sample}. Loading would silently corrupt ` +
`search (cross-dimension cosine returns 0). Choose one:\n` +
` - Re-embed the existing index against the new provider, then start.\n` +
`search (cross-dimension cosine returns 0). The checkpoint is kept ` +
`and vector recovery stays paused; it is not retried. Choose one, then restart:\n` +
` - Re-embed the existing index against the new provider.\n` +
` - Set AGENTMEMORY_DROP_STALE_INDEX=true to discard the persisted ` +
`vectors and rebuild from live observations.\n` +
` - Switch the embedding provider back to the one that wrote the index.`,
);
await indexPersistence.prepareBackfill(null);
indexPersistence.pauseForUnusableCheckpoint(
`persisted vector checkpoint has ${mismatches.length} of ${loaded.vector.size} vectors with the wrong dimension`,
);
checkpointBackfillSince = undefined;
mismatchBlocked = true;
}
} else if (boot) {
Expand Down Expand Up @@ -732,23 +737,36 @@ async function main() {
capture.start();
})();

// An unreadable boot checkpoint is retried on the runtime tick with a
// doubling delay; a checkpoint that loads but cannot be used stops retries.
// Neither case is fixed by rebuilding the keyword index, so they do not
// count as a paused recovery that needs one.
const checkpointRetry = createRetryBackoff({ initialMs: 30_000, maxMs: 30 * 60_000 });
const checkpointUnusable = () => indexPersistence.isPausedForUnavailable() || checkpointRetry.stopped;
const rearm = onStateWorkerReturn(sdk, bindings, async () => {
try {
await rearmIndexRecovery({
retryCheckpoint: async () => {
if (!bootCheckpointUnavailable || !indexPersistence.resumeAfterUnavailable()) return false;
if (!bootCheckpointUnavailable || checkpointRetry.stopped || !indexPersistence.resumeAfterUnavailable()) return false;
const retry = await restoreVectorCheckpoint(false);
if (retry.mismatchBlocked) {
checkpointRetry.stop();
return false;
}
checkpointLoaded = retry.loaded;
vectorBackfillSince = retry.vectorBackfillSince;
if (!retry.loaded || retry.loaded.state === "unavailable") return false;
if (!retry.loaded || retry.loaded.state === "unavailable") {
checkpointRetry.failed();
return false;
}
vectorIndex?.finishLiveTracking();
return true;
},
indexNeedsRearm: () => shouldRearmIndex({
bm25Incomplete: isBm25RebuildIncomplete(),
persistencePaused: indexPersistence.isRecoveryPaused(),
persistencePaused: indexPersistence.isRecoveryPaused() && !checkpointUnusable(),
lastRecoveryFailed: indexRecoveryFailed,
checkpointSaveFailed: Boolean(indexPersistence.status().vector?.lastError),
checkpointSaveFailed: indexPersistence.checkpointSaveFailed(),
}),
runIndexRecovery,
});
Expand All @@ -760,10 +778,16 @@ async function main() {
} catch (err) {
bootWarn(`State-worker re-arm failed: ${err instanceof Error ? err.message : String(err)}`);
}
}, { pending: () => bootCheckpointUnavailable && indexPersistence.isPausedForUnavailable() });
}, {
pending: () => bootCheckpointUnavailable && indexPersistence.isPausedForUnavailable() && checkpointRetry.due(),
onWorkersChanged: () => { void checkDuplicateWorkers(sdk); },
});
// Worker connect/disconnect events drive the duplicate-worker check, plus one
// check whenever that observer becomes active again. While it is not active
// (pending, rejected, out of conflict retries) the tick checks instead.
const rearmTick = setInterval(() => {
void rearm.probeTick();
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
Resolved
void checkDuplicateWorkers(sdk);
if (!bindings.runtimeActive("runtime::on-workers-changed")) void checkDuplicateWorkers(sdk);
}, 30_000);
rearmTick.unref();
void checkDuplicateWorkers(sdk);
Expand Down Expand Up @@ -908,6 +932,9 @@ async function main() {
process.exit(await runStartupCleanup(1).catch(() => 1));
}, SHUTDOWN_HARD_EXIT_MS);
hardExit.unref();
clearInterval(rearmTick);
if (sweepTimer) clearInterval(sweepTimer);
rearm.stop();
healthMonitor.stop();
dedupMap.stop();
capture.stop();
Expand Down
78 changes: 64 additions & 14 deletions src/runtime/bindings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,15 @@ interface SdkLike {
observe?: (source: ObserveSource, target: string, options?: ObserverOptions) => ObserverHandle;
}

type BindingKind = "http" | "state" | "pipeline";
type BindingKind = "http" | "state" | "pipeline" | "runtime";

interface BoundObserver {
handle: ObserverHandle;
kind: BindingKind;
target: string;
// Pipeline work that also runs as a direct call while its observer is not active.
fallback: boolean;
}

const instances = new WeakMap<object, Bindings>();

Expand All @@ -47,11 +55,11 @@ const ID_CONFLICT_RETRY_DELAYS_MS = [10_000, 20_000, 40_000, 80_000];

export class Bindings {
private readonly sdk: SdkLike;
private readonly bound: Array<{ handle: ObserverHandle; kind: BindingKind; target: string }> = [];
private readonly bound: BoundObserver[] = [];
private readonly reasons = new Set<string>();
private readonly inline: boolean;
private failedObserverCount = 0;
private readonly boundWatchers = new Set<(entry: { handle: ObserverHandle; kind: BindingKind }) => void>();
private readonly boundWatchers = new Set<(entry: BoundObserver) => void>();

constructor(sdk: SdkLike) {
this.sdk = sdk;
Expand Down Expand Up @@ -87,6 +95,18 @@ export class Bindings {
return this.bound.some((entry) => entry.kind === "pipeline" && entry.target === target && entry.handle.state === "active");
}

onRuntimeHandle(target: string, listener: (handle: ObserverHandle) => void): () => void {
const watch = (entry: BoundObserver) => {
if (entry.kind === "runtime" && entry.target === target) listener(entry.handle);
};
this.boundWatchers.add(watch);
return () => this.boundWatchers.delete(watch);
}

runtimeActive(target: string): boolean {
return this.bound.some((entry) => entry.kind === "runtime" && entry.target === target && entry.handle.state === "active");
}

get modes(): BindingModes {
return { state: this.modeOf("state"), http: this.modeOf("http") };
}
Expand All @@ -107,13 +127,21 @@ export class Bindings {
return parsed;
}

private observe(kind: BindingKind, label: string, source: ObserveSource, target: string, options?: ObserverOptions, attempt = 0): ObserverHandle | null {
private observe(
kind: BindingKind,
label: string,
source: ObserveSource,
target: string,
options?: ObserverOptions,
extra: { fallback?: boolean; attempt?: number } = {},
): ObserverHandle | null {
const attempt = extra.attempt ?? 0;
if (typeof this.sdk.observe !== "function") {
this.reasons.add("client has no observe(); bindings skipped");
return null;
}
const handle = this.sdk.observe(source, target, options);
const entry = { handle, kind, target };
const entry: BoundObserver = { handle, kind, target, fallback: extra.fallback ?? false };
this.bound.push(entry);
for (const watch of this.boundWatchers) watch(entry);
void handle.ready().catch((err: unknown) => {
Expand All @@ -125,7 +153,7 @@ export class Bindings {
void handle.dispose().catch(() => {}).finally(() => {
const index = this.bound.indexOf(entry);
if (index >= 0) this.bound.splice(index, 1);
this.observe(kind, label, source, target, options, attempt + 1);
this.observe(kind, label, source, target, options, { ...extra, attempt: attempt + 1 });
});
}, delay);
timer.unref?.();
Expand Down Expand Up @@ -179,9 +207,23 @@ export class Bindings {
);
}

observePipeline(source: ObserveSource, target: string, options?: ObserverOptions): ObserverHandle | null {
/**
* Binds a pipeline step. `fallback: true` (the default) means the caller
* checks `observing(target)` and calls the step directly while the observer
* is not active; pass `fallback: false` for work that only runs through it.
*/
observePipeline(
source: ObserveSource,
target: string,
options?: ObserverOptions,
opts: { fallback?: boolean } = {},
): ObserverHandle | null {
if (this.inline) return null;
return this.observe("pipeline", `pipeline ${target}`, source, target, options);
return this.observe("pipeline", `pipeline ${target}`, source, target, options, { fallback: opts.fallback ?? true });
}

observeRuntime(source: ObserveSource, target: string, options?: ObserverOptions): ObserverHandle | null {
return this.observe("runtime", `runtime ${target}`, source, target, options);
}

observeStateChanges(filters: Record<string, unknown>, target: string, options?: ObserverOptions): ObserverHandle | null {
Expand All @@ -208,7 +250,7 @@ export class Bindings {
const timer = setTimeout(finish, timeoutMs);
timer.unref?.();
const watch = ({ handle, kind }: { handle: ObserverHandle; kind: BindingKind }) => {
if (done || kind === "pipeline") return;
if (done || (kind !== "state" && kind !== "http")) return;
unsubscribes.push(handle.onStateChange(() => { if (settled()) finish(); }));
if (settled()) finish();
};
Expand All @@ -223,11 +265,19 @@ export class Bindings {
const reasons = new Set(this.reasons);
if (modes.state === "pending" && this.handlesOf("state").length > 0) reasons.add("state::on-changed observers pending until the state worker connects");
if (modes.http === "pending" && this.handlesOf("http").length > 0) reasons.add("http::on-request observers pending until the http worker connects");
if (!this.inline) {
const inactive = this.bound.filter((entry) => entry.kind === "pipeline" && entry.handle.state !== "active");
if (inactive.length > 0) {
reasons.add(`${inactive.length} pipeline observers are not active, so their work runs as direct calls: ${inactive.map((entry) => entry.target).join(", ")}`);
}
const inactive = this.bound.filter((entry) => entry.handle.state !== "active");
const targets = (entries: BoundObserver[]) => [...new Set(entries.map((entry) => entry.target))].join(", ");
const directCalls = inactive.filter((entry) => entry.kind === "pipeline" && entry.fallback);
if (!this.inline && directCalls.length > 0) {
reasons.add(`${directCalls.length} pipeline observers are not active, so their work runs as direct calls: ${targets(directCalls)}`);
}
const stalled = inactive.filter((entry) => entry.kind === "pipeline" && !entry.fallback);
if (stalled.length > 0) {
reasons.add(`${stalled.length} observers have no direct-call fallback and are not active, so their work does not run until they activate: ${targets(stalled)}`);
}
const runtime = inactive.filter((entry) => entry.kind === "runtime");
if (runtime.length > 0) {
reasons.add(`${runtime.length} runtime observers are not active: ${targets(runtime)}`);
}
for (const kind of ["state", "http"] as const) {
if (modes[kind] !== "observe") continue;
Expand Down
9 changes: 9 additions & 0 deletions src/runtime/duplicates.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ export interface DuplicateWorkerInfo {

let last: DuplicateWorkerInfo | null = null;
let lastKey = "";
// Checks can overlap (worker events, the fallback tick, a state-worker
// return). A check only publishes when no check started after it already has.
let started = 0;
let applied = 0;

export function getDuplicateWorkers(): DuplicateWorkerInfo | null {
return last;
Expand All @@ -16,11 +20,16 @@ export function getDuplicateWorkers(): DuplicateWorkerInfo | null {
export function resetDuplicateWorkersForTest(): void {
last = null;
lastKey = "";
started = 0;
applied = 0;
}

export async function checkDuplicateWorkers(sdk: IIIClient): Promise<DuplicateWorkerInfo | null> {
const seq = ++started;
try {
const listed = await sdk.trigger({ function_id: "engine::workers::list", payload: {} });
if (seq < applied) return last;
applied = seq;
const entries = Array.isArray(listed)
? listed
: (((listed as { workers?: unknown[] })?.workers ?? []) as unknown[]);
Expand Down
3 changes: 3 additions & 0 deletions src/runtime/pipeline-observers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export function registerPipelineObservers(sdk: IIIClient): void {
{ function_id: "mem::remember", capture: ["input", "output"] },
"mem::post-remember-graph",
{ id: "agentmemory:pipeline:post-remember-graph", max_in_flight: 1 },
{ fallback: false },
);
}
for (const fn of ["mem::compress", "mem::summarize", "mem::graph-extract", "mem::consolidate-pipeline", "mem::reflect", "mem::skill-extract"]) {
Expand All @@ -57,13 +58,15 @@ export function registerPipelineObservers(sdk: IIIClient): void {
{ function_id: fn, filters: failure.filters, capture: failure.capture },
"mem::record-failure",
{ id: `agentmemory:pipeline:failure:${fn}:${failure.suffix}`, max_in_flight: 4, metadata: { source_fn: fn } },
{ fallback: false },
);
}
}
bindings.observePipeline(
{ function_id: "mem::graph-extract", filters: { "output.partial": { $eq: true } }, capture: ["input", "output"] },
"mem::record-failure",
{ id: "agentmemory:pipeline:failure:mem::graph-extract:partial", max_in_flight: 4, metadata: { source_fn: "mem::graph-extract" } },
{ fallback: false },
);
bindings.observeStateChanges(
{
Expand Down
30 changes: 30 additions & 0 deletions src/runtime/recovery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,36 @@ export async function rearmIndexRecovery(steps: RearmSteps): Promise<void> {
if (restored || steps.indexNeedsRearm()) await steps.runIndexRecovery();
}

export interface RetryBackoff {
readonly stopped: boolean;
/** True once the current delay has passed and retries were not stopped. */
due(): boolean;
/** Records a failed attempt and doubles the delay, up to maxMs. */
failed(): void;
/** Stops retrying for the life of the process. */
stop(): void;
}

export function createRetryBackoff(opts: { initialMs: number; maxMs: number; now?: () => number }): RetryBackoff {
const now = opts.now ?? (() => Date.now());
let delay = opts.initialMs;
let nextAt = now() + delay;
let stopped = false;
return {
get stopped() {
return stopped;
},
due: () => !stopped && now() >= nextAt,
failed: () => {
delay = Math.min(delay * 2, opts.maxMs);
nextAt = now() + delay;
},
stop: () => {
stopped = true;
},
};
}

export function backfillFailedOutright(result: { added: number; failed: number }): boolean {
return result.failed > 0 && result.added === 0;
}
Expand Down
Loading