From b0f4afa196b375161cc91f8090bf647fe0fa71b1 Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Fri, 25 Sep 2026 20:15:39 +0530 Subject: [PATCH 1/7] feat(state): add opt-in redis backend for iii-state and iii-stream The bundled file-based kv adapter rewrites a whole scope on every flush and holds the entire store in the engine's heap, which does not scale for a shared daemon with several concurrent writers. iii-engine 0.22.1 already ships a redis adapter for both iii-state and iii-stream that writes per key (HSET) instead of rewriting a whole scope, so expose it as an opt-in instead of building a new storage layer. AGENTMEMORY_STATE_BACKEND=redis (plus AGENTMEMORY_REDIS_URL) switches both iii-state and iii-stream to that adapter when agentmemory renders its runtime iii-config. The file-based store stays the default; leaving the variable unset renders byte-identical config to before this change. Selecting redis without a URL fails fast with a clear error before the engine is launched, both from renderEngineConfig directly and from the CLI's startEngine() check. rewriteBundledConfig (src/cli/engine-launch.ts) is the path a real npm install actually takes; it previously rebuilt its own options object for renderEngineConfig and silently dropped anything beyond dataDir and ports, so a live redis: false-negative was live-verified via docker (iii-state/iii-stream kept rendering to the file adapter under the bundled config until this was fixed) before landing. README documents the switch plus the export/import commands for moving data between backends, since switching starts from an empty store. --- .env.example | 3 + README.md | 29 +++++++++ plugin/skills/agentmemory-config/REFERENCE.md | 4 +- src/cli.ts | 10 ++- src/cli/engine-config.ts | 64 +++++++++++++++++-- src/cli/engine-launch.ts | 1 + src/config.ts | 14 ++++ test/engine-config.test.ts | 54 ++++++++++++++++ test/engine-launch.test.ts | 18 ++++++ test/state-backend-config.test.ts | 63 ++++++++++++++++++ 10 files changed, 253 insertions(+), 7 deletions(-) create mode 100644 test/state-backend-config.test.ts diff --git a/.env.example b/.env.example index 9219a338a1..854a05df01 100644 --- a/.env.example +++ b/.env.example @@ -150,6 +150,9 @@ # AGENTMEMORY_USE_DOCKER=1 # Prefer bundled Docker even when a compatible native iii binary exists # AGENTMEMORY_EXPORT_ROOT=~/agentmemory-backup # Default destination for `agentmemory export` +# AGENTMEMORY_STATE_BACKEND=redis # file (default) | redis — storage adapter for iii-state and iii-stream, see README "Storage backend" +# AGENTMEMORY_REDIS_URL=redis://localhost:6379 # Required when AGENTMEMORY_STATE_BACKEND=redis + # STANDALONE_MCP=1 # MCP shim only — bypass the worker and run @agentmemory/mcp in-process # STANDALONE_PERSIST_PATH=~/.agentmemory/local.db # Path used by the standalone MCP shim's local fallback store diff --git a/README.md b/README.md index 1a84a0139b..890cae86f6 100644 --- a/README.md +++ b/README.md @@ -1288,6 +1288,35 @@ On engine 0.22.x keep the `iii-` prefixed names for the builtins above; the unpr Full registry: [workers.iii.dev](https://workers.iii.dev). Every worker there composes through the same primitives agentmemory uses, and the agentmemory you already have is one of them. +### Storage backend: file (default) vs redis + +`iii-state` and `iii-stream` default to iii-engine's bundled file-based KV store: one JSON file per scope, held in the engine process's memory and rewritten to disk on a timer. That's the right default for a single-user local install, but a shared daemon with several concurrent writers benefits from real per-key writes instead. + +Set `AGENTMEMORY_STATE_BACKEND=redis` (plus `AGENTMEMORY_REDIS_URL`) to switch both workers to iii-engine's built-in `redis` adapter, which stores each key as a Redis hash field (`HSET`) instead of rewriting a whole scope on every write: + +```env +# ~/.agentmemory/.env +AGENTMEMORY_STATE_BACKEND=redis +AGENTMEMORY_REDIS_URL=redis://localhost:6379 +``` + +`AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged. Only a native (non-Docker) start reads these two variables and renders them into the launched `iii-config`. A Docker Compose deployment mounts `iii-config.docker.yaml` read-only and never goes through this render step — switch it by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. + +**Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side — nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to: + +```bash +# 1. On the old backend, while agentmemory is still running on it: +curl -s http://localhost:3111/agentmemory/export > backup.json + +# 2. Switch AGENTMEMORY_STATE_BACKEND (and AGENTMEMORY_REDIS_URL if needed), +# restart agentmemory against the new backend, then: +jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \ + curl -s -X POST http://localhost:3111/agentmemory/import \ + -H 'Content-Type: application/json' -d @- +``` + +`/agentmemory/export` also accepts `?maxSessions=` and `?offset=` for chunking a large corpus across several calls; `strategy` on import is `merge` (default-safe), `replace`, or `skip`. + ### What iii replaces | Traditional stack | agentmemory uses | diff --git a/plugin/skills/agentmemory-config/REFERENCE.md b/plugin/skills/agentmemory-config/REFERENCE.md index d152a7c62b..43ca60c548 100644 --- a/plugin/skills/agentmemory-config/REFERENCE.md +++ b/plugin/skills/agentmemory-config/REFERENCE.md @@ -3,7 +3,7 @@ Generated by scanning `src/` for `AGENTMEMORY_*` usage. Do not edit the block below by hand; run `npm run skills:gen` after adding or removing a variable. Internal markers ending in two underscores are excluded. -Configuration is read from the environment and from `~/.agentmemory/.env` (no `export` prefix). 43 recognized variables: +Configuration is read from the environment and from `~/.agentmemory/.env` (no `export` prefix). 45 recognized variables: - `AGENTMEMORY_AGENT_SCOPE` - `AGENTMEMORY_ALLOW_AGENT_SDK` @@ -35,12 +35,14 @@ Configuration is read from the environment and from `~/.agentmemory/.env` (no `e - `AGENTMEMORY_PROBE_TIMEOUT_MS` - `AGENTMEMORY_PROJECT_NAME` - `AGENTMEMORY_PROVIDER` +- `AGENTMEMORY_REDIS_URL` - `AGENTMEMORY_REFLECT` - `AGENTMEMORY_RUNTIME_DIR` - `AGENTMEMORY_SDK_CHILD` - `AGENTMEMORY_SECRET` - `AGENTMEMORY_SESSION_ID` - `AGENTMEMORY_SLOTS` +- `AGENTMEMORY_STATE_BACKEND` - `AGENTMEMORY_SUPPRESS_COST_WARNING` - `AGENTMEMORY_TOOLS` - `AGENTMEMORY_URL` diff --git a/src/cli.ts b/src/cli.ts index 832428ebcf..4f3e47809e 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -78,7 +78,7 @@ import { renderSplash } from "./cli/splash.js"; import { isFirstRun, readPrefs, resetPrefs, writePrefs } from "./cli/preferences.js"; import { runOnboarding } from "./cli/onboarding.js"; import { setBootVerbose } from "./logger.js"; -import { hydrateProcessEnvFromFile } from "./config.js"; +import { getRedisUrl, getStateBackend, hydrateProcessEnvFromFile } from "./config.js"; import { III_PINNED_VERSION, VERSION } from "./version.js"; import { getAllTools, ESSENTIAL_TOOLS } from "./mcp/tools-registry.js"; import { knownAgents } from "./cli/connect/index.js"; @@ -1465,6 +1465,7 @@ function prepareEngineLaunch(configPath: string): { viewerPort: getConfiguredViewerPort(), enginePort: getEnginePort(), }, + stateBackend: { kind: getStateBackend(), redisUrl: getRedisUrl() }, }; const rewritten = bundledConfig ? rewriteBundledConfig( @@ -1552,6 +1553,13 @@ function pickCompatibleIii(candidates: Array): string } async function startEngine(): Promise { + if (getStateBackend() === "redis" && !getRedisUrl()) { + p.log.error( + "AGENTMEMORY_STATE_BACKEND=redis requires AGENTMEMORY_REDIS_URL to be set (e.g. redis://localhost:6379). " + + "Set both in ~/.agentmemory/.env, or unset AGENTMEMORY_STATE_BACKEND to keep the default file store.", + ); + process.exit(1); + } await assertRuntimePortOwnership(); const persistedState = readEngineState(); if (persistedState?.kind === "docker") { diff --git a/src/cli/engine-config.ts b/src/cli/engine-config.ts index b48f09dfa5..6c1ee1deab 100644 --- a/src/cli/engine-config.ts +++ b/src/cli/engine-config.ts @@ -83,6 +83,7 @@ export function clearPersistedBuiltinConfig( export interface EngineConfigOptions { dataDir: string; ports?: EngineRuntimePorts; + stateBackend?: StateBackendOptions; } export interface EngineRuntimePorts { @@ -92,6 +93,11 @@ export interface EngineRuntimePorts { enginePort: number; } +export interface StateBackendOptions { + kind: "file" | "redis"; + redisUrl?: string; +} + function yamlSingleQuote(value: string): string { return `'${value.replace(/'/g, "''")}'`; } @@ -154,6 +160,40 @@ function setWorkerPort(lines: string[], name: string, port: number): void { } } +function replaceKvAdapterWithRedis( + lines: string[], + workerName: string, + redisUrl: string, +): void { + const block = workerBlock(lines, workerName); + if (!block) return; + const adapterIndex = lines.findIndex( + (line, index) => + index > block.start && index < block.end && line.trim() === "adapter:", + ); + if (adapterIndex === -1) return; + const adapterIndent = lines[adapterIndex]!.match(/^\s*/)?.[0] ?? ""; + let end = block.end; + for (let i = adapterIndex + 1; i < block.end; i++) { + const line = lines[i]!; + if (line.trim() === "") continue; + const indent = line.match(/^\s*/)?.[0] ?? ""; + if (indent.length <= adapterIndent.length) { + end = i; + break; + } + } + const childIndent = `${adapterIndent} `; + const grandchildIndent = `${adapterIndent} `; + lines.splice( + adapterIndex + 1, + end - (adapterIndex + 1), + `${childIndent}name: redis`, + `${childIndent}config:`, + `${grandchildIndent}redis_url: ${yamlSingleQuote(redisUrl)}`, + ); +} + function setManagedCorsOrigins( lines: string[], restPort: number, @@ -181,6 +221,12 @@ export function renderEngineConfig( template: string, options: EngineConfigOptions, ): string { + if (options.stateBackend?.kind === "redis" && !options.stateBackend.redisUrl) { + throw new Error( + "AGENTMEMORY_STATE_BACKEND=redis requires AGENTMEMORY_REDIS_URL to be set (e.g. redis://localhost:6379).", + ); + } + const rendered = template .replace( "file_path: ./data/state_store.db", @@ -190,12 +236,20 @@ export function renderEngineConfig( "file_path: ./data/stream_store", `file_path: ${yamlSingleQuote(join(options.dataDir, "stream_store"))}`, ); - if (!options.ports) return rendered; + + const usesRedis = options.stateBackend?.kind === "redis"; + if (!options.ports && !usesRedis) return rendered; const lines = rendered.split("\n"); - setWorkerPort(lines, "iii-http", options.ports.restPort); - setWorkerPort(lines, "iii-stream", options.ports.streamPort); - setWorkerPort(lines, "iii-worker-manager", options.ports.enginePort); - setManagedCorsOrigins(lines, options.ports.restPort, options.ports.viewerPort); + if (options.ports) { + setWorkerPort(lines, "iii-http", options.ports.restPort); + setWorkerPort(lines, "iii-stream", options.ports.streamPort); + setWorkerPort(lines, "iii-worker-manager", options.ports.enginePort); + setManagedCorsOrigins(lines, options.ports.restPort, options.ports.viewerPort); + } + if (usesRedis) { + replaceKvAdapterWithRedis(lines, "iii-state", options.stateBackend!.redisUrl!); + replaceKvAdapterWithRedis(lines, "iii-stream", options.stateBackend!.redisUrl!); + } return lines.join("\n"); } diff --git a/src/cli/engine-launch.ts b/src/cli/engine-launch.ts index d75173600e..43ee801131 100644 --- a/src/cli/engine-launch.ts +++ b/src/cli/engine-launch.ts @@ -162,6 +162,7 @@ export function rewriteBundledConfig( const rendered = renderEngineConfig(raw, { dataDir: options?.dataDir ?? join(agentmemoryHome(home), "data"), ...(options?.ports ? { ports: options.ports } : {}), + ...(options?.stateBackend ? { stateBackend: options.stateBackend } : {}), }); return removeAgentmemoryExecCommand(rendered, nodeBin, workerEntry); } diff --git a/src/config.ts b/src/config.ts index a6c62ce55e..51bd63e64e 100644 --- a/src/config.ts +++ b/src/config.ts @@ -235,6 +235,20 @@ export function isDropStaleIndexEnabled(): boolean { return getMergedEnv()["AGENTMEMORY_DROP_STALE_INDEX"] === "true"; } +const VALID_STATE_BACKENDS = new Set(["file", "redis"]); + +export function getStateBackend(): "file" | "redis" { + const raw = (getMergedEnv()["AGENTMEMORY_STATE_BACKEND"] || "file") + .trim() + .toLowerCase(); + return VALID_STATE_BACKENDS.has(raw) ? (raw as "file" | "redis") : "file"; +} + +export function getRedisUrl(): string | undefined { + const value = getMergedEnv()["AGENTMEMORY_REDIS_URL"]; + return hasRealValue(value) ? value.trim() : undefined; +} + export function detectLlmProviderKind(): "llm" | "noop" { const env = getMergedEnv(); if ( diff --git a/test/engine-config.test.ts b/test/engine-config.test.ts index 59919a72cc..bdbd729608 100644 --- a/test/engine-config.test.ts +++ b/test/engine-config.test.ts @@ -73,6 +73,60 @@ describe("renderEngineConfig", () => { ); }); + it("is byte-identical to the file-default render when stateBackend is omitted or 'file'", () => { + const source = readFileSync( + join(import.meta.dirname, "..", "iii-config.yaml"), + "utf8", + ); + const dataDir = "/tmp/agentmemory-fixture"; + + const withoutOption = renderEngineConfig(source, { dataDir }); + const withFileKind = renderEngineConfig(source, { + dataDir, + stateBackend: { kind: "file" }, + }); + + expect(withFileKind).toBe(withoutOption); + expect(withFileKind).toContain("name: kv"); + expect(withFileKind).not.toContain("name: redis"); + }); + + it("switches both iii-state and iii-stream to the redis adapter", () => { + const source = readFileSync( + join(import.meta.dirname, "..", "iii-config.yaml"), + "utf8", + ); + + const rendered = renderEngineConfig(source, { + dataDir: "/tmp/agentmemory-fixture", + stateBackend: { kind: "redis", redisUrl: "redis://localhost:6390" }, + }); + + expect(rendered).toMatch( + /- name: iii-state\n {4}config:\n {6}adapter:\n {8}name: redis\n {8}config:\n {10}redis_url: 'redis:\/\/localhost:6390'/, + ); + expect(rendered).toMatch( + /- name: iii-stream\n {4}config:\n {6}port: 3112\n {6}host: 127\.0\.0\.1\n {6}adapter:\n {8}name: redis\n {8}config:\n {10}redis_url: 'redis:\/\/localhost:6390'/, + ); + expect(rendered).not.toContain("store_method: file_based"); + expect(rendered).not.toContain("state_store.db"); + expect(rendered).not.toContain("stream_store"); + }); + + it("throws a clear error when redis is selected without a URL", () => { + const source = readFileSync( + join(import.meta.dirname, "..", "iii-config.yaml"), + "utf8", + ); + + expect(() => + renderEngineConfig(source, { + dataDir: "/tmp/agentmemory-fixture", + stateBackend: { kind: "redis" }, + }), + ).toThrow(/AGENTMEMORY_REDIS_URL/); + }); + it("keeps the iii- prefixed builtin names: on 0.22.1 the unprefixed names resolve to registry workers", () => { const source = readFileSync( join(import.meta.dirname, "..", "iii-config.yaml"), diff --git a/test/engine-launch.test.ts b/test/engine-launch.test.ts index cf12f687cd..a5d0b21253 100644 --- a/test/engine-launch.test.ts +++ b/test/engine-launch.test.ts @@ -175,4 +175,22 @@ describe("rewriteBundledConfig", () => { expect(out).toContain("port: 3212"); expect(out).toContain("port: 49234"); }); + + it("forwards the redis state backend through to renderEngineConfig", () => { + const raw = readFileSync(join(import.meta.dirname, "..", "iii-config.yaml"), "utf-8"); + const out = rewriteBundledConfig( + raw, + HOME, + process.execPath, + "/opt/pkg/dist/index.mjs", + { + dataDir: "/var/lib/agentmemory", + stateBackend: { kind: "redis", redisUrl: "redis://localhost:6379" }, + }, + ); + + expect(out).toMatch(/- name: iii-state\n {4}config:\n {6}adapter:\n {8}name: redis/); + expect(out).toMatch(/- name: iii-stream\n {4}config:\n {6}port: 3112\n {6}host: 127\.0\.0\.1\n {6}adapter:\n {8}name: redis/); + expect(out).not.toContain("store_method: file_based"); + }); }); diff --git a/test/state-backend-config.test.ts b/test/state-backend-config.test.ts new file mode 100644 index 0000000000..05b6600f22 --- /dev/null +++ b/test/state-backend-config.test.ts @@ -0,0 +1,63 @@ +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { getRedisUrl, getStateBackend, __resetEnvFileCache } from "../src/config.js"; + +const KEYS = ["AGENTMEMORY_STATE_BACKEND", "AGENTMEMORY_REDIS_URL"] as const; + +describe("getStateBackend / getRedisUrl", () => { + const saved: Record = {}; + let sandboxHome: string; + let savedHome: string | undefined; + let savedUserProfile: string | undefined; + + beforeEach(() => { + sandboxHome = mkdtempSync(join(tmpdir(), "am-statecfg-")); + savedHome = process.env["HOME"]; + savedUserProfile = process.env["USERPROFILE"]; + process.env["HOME"] = sandboxHome; + process.env["USERPROFILE"] = sandboxHome; + __resetEnvFileCache(); + for (const k of KEYS) { + saved[k] = process.env[k]; + delete process.env[k]; + } + }); + + afterEach(() => { + for (const k of KEYS) { + if (saved[k] === undefined) delete process.env[k]; + else process.env[k] = saved[k]; + } + if (savedHome === undefined) delete process.env["HOME"]; + else process.env["HOME"] = savedHome; + if (savedUserProfile === undefined) delete process.env["USERPROFILE"]; + else process.env["USERPROFILE"] = savedUserProfile; + __resetEnvFileCache(); + rmSync(sandboxHome, { recursive: true, force: true }); + }); + + it("defaults to file when unset", () => { + expect(getStateBackend()).toBe("file"); + }); + + it("returns file for an unrecognized value", () => { + process.env["AGENTMEMORY_STATE_BACKEND"] = "sqlite"; + expect(getStateBackend()).toBe("file"); + }); + + it("returns redis when explicitly set, case-insensitively", () => { + process.env["AGENTMEMORY_STATE_BACKEND"] = "Redis"; + expect(getStateBackend()).toBe("redis"); + }); + + it("returns undefined for AGENTMEMORY_REDIS_URL when unset", () => { + expect(getRedisUrl()).toBeUndefined(); + }); + + it("returns the trimmed AGENTMEMORY_REDIS_URL when set", () => { + process.env["AGENTMEMORY_REDIS_URL"] = " redis://localhost:6379 "; + expect(getRedisUrl()).toBe("redis://localhost:6379"); + }); +}); From 7b7cac5e8cfe4a1ba852eb87b59f80a821e1d98e Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Sat, 26 Sep 2026 11:45:22 +0530 Subject: [PATCH 2/7] fix(state): fail loudly on unswitchable redis config, harden migration docs and test isolation renderEngineConfig silently left a worker on its default file/kv adapter when a custom iii-config.yaml was missing the iii-state/iii-stream worker block or its adapter block, even though AGENTMEMORY_STATE_BACKEND=redis had been requested. It now throws so the mismatch surfaces instead of the engine starting on a different backend than the user selected. The README migration commands for export/import omitted the bearer token required when AGENTMEMORY_SECRET is set, used curl -s (which can write an HTTP error body to backup.json without failing), and never checked that backup.json was a usable export before telling the user to switch backends and restart. All three are fixed. test/state-backend-config.test.ts imported src/config.ts at module scope, so DATA_DIR/ENV_FILE were computed from the real HOME before beforeEach sandboxed it, letting a developer's real ~/.agentmemory/.env leak into the test. It now sandboxes HOME before a fresh dynamic import of the module, matching the pattern already used in test/env-loader.test.ts. --- README.md | 15 +++++++++++--- src/cli/engine-config.ts | 12 +++++++++-- test/engine-config.test.ts | 33 +++++++++++++++++++++++++++++++ test/state-backend-config.test.ts | 25 +++++++++++++---------- 4 files changed, 70 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 890cae86f6..3e1dc5dde7 100644 --- a/README.md +++ b/README.md @@ -1305,13 +1305,22 @@ AGENTMEMORY_REDIS_URL=redis://localhost:6379 **Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side — nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to: ```bash +# 0. If AGENTMEMORY_SECRET is set, both requests need it: +AUTH=(${AGENTMEMORY_SECRET:+-H "Authorization: Bearer $AGENTMEMORY_SECRET"}) + # 1. On the old backend, while agentmemory is still running on it: -curl -s http://localhost:3111/agentmemory/export > backup.json +curl -fsS "${AUTH[@]}" http://localhost:3111/agentmemory/export > backup.json + +# 2. Confirm backup.json is a usable export before switching backends: +jq -e '.version and .exportedAt' backup.json > /dev/null || { + echo "backup.json is not a valid export; do not switch backends" >&2 + exit 1 +} -# 2. Switch AGENTMEMORY_STATE_BACKEND (and AGENTMEMORY_REDIS_URL if needed), +# 3. Switch AGENTMEMORY_STATE_BACKEND (and AGENTMEMORY_REDIS_URL if needed), # restart agentmemory against the new backend, then: jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \ - curl -s -X POST http://localhost:3111/agentmemory/import \ + curl -fsS "${AUTH[@]}" -X POST http://localhost:3111/agentmemory/import \ -H 'Content-Type: application/json' -d @- ``` diff --git a/src/cli/engine-config.ts b/src/cli/engine-config.ts index 6c1ee1deab..e8d6ce4606 100644 --- a/src/cli/engine-config.ts +++ b/src/cli/engine-config.ts @@ -166,12 +166,20 @@ function replaceKvAdapterWithRedis( redisUrl: string, ): void { const block = workerBlock(lines, workerName); - if (!block) return; + if (!block) { + throw new Error( + `AGENTMEMORY_STATE_BACKEND=redis requires a "${workerName}" worker in the engine config, but none was found.`, + ); + } const adapterIndex = lines.findIndex( (line, index) => index > block.start && index < block.end && line.trim() === "adapter:", ); - if (adapterIndex === -1) return; + if (adapterIndex === -1) { + throw new Error( + `AGENTMEMORY_STATE_BACKEND=redis requires an "adapter:" block under the "${workerName}" worker in the engine config, but none was found.`, + ); + } const adapterIndent = lines[adapterIndex]!.match(/^\s*/)?.[0] ?? ""; let end = block.end; for (let i = adapterIndex + 1; i < block.end; i++) { diff --git a/test/engine-config.test.ts b/test/engine-config.test.ts index bdbd729608..34de4de20b 100644 --- a/test/engine-config.test.ts +++ b/test/engine-config.test.ts @@ -127,6 +127,39 @@ describe("renderEngineConfig", () => { ).toThrow(/AGENTMEMORY_REDIS_URL/); }); + it("throws when redis is selected but the iii-state worker block is missing", () => { + const source = ["workers:", " - name: iii-http", " config:", " port: 3111"].join( + "\n", + ); + + expect(() => + renderEngineConfig(source, { + dataDir: "/tmp/agentmemory-fixture", + stateBackend: { kind: "redis", redisUrl: "redis://localhost:6390" }, + }), + ).toThrow(/iii-state/); + }); + + it("throws when redis is selected but iii-state has no adapter block to replace", () => { + const source = [ + "workers:", + " - name: iii-state", + " config: {}", + " - name: iii-stream", + " config:", + " adapter:", + " name: kv", + " config: {}", + ].join("\n"); + + expect(() => + renderEngineConfig(source, { + dataDir: "/tmp/agentmemory-fixture", + stateBackend: { kind: "redis", redisUrl: "redis://localhost:6390" }, + }), + ).toThrow(/adapter/); + }); + it("keeps the iii- prefixed builtin names: on 0.22.1 the unprefixed names resolve to registry workers", () => { const source = readFileSync( join(import.meta.dirname, "..", "iii-config.yaml"), diff --git a/test/state-backend-config.test.ts b/test/state-backend-config.test.ts index 05b6600f22..a406688e18 100644 --- a/test/state-backend-config.test.ts +++ b/test/state-backend-config.test.ts @@ -1,8 +1,7 @@ -import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { getRedisUrl, getStateBackend, __resetEnvFileCache } from "../src/config.js"; const KEYS = ["AGENTMEMORY_STATE_BACKEND", "AGENTMEMORY_REDIS_URL"] as const; @@ -11,18 +10,24 @@ describe("getStateBackend / getRedisUrl", () => { let sandboxHome: string; let savedHome: string | undefined; let savedUserProfile: string | undefined; + let config: typeof import("../src/config.js"); - beforeEach(() => { + beforeEach(async () => { sandboxHome = mkdtempSync(join(tmpdir(), "am-statecfg-")); savedHome = process.env["HOME"]; savedUserProfile = process.env["USERPROFILE"]; process.env["HOME"] = sandboxHome; process.env["USERPROFILE"] = sandboxHome; - __resetEnvFileCache(); for (const k of KEYS) { saved[k] = process.env[k]; delete process.env[k]; } + // src/config.ts computes its DATA_DIR/env-file path from HOME at module + // import time, so HOME must be sandboxed before the first import, not + // just before each getStateBackend()/getRedisUrl() call. + vi.resetModules(); + config = await import("../src/config.js"); + config.__resetEnvFileCache(); }); afterEach(() => { @@ -34,30 +39,30 @@ describe("getStateBackend / getRedisUrl", () => { else process.env["HOME"] = savedHome; if (savedUserProfile === undefined) delete process.env["USERPROFILE"]; else process.env["USERPROFILE"] = savedUserProfile; - __resetEnvFileCache(); + config.__resetEnvFileCache(); rmSync(sandboxHome, { recursive: true, force: true }); }); it("defaults to file when unset", () => { - expect(getStateBackend()).toBe("file"); + expect(config.getStateBackend()).toBe("file"); }); it("returns file for an unrecognized value", () => { process.env["AGENTMEMORY_STATE_BACKEND"] = "sqlite"; - expect(getStateBackend()).toBe("file"); + expect(config.getStateBackend()).toBe("file"); }); it("returns redis when explicitly set, case-insensitively", () => { process.env["AGENTMEMORY_STATE_BACKEND"] = "Redis"; - expect(getStateBackend()).toBe("redis"); + expect(config.getStateBackend()).toBe("redis"); }); it("returns undefined for AGENTMEMORY_REDIS_URL when unset", () => { - expect(getRedisUrl()).toBeUndefined(); + expect(config.getRedisUrl()).toBeUndefined(); }); it("returns the trimmed AGENTMEMORY_REDIS_URL when set", () => { process.env["AGENTMEMORY_REDIS_URL"] = " redis://localhost:6379 "; - expect(getRedisUrl()).toBe("redis://localhost:6379"); + expect(config.getRedisUrl()).toBe("redis://localhost:6379"); }); }); From 27076cbf1457dbbfe6241f21a691a75d55cac3a5 Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Sat, 26 Sep 2026 13:15:56 +0530 Subject: [PATCH 3/7] fix(state): make a failed redis launch fatal and stop leaking the URL to disk A failed redis config rewrite in prepareEngineLaunch was caught and silently discarded, so the CLI would start the engine on the file (or in-memory) store while the user believed it was on Redis. The catch now runs resolveLaunchRenderFailure and exits loudly when the state backend is redis, while the file backend keeps its existing silent fallback to the bundled config. The rendered runtime config no longer embeds the literal Redis URL. iii 0.22.1 expands ${VAR} references in config values at boot (see engine/src/workers/config.rs expand_env_vars, called from config_file), so the adapter block now renders redis_url: '${AGENTMEMORY_REDIS_URL}' and the CLI already forwards process.env (hydrated from ~/.agentmemory/.env) to the spawned engine process, so the reference resolves the same way the literal used to. The engine's own configuration worker still persists the expanded value under ~/.agentmemory/config/*.yaml (fs.rs writes the resolved struct, not the source text), so the README documents that residual exposure and recommends chmod 700 on shared hosts. Every engine stderr surface the CLI prints (captured startup stderr, docker logs) now runs through redactCredentialUrls first. getStateBackend() now throws on an unrecognized AGENTMEMORY_STATE_BACKEND value instead of silently defaulting to file; startEngine validates it before touching the engine and exits with the same clear error style already used for a missing AGENTMEMORY_REDIS_URL. Both Docker paths (resuming a persisted docker-owned engine, and the AGENTMEMORY_USE_DOCKER compose path) now warn when redis was requested, since neither renders iii-config.docker.yaml. /agentmemory/status reports the active stateBackend so an operator can confirm which store is live without reading the URL. README: replaced the AUTH=(...) migration snippet, which produces a different (and wrong) curl argv under zsh than under bash, with an if/else that is not shell-dependent; documented plain redis:// only (the pinned engine has no TLS), one Redis DB per --instance, the appendonly/noeviction recommendation, and corrected the claim that only a native start renders the two variables (the one-click deploy templates overwrite the bundled config and start natively too, so they render it as well; only this repo's own Docker Compose path does not). deploy/README.md documents the same two variables. --- README.md | 37 ++++++++++++++++++++++--------- deploy/README.md | 13 +++++++++++ src/cli.ts | 35 ++++++++++++++++++++++++----- src/cli/engine-config.ts | 9 ++++---- src/cli/engine-launch.ts | 25 +++++++++++++++++++++ src/config.ts | 10 +++++++-- src/functions/status.ts | 4 ++++ src/triggers/api.ts | 2 ++ test/cli-engine-startup.test.ts | 35 +++++++++++++++++++++++++++++ test/engine-child-env.test.ts | 5 +++++ test/engine-config.test.ts | 5 +++-- test/engine-launch.test.ts | 34 ++++++++++++++++++++++++++++ test/state-backend-config.test.ts | 7 +++++- test/status.test.ts | 11 +++++++++ 14 files changed, 208 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 3e1dc5dde7..baacfdc8e8 100644 --- a/README.md +++ b/README.md @@ -1290,7 +1290,7 @@ Full registry: [workers.iii.dev](https://workers.iii.dev). Every worker there co ### Storage backend: file (default) vs redis -`iii-state` and `iii-stream` default to iii-engine's bundled file-based KV store: one JSON file per scope, held in the engine process's memory and rewritten to disk on a timer. That's the right default for a single-user local install, but a shared daemon with several concurrent writers benefits from real per-key writes instead. +`iii-state` and `iii-stream` default to iii-engine's bundled file-based KV store: one JSON file per scope, held in the engine process's memory and rewritten to disk on a timer. That's the right default for a single-user local install; a shared daemon with several concurrent writers gets real per-key writes from Redis instead, at the cost of a network round trip per operation (every `state::*` call still serializes on one Redis connection, so this trades the file store's lock for a socket, not for parallelism). Set `AGENTMEMORY_STATE_BACKEND=redis` (plus `AGENTMEMORY_REDIS_URL`) to switch both workers to iii-engine's built-in `redis` adapter, which stores each key as a Redis hash field (`HSET`) instead of rewriting a whole scope on every write: @@ -1300,16 +1300,27 @@ AGENTMEMORY_STATE_BACKEND=redis AGENTMEMORY_REDIS_URL=redis://localhost:6379 ``` -`AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged. Only a native (non-Docker) start reads these two variables and renders them into the launched `iii-config`. A Docker Compose deployment mounts `iii-config.docker.yaml` read-only and never goes through this render step — switch it by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. +`AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged, and an unrecognized value (anything other than `file` or `redis`) is a startup error rather than a silent fallback. `/agentmemory/status` reports which backend is active (never the URL). -**Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side — nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to: +**Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings — Upstash, Redis Cloud, ElastiCache with in-transit encryption — default to TLS-only) fails to connect. Point at a local Redis, or a plain-TCP one reached through an SSH/stunnel tunnel. -```bash -# 0. If AGENTMEMORY_SECRET is set, both requests need it: -AUTH=(${AGENTMEMORY_SECRET:+-H "Authorization: Bearer $AGENTMEMORY_SECRET"}) +**One Redis DB per `--instance`.** The engine's Redis key prefixes (`state:`, `stream::`) are fixed, so two agentmemory instances (`--instance 1`, `--instance 2`, ...) pointed at the same database collide and cross-feed each other's viewer stream. Give each instance its own database index, e.g. `redis://localhost:6379/1`. + +**Recommended Redis settings.** The default `save 3600 1 300 100 60 10000` snapshot policy can lose minutes of writes on a crash, worse than the file store's 5s flush window — set `appendonly yes` for anything you'd mind losing. Set `maxmemory-policy noeviction`; `allkeys-lru` or similar silently drops memories once Redis hits its memory limit. + +A native (non-Docker) start, and every one-click [deploy template](deploy/) (they overwrite the bundled `iii-config.yaml` and start natively), read `AGENTMEMORY_STATE_BACKEND`/`AGENTMEMORY_REDIS_URL` and render them into the launched `iii-config` — the URL itself is never written to that rendered file, only a `${AGENTMEMORY_REDIS_URL}` reference that the engine process expands from its own environment at boot. Only this repo's own Docker Compose path (`AGENTMEMORY_USE_DOCKER=1`, or resuming an engine already started that way) mounts `iii-config.docker.yaml` read-only and never renders — `agentmemory start` warns when it detects that combination. Switch that file by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. +The rendered config keeps the URL out of `~/.agentmemory/data/iii-config.runtime.yaml`, but the engine's own configuration worker still persists the *expanded* value to `~/.agentmemory/config/iii-state.yaml` and `iii-stream.yaml` once it boots (iii-engine's `${VAR}` expansion happens before that worker stores its seed, and it stores the resolved value, not the reference). Treat that directory as holding a credential: `chmod 700 ~/.agentmemory` on any shared host, and prefer a Redis ACL user scoped to what agentmemory needs over the database's admin credentials. + +**Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side — nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to. This runs identically under bash and zsh (including `bash -u`) — an array like `AUTH=(${AGENTMEMORY_SECRET:+-H "Authorization: Bearer $AGENTMEMORY_SECRET"})` does not: zsh keeps the header as one malformed word where bash splits it into two, so both requests 401 whenever `AGENTMEMORY_SECRET` is set: + +```bash # 1. On the old backend, while agentmemory is still running on it: -curl -fsS "${AUTH[@]}" http://localhost:3111/agentmemory/export > backup.json +if [ -n "${AGENTMEMORY_SECRET:-}" ]; then + curl -fsS -H "Authorization: Bearer $AGENTMEMORY_SECRET" http://localhost:3111/agentmemory/export > backup.json +else + curl -fsS http://localhost:3111/agentmemory/export > backup.json +fi # 2. Confirm backup.json is a usable export before switching backends: jq -e '.version and .exportedAt' backup.json > /dev/null || { @@ -1319,9 +1330,15 @@ jq -e '.version and .exportedAt' backup.json > /dev/null || { # 3. Switch AGENTMEMORY_STATE_BACKEND (and AGENTMEMORY_REDIS_URL if needed), # restart agentmemory against the new backend, then: -jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \ - curl -fsS "${AUTH[@]}" -X POST http://localhost:3111/agentmemory/import \ - -H 'Content-Type: application/json' -d @- +if [ -n "${AGENTMEMORY_SECRET:-}" ]; then + jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \ + curl -fsS -H "Authorization: Bearer $AGENTMEMORY_SECRET" -X POST http://localhost:3111/agentmemory/import \ + -H 'Content-Type: application/json' -d @- +else + jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \ + curl -fsS -X POST http://localhost:3111/agentmemory/import \ + -H 'Content-Type: application/json' -d @- +fi ``` `/agentmemory/export` also accepts `?maxSessions=` and `?offset=` for chunking a large corpus across several calls; `strategy` on import is `merge` (default-safe), `replace`, or `skip`. diff --git a/deploy/README.md b/deploy/README.md index 91aa199e28..597bd2b75a 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -80,6 +80,19 @@ absent (no third-party calls), `AGENTMEMORY_AUTO_COMPRESS` is off, and `AGENTMEMORY_INJECT_CONTEXT` is off. Opt in only after you've confirmed your provider quota can absorb the workload. +## Optional: Redis state backend + +Every entrypoint here overwrites the npm-bundled `iii-config.yaml` and +execs `agentmemory` directly (no Docker Compose in the loop), so the +same two variables the main README's "Storage backend" section +documents work unchanged: set `AGENTMEMORY_STATE_BACKEND=redis` and +`AGENTMEMORY_REDIS_URL=redis://host:6379` alongside your platform's +other secrets to point `iii-state`/`iii-stream` at an external Redis +instead of the `/data` volume. The pinned engine only supports plain +`redis://` (no TLS), and a Redis reachable from these platforms is +almost always run by you elsewhere — a managed TLS-only Redis will not +work here without a plain-TCP tunnel in front of it. + ## Cold-start budget Measured against fly.io's `iad` region with a 1 GB volume: diff --git a/src/cli.ts b/src/cli.ts index 4f3e47809e..9f9d668f49 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -64,6 +64,8 @@ import { rewriteBundledConfig, runtimeConfigPath, engineChildEnv, + resolveLaunchRenderFailure, + redactCredentialUrls, } from "./cli/engine-launch.js"; import { runtimeMetadataPath } from "./runtime-paths.js"; import { createStartupStderrCapture } from "./cli/startup-stderr.js"; @@ -1352,7 +1354,7 @@ let startupFailure: StartupFailure | null = null; let activeStartupStderr = createStartupStderrCapture(); function printCapturedStartupStderr(): void { - const stderr = activeStartupStderr.text().trim(); + const stderr = redactCredentialUrls(activeStartupStderr.text().trim()); if (IS_VERBOSE && stderr) { p.note(stderr, "engine stderr"); } @@ -1380,7 +1382,7 @@ function printDockerStartupLogs(): void { maxBuffer: 64 * 1024, }, ); - const output = `${result.stdout ?? ""}${result.stderr ?? ""}`.trim(); + const output = redactCredentialUrls(`${result.stdout ?? ""}${result.stderr ?? ""}`.trim()); if (output) p.note(output.slice(-16 * 1024), "docker engine logs"); } @@ -1414,7 +1416,7 @@ function spawnEngineBackground( const abnormal = (code !== null && code !== 0) || (code === null && signal !== null); if (abnormal) { - const stderr = activeStartupStderr.text(); + const stderr = redactCredentialUrls(activeStartupStderr.text()); startupFailure = { kind: isDocker ? "docker-crashed" : "engine-crashed", stderr: @@ -1455,6 +1457,7 @@ function prepareEngineLaunch(configPath: string): { } catch { return { configPath, cwd: process.cwd() }; } + const stateBackendKind = getStateBackend(); try { const rawConfig = readFileSync(configPath, "utf-8"); const options = { @@ -1465,7 +1468,7 @@ function prepareEngineLaunch(configPath: string): { viewerPort: getConfiguredViewerPort(), enginePort: getEnginePort(), }, - stateBackend: { kind: getStateBackend(), redisUrl: getRedisUrl() }, + stateBackend: { kind: stateBackendKind, redisUrl: getRedisUrl() }, }; const rewritten = bundledConfig ? rewriteBundledConfig( @@ -1514,6 +1517,11 @@ function prepareEngineLaunch(configPath: string): { cwd, }; } catch (err) { + const failure = resolveLaunchRenderFailure(stateBackendKind, err); + if (failure.fatal) { + p.log.error(failure.message); + process.exit(1); + } vlog(`runtime config generation failed, using bundled config verbatim: ${String(err)}`); return { configPath, cwd }; } @@ -1552,8 +1560,23 @@ function pickCompatibleIii(candidates: Array): string return null; } +function warnIfDockerIgnoresStateBackend(stateBackendKind: "file" | "redis"): void { + if (stateBackendKind !== "redis") return; + p.log.warn( + "AGENTMEMORY_STATE_BACKEND=redis is not applied on the Docker path: docker-compose.yml mounts iii-config.docker.yaml read-only, so this start will not render it. " + + "Edit iii-config.docker.yaml by hand to point iii-state/iii-stream at redis (see README \"Storage backend\"), or it keeps using the file store.", + ); +} + async function startEngine(): Promise { - if (getStateBackend() === "redis" && !getRedisUrl()) { + let stateBackendKind: "file" | "redis"; + try { + stateBackendKind = getStateBackend(); + } catch (err) { + p.log.error(err instanceof Error ? err.message : String(err)); + process.exit(1); + } + if (stateBackendKind === "redis" && !getRedisUrl()) { p.log.error( "AGENTMEMORY_STATE_BACKEND=redis requires AGENTMEMORY_REDIS_URL to be set (e.g. redis://localhost:6379). " + "Set both in ~/.agentmemory/.env, or unset AGENTMEMORY_STATE_BACKEND to keep the default file store.", @@ -1563,6 +1586,7 @@ async function startEngine(): Promise { await assertRuntimePortOwnership(); const persistedState = readEngineState(); if (persistedState?.kind === "docker") { + warnIfDockerIgnoresStateBackend(stateBackendKind); const inspection = inspectOwnedDockerEngine(persistedState); if (inspection.status === "unavailable" || inspection.status === "missing") { const detail = inspection.status === "unavailable" @@ -1708,6 +1732,7 @@ async function startEngine(): Promise { } if (choice === "docker" && dockerBin && composeFile) { + warnIfDockerIgnoresStateBackend(stateBackendKind); const s = p.spinner(); s.start("Starting iii-engine via Docker..."); configureDockerHostUser(); diff --git a/src/cli/engine-config.ts b/src/cli/engine-config.ts index e8d6ce4606..40d55ab052 100644 --- a/src/cli/engine-config.ts +++ b/src/cli/engine-config.ts @@ -160,10 +160,11 @@ function setWorkerPort(lines: string[], name: string, port: number): void { } } +const REDIS_URL_ENV_REF = "${AGENTMEMORY_REDIS_URL}"; + function replaceKvAdapterWithRedis( lines: string[], workerName: string, - redisUrl: string, ): void { const block = workerBlock(lines, workerName); if (!block) { @@ -198,7 +199,7 @@ function replaceKvAdapterWithRedis( end - (adapterIndex + 1), `${childIndent}name: redis`, `${childIndent}config:`, - `${grandchildIndent}redis_url: ${yamlSingleQuote(redisUrl)}`, + `${grandchildIndent}redis_url: ${yamlSingleQuote(REDIS_URL_ENV_REF)}`, ); } @@ -256,8 +257,8 @@ export function renderEngineConfig( setManagedCorsOrigins(lines, options.ports.restPort, options.ports.viewerPort); } if (usesRedis) { - replaceKvAdapterWithRedis(lines, "iii-state", options.stateBackend!.redisUrl!); - replaceKvAdapterWithRedis(lines, "iii-stream", options.stateBackend!.redisUrl!); + replaceKvAdapterWithRedis(lines, "iii-state"); + replaceKvAdapterWithRedis(lines, "iii-stream"); } return lines.join("\n"); } diff --git a/src/cli/engine-launch.ts b/src/cli/engine-launch.ts index 43ee801131..d96c43719c 100644 --- a/src/cli/engine-launch.ts +++ b/src/cli/engine-launch.ts @@ -192,3 +192,28 @@ export function legacyDataMigrations( export function engineChildEnv(base: NodeJS.ProcessEnv): NodeJS.ProcessEnv { return { ...base, III_TELEMETRY_ENABLED: base["III_TELEMETRY_ENABLED"] ?? "false" }; } + +export type LaunchRenderFailure = + | { fatal: true; message: string } + | { fatal: false }; + +export function resolveLaunchRenderFailure( + stateBackendKind: "file" | "redis", + err: unknown, +): LaunchRenderFailure { + if (stateBackendKind !== "redis") return { fatal: false }; + const reason = err instanceof Error ? err.message : String(err); + return { + fatal: true, + message: + `Failed to render the Redis state backend into the engine config: ${reason} ` + + "Refusing to start the engine, since it would silently fall back to the file (or in-memory) store instead of Redis. " + + "Fix the config or unset AGENTMEMORY_STATE_BACKEND to use the default file store.", + }; +} + +const CREDENTIAL_URL_PATTERN = /:\/\/[^@\s/]*@/g; + +export function redactCredentialUrls(text: string): string { + return text.replace(CREDENTIAL_URL_PATTERN, "://***@"); +} diff --git a/src/config.ts b/src/config.ts index 51bd63e64e..03ccc72b71 100644 --- a/src/config.ts +++ b/src/config.ts @@ -238,10 +238,16 @@ export function isDropStaleIndexEnabled(): boolean { const VALID_STATE_BACKENDS = new Set(["file", "redis"]); export function getStateBackend(): "file" | "redis" { - const raw = (getMergedEnv()["AGENTMEMORY_STATE_BACKEND"] || "file") + const raw = (getMergedEnv()["AGENTMEMORY_STATE_BACKEND"] || "") .trim() .toLowerCase(); - return VALID_STATE_BACKENDS.has(raw) ? (raw as "file" | "redis") : "file"; + if (!raw) return "file"; + if (!VALID_STATE_BACKENDS.has(raw)) { + throw new Error( + `AGENTMEMORY_STATE_BACKEND="${raw}" is not a recognized state backend. Use "file" (the default) or "redis", or unset it to keep the default file store.`, + ); + } + return raw as "file" | "redis"; } export function getRedisUrl(): string | undefined { diff --git a/src/functions/status.ts b/src/functions/status.ts index 552fe6c07a..52ff704c14 100644 --- a/src/functions/status.ts +++ b/src/functions/status.ts @@ -37,6 +37,7 @@ export interface StatusInputs { version: string; engineVersion: string; uptimeSeconds: number; + stateBackend: "file" | "redis"; ports: { rest: number | null; streams: number | null; viewer: number | null }; health: { status?: string; @@ -67,6 +68,7 @@ export interface StatusReport { version: string; engineVersion: string; uptimeSeconds: number; + stateBackend: StatusInputs["stateBackend"]; ports: StatusInputs["ports"]; }; health: StatusInputs["health"]; @@ -209,6 +211,7 @@ export function evaluateStatus(input: StatusInputs): StatusReport { version: input.version, engineVersion: input.engineVersion, uptimeSeconds: input.uptimeSeconds, + stateBackend: input.stateBackend, ports: input.ports, }, health: input.health, @@ -313,6 +316,7 @@ a{color:inherit}

Problems

    ${problems}

Service

${row("Uptime", escapeHtml(formatDuration(report.service.uptimeSeconds)))} +${row("State backend", escapeHtml(report.service.stateBackend))} ${row("REST port", escapeHtml(ports.rest ?? "unknown"))} ${row("Streams port", escapeHtml(ports.streams ?? "unknown"))} ${row("Viewer port", escapeHtml(ports.viewer ?? "not running"))} diff --git a/src/triggers/api.ts b/src/triggers/api.ts index d8a447f2e0..0afb23d104 100644 --- a/src/triggers/api.ts +++ b/src/triggers/api.ts @@ -27,6 +27,7 @@ import { detectEmbeddingProvider, detectLlmProviderKind, getAgentId, + getStateBackend, isAgentScopeIsolated, loadConfig, } from "../config.js"; @@ -346,6 +347,7 @@ export function registerApiTriggers( version: VERSION, engineVersion: III_PINNED_VERSION, uptimeSeconds: Math.round(process.uptime()), + stateBackend: getStateBackend(), ports: { rest: loadConfig().restPort ?? null, streams: bootStreamsPort ?? null, diff --git a/test/cli-engine-startup.test.ts b/test/cli-engine-startup.test.ts index ceaeeacbe6..3a2fef81f9 100644 --- a/test/cli-engine-startup.test.ts +++ b/test/cli-engine-startup.test.ts @@ -146,4 +146,39 @@ describe("fresh native engine startup", () => { expect(spawnBody).toContain("clearEngineState()"); expect(spawnBody).not.toContain("if (!isDocker) clearEnginePidfile();\n clearEngineState();"); }); + + it("stops the launch instead of silently falling back to file when the redis rewrite fails", () => { + const prepareStart = source.indexOf("function prepareEngineLaunch"); + const prepareEnd = source.indexOf("function startIiiBin", prepareStart); + const prepareBody = source.slice(prepareStart, prepareEnd); + expect(prepareBody).toContain("const stateBackendKind = getStateBackend();"); + expect(prepareBody).toMatch( + /catch \(err\) \{\s*const failure = resolveLaunchRenderFailure\(stateBackendKind, err\);\s*if \(failure\.fatal\) \{\s*p\.log\.error\(failure\.message\);\s*process\.exit\(1\);\s*\}/, + ); + expect(prepareBody).toContain( + 'vlog(`runtime config generation failed, using bundled config verbatim: ${String(err)}`);', + ); + }); + + it("rejects an unrecognized AGENTMEMORY_STATE_BACKEND before touching the engine", () => { + const engineStart = source.indexOf("async function startEngine"); + const engineEnd = source.indexOf("async function waitForEngine", engineStart); + const engineBody = source.slice(engineStart, engineEnd); + expect(engineBody).toMatch( + /try \{\s*stateBackendKind = getStateBackend\(\);\s*\} catch \(err\) \{\s*p\.log\.error\(err instanceof Error \? err\.message : String\(err\)\);\s*process\.exit\(1\);\s*\}/, + ); + expect(engineBody.match(/warnIfDockerIgnoresStateBackend\(stateBackendKind\)/g)).toHaveLength(2); + }); + + it("redacts credentials out of every engine stderr surface the CLI prints", () => { + expect(source).toContain( + "const stderr = redactCredentialUrls(activeStartupStderr.text().trim());", + ); + expect(source).toContain( + "const stderr = redactCredentialUrls(activeStartupStderr.text());", + ); + expect(source).toContain( + 'const output = redactCredentialUrls(`${result.stdout ?? ""}${result.stderr ?? ""}`.trim());', + ); + }); }); diff --git a/test/engine-child-env.test.ts b/test/engine-child-env.test.ts index bdbb67d0ae..11088389a0 100644 --- a/test/engine-child-env.test.ts +++ b/test/engine-child-env.test.ts @@ -13,4 +13,9 @@ describe("engineChildEnv", () => { expect(engineChildEnv({ III_TELEMETRY_ENABLED: "true" })["III_TELEMETRY_ENABLED"]).toBe("true"); expect(engineChildEnv({ III_TELEMETRY_ENABLED: "false" })["III_TELEMETRY_ENABLED"]).toBe("false"); }); + + it("passes AGENTMEMORY_REDIS_URL through so the engine can expand ${AGENTMEMORY_REDIS_URL} itself", () => { + const env = engineChildEnv({ AGENTMEMORY_REDIS_URL: "redis://user:pass@localhost:6379" }); + expect(env["AGENTMEMORY_REDIS_URL"]).toBe("redis://user:pass@localhost:6379"); + }); }); diff --git a/test/engine-config.test.ts b/test/engine-config.test.ts index 34de4de20b..2a0e86be02 100644 --- a/test/engine-config.test.ts +++ b/test/engine-config.test.ts @@ -103,11 +103,12 @@ describe("renderEngineConfig", () => { }); expect(rendered).toMatch( - /- name: iii-state\n {4}config:\n {6}adapter:\n {8}name: redis\n {8}config:\n {10}redis_url: 'redis:\/\/localhost:6390'/, + /- name: iii-state\n {4}config:\n {6}adapter:\n {8}name: redis\n {8}config:\n {10}redis_url: '\$\{AGENTMEMORY_REDIS_URL\}'/, ); expect(rendered).toMatch( - /- name: iii-stream\n {4}config:\n {6}port: 3112\n {6}host: 127\.0\.0\.1\n {6}adapter:\n {8}name: redis\n {8}config:\n {10}redis_url: 'redis:\/\/localhost:6390'/, + /- name: iii-stream\n {4}config:\n {6}port: 3112\n {6}host: 127\.0\.0\.1\n {6}adapter:\n {8}name: redis\n {8}config:\n {10}redis_url: '\$\{AGENTMEMORY_REDIS_URL\}'/, ); + expect(rendered).not.toContain("redis://localhost:6390"); expect(rendered).not.toContain("store_method: file_based"); expect(rendered).not.toContain("state_store.db"); expect(rendered).not.toContain("stream_store"); diff --git a/test/engine-launch.test.ts b/test/engine-launch.test.ts index a5d0b21253..86748aa55d 100644 --- a/test/engine-launch.test.ts +++ b/test/engine-launch.test.ts @@ -7,7 +7,9 @@ import { dockerProjectName, isBundledConfig, legacyDataMigrations, + redactCredentialUrls, resolveEngineCwd, + resolveLaunchRenderFailure, rewriteBundledConfig, runtimeConfigPath, } from "../src/cli/engine-launch.js"; @@ -194,3 +196,35 @@ describe("rewriteBundledConfig", () => { expect(out).not.toContain("store_method: file_based"); }); }); + +describe("resolveLaunchRenderFailure", () => { + it("is fatal when the redis backend fails to render, so the launch never silently falls back", () => { + const result = resolveLaunchRenderFailure("redis", new Error("no adapter block")); + expect(result.fatal).toBe(true); + expect(result.fatal && result.message).toMatch(/no adapter block/); + }); + + it("stays non-fatal for the file backend, keeping the bundled-config fallback", () => { + const result = resolveLaunchRenderFailure("file", new Error("disk full")); + expect(result).toEqual({ fatal: false }); + }); + + it("stringifies a non-Error throw", () => { + const result = resolveLaunchRenderFailure("redis", "permission denied"); + expect(result.fatal && result.message).toMatch(/permission denied/); + }); +}); + +describe("redactCredentialUrls", () => { + it("masks userinfo in a redis URL without touching the rest of the message", () => { + const out = redactCredentialUrls( + "Failed to connect to redis://user:s3cr3t@localhost:6379: connection refused", + ); + expect(out).toBe("Failed to connect to redis://***@localhost:6379: connection refused"); + expect(out).not.toContain("s3cr3t"); + }); + + it("leaves text with no credentials untouched", () => { + expect(redactCredentialUrls("engine exited with code 1")).toBe("engine exited with code 1"); + }); +}); diff --git a/test/state-backend-config.test.ts b/test/state-backend-config.test.ts index a406688e18..f490908625 100644 --- a/test/state-backend-config.test.ts +++ b/test/state-backend-config.test.ts @@ -47,8 +47,13 @@ describe("getStateBackend / getRedisUrl", () => { expect(config.getStateBackend()).toBe("file"); }); - it("returns file for an unrecognized value", () => { + it("throws a clear error for an unrecognized value instead of silently using file", () => { process.env["AGENTMEMORY_STATE_BACKEND"] = "sqlite"; + expect(() => config.getStateBackend()).toThrow(/AGENTMEMORY_STATE_BACKEND="sqlite"/); + }); + + it("treats an empty string the same as unset", () => { + process.env["AGENTMEMORY_STATE_BACKEND"] = ""; expect(config.getStateBackend()).toBe("file"); }); diff --git a/test/status.test.ts b/test/status.test.ts index 6ecfb50ca5..8a7144fba6 100644 --- a/test/status.test.ts +++ b/test/status.test.ts @@ -14,6 +14,7 @@ function inputs(overrides: Partial = {}): StatusInputs { version: "0.9.29", engineVersion: "0.22.1", uptimeSeconds: 125, + stateBackend: "file", ports: { rest: 3111, streams: 3112, viewer: 3113 }, health: { status: "healthy", alerts: [], notes: [], connectionState: "connected" }, circuitBreaker: { state: "closed", failures: 0 }, @@ -52,6 +53,12 @@ describe("evaluateStatus", () => { expect(report.service.engineVersion).toBe("0.22.1"); }); + it("surfaces the active state backend, never the URL", () => { + const report = evaluateStatus(inputs({ stateBackend: "redis" })); + expect(report.service.stateBackend).toBe("redis"); + expect(JSON.stringify(report)).not.toMatch(/redis:\/\//); + }); + it("flags a function failing at least 20% of 5+ calls with a provider-specific fix", () => { const report = evaluateStatus( inputs({ @@ -227,6 +234,10 @@ describe("status wiring", () => { expect(api.match(/key: "GRAPH_EXTRACTION_ENABLED"/g)?.length).toBe(1); }); + it("reports the active state backend from config, not a hardcoded value", () => { + expect(api).toMatch(/stateBackend: getStateBackend\(\)/); + }); + it("the viewer has a Health tab that reads the status report", () => { expect(viewer).toContain(''); expect(viewer).toContain('
'); From 40049e02da5b3e81305a9385c08a485bdad6662e Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Mon, 28 Sep 2026 11:13:09 +0530 Subject: [PATCH 4/7] fix(redis): reject unloadable URLs and close the file-store fallback The engine expands ${AGENTMEMORY_REDIS_URL} into the YAML text before it parses the config, so a single quote or a control character in the URL breaks the single-quoted scalar and the engine does not start. The render now rejects those characters with a message to percent-encode them. A failure to create the engine working directory returned the original file-backed config before the Redis render ran, so a Redis selection could silently start on the file store. That path now uses the same fatal handling as a failed Redis render. Docker Compose forwards AGENTMEMORY_REDIS_URL to the engine container, so a hand-edited iii-config.docker.yaml can reference it instead of holding a literal URL. The README and deploy notes say plain redis:// is unencrypted and must stay on a trusted network or behind an encrypted tunnel. --- README.md | 4 ++-- deploy/README.md | 11 ++++++++--- docker-compose.yml | 1 + src/cli.ts | 12 +++++++++--- src/cli/engine-config.ts | 9 +++++++++ test/engine-config.test.ts | 31 +++++++++++++++++++++++++++++++ 6 files changed, 60 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index baacfdc8e8..ce10128142 100644 --- a/README.md +++ b/README.md @@ -1302,13 +1302,13 @@ AGENTMEMORY_REDIS_URL=redis://localhost:6379 `AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged, and an unrecognized value (anything other than `file` or `redis`) is a startup error rather than a silent fallback. `/agentmemory/status` reports which backend is active (never the URL). -**Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings — Upstash, Redis Cloud, ElastiCache with in-transit encryption — default to TLS-only) fails to connect. Point at a local Redis, or a plain-TCP one reached through an SSH/stunnel tunnel. +**Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings — Upstash, Redis Cloud, ElastiCache with in-transit encryption — default to TLS-only) fails to connect. The connection is unencrypted, so the Redis password and every stored memory cross the wire in clear text: point at a local Redis or one on a private network you trust. For any other Redis, run an encrypted tunnel (stunnel, SSH, or a VPN) on the agentmemory host, so the plain `redis://` hop stays on that host and the tunnel's upstream connection is encrypted and authenticated. If a Redis password contains a single quote, percent-encode it (`%27`); the engine expands the URL into its YAML config before parsing it. **One Redis DB per `--instance`.** The engine's Redis key prefixes (`state:`, `stream::`) are fixed, so two agentmemory instances (`--instance 1`, `--instance 2`, ...) pointed at the same database collide and cross-feed each other's viewer stream. Give each instance its own database index, e.g. `redis://localhost:6379/1`. **Recommended Redis settings.** The default `save 3600 1 300 100 60 10000` snapshot policy can lose minutes of writes on a crash, worse than the file store's 5s flush window — set `appendonly yes` for anything you'd mind losing. Set `maxmemory-policy noeviction`; `allkeys-lru` or similar silently drops memories once Redis hits its memory limit. -A native (non-Docker) start, and every one-click [deploy template](deploy/) (they overwrite the bundled `iii-config.yaml` and start natively), read `AGENTMEMORY_STATE_BACKEND`/`AGENTMEMORY_REDIS_URL` and render them into the launched `iii-config` — the URL itself is never written to that rendered file, only a `${AGENTMEMORY_REDIS_URL}` reference that the engine process expands from its own environment at boot. Only this repo's own Docker Compose path (`AGENTMEMORY_USE_DOCKER=1`, or resuming an engine already started that way) mounts `iii-config.docker.yaml` read-only and never renders — `agentmemory start` warns when it detects that combination. Switch that file by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. +A native (non-Docker) start, and every one-click [deploy template](deploy/) (they overwrite the bundled `iii-config.yaml` and start natively), read `AGENTMEMORY_STATE_BACKEND`/`AGENTMEMORY_REDIS_URL` and render them into the launched `iii-config` — the URL itself is never written to that rendered file, only a `${AGENTMEMORY_REDIS_URL}` reference that the engine process expands from its own environment at boot. Only this repo's own Docker Compose path (`AGENTMEMORY_USE_DOCKER=1`, or resuming an engine already started that way) mounts `iii-config.docker.yaml` read-only and never renders — `agentmemory start` warns when it detects that combination. Switch that file by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. `docker-compose.yml` passes `AGENTMEMORY_REDIS_URL` into the engine container, so `redis_url: '${AGENTMEMORY_REDIS_URL}'` works there and keeps the URL out of the mounted file. The rendered config keeps the URL out of `~/.agentmemory/data/iii-config.runtime.yaml`, but the engine's own configuration worker still persists the *expanded* value to `~/.agentmemory/config/iii-state.yaml` and `iii-stream.yaml` once it boots (iii-engine's `${VAR}` expansion happens before that worker stores its seed, and it stores the resolved value, not the reference). Treat that directory as holding a credential: `chmod 700 ~/.agentmemory` on any shared host, and prefer a Redis ACL user scoped to what agentmemory needs over the database's admin credentials. diff --git a/deploy/README.md b/deploy/README.md index 597bd2b75a..dc96076aaf 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -89,9 +89,14 @@ documents work unchanged: set `AGENTMEMORY_STATE_BACKEND=redis` and `AGENTMEMORY_REDIS_URL=redis://host:6379` alongside your platform's other secrets to point `iii-state`/`iii-stream` at an external Redis instead of the `/data` volume. The pinned engine only supports plain -`redis://` (no TLS), and a Redis reachable from these platforms is -almost always run by you elsewhere — a managed TLS-only Redis will not -work here without a plain-TCP tunnel in front of it. +`redis://` (no TLS), so the connection, including the Redis password +and every stored memory, travels unencrypted. Only use it over a +private network you trust (the platform's private networking, or a +Redis in the same machine or VPC). To reach any other Redis, including +a managed TLS-only one, run an encrypted tunnel next to agentmemory +(stunnel, an SSH tunnel, or a VPN such as WireGuard or Tailscale) so +the plain `redis://` hop never leaves the host, and the tunnel's own +upstream connection is encrypted and authenticated. ## Cold-start budget diff --git a/docker-compose.yml b/docker-compose.yml index 8b195e4c67..0c06226a5b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -38,6 +38,7 @@ services: III_STREAM_PORT: "${III_STREAM_PORT:-3112}" III_VIEWER_PORT: "${III_VIEWER_PORT:-3113}" III_ENGINE_PORT: "${III_ENGINE_PORT:-49134}" + AGENTMEMORY_REDIS_URL: "${AGENTMEMORY_REDIS_URL:-}" ports: - "127.0.0.1:${III_ENGINE_PORT:-49134}:${III_ENGINE_PORT:-49134}" - "127.0.0.1:${III_REST_PORT:-3111}:${III_REST_PORT:-3111}" diff --git a/src/cli.ts b/src/cli.ts index 9f9d668f49..2b9fbad606 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1452,12 +1452,17 @@ function prepareEngineLaunch(configPath: string): { home, bundledConfig, ); + const stateBackendKind = getStateBackend(); try { mkdirSync(cwd, { recursive: true }); - } catch { + } catch (err) { + const failure = resolveLaunchRenderFailure(stateBackendKind, err); + if (failure.fatal) { + p.log.error(failure.message); + process.exit(1); + } return { configPath, cwd: process.cwd() }; } - const stateBackendKind = getStateBackend(); try { const rawConfig = readFileSync(configPath, "utf-8"); const options = { @@ -1564,7 +1569,8 @@ function warnIfDockerIgnoresStateBackend(stateBackendKind: "file" | "redis"): vo if (stateBackendKind !== "redis") return; p.log.warn( "AGENTMEMORY_STATE_BACKEND=redis is not applied on the Docker path: docker-compose.yml mounts iii-config.docker.yaml read-only, so this start will not render it. " + - "Edit iii-config.docker.yaml by hand to point iii-state/iii-stream at redis (see README \"Storage backend\"), or it keeps using the file store.", + "Edit iii-config.docker.yaml by hand to point iii-state/iii-stream at redis (see README \"Storage backend\"), or it keeps using the file store. " + + "docker-compose.yml passes AGENTMEMORY_REDIS_URL into the engine container, so redis_url: '${AGENTMEMORY_REDIS_URL}' works in that file.", ); } diff --git a/src/cli/engine-config.ts b/src/cli/engine-config.ts index 40d55ab052..9462bb8c3e 100644 --- a/src/cli/engine-config.ts +++ b/src/cli/engine-config.ts @@ -161,6 +161,7 @@ function setWorkerPort(lines: string[], name: string, port: number): void { } const REDIS_URL_ENV_REF = "${AGENTMEMORY_REDIS_URL}"; +const UNSAFE_REDIS_URL_CHARS = /['\u0000-\u001f\u007f]/; function replaceKvAdapterWithRedis( lines: string[], @@ -235,6 +236,14 @@ export function renderEngineConfig( "AGENTMEMORY_STATE_BACKEND=redis requires AGENTMEMORY_REDIS_URL to be set (e.g. redis://localhost:6379).", ); } + if ( + options.stateBackend?.kind === "redis" && + UNSAFE_REDIS_URL_CHARS.test(options.stateBackend.redisUrl ?? "") + ) { + throw new Error( + "AGENTMEMORY_REDIS_URL contains a single quote or a control character, which breaks the engine config once the engine expands it. Percent-encode those characters in the URL (for example a single quote as %27).", + ); + } const rendered = template .replace( diff --git a/test/engine-config.test.ts b/test/engine-config.test.ts index 2a0e86be02..7affe48607 100644 --- a/test/engine-config.test.ts +++ b/test/engine-config.test.ts @@ -128,6 +128,37 @@ describe("renderEngineConfig", () => { ).toThrow(/AGENTMEMORY_REDIS_URL/); }); + it.each(["redis://:pa'ss@localhost:6379", "redis://localhost:6379/0\n", "redis://:a\tb@localhost:6379"])( + "rejects a redis URL the engine could not load after expansion: %j", + (redisUrl) => { + const source = readFileSync( + join(import.meta.dirname, "..", "iii-config.yaml"), + "utf8", + ); + + expect(() => + renderEngineConfig(source, { + dataDir: "/tmp/agentmemory-fixture", + stateBackend: { kind: "redis", redisUrl }, + }), + ).toThrow(/percent-encode/i); + }, + ); + + it("accepts a percent-encoded single quote in the redis password", () => { + const source = readFileSync( + join(import.meta.dirname, "..", "iii-config.yaml"), + "utf8", + ); + + expect(() => + renderEngineConfig(source, { + dataDir: "/tmp/agentmemory-fixture", + stateBackend: { kind: "redis", redisUrl: "redis://:pa%27ss@localhost:6379" }, + }), + ).not.toThrow(); + }); + it("throws when redis is selected but the iii-state worker block is missing", () => { const source = ["workers:", " - name: iii-http", " config:", " port: 3111"].join( "\n", From 1c86a4c7e34f8fbf36fdcb88a51d107ba429a7de Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Mon, 28 Sep 2026 11:13:52 +0530 Subject: [PATCH 5/7] docs(redis): drop em-dashes from the state backend section --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index ce10128142..95b3e43877 100644 --- a/README.md +++ b/README.md @@ -1302,17 +1302,17 @@ AGENTMEMORY_REDIS_URL=redis://localhost:6379 `AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged, and an unrecognized value (anything other than `file` or `redis`) is a startup error rather than a silent fallback. `/agentmemory/status` reports which backend is active (never the URL). -**Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings — Upstash, Redis Cloud, ElastiCache with in-transit encryption — default to TLS-only) fails to connect. The connection is unencrypted, so the Redis password and every stored memory cross the wire in clear text: point at a local Redis or one on a private network you trust. For any other Redis, run an encrypted tunnel (stunnel, SSH, or a VPN) on the agentmemory host, so the plain `redis://` hop stays on that host and the tunnel's upstream connection is encrypted and authenticated. If a Redis password contains a single quote, percent-encode it (`%27`); the engine expands the URL into its YAML config before parsing it. +**Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings, such as Upstash, Redis Cloud, and ElastiCache with in-transit encryption, default to TLS-only) fails to connect. The connection is unencrypted, so the Redis password and every stored memory cross the wire in clear text: point at a local Redis or one on a private network you trust. For any other Redis, run an encrypted tunnel (stunnel, SSH, or a VPN) on the agentmemory host, so the plain `redis://` hop stays on that host and the tunnel's upstream connection is encrypted and authenticated. If a Redis password contains a single quote, percent-encode it (`%27`); the engine expands the URL into its YAML config before parsing it. **One Redis DB per `--instance`.** The engine's Redis key prefixes (`state:`, `stream::`) are fixed, so two agentmemory instances (`--instance 1`, `--instance 2`, ...) pointed at the same database collide and cross-feed each other's viewer stream. Give each instance its own database index, e.g. `redis://localhost:6379/1`. -**Recommended Redis settings.** The default `save 3600 1 300 100 60 10000` snapshot policy can lose minutes of writes on a crash, worse than the file store's 5s flush window — set `appendonly yes` for anything you'd mind losing. Set `maxmemory-policy noeviction`; `allkeys-lru` or similar silently drops memories once Redis hits its memory limit. +**Recommended Redis settings.** The default `save 3600 1 300 100 60 10000` snapshot policy can lose minutes of writes on a crash, worse than the file store's 5s flush window. Set `appendonly yes` for anything you'd mind losing. Set `maxmemory-policy noeviction`; `allkeys-lru` or similar silently drops memories once Redis hits its memory limit. -A native (non-Docker) start, and every one-click [deploy template](deploy/) (they overwrite the bundled `iii-config.yaml` and start natively), read `AGENTMEMORY_STATE_BACKEND`/`AGENTMEMORY_REDIS_URL` and render them into the launched `iii-config` — the URL itself is never written to that rendered file, only a `${AGENTMEMORY_REDIS_URL}` reference that the engine process expands from its own environment at boot. Only this repo's own Docker Compose path (`AGENTMEMORY_USE_DOCKER=1`, or resuming an engine already started that way) mounts `iii-config.docker.yaml` read-only and never renders — `agentmemory start` warns when it detects that combination. Switch that file by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. `docker-compose.yml` passes `AGENTMEMORY_REDIS_URL` into the engine container, so `redis_url: '${AGENTMEMORY_REDIS_URL}'` works there and keeps the URL out of the mounted file. +A native (non-Docker) start, and every one-click [deploy template](deploy/) (they overwrite the bundled `iii-config.yaml` and start natively), read `AGENTMEMORY_STATE_BACKEND`/`AGENTMEMORY_REDIS_URL` and render them into the launched `iii-config`. The URL itself is never written to that rendered file, only a `${AGENTMEMORY_REDIS_URL}` reference that the engine process expands from its own environment at boot. Only this repo's own Docker Compose path (`AGENTMEMORY_USE_DOCKER=1`, or resuming an engine already started that way) mounts `iii-config.docker.yaml` read-only and never renders; `agentmemory start` warns when it detects that combination. Switch that file by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. `docker-compose.yml` passes `AGENTMEMORY_REDIS_URL` into the engine container, so `redis_url: '${AGENTMEMORY_REDIS_URL}'` works there and keeps the URL out of the mounted file. The rendered config keeps the URL out of `~/.agentmemory/data/iii-config.runtime.yaml`, but the engine's own configuration worker still persists the *expanded* value to `~/.agentmemory/config/iii-state.yaml` and `iii-stream.yaml` once it boots (iii-engine's `${VAR}` expansion happens before that worker stores its seed, and it stores the resolved value, not the reference). Treat that directory as holding a credential: `chmod 700 ~/.agentmemory` on any shared host, and prefer a Redis ACL user scoped to what agentmemory needs over the database's admin credentials. -**Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side — nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to. This runs identically under bash and zsh (including `bash -u`) — an array like `AUTH=(${AGENTMEMORY_SECRET:+-H "Authorization: Bearer $AGENTMEMORY_SECRET"})` does not: zsh keeps the header as one malformed word where bash splits it into two, so both requests 401 whenever `AGENTMEMORY_SECRET` is set: +**Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side; nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to. This runs identically under bash and zsh (including `bash -u`). An array like `AUTH=(${AGENTMEMORY_SECRET:+-H "Authorization: Bearer $AGENTMEMORY_SECRET"})` does not: zsh keeps the header as one malformed word where bash splits it into two, so both requests 401 whenever `AGENTMEMORY_SECRET` is set: ```bash # 1. On the old backend, while agentmemory is still running on it: From 6d5ff6de9f7b21e017a51aea10404a8becc63a5e Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Mon, 28 Sep 2026 21:05:34 +0530 Subject: [PATCH 6/7] fix(redis): make every state and stream path behave the same on redis The engine's redis adapter differs from the file store in ways that broke or degraded agentmemory beyond basic writes: - state::list is HGETALL collected into a HashMap, so records came back in arbitrary order (export paging, viewer stream seeding, anything that slices a list). StateKV now sorts redis listings oldest first by the creation time in generated ids, then timestamps, then id. - state::update runs a Lua script whose cjson round trip turns empty arrays into empty objects (session tags, commitShas). On redis StateKV applies set/remove/merge itself under a per-key lock and writes with state::set, which fires the same created/updated triggers. - The legacy audit migration probed the file store's scope file on disk, so a fresh redis store was marked unreadable and the audit log showed as frozen. On redis it sizes the legacy scope from the hash itself and skips the file flush wait. - After a redis restart the engine's pub/sub pump ends and live viewer events stop until restart. The health monitor now sends a probe event through the stream every 30 seconds on redis and raises stream_relay_down with a restart fix. - /status probes the state store and reports state-store-unreachable with a backend specific fix; the Health page shows the backend in the State store row and the relay state. --- src/functions/audit.ts | 22 +- src/functions/status.ts | 30 +- src/health/monitor.ts | 12 +- src/health/stream-relay-probe.ts | 140 ++++++++++ src/health/thresholds.ts | 5 + src/index.ts | 17 +- src/state/kv.ts | 134 ++++++++- src/state/viewer-stream.ts | 29 +- src/triggers/api.ts | 17 +- src/types.ts | 1 + src/viewer/index.html | 9 +- test/redis-state-backend.test.ts | 457 +++++++++++++++++++++++++++++++ test/status.test.ts | 4 +- 13 files changed, 849 insertions(+), 28 deletions(-) create mode 100644 src/health/stream-relay-probe.ts create mode 100644 test/redis-state-backend.test.ts diff --git a/src/functions/audit.ts b/src/functions/audit.ts index c629ff0303..f0f9b52425 100644 --- a/src/functions/audit.ts +++ b/src/functions/audit.ts @@ -99,6 +99,21 @@ export async function probeLegacyAuditFile(): Promise { return { safe: true, empty: false, sizeBytes: fileStat.size }; } +export async function probeLegacyAuditScope(kv: StateKV): Promise { + let rows: AuditEntry[]; + try { + rows = await kv.list(KV.audit); + } catch { + return { safe: false, reason: "unreadable" }; + } + if (!Array.isArray(rows) || rows.length === 0) return { safe: true, empty: true }; + const sizeBytes = Buffer.byteLength(JSON.stringify(rows), "utf8"); + if (sizeBytes > getAuditMigrateMaxBytes()) { + return { safe: false, reason: "too-large", sizeBytes }; + } + return { safe: true, empty: false, sizeBytes }; +} + async function auditMigrationDeleteDelayMs(): Promise { const dataDir = getEnvVar("AGENTMEMORY_DATA_DIR"); if (!dataDir) return DEFAULT_SAVE_INTERVAL_MS * AUDIT_MIGRATION_DELETE_WAIT_INTERVALS; @@ -334,8 +349,11 @@ export async function startAuditMigration( kv: StateKV, deps: AuditMigrationDeps = {}, ): Promise { - const runProbe = deps.probe ?? probeLegacyAuditFile; - const runDeleteDelayMs = deps.deleteDelayMs ?? auditMigrationDeleteDelayMs; + const onRedis = kv.backend === "redis"; + const runProbe = + deps.probe ?? (onRedis ? () => probeLegacyAuditScope(kv) : probeLegacyAuditFile); + const runDeleteDelayMs = + deps.deleteDelayMs ?? (onRedis ? async () => 0 : auditMigrationDeleteDelayMs); const marker = await readAuditMigrationState(kv); if (marker?.status === "done") return; diff --git a/src/functions/status.ts b/src/functions/status.ts index 20d8f4f342..fbc24f24fe 100644 --- a/src/functions/status.ts +++ b/src/functions/status.ts @@ -70,6 +70,7 @@ export interface StatusInputs { graphExtractionEnabled: boolean; auditLegacy: { status: string; sizeBytes?: number } | null; indexPersistence?: IndexPersistenceStatus | null; + stateStore?: { ok: boolean; latencyMs?: number } | null; } export interface IndexBreakdown { @@ -87,6 +88,7 @@ export interface StatusReport { engineVersion: string; uptimeSeconds: number; stateBackend: StatusInputs["stateBackend"]; + stateStore: { ok: boolean; latencyMs?: number } | null; ports: StatusInputs["ports"]; }; health: StatusInputs["health"]; @@ -155,6 +157,13 @@ export function describeHealthAlert(slug: string): { level: "warn" | "error"; me fix: "It usually clears after an import or index rebuild finishes. If it persists, restart agentmemory.", }; } + if (slug === "stream_relay_down") { + return { + level: "warn", + message: "Live updates are not reaching the viewer: the engine stopped relaying stream events, which happens after Redis restarts or drops the connection. Data is still saved.", + fix: "Restart agentmemory (npx @agentmemory/agentmemory stop, then start) so the engine subscribes to Redis again, then reload the viewer.", + }; + } if (slug === "connection_reconnecting") { return { level: "warn", @@ -363,7 +372,25 @@ export function evaluateStatus(input: StatusInputs): StatusReport { level: "info", code: "audit-legacy-frozen", message: `An older audit log${sizeText} was left in place instead of being migrated into monthly scopes, so it is no longer rewritten but its rows do not show up in audit queries.`, - fix: "Safe to ignore. To remove it, stop agentmemory, delete mem%3Aaudit.bin from the state store directory, then start it again.", + fix: + input.stateBackend === "redis" + ? "Safe to ignore. To remove it, run redis-cli -u \"$AGENTMEMORY_REDIS_URL\" DEL state:mem:audit, then restart agentmemory." + : "Safe to ignore. To remove it, stop agentmemory, delete mem%3Aaudit.bin from the state store directory, then start it again.", + }); + } + + if (input.stateStore && !input.stateStore.ok) { + problems.push({ + level: "error", + code: "state-store-unreachable", + message: + input.stateBackend === "redis" + ? "The state store is not answering: the engine cannot reach Redis, so nothing is being saved or read." + : "The state store is not answering: the engine did not complete a state read in time, so nothing is being saved or read.", + fix: + input.stateBackend === "redis" + ? "Check that Redis is running and reachable at AGENTMEMORY_REDIS_URL (redis-cli -u \"$AGENTMEMORY_REDIS_URL\" ping should answer PONG). Once Redis is back, restart agentmemory: the engine does not resume the live viewer stream after a Redis restart." + : "Check the server log for engine errors, then restart agentmemory.", }); } @@ -381,6 +408,7 @@ export function evaluateStatus(input: StatusInputs): StatusReport { engineVersion: input.engineVersion, uptimeSeconds: input.uptimeSeconds, stateBackend: input.stateBackend, + stateStore: input.stateStore ?? null, ports: input.ports, }, health: input.health, diff --git a/src/health/monitor.ts b/src/health/monitor.ts index 35a9de9e25..71b7733a35 100644 --- a/src/health/monitor.ts +++ b/src/health/monitor.ts @@ -4,10 +4,12 @@ import type { HealthSnapshot } from "../types.js"; import type { StateKV } from "../state/kv.js"; import { KV } from "../state/schema.js"; import { evaluateHealth } from "./thresholds.js"; +import type { StreamRelayProbe } from "./stream-relay-probe.js"; export function registerHealthMonitor( sdk: IIIClient, kv: StateKV, + options: { streamRelayProbe?: StreamRelayProbe } = {}, ): { stop: () => void } { let connectionState = "connected"; let prevCpuUsage = process.cpuUsage(); @@ -64,8 +66,13 @@ export function registerHealthMonitor( kvConnectivity = { status: "error", error: "kv_probe_failed", latencyMs: Math.round((performance.now() - kvStart) * 100) / 100 }; } + const streamRelay = options.streamRelayProbe + ? await options.streamRelayProbe.check().catch(() => "unknown" as const) + : undefined; + const snapshot: HealthSnapshot = { connectionState, + ...(streamRelay ? { streamRelay } : {}), workers, memory: { heapUsed: mem.heapUsed, @@ -102,7 +109,10 @@ export function registerHealthMonitor( interval.unref(); return { - stop: () => clearInterval(interval), + stop: () => { + clearInterval(interval); + options.streamRelayProbe?.close(); + }, }; } diff --git a/src/health/stream-relay-probe.ts b/src/health/stream-relay-probe.ts new file mode 100644 index 0000000000..7350744c18 --- /dev/null +++ b/src/health/stream-relay-probe.ts @@ -0,0 +1,140 @@ +import type { IIIClient } from "iii-sdk"; +import { STREAM } from "../state/schema.js"; + +export type StreamRelayState = "ok" | "down" | "unknown"; + +export interface StreamRelayProbe { + check(): Promise; + close(): void; +} + +interface ProbeSocket { + readyState: number; + onopen: (() => void) | null; + onmessage: ((event: { data: unknown }) => void) | null; + onclose: (() => void) | null; + onerror: (() => void) | null; + send(data: string): void; + close(): void; +} + +export interface StreamRelayProbeOptions { + url: string; + timeoutMs?: number; + createSocket?: (url: string) => ProbeSocket; +} + +const PROBE_GROUP = "relay-probe"; +const PROBE_EVENT = "relay.probe"; +const OPEN = 1; + +function defaultSocket(url: string): ProbeSocket { + return new WebSocket(url) as unknown as ProbeSocket; +} + +export function createStreamRelayProbe( + sdk: IIIClient, + options: StreamRelayProbeOptions, +): StreamRelayProbe { + const timeoutMs = options.timeoutMs ?? 5000; + const createSocket = options.createSocket ?? defaultSocket; + let socket: ProbeSocket | null = null; + let ready: Promise | null = null; + const waiters = new Map void>(); + + function connect(): Promise { + if (socket && socket.readyState === OPEN && ready) return ready; + try { + socket?.close(); + } catch {} + let current: ProbeSocket; + try { + current = createSocket(options.url); + } catch { + socket = null; + ready = null; + return Promise.resolve(false); + } + socket = current; + ready = new Promise((resolve) => { + const timer = setTimeout(() => resolve(false), timeoutMs); + current.onopen = () => { + clearTimeout(timer); + current.send( + JSON.stringify({ + type: "join", + data: { + subscriptionId: `relay-probe-${process.pid}-${Date.now()}`, + streamName: STREAM.name, + groupId: PROBE_GROUP, + }, + }), + ); + resolve(true); + }; + current.onerror = () => { + clearTimeout(timer); + resolve(false); + }; + current.onclose = () => { + clearTimeout(timer); + if (socket === current) { + socket = null; + ready = null; + } + resolve(false); + }; + }); + current.onmessage = (event) => { + try { + const raw = typeof event.data === "string" ? event.data : String(event.data); + const nonce = JSON.parse(raw)?.event?.event?.data?.nonce; + if (typeof nonce === "string") waiters.get(nonce)?.(); + } catch {} + }; + return ready; + } + + async function check(): Promise { + if (!(await connect())) return "unknown"; + const nonce = `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`; + let timer: ReturnType | undefined; + const received = new Promise((resolve) => { + waiters.set(nonce, () => resolve(true)); + timer = setTimeout(() => resolve(false), timeoutMs); + }); + try { + await sdk.trigger({ + function_id: "stream::send", + payload: { + stream_name: STREAM.name, + group_id: PROBE_GROUP, + id: nonce, + type: PROBE_EVENT, + data: { nonce }, + }, + timeoutMs, + }); + } catch { + waiters.delete(nonce); + clearTimeout(timer); + return "down"; + } + const ok = await received; + waiters.delete(nonce); + clearTimeout(timer); + return ok ? "ok" : "down"; + } + + return { + check, + close() { + waiters.clear(); + try { + socket?.close(); + } catch {} + socket = null; + ready = null; + }, + }; +} diff --git a/src/health/thresholds.ts b/src/health/thresholds.ts index 9abf35eccf..70f85a577c 100644 --- a/src/health/thresholds.ts +++ b/src/health/thresholds.ts @@ -41,6 +41,11 @@ export function evaluateHealth( degraded = true; } + if (snapshot.streamRelay === "down") { + alerts.push("stream_relay_down"); + degraded = true; + } + if (snapshot.eventLoopLagMs > cfg.eventLoopLagCriticalMs) { alerts.push( `event_loop_lag_critical_${Math.round(snapshot.eventLoopLagMs)}ms`, diff --git a/src/index.ts b/src/index.ts index 1111adf12b..e85f16217f 100644 --- a/src/index.ts +++ b/src/index.ts @@ -15,6 +15,7 @@ import { isContextInjectionEnabled, isDropStaleIndexEnabled, getAuditRetentionMonths, + getStateBackend, } from "./config.js"; import { createProvider, @@ -108,6 +109,7 @@ import { startViewerServer } from "./viewer/server.js"; import { MetricsStore } from "./eval/metrics-store.js"; import { DedupMap } from "./functions/dedup.js"; import { registerHealthMonitor } from "./health/monitor.js"; +import { createStreamRelayProbe } from "./health/stream-relay-probe.js"; import { initMetrics, OTEL_CONFIG } from "./telemetry/setup.js"; import { VERSION } from "./version.js"; import { bootLog } from "./logger.js"; @@ -226,7 +228,11 @@ async function main() { writeWorkerPidfile(); - const kv = new StateKV(sdk); + let stateBackend: "file" | "redis" = "file"; + try { + stateBackend = getStateBackend(); + } catch {} + const kv = new StateKV(sdk, { backend: stateBackend }); const secret = getEnvVar("AGENTMEMORY_SECRET"); const metricsStore = new MetricsStore(kv); const dedupMap = new DedupMap(); @@ -413,7 +419,12 @@ async function main() { registerViewerStreamTriggers(sdk, kv, { secret, metricsStore, provider }); registerMcpEndpoints(sdk, kv, secret); - const healthMonitor = registerHealthMonitor(sdk, kv); + const healthMonitor = registerHealthMonitor(sdk, kv, { + streamRelayProbe: + kv.backend === "redis" + ? createStreamRelayProbe(sdk, { url: `ws://localhost:${config.streamsPort}` }) + : undefined, + }); const indexPersistence = new IndexPersistence(kv, vectorIndex); setIndexPersistence(indexPersistence); @@ -601,7 +612,7 @@ async function main() { }, 60 * 60 * 1000); recentSearchesSweepTimer.unref(); - void seedViewerStreamTracker(sdk).catch(() => {}); + void seedViewerStreamTracker(sdk, { unorderedListing: kv.backend === "redis" }).catch(() => {}); if (isConsolidationEnabled()) { const consolidationTimer = setInterval(async () => { diff --git a/src/state/kv.ts b/src/state/kv.ts index a8a1707ac9..a1b1ddc115 100644 --- a/src/state/kv.ts +++ b/src/state/kv.ts @@ -1,7 +1,115 @@ import type { IIIClient } from 'iii-sdk' +import { withKeyedLock } from './keyed-mutex.js' + +export type StateBackend = 'file' | 'redis' + +export interface StateKVOptions { + backend?: StateBackend +} + +type UpdateOp = { type: string; path: string; value?: unknown } + +type UpdateResult = { old_value: unknown; new_value: unknown; errors: unknown[] } + +const LOCAL_UPDATE_OPS = new Set(['set', 'remove', 'merge']) +const UNSAFE_PATHS = new Set(['__proto__', 'constructor', 'prototype']) +const ORDER_FIELDS = ['createdAt', 'timestamp', 'startedAt', 'updatedAt'] as const + +function isPlainObject(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value) +} + +function canApplyLocally(ops: UpdateOp[]): boolean { + return ops.every( + (op) => + LOCAL_UPDATE_OPS.has(op.type) && + typeof (op.path ?? '') === 'string' && + !UNSAFE_PATHS.has(op.path ?? '') && + (op.type !== 'merge' || isPlainObject(op.value)), + ) +} + +export function applyUpdateOps(oldValue: unknown, ops: UpdateOp[]): unknown { + let current: unknown = + oldValue === null || oldValue === undefined ? {} : structuredClone(oldValue) + for (const op of ops) { + const path = op.path ?? '' + if (op.type === 'set') { + if (path === '') current = op.value ?? null + else if (isPlainObject(current)) current[path] = op.value ?? null + } else if (op.type === 'remove') { + if (path === '') current = null + else if (isPlainObject(current)) delete current[path] + } else if (op.type === 'merge' && isPlainObject(op.value)) { + if (path === '') { + if (isPlainObject(current)) Object.assign(current, op.value) + } else { + const root: Record = isPlainObject(current) ? current : {} + const existing = root[path] + const target = isPlainObject(existing) ? existing : {} + Object.assign(target, op.value) + root[path] = target + current = root + } + } + } + return current +} + +const MIN_ID_TIME = Date.UTC(2020, 0, 1) +const MAX_ID_TIME = Date.UTC(2100, 0, 1) + +export function generatedIdTime(id: unknown): number | null { + if (typeof id !== 'string') return null + const parts = id.split('_') + if (parts.length < 3) return null + const segment = parts[parts.length - 2]! + if (!/^[0-9a-z]{6,10}$/.test(segment)) return null + const ms = parseInt(segment, 36) + return ms >= MIN_ID_TIME && ms < MAX_ID_TIME ? ms : null +} + +function orderKey(value: unknown): number { + if (!isPlainObject(value)) return Number.POSITIVE_INFINITY + const fromId = generatedIdTime(value.id) + if (fromId !== null) return fromId + for (const field of ORDER_FIELDS) { + const raw = value[field] + if (typeof raw === 'number' && Number.isFinite(raw)) return raw + if (typeof raw === 'string') { + const parsed = Date.parse(raw) + if (!Number.isNaN(parsed)) return parsed + } + } + return Number.POSITIVE_INFINITY +} + +function idKey(value: unknown): string { + if (!isPlainObject(value)) return '' + const id = value.id ?? value.key + return typeof id === 'string' ? id : typeof id === 'number' ? String(id) : '' +} + +export function orderLikeInsertion(values: T[]): T[] { + return values + .map((value) => ({ value, at: orderKey(value), id: idKey(value) })) + .sort((a, b) => { + if (a.at !== b.at) return a.at < b.at ? -1 : 1 + if (a.id === b.id) return 0 + return a.id < b.id ? -1 : 1 + }) + .map((entry) => entry.value) +} export class StateKV { - constructor(private sdk: IIIClient) {} + readonly backend: StateBackend + + constructor( + private sdk: IIIClient, + options: StateKVOptions = {}, + ) { + this.backend = options.backend ?? 'file' + } async get(scope: string, key: string): Promise { return this.sdk.trigger<{ scope: string; key: string }, T | null>({ @@ -17,15 +125,17 @@ export class StateKV { }) } - async update( - scope: string, - key: string, - ops: Array<{ type: string; path: string; value?: unknown }>, - ): Promise { - return this.sdk.trigger< - { scope: string; key: string; ops: Array<{ type: string; path: string; value?: unknown }> }, - T - >({ + async update(scope: string, key: string, ops: UpdateOp[]): Promise { + if (this.backend === 'redis' && canApplyLocally(ops)) { + return withKeyedLock(`state-update:${scope}\u0000${key}`, async () => { + const oldValue = await this.get(scope, key) + const newValue = applyUpdateOps(oldValue, ops) + await this.set(scope, key, newValue) + const result: UpdateResult = { old_value: oldValue, new_value: newValue, errors: [] } + return result as T + }) + } + return this.sdk.trigger<{ scope: string; key: string; ops: UpdateOp[] }, T>({ function_id: 'state::update', payload: { scope, key, ops }, }) @@ -39,9 +149,11 @@ export class StateKV { } async list(scope: string): Promise { - return this.sdk.trigger<{ scope: string }, T[]>({ + const values = await this.sdk.trigger<{ scope: string }, T[]>({ function_id: 'state::list', payload: { scope }, }) + if (this.backend !== 'redis' || !Array.isArray(values)) return values + return orderLikeInsertion(values) } } diff --git a/src/state/viewer-stream.ts b/src/state/viewer-stream.ts index d89d65a9d6..612e6f3526 100644 --- a/src/state/viewer-stream.ts +++ b/src/state/viewer-stream.ts @@ -1,5 +1,6 @@ import type { IIIClient } from "iii-sdk"; import { STREAM } from "./schema.js"; +import { generatedIdTime } from "./kv.js"; import { getViewerStreamMax } from "../config.js"; import { logger } from "../logger.js"; @@ -92,10 +93,32 @@ export async function pruneViewerStreamIfDue(sdk: IIIClient): Promise { } type StoredViewerItem = { - observation?: { id?: unknown }; + observation?: { id?: unknown; timestamp?: unknown }; }; -export async function seedViewerStreamTracker(sdk: IIIClient): Promise { +function itemTime(item: StoredViewerItem): number { + const fromId = generatedIdTime(item?.observation?.id); + if (fromId !== null) return fromId; + const raw = item?.observation?.timestamp; + const parsed = typeof raw === "string" ? Date.parse(raw) : typeof raw === "number" ? raw : NaN; + return Number.isFinite(parsed) ? parsed : Number.NEGATIVE_INFINITY; +} + +function oldestFirst(items: StoredViewerItem[]): StoredViewerItem[] { + return items + .map((item, index) => ({ item, index, at: itemTime(item) })) + .sort((a, b) => (a.at !== b.at ? (a.at < b.at ? -1 : 1) : a.index - b.index)) + .map((entry) => entry.item); +} + +export interface SeedViewerStreamOptions { + unorderedListing?: boolean; +} + +export async function seedViewerStreamTracker( + sdk: IIIClient, + options: SeedViewerStreamOptions = {}, +): Promise { let items: StoredViewerItem[]; try { items = await sdk.trigger({ @@ -111,7 +134,7 @@ export async function seedViewerStreamTracker(sdk: IIIClient): Promise { } if (!Array.isArray(items)) return 0; const stored: string[] = []; - for (const item of items) { + for (const item of options.unorderedListing ? oldestFirst(items) : items) { const id = item?.observation?.id; if (typeof id !== "string" || trackedItemIdSet.has(id)) continue; trackedItemIdSet.add(id); diff --git a/src/triggers/api.ts b/src/triggers/api.ts index dcb5ea944f..8ae72d43c5 100644 --- a/src/triggers/api.ts +++ b/src/triggers/api.ts @@ -52,7 +52,6 @@ import { detectEmbeddingProvider, detectLlmProviderKind, getAgentId, - getStateBackend, isAgentScopeIsolated, loadConfig, } from "../config.js"; @@ -292,6 +291,16 @@ async function valueWithin(work: Promise, ms: number): Promise { } } +async function probeStateStore(kv: StateKV): Promise<{ ok: boolean; latencyMs?: number }> { + const started = performance.now(); + const answered = await valueWithin( + kv.get(KV.health, "_probe").then(() => true), + STATUS_CHECK_TIMEOUT_MS, + ); + if (!answered) return { ok: false }; + return { ok: true, latencyMs: Math.round((performance.now() - started) * 100) / 100 }; +} + type UnindexedScan = Awaited>; const unindexedScans = new WeakMap Promise; last: { at: number; value: UnindexedScan } | null }>(); @@ -324,7 +333,7 @@ export function createStatusReporter(sdk: IIIClient, kv: StateKV, deps: StatusRe const cached = scan.last && options.scanMaxAgeMs !== undefined && Date.now() - scan.last.at < options.scanMaxAgeMs ? scan.last.value : null; - const [health, functionMetrics, graph, unindexed, auditMigrationState] = await Promise.all([ + const [health, functionMetrics, graph, unindexed, auditMigrationState, stateStore] = await Promise.all([ options.health !== undefined ? Promise.resolve(options.health) : valueWithin(getLatestHealth(kv), STATUS_CHECK_TIMEOUT_MS), deps.metricsStore ? valueWithin(deps.metricsStore.getAll(), STATUS_CHECK_TIMEOUT_MS) : Promise.resolve([]), valueWithin( @@ -336,6 +345,7 @@ export function createStatusReporter(sdk: IIIClient, kv: StateKV, deps: StatusRe kv.get(KV.auditMonths, AUDIT_MIGRATION_STATE_KEY), STATUS_CHECK_TIMEOUT_MS, ), + probeStateStore(kv), ]); const observationsIndexed = [...idx.observationCountsBySession().values()].reduce((a, n) => a + n, 0); const documentKinds = idx.documentKindCounts(); @@ -348,7 +358,7 @@ export function createStatusReporter(sdk: IIIClient, kv: StateKV, deps: StatusRe version: VERSION, engineVersion: III_PINNED_VERSION, uptimeSeconds: Math.round(process.uptime()), - stateBackend: getStateBackend(), + stateBackend: kv.backend === "redis" ? "redis" : "file", ports: { rest: loadConfig().restPort ?? null, streams: loadConfig().streamsPort ?? null, @@ -387,6 +397,7 @@ export function createStatusReporter(sdk: IIIClient, kv: StateKV, deps: StatusRe ? { status: auditMigrationState.status, sizeBytes: auditMigrationState.legacySizeBytes } : null, indexPersistence: getIndexPersistenceStatus(), + stateStore, }); }; } diff --git a/src/types.ts b/src/types.ts index 909675acab..38af3d2d8a 100644 --- a/src/types.ts +++ b/src/types.ts @@ -239,6 +239,7 @@ export interface HealthSnapshot { eventLoopLagMs: number; uptimeSeconds: number; kvConnectivity?: { status: string; latencyMs?: number; error?: string }; + streamRelay?: "ok" | "down" | "unknown"; status: "healthy" | "degraded" | "critical"; alerts: string[]; notes?: string[]; diff --git a/src/viewer/index.html b/src/viewer/index.html index 39a97b6f26..7a2e071932 100644 --- a/src/viewer/index.html +++ b/src/viewer/index.html @@ -1660,6 +1660,7 @@

agentmemory

eventLoop: { term: 'Event loop delay', text: 'How far behind the worker is in handling work. Under 20 ms is healthy; a large value means it is busy or blocked.' }, uptime: { term: 'Uptime', text: 'How long the agentmemory worker has been running since it last started.' }, health: { term: 'Health status', text: 'Healthy means everything is normal. Degraded means something needs attention but memory still works. Critical means a core part is failing.' }, + stateBackend: { term: 'State store', text: 'Where the engine keeps every record: file (local files on this machine, the default) or redis (a Redis server set by AGENTMEMORY_STATE_BACKEND and AGENTMEMORY_REDIS_URL). ok means a test write and read just succeeded; the number is how long it took.' }, engineConnection: { term: 'Engine connection', text: 'Whether the agentmemory worker is connected to the iii engine that stores its data and runs its functions.' }, live: { term: 'Live updates', text: 'The viewer keeps one streaming connection open and updates every page the moment data changes. Nothing is refreshed on a timer.' }, flags: { term: 'Feature flags', text: 'Optional features that are on or off in this install. Off features leave some pages empty; each one lists the setting that turns it on.' }, @@ -7084,8 +7085,8 @@

agentmemory

} if (store.lastHealthAt && Date.now() - store.lastHealthAt > HEALTH_STALE_MS) { var secs = Math.round((Date.now() - store.lastHealthAt) / 1000); - return '
Stale: no health report for ' + secs + 's. Reports normally arrive every 30 seconds. The stream is connected, so the worker is busy or the iii engine is paused. The numbers below are from ' + esc(last) + '.' + - '
Fix: wait a minute; if it stays stale, check the server log and restart agentmemory.
'; + return '
Stale: no health report for ' + secs + 's. Reports normally arrive every 30 seconds. The stream is connected, so the worker is busy or the iii engine is paused' + (r && r.service && r.service.stateBackend === 'redis' ? ', or Redis restarted: the engine stops relaying live events after a Redis restart until agentmemory restarts' : '') + '. The numbers below are from ' + esc(last) + '.' + + '
Fix: wait a minute; if it stays stale, press Re-check now for the full report, check the server log and restart agentmemory.
'; } return ''; } @@ -7128,11 +7129,13 @@

agentmemory

function engineCardHtml(r, snap) { var ports = (r && r.service && r.service.ports) || {}; var kv = snap.kvConnectivity; + var backend = r && r.service ? r.service.stateBackend : null; var streamText = conn.state === 'live' ? 'live' : conn.state; return '
Engine and connection ' + help('engineConnection') + '
' + hrow('Engine connection', esc(snap.connectionState || 'unknown'), 'engineConnection', 'The worker\'s link to the iii engine, which stores the data and runs the functions.') + - (kv ? hrow('State store', esc(kv.status + (kv.latencyMs !== undefined ? ' · ' + kv.latencyMs + ' ms' : '') + (kv.error ? ' · ' + kv.error : ''))) : '') + + ((kv || backend) ? hrow('State store', esc((backend ? backend + (kv ? ' · ' : '') : '') + (kv ? kv.status + (kv.latencyMs !== undefined ? ' · ' + kv.latencyMs + ' ms' : '') + (kv.error ? ' · ' + kv.error : '') : '')), 'stateBackend', backend === 'redis' ? 'Stored in Redis (AGENTMEMORY_STATE_BACKEND=redis). If Redis restarts, restart agentmemory so the live stream resumes.' : 'Stored in the engine\'s local files (the default). Set AGENTMEMORY_STATE_BACKEND=redis to use Redis.') : '') + hrow('Viewer stream', esc(streamText), 'live', 'This page\'s own connection. Offline means the numbers here stop changing.') + + (snap.streamRelay ? hrow('Redis event relay', esc(snap.streamRelay), 'live', snap.streamRelay === 'down' ? 'The engine stopped relaying live events through Redis, usually after Redis restarted. Data is still saved. Restart agentmemory to resume live updates.' : 'A test event sent through Redis every 30 seconds came back to the worker.') : '') + hrow('Uptime', esc(formatUptime(snap.uptimeSeconds)), 'uptime') + hrow('Version', esc('v' + (store.version || (r && r.service.version) || '?') + (r ? ' · engine ' + r.service.engineVersion : ''))) + hrow('Ports', esc('REST ' + (ports.rest || '?') + ' · streams ' + (ports.streams || '?') + ' · viewer ' + (ports.viewer || 'off'))) + diff --git a/test/redis-state-backend.test.ts b/test/redis-state-backend.test.ts new file mode 100644 index 0000000000..1926409070 --- /dev/null +++ b/test/redis-state-backend.test.ts @@ -0,0 +1,457 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; + +vi.mock("../src/logger.js", () => ({ + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }, +})); + +import { StateKV, applyUpdateOps, generatedIdTime, orderLikeInsertion } from "../src/state/kv.js"; +import { + AUDIT_MIGRATION_STATE_KEY, + probeLegacyAuditScope, + queryAudit, + startAuditMigration, +} from "../src/functions/audit.js"; +import { KV, generateId } from "../src/state/schema.js"; +import { + resetViewerStreamTracker, + seedViewerStreamTracker, +} from "../src/state/viewer-stream.js"; +import { evaluateStatus, type StatusInputs } from "../src/functions/status.js"; + +type Call = { function_id: string; payload: Record }; +type StateEvent = { type: "created" | "updated" | "deleted"; scope: string; key: string; old: unknown; next: unknown }; + +function cjsonRoundTrip(value: unknown): unknown { + return JSON.parse( + JSON.stringify(value, (_key, v) => (Array.isArray(v) && v.length === 0 ? {} : v)), + ); +} + +function shuffled(values: T[]): T[] { + const out = [...values]; + for (let i = out.length - 1; i > 0; i--) { + const j = (i * 7919 + 13) % (i + 1); + [out[i], out[j]] = [out[j]!, out[i]!]; + } + return out; +} + +function fakeRedisEngine() { + const hashes = new Map>(); + const calls: Call[] = []; + const events: StateEvent[] = []; + const hash = (scope: string) => { + let h = hashes.get(`state:${scope}`); + if (!h) { + h = new Map(); + hashes.set(`state:${scope}`, h); + } + return h; + }; + const read = (scope: string, key: string) => { + const raw = hashes.get(`state:${scope}`)?.get(key); + return raw === undefined ? null : JSON.parse(raw); + }; + const sdk = { + trigger: vi.fn(async (req: Call) => { + calls.push(req); + const p = req.payload as { scope: string; key: string; value?: unknown; ops?: never[] }; + switch (req.function_id) { + case "state::get": + return read(p.scope, p.key); + case "state::set": { + const old = read(p.scope, p.key); + hash(p.scope).set(p.key, JSON.stringify(p.value)); + events.push({ type: old === null ? "created" : "updated", scope: p.scope, key: p.key, old, next: p.value }); + return { old_value: old, new_value: p.value }; + } + case "state::update": { + const old = read(p.scope, p.key); + const next = cjsonRoundTrip(applyUpdateOps(old, p.ops ?? [])); + hash(p.scope).set(p.key, JSON.stringify(next)); + events.push({ type: old === null ? "created" : "updated", scope: p.scope, key: p.key, old, next }); + return { old_value: old, new_value: next, errors: [] }; + } + case "state::delete": { + const old = read(p.scope, p.key); + hashes.get(`state:${p.scope}`)?.delete(p.key); + events.push({ type: "deleted", scope: p.scope, key: p.key, old, next: null }); + return old; + } + case "state::list": { + const h = hashes.get(`state:${p.scope}`); + return h ? shuffled([...h.values()].map((raw) => JSON.parse(raw))) : []; + } + default: + return null; + } + }), + }; + return { sdk, hashes, calls, events, read }; +} + +describe("StateKV on the redis backend", () => { + it("keeps empty arrays through an update instead of letting the engine's Lua path turn them into objects", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + await kv.set(KV.sessions, "ses_1", { id: "ses_1", tags: [], commitShas: [], observationCount: 0 }); + + await kv.update(KV.sessions, "ses_1", [ + { type: "set", path: "status", value: "completed" }, + { type: "set", path: "observationCount", value: 3 }, + ]); + + expect(engine.read(KV.sessions, "ses_1")).toEqual({ + id: "ses_1", + tags: [], + commitShas: [], + observationCount: 3, + status: "completed", + }); + expect(engine.calls.some((c) => c.function_id === "state::update")).toBe(false); + const last = engine.events.at(-1)!; + expect(last.type).toBe("updated"); + expect((last.old as { tags: unknown }).tags).toEqual([]); + expect((last.next as { status: string }).status).toBe("completed"); + }); + + it("shows the corruption the local path avoids when the engine applies the update", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "file" }); + await kv.set(KV.sessions, "ses_1", { id: "ses_1", tags: [] }); + await kv.update(KV.sessions, "ses_1", [{ type: "set", path: "status", value: "completed" }]); + expect(engine.read(KV.sessions, "ses_1")).toEqual({ id: "ses_1", tags: {}, status: "completed" }); + }); + + it("creates a missing key like the engine does and reports it as created", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + const result = await kv.update<{ old_value: unknown; new_value: unknown }>(KV.sessions, "ghost", [ + { type: "set", path: "status", value: "completed" }, + ]); + expect(result).toEqual({ old_value: null, new_value: { status: "completed" }, errors: [] }); + expect(engine.events.at(-1)!.type).toBe("created"); + }); + + it("serializes concurrent updates to one key so neither write is lost", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + await kv.set("scope", "k", { id: "k" }); + await Promise.all([ + kv.update("scope", "k", [{ type: "set", path: "a", value: 1 }]), + kv.update("scope", "k", [{ type: "set", path: "b", value: 2 }]), + kv.update("scope", "k", [{ type: "merge", path: "", value: { c: 3 } }]), + ]); + expect(engine.read("scope", "k")).toEqual({ id: "k", a: 1, b: 2, c: 3 }); + }); + + it("hands ops it does not mirror to the engine", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + await kv.update("scope", "k", [{ type: "increment", path: "n", value: 1 }]); + await kv.update("scope", "k", [{ type: "set", path: "__proto__", value: 1 }]); + expect(engine.calls.filter((c) => c.function_id === "state::update")).toHaveLength(2); + }); + + it("mirrors the engine's set, remove and merge semantics", () => { + expect(applyUpdateOps(null, [{ type: "set", path: "a", value: 1 }])).toEqual({ a: 1 }); + expect(applyUpdateOps({ a: 1, b: 2 }, [{ type: "remove", path: "a" }])).toEqual({ b: 2 }); + expect(applyUpdateOps({ a: 1 }, [{ type: "set", path: "", value: [] }])).toEqual([]); + expect(applyUpdateOps({ m: { x: 1 } }, [{ type: "merge", path: "m", value: { y: 2 } }])).toEqual({ m: { x: 1, y: 2 } }); + expect(applyUpdateOps({ m: 5 }, [{ type: "merge", path: "m", value: { y: 2 } }])).toEqual({ m: { y: 2 } }); + expect(applyUpdateOps("text", [{ type: "set", path: "a", value: 1 }])).toBe("text"); + const original = { list: [1] }; + applyUpdateOps(original, [{ type: "set", path: "list", value: [] }]); + expect(original).toEqual({ list: [1] }); + }); + + it("returns list results oldest first on redis, the order the file store keeps", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + const base = Date.UTC(2026, 8, 1); + const ids = Array.from({ length: 30 }, (_, i) => `mem_${(base + i * 1000).toString(36)}_${"a".repeat(12)}`); + for (const id of shuffled(ids)) await kv.set(KV.memories, id, { id, createdAt: "2020-01-01T00:00:00.000Z" }); + const listed = await kv.list<{ id: string }>(KV.memories); + expect(listed.map((m) => m.id)).toEqual(ids); + const again = await kv.list<{ id: string }>(KV.memories); + expect(again.map((m) => m.id)).toEqual(ids); + }); + + it("orders records without generated ids by their timestamp fields, then id", () => { + const rows = [ + { id: "b", startedAt: "2026-09-02T00:00:00.000Z" }, + { id: "c", timestamp: 1 }, + { id: "a", startedAt: "2026-09-02T00:00:00.000Z" }, + { id: "z" }, + { id: "d", createdAt: "2026-09-01T00:00:00.000Z" }, + ]; + expect(orderLikeInsertion(rows).map((r) => r.id)).toEqual(["c", "d", "a", "b", "z"]); + }); + + it("reads the creation time generateId embeds", () => { + const before = Date.now(); + const at = generatedIdTime(generateId("obs")); + expect(at).not.toBeNull(); + expect(at!).toBeGreaterThanOrEqual(before - 1); + expect(generatedIdTime("mem_0123456789abcdef")).toBeNull(); + expect(generatedIdTime("session-uuid")).toBeNull(); + }); + + it("leaves list order alone on the file backend", async () => { + const values = [{ id: "z" }, { id: "a" }]; + const sdk = { trigger: vi.fn(async () => values) }; + const kv = new StateKV(sdk as never); + expect(kv.backend).toBe("file"); + expect(await kv.list("scope")).toBe(values); + }); +}); + +describe("audit migration on the redis backend", () => { + const savedDataDir = process.env.AGENTMEMORY_DATA_DIR; + + beforeEach(() => { + process.env.AGENTMEMORY_DATA_DIR = "/nonexistent/agentmemory-redis-test"; + }); + + afterEach(() => { + if (savedDataDir === undefined) delete process.env.AGENTMEMORY_DATA_DIR; + else process.env.AGENTMEMORY_DATA_DIR = savedDataDir; + }); + + it("marks a fresh redis store done instead of freezing it for a missing file", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + await startAuditMigration(kv); + const state = await kv.get<{ status: string; safeToListLegacy: boolean }>(KV.auditMonths, AUDIT_MIGRATION_STATE_KEY); + expect(state?.status).toBe("done"); + expect(state?.safeToListLegacy).toBe(true); + const result = await queryAudit(kv); + expect(result.legacyFrozen).toBe(false); + }); + + it("the file probe would have frozen the same store, which is the bug this avoids", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "file" }); + await startAuditMigration(kv, { deleteDelayMs: async () => 0 }); + const state = await kv.get<{ status: string }>(KV.auditMonths, AUDIT_MIGRATION_STATE_KEY); + expect(state?.status).toBe("unreadable"); + }); + + it("moves a legacy redis audit hash into month scopes without waiting for a file flush", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + const rows = [ + { id: "aud_1", timestamp: "2026-08-10T00:00:00.000Z", operation: "forget", functionId: "mem::forget", targetIds: ["mem_1"], details: {} }, + { id: "aud_2", timestamp: "2026-09-10T00:00:00.000Z", operation: "delete", functionId: "mem::governance-delete", targetIds: ["mem_2"], details: {} }, + { id: "aud_3", timestamp: "2026-09-11T00:00:00.000Z", operation: "index_persist", functionId: "x", targetIds: [], details: {} }, + ]; + for (const row of rows) await kv.set(KV.audit, row.id, row); + const started = Date.now(); + await startAuditMigration(kv); + expect(Date.now() - started).toBeLessThan(2000); + expect(engine.hashes.get(`state:${KV.audit}`)?.size ?? 0).toBe(0); + expect(await kv.get(KV.auditMonth("2026-08"), "aud_1")).not.toBeNull(); + expect(await kv.get(KV.auditMonth("2026-09"), "aud_2")).not.toBeNull(); + expect(await kv.get(KV.auditMonth("2026-09"), "aud_3")).toBeNull(); + const state = await kv.get<{ status: string; migrated: number; purged: number }>(KV.auditMonths, AUDIT_MIGRATION_STATE_KEY); + expect(state).toMatchObject({ status: "done", migrated: 2, purged: 1 }); + }); + + it("sizes the legacy redis scope from its rows and freezes it past the cap", async () => { + const engine = fakeRedisEngine(); + const kv = new StateKV(engine.sdk as never, { backend: "redis" }); + const saved = process.env.AGENTMEMORY_AUDIT_MIGRATE_MAX_BYTES; + process.env.AGENTMEMORY_AUDIT_MIGRATE_MAX_BYTES = "100"; + try { + await kv.set(KV.audit, "aud_big", { id: "aud_big", timestamp: "2026-09-01T00:00:00.000Z", details: { blob: "x".repeat(500) } }); + const probe = await probeLegacyAuditScope(kv); + expect(probe).toMatchObject({ safe: false, reason: "too-large" }); + } finally { + if (saved === undefined) delete process.env.AGENTMEMORY_AUDIT_MIGRATE_MAX_BYTES; + else process.env.AGENTMEMORY_AUDIT_MIGRATE_MAX_BYTES = saved; + } + }); + + it("reports an unreachable redis as unreadable rather than empty", async () => { + const kv = new StateKV({ trigger: vi.fn(async () => { throw new Error("Failed to get group from Redis"); }) } as never, { backend: "redis" }); + expect(await probeLegacyAuditScope(kv)).toEqual({ safe: false, reason: "unreadable" }); + }); +}); + +describe("viewer stream seeding on the redis backend", () => { + beforeEach(() => resetViewerStreamTracker()); + afterEach(() => { + delete process.env.AGENTMEMORY_VIEWER_STREAM_MAX; + resetViewerStreamTracker(); + }); + + it("prunes the oldest items even though redis lists them in arbitrary order", async () => { + process.env.AGENTMEMORY_VIEWER_STREAM_MAX = "200"; + const base = Date.UTC(2026, 8, 1); + const ids = Array.from({ length: 203 }, (_, i) => `obs_${(base + i * 1000).toString(36)}_${"b".repeat(12)}`); + const items = shuffled(ids).map((id) => ({ observation: { id, timestamp: "not-a-date" } })); + const deleted: string[] = []; + const sdk = { + trigger: vi.fn(async (req: { function_id: string; payload: { item_id?: string } }) => { + if (req.function_id === "stream::list") return items; + if (req.payload.item_id) deleted.push(req.payload.item_id); + return {}; + }), + }; + await expect(seedViewerStreamTracker(sdk as never, { unorderedListing: true })).resolves.toBe(203); + expect(deleted.sort()).toEqual(ids.slice(0, 3).sort()); + }); +}); + +describe("status on the redis backend", () => { + function inputs(overrides: Partial): StatusInputs { + return { + now: new Date("2026-09-28T00:00:00.000Z"), + version: "0.0.0", + engineVersion: "0.22.1", + uptimeSeconds: 10, + stateBackend: "redis", + ports: { rest: 3111, streams: 3112, viewer: 3113 }, + health: { status: "healthy", alerts: [] }, + circuitBreaker: null, + functionMetrics: [], + provider: "anthropic", + embeddingProvider: "none", + flags: [], + index: { + bm25Documents: 0, + vectorDocuments: null, + observationsIndexed: 0, + missingObservations: 0, + sessions: 0, + bm25Incomplete: false, + pendingVectorBackfill: 0, + }, + graph: null, + graphExtractionEnabled: false, + auditLegacy: null, + ...overrides, + }; + } + + it("flags an unreachable redis as an error with a redis fix and never shows the URL", () => { + const report = evaluateStatus(inputs({ stateStore: { ok: false } })); + const problem = report.problems.find((p) => p.code === "state-store-unreachable"); + expect(problem?.level).toBe("error"); + expect(problem?.fix).toContain("AGENTMEMORY_REDIS_URL"); + expect(problem?.fix).toContain("restart agentmemory"); + expect(JSON.stringify(report)).not.toContain("redis://"); + expect(report.service.stateBackend).toBe("redis"); + expect(report.service.stateStore).toEqual({ ok: false }); + }); + + it("gives a file-store fix on the default backend", () => { + const report = evaluateStatus(inputs({ stateBackend: "file", stateStore: { ok: false } })); + const problem = report.problems.find((p) => p.code === "state-store-unreachable"); + expect(problem?.fix).not.toContain("Redis"); + }); + + it("tells redis users how to drop a frozen legacy audit hash", () => { + const report = evaluateStatus(inputs({ auditLegacy: { status: "too-large", sizeBytes: 1 } })); + const problem = report.problems.find((p) => p.code === "audit-legacy-frozen"); + expect(problem?.fix).toContain("DEL state:mem:audit"); + }); +}); + +describe("redis event relay probe", () => { + function fakeSocketFactory(behaviour: { open: boolean; relay: boolean }) { + const sockets: Array<{ joined: unknown[]; deliver: (data: unknown) => void }> = []; + const create = () => { + const socket = { + readyState: 0, + onopen: null as null | (() => void), + onmessage: null as null | ((e: { data: unknown }) => void), + onclose: null as null | (() => void), + onerror: null as null | (() => void), + joined: [] as unknown[], + send(data: string) { + socket.joined.push(JSON.parse(data)); + }, + close() { + socket.readyState = 3; + }, + deliver(data: unknown) { + socket.onmessage?.({ data: JSON.stringify(data) }); + }, + }; + sockets.push(socket); + setTimeout(() => { + if (behaviour.open) { + socket.readyState = 1; + socket.onopen?.(); + } else { + socket.onerror?.(); + } + }, 1); + return socket; + }; + const sdk = { + trigger: vi.fn(async (req: { function_id: string; payload: { data: { nonce: string } } }) => { + if (behaviour.relay) { + setTimeout(() => { + for (const s of sockets) s.deliver({ type: "stream", event: { type: "event", event: { type: "relay.probe", data: req.payload.data } } }); + }, 1); + } + return null; + }), + }; + return { create, sdk, sockets }; + } + + it("reports ok when the probe event comes back over the stream", async () => { + const { createStreamRelayProbe } = await import("../src/health/stream-relay-probe.js"); + const f = fakeSocketFactory({ open: true, relay: true }); + const probe = createStreamRelayProbe(f.sdk as never, { url: "ws://x", timeoutMs: 200, createSocket: f.create as never }); + expect(await probe.check()).toBe("ok"); + expect(await probe.check()).toBe("ok"); + expect(f.sockets).toHaveLength(1); + expect(f.sockets[0]!.joined[0]).toMatchObject({ type: "join", data: { streamName: "mem-live", groupId: "relay-probe" } }); + probe.close(); + }); + + it("reports down when the engine accepts the event but never relays it", async () => { + const { createStreamRelayProbe } = await import("../src/health/stream-relay-probe.js"); + const f = fakeSocketFactory({ open: true, relay: false }); + const probe = createStreamRelayProbe(f.sdk as never, { url: "ws://x", timeoutMs: 100, createSocket: f.create as never }); + expect(await probe.check()).toBe("down"); + probe.close(); + }); + + it("reports down when the send itself fails and unknown when the stream port is closed", async () => { + const { createStreamRelayProbe } = await import("../src/health/stream-relay-probe.js"); + const f = fakeSocketFactory({ open: true, relay: false }); + f.sdk.trigger.mockRejectedValueOnce(new Error("Failed to publish event to Redis")); + const probe = createStreamRelayProbe(f.sdk as never, { url: "ws://x", timeoutMs: 100, createSocket: f.create as never }); + expect(await probe.check()).toBe("down"); + const closed = fakeSocketFactory({ open: false, relay: true }); + const probe2 = createStreamRelayProbe(closed.sdk as never, { url: "ws://x", timeoutMs: 100, createSocket: closed.create as never }); + expect(await probe2.check()).toBe("unknown"); + probe.close(); + probe2.close(); + }); + + it("turns a dead relay into a degraded health alert with a restart fix", async () => { + const { evaluateHealth } = await import("../src/health/thresholds.js"); + const { describeHealthAlert } = await import("../src/functions/status.js"); + const result = evaluateHealth({ + connectionState: "connected", + workers: [], + memory: { heapUsed: 1, heapTotal: 100, rss: 1, external: 0 }, + cpu: { userMicros: 0, systemMicros: 0, percent: 1 }, + eventLoopLagMs: 1, + uptimeSeconds: 1, + streamRelay: "down", + status: "healthy", + alerts: [], + } as never); + expect(result.status).toBe("degraded"); + expect(result.alerts).toContain("stream_relay_down"); + const described = describeHealthAlert("stream_relay_down"); + expect(described.fix).toContain("Restart agentmemory"); + }); +}); diff --git a/test/status.test.ts b/test/status.test.ts index c057f9c070..a5cad34e28 100644 --- a/test/status.test.ts +++ b/test/status.test.ts @@ -411,6 +411,7 @@ describe("prefersHtml", () => { describe("status wiring", () => { const api = readFileSync("src/triggers/api.ts", "utf-8"); const viewer = readFileSync("src/viewer/index.html", "utf-8"); + const index = readFileSync("src/index.ts", "utf-8"); it("registers GET /agentmemory/status behind the same auth check as the other endpoints", () => { expect(api).toMatch(/registerFunction\("api::status",\s*async \(req: HttpRequest\): Promise => \{\s*const authErr = checkAuth\(req, secret\);/); @@ -446,7 +447,8 @@ describe("status wiring", () => { }); it("reports the active state backend from config, not a hardcoded value", () => { - expect(api).toMatch(/stateBackend: getStateBackend\(\)/); + expect(api).toMatch(/stateBackend: kv.backend === "redis" \? "redis" : "file"/); + expect(index).toMatch(/new StateKV\(sdk, \{ backend: stateBackend \}\)/); }); it("the viewer has a Health tab that reads the status report", () => { From f87fec3e6415244569a030bde3df450f07968d90 Mon Sep 17 00:00:00 2001 From: Rohit Ghumare Date: Mon, 28 Sep 2026 21:05:35 +0530 Subject: [PATCH 7/7] docs(redis): document redis behaviour, one server per instance and the restart caveat Redis pub/sub ignores the database index, so separate DB numbers keep data apart but still cross-feed live viewer events between instances. List what works unchanged, what agentmemory handles for the engine's redis differences, and what to do after Redis restarts. --- README.md | 12 ++++++++++-- plugin/skills/agentmemory-config/REFERENCE.md | 7 +++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7ef8cead54..120ac49fa0 100644 --- a/README.md +++ b/README.md @@ -1300,11 +1300,19 @@ AGENTMEMORY_STATE_BACKEND=redis AGENTMEMORY_REDIS_URL=redis://localhost:6379 ``` -`AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged, and an unrecognized value (anything other than `file` or `redis`) is a startup error rather than a silent fallback. `/agentmemory/status` reports which backend is active (never the URL). +`AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged, and an unrecognized value (anything other than `file` or `redis`) is a startup error rather than a silent fallback. `/agentmemory/status` and the viewer's Health page (the State store row) report which backend is active and whether it answers, never the URL. **Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings, such as Upstash, Redis Cloud, and ElastiCache with in-transit encryption, default to TLS-only) fails to connect. The connection is unencrypted, so the Redis password and every stored memory cross the wire in clear text: point at a local Redis or one on a private network you trust. For any other Redis, run an encrypted tunnel (stunnel, SSH, or a VPN) on the agentmemory host, so the plain `redis://` hop stays on that host and the tunnel's upstream connection is encrypted and authenticated. If a Redis password contains a single quote, percent-encode it (`%27`); the engine expands the URL into its YAML config before parsing it. -**One Redis DB per `--instance`.** The engine's Redis key prefixes (`state:`, `stream::`) are fixed, so two agentmemory instances (`--instance 1`, `--instance 2`, ...) pointed at the same database collide and cross-feed each other's viewer stream. Give each instance its own database index, e.g. `redis://localhost:6379/1`. +**One Redis server per `--instance`.** The engine's Redis key prefixes (`state:`, `stream::`) are fixed, so two agentmemory instances (`--instance 1`, `--instance 2`, ...) pointed at the same database overwrite each other's data. A separate database index (`redis://localhost:6379/1`) keeps the stored data apart, but the engine relays live viewer events over one Redis pub/sub channel (`stream::events`), and Redis pub/sub ignores the database index, so each instance's viewer would still show the other's live events. Give each instance its own Redis server (or port) when you run more than one. + +**What stays the same, and what differs.** Every agentmemory feature works on Redis: sessions, observations, memories (remember, supersede, evolve, forget), search and the index buckets, lessons, the graph, the audit log and its monthly scopes, export and import, governance deletes, consolidation status, the viewer snapshot and its live stream, and the health monitor. The engine stores each scope as one Redis hash (`HSET`/`HGET`/`HGETALL`) and fires the same state triggers as the file store. Three engine differences are handled inside agentmemory: + +- Redis returns a scope's records in no fixed order. agentmemory sorts them oldest first (by the creation time in the record id, then its timestamp) so lists, paging and export chunks come back in the same order as on the file store. +- The engine applies partial updates on Redis in a Lua script that turns empty arrays into empty objects. agentmemory applies those updates itself (read, change, write under a per-key lock) on Redis, so fields like `tags: []` stay arrays. +- The legacy audit log check reads the old scope from Redis instead of looking for the file store's file on disk. + +One difference needs you: **after Redis restarts, the engine stops relaying live events** to the viewer until agentmemory restarts. Data is still saved and read normally. The health monitor sends a test event through Redis every 30 seconds; when it does not come back, `/agentmemory/status` and the viewer's Health page show "Live updates are not reaching the viewer" with the fix: restart agentmemory. If Redis is down, the status report shows "The state store is not answering" and how to check it (`redis-cli -u "$AGENTMEMORY_REDIS_URL" ping`). Listing a very large scope reads the whole hash in one `HGETALL`, the same cost as the file store holding it in memory. **Recommended Redis settings.** The default `save 3600 1 300 100 60 10000` snapshot policy can lose minutes of writes on a crash, worse than the file store's 5s flush window. Set `appendonly yes` for anything you'd mind losing. Set `maxmemory-policy noeviction`; `allkeys-lru` or similar silently drops memories once Redis hits its memory limit. diff --git a/plugin/skills/agentmemory-config/REFERENCE.md b/plugin/skills/agentmemory-config/REFERENCE.md index 1c1f8816cc..f7e3b60f22 100644 --- a/plugin/skills/agentmemory-config/REFERENCE.md +++ b/plugin/skills/agentmemory-config/REFERENCE.md @@ -58,3 +58,10 @@ Configuration is read from the environment and from `~/.agentmemory/.env` (no `e - `AGENTMEMORY_VIEWER_STREAM_MAX` - `AGENTMEMORY_VIEWER_URL` + +## State backend + +- `AGENTMEMORY_STATE_BACKEND`: `file` (default) or `redis`. Anything else stops startup with an error. +- `AGENTMEMORY_REDIS_URL`: required with `redis`, plain `redis://` only (the pinned engine has no TLS). The URL never appears in `/agentmemory/status` or the viewer. + +Switching backends starts from an empty store; move data with `/agentmemory/export` and `/agentmemory/import`. Run one Redis server per `--instance`: live viewer events travel over a Redis pub/sub channel that ignores the database index. After Redis restarts, restart agentmemory so live viewer updates resume; the Health page flags this. See the README section "Storage backend: file (default) vs redis".