Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,9 @@
# AGENTMEMORY_USE_DOCKER=1 # Prefer bundled Docker even when a compatible native iii binary exists
# AGENTMEMORY_EXPORT_ROOT=~/agentmemory-backup # Default destination for `agentmemory export`

# AGENTMEMORY_STATE_BACKEND=redis # file (default) | redis — storage adapter for iii-state and iii-stream, see README "Storage backend"
# AGENTMEMORY_REDIS_URL=redis://localhost:6379 # Required when AGENTMEMORY_STATE_BACKEND=redis

# STANDALONE_MCP=1 # MCP shim only — bypass the worker and run @agentmemory/mcp in-process
# STANDALONE_PERSIST_PATH=~/.agentmemory/local.db # Path used by the standalone MCP shim's local fallback store

Expand Down
63 changes: 63 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1288,6 +1288,69 @@ On engine 0.22.x keep the `iii-` prefixed names for the builtins above; the unpr

Full registry: [workers.iii.dev](https://workers.iii.dev). Every worker there composes through the same primitives agentmemory uses, and the agentmemory you already have is one of them.

### Storage backend: file (default) vs redis

`iii-state` and `iii-stream` default to iii-engine's bundled file-based KV store: one JSON file per scope, held in the engine process's memory and rewritten to disk on a timer. That's the right default for a single-user local install; a shared daemon with several concurrent writers gets real per-key writes from Redis instead, at the cost of a network round trip per operation (every `state::*` call still serializes on one Redis connection, so this trades the file store's lock for a socket, not for parallelism).

Set `AGENTMEMORY_STATE_BACKEND=redis` (plus `AGENTMEMORY_REDIS_URL`) to switch both workers to iii-engine's built-in `redis` adapter, which stores each key as a Redis hash field (`HSET`) instead of rewriting a whole scope on every write:

```env
# ~/.agentmemory/.env
AGENTMEMORY_STATE_BACKEND=redis
AGENTMEMORY_REDIS_URL=redis://localhost:6379
```

`AGENTMEMORY_STATE_BACKEND` defaults to `file`; leaving it unset keeps today's behavior unchanged, and an unrecognized value (anything other than `file` or `redis`) is a startup error rather than a silent fallback. `/agentmemory/status` and the viewer's Health page (the State store row) report which backend is active and whether it answers, never the URL.

**Plain `redis://` only.** The pinned engine (0.22.1) builds its Redis client without TLS support, so a `rediss://` URL (most managed Redis offerings, such as Upstash, Redis Cloud, and ElastiCache with in-transit encryption, default to TLS-only) fails to connect. The connection is unencrypted, so the Redis password and every stored memory cross the wire in clear text: point at a local Redis or one on a private network you trust. For any other Redis, run an encrypted tunnel (stunnel, SSH, or a VPN) on the agentmemory host, so the plain `redis://` hop stays on that host and the tunnel's upstream connection is encrypted and authenticated. If a Redis password contains a single quote, percent-encode it (`%27`); the engine expands the URL into its YAML config before parsing it.

**One Redis server per `--instance`.** The engine's Redis key prefixes (`state:<scope>`, `stream:<name>:<group>`) are fixed, so two agentmemory instances (`--instance 1`, `--instance 2`, ...) pointed at the same database overwrite each other's data. A separate database index (`redis://localhost:6379/1`) keeps the stored data apart, but the engine relays live viewer events over one Redis pub/sub channel (`stream::events`), and Redis pub/sub ignores the database index, so each instance's viewer would still show the other's live events. Give each instance its own Redis server (or port) when you run more than one.

**What stays the same, and what differs.** Every agentmemory feature works on Redis: sessions, observations, memories (remember, supersede, evolve, forget), search and the index buckets, lessons, the graph, the audit log and its monthly scopes, export and import, governance deletes, consolidation status, the viewer snapshot and its live stream, and the health monitor. The engine stores each scope as one Redis hash (`HSET`/`HGET`/`HGETALL`) and fires the same state triggers as the file store. Three engine differences are handled inside agentmemory:

- Redis returns a scope's records in no fixed order. agentmemory sorts them oldest first (by the creation time in the record id, then its timestamp) so lists, paging and export chunks come back in the same order as on the file store.
- The engine applies partial updates on Redis in a Lua script that turns empty arrays into empty objects. agentmemory applies those updates itself (read, change, write under a per-key lock) on Redis, so fields like `tags: []` stay arrays.
- The legacy audit log check reads the old scope from Redis instead of looking for the file store's file on disk.

One difference needs you: **after Redis restarts, the engine stops relaying live events** to the viewer until agentmemory restarts. Data is still saved and read normally. The health monitor sends a test event through Redis every 30 seconds; when it does not come back, `/agentmemory/status` and the viewer's Health page show "Live updates are not reaching the viewer" with the fix: restart agentmemory. If Redis is down, the status report shows "The state store is not answering" and how to check it (`redis-cli -u "$AGENTMEMORY_REDIS_URL" ping`). Listing a very large scope reads the whole hash in one `HGETALL`, the same cost as the file store holding it in memory.

**Recommended Redis settings.** The default `save 3600 1 300 100 60 10000` snapshot policy can lose minutes of writes on a crash, worse than the file store's 5s flush window. Set `appendonly yes` for anything you'd mind losing. Set `maxmemory-policy noeviction`; `allkeys-lru` or similar silently drops memories once Redis hits its memory limit.

A native (non-Docker) start, and every one-click [deploy template](deploy/) (they overwrite the bundled `iii-config.yaml` and start natively), read `AGENTMEMORY_STATE_BACKEND`/`AGENTMEMORY_REDIS_URL` and render them into the launched `iii-config`. The URL itself is never written to that rendered file, only a `${AGENTMEMORY_REDIS_URL}` reference that the engine process expands from its own environment at boot. Only this repo's own Docker Compose path (`AGENTMEMORY_USE_DOCKER=1`, or resuming an engine already started that way) mounts `iii-config.docker.yaml` read-only and never renders; `agentmemory start` warns when it detects that combination. Switch that file by hand, following the same `name: redis` / `config: redis_url: ...` shape shown in the [iii-state](https://workers.iii.dev/workers/iii-state) and [iii-stream](https://workers.iii.dev/workers/iii-stream) worker docs, and point `redis_url` at a Redis reachable from the container. `docker-compose.yml` passes `AGENTMEMORY_REDIS_URL` into the engine container, so `redis_url: '${AGENTMEMORY_REDIS_URL}'` works there and keeps the URL out of the mounted file.

The rendered config keeps the URL out of `~/.agentmemory/data/iii-config.runtime.yaml`, but the engine's own configuration worker still persists the *expanded* value to `~/.agentmemory/config/iii-state.yaml` and `iii-stream.yaml` once it boots (iii-engine's `${VAR}` expansion happens before that worker stores its seed, and it stores the resolved value, not the reference). Treat that directory as holding a credential: `chmod 700 ~/.agentmemory` on any shared host, and prefer a Redis ACL user scoped to what agentmemory needs over the database's admin credentials.

**Migration is not automatic.** Switching `AGENTMEMORY_STATE_BACKEND` starts from an empty store on either side; nothing copies existing data from file to Redis or back. Export from the backend you're leaving and import into the one you're moving to. This runs identically under bash and zsh (including `bash -u`). An array like `AUTH=(${AGENTMEMORY_SECRET:+-H "Authorization: Bearer $AGENTMEMORY_SECRET"})` does not: zsh keeps the header as one malformed word where bash splits it into two, so both requests 401 whenever `AGENTMEMORY_SECRET` is set:

```bash
# 1. On the old backend, while agentmemory is still running on it:
if [ -n "${AGENTMEMORY_SECRET:-}" ]; then
curl -fsS -H "Authorization: Bearer $AGENTMEMORY_SECRET" http://localhost:3111/agentmemory/export > backup.json
else
curl -fsS http://localhost:3111/agentmemory/export > backup.json
fi

# 2. Confirm backup.json is a usable export before switching backends:
jq -e '.version and .exportedAt' backup.json > /dev/null || {
echo "backup.json is not a valid export; do not switch backends" >&2
exit 1
}

# 3. Switch AGENTMEMORY_STATE_BACKEND (and AGENTMEMORY_REDIS_URL if needed),
# restart agentmemory against the new backend, then:
if [ -n "${AGENTMEMORY_SECRET:-}" ]; then
jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \
curl -fsS -H "Authorization: Bearer $AGENTMEMORY_SECRET" -X POST http://localhost:3111/agentmemory/import \
-H 'Content-Type: application/json' -d @-
else
jq -n --slurpfile d backup.json '{exportData: $d[0], strategy: "merge"}' | \
curl -fsS -X POST http://localhost:3111/agentmemory/import \
-H 'Content-Type: application/json' -d @-
fi
```

`/agentmemory/export` also accepts `?maxSessions=` and `?offset=` for chunking a large corpus across several calls; `strategy` on import is `merge` (default-safe), `replace`, or `skip`.

### What iii replaces

| Traditional stack | agentmemory uses |
Expand Down
18 changes: 18 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,24 @@ absent (no third-party calls), `AGENTMEMORY_AUTO_COMPRESS` is off,
and `AGENTMEMORY_INJECT_CONTEXT` is off. Opt in only after you've
confirmed your provider quota can absorb the workload.

## Optional: Redis state backend

Every entrypoint here overwrites the npm-bundled `iii-config.yaml` and
execs `agentmemory` directly (no Docker Compose in the loop), so the
same two variables the main README's "Storage backend" section
documents work unchanged: set `AGENTMEMORY_STATE_BACKEND=redis` and
`AGENTMEMORY_REDIS_URL=redis://host:6379` alongside your platform's
other secrets to point `iii-state`/`iii-stream` at an external Redis
instead of the `/data` volume. The pinned engine only supports plain
`redis://` (no TLS), so the connection, including the Redis password
and every stored memory, travels unencrypted. Only use it over a
private network you trust (the platform's private networking, or a
Redis in the same machine or VPC). To reach any other Redis, including
a managed TLS-only one, run an encrypted tunnel next to agentmemory
(stunnel, an SSH tunnel, or a VPN such as WireGuard or Tailscale) so
the plain `redis://` hop never leaves the host, and the tunnel's own
upstream connection is encrypted and authenticated.

## Cold-start budget

Measured against fly.io's `iad` region with a 1 GB volume:
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ services:
III_STREAM_PORT: "${III_STREAM_PORT:-3112}"
III_VIEWER_PORT: "${III_VIEWER_PORT:-3113}"
III_ENGINE_PORT: "${III_ENGINE_PORT:-49134}"
AGENTMEMORY_REDIS_URL: "${AGENTMEMORY_REDIS_URL:-}"
ports:
- "127.0.0.1:${III_ENGINE_PORT:-49134}:${III_ENGINE_PORT:-49134}"
- "127.0.0.1:${III_REST_PORT:-3111}:${III_REST_PORT:-3111}"
Expand Down
11 changes: 10 additions & 1 deletion plugin/skills/agentmemory-config/REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Generated by scanning `src/` for `AGENTMEMORY_*` usage. Do not edit the block below by hand; run `npm run skills:gen` after adding or removing a variable. Internal markers ending in two underscores are excluded.

<!-- AUTOGEN:env START - generated by scripts/skills/generate.ts, do not edit by hand -->
Configuration is read from the environment and from `~/.agentmemory/.env` (no `export` prefix). 50 recognized variables:
Configuration is read from the environment and from `~/.agentmemory/.env` (no `export` prefix). 52 recognized variables:

- `AGENTMEMORY_AGENT_SCOPE`
- `AGENTMEMORY_ALLOW_AGENT_SDK`
Expand Down Expand Up @@ -38,12 +38,14 @@ Configuration is read from the environment and from `~/.agentmemory/.env` (no `e
- `AGENTMEMORY_PROBE_TIMEOUT_MS`
- `AGENTMEMORY_PROJECT_NAME`
- `AGENTMEMORY_PROVIDER`
- `AGENTMEMORY_REDIS_URL`
- `AGENTMEMORY_REFLECT`
- `AGENTMEMORY_RUNTIME_DIR`
- `AGENTMEMORY_SDK_CHILD`
- `AGENTMEMORY_SECRET`
- `AGENTMEMORY_SESSION_ID`
- `AGENTMEMORY_SLOTS`
- `AGENTMEMORY_STATE_BACKEND`
- `AGENTMEMORY_SUPPRESS_COST_WARNING`
- `AGENTMEMORY_TOOLS`
- `AGENTMEMORY_URL`
Expand All @@ -56,3 +58,10 @@ Configuration is read from the environment and from `~/.agentmemory/.env` (no `e
- `AGENTMEMORY_VIEWER_STREAM_MAX`
- `AGENTMEMORY_VIEWER_URL`
<!-- AUTOGEN:env END -->

## State backend

- `AGENTMEMORY_STATE_BACKEND`: `file` (default) or `redis`. Anything else stops startup with an error.
- `AGENTMEMORY_REDIS_URL`: required with `redis`, plain `redis://` only (the pinned engine has no TLS). The URL never appears in `/agentmemory/status` or the viewer.

Switching backends starts from an empty store; move data with `/agentmemory/export` and `/agentmemory/import`. Run one Redis server per `--instance`: live viewer events travel over a Redis pub/sub channel that ignores the database index. After Redis restarts, restart agentmemory so live viewer updates resume; the Health page flags this. See the README section "Storage backend: file (default) vs redis".
49 changes: 44 additions & 5 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ import {
rewriteBundledConfig,
runtimeConfigPath,
engineChildEnv,
resolveLaunchRenderFailure,
redactCredentialUrls,
} from "./cli/engine-launch.js";
import { runtimeMetadataPath } from "./runtime-paths.js";
import { createStartupStderrCapture } from "./cli/startup-stderr.js";
Expand All @@ -78,7 +80,7 @@ import { renderSplash } from "./cli/splash.js";
import { isFirstRun, readPrefs, resetPrefs, writePrefs } from "./cli/preferences.js";
import { runOnboarding } from "./cli/onboarding.js";
import { setBootVerbose } from "./logger.js";
import { hydrateProcessEnvFromFile } from "./config.js";
import { getRedisUrl, getStateBackend, hydrateProcessEnvFromFile } from "./config.js";
import { III_PINNED_VERSION, VERSION } from "./version.js";
import { getAllTools, ESSENTIAL_TOOLS } from "./mcp/tools-registry.js";
import { knownAgents } from "./cli/connect/index.js";
Expand Down Expand Up @@ -1352,7 +1354,7 @@ let startupFailure: StartupFailure | null = null;
let activeStartupStderr = createStartupStderrCapture();

function printCapturedStartupStderr(): void {
const stderr = activeStartupStderr.text().trim();
const stderr = redactCredentialUrls(activeStartupStderr.text().trim());
if (IS_VERBOSE && stderr) {
p.note(stderr, "engine stderr");
}
Expand Down Expand Up @@ -1380,7 +1382,7 @@ function printDockerStartupLogs(): void {
maxBuffer: 64 * 1024,
},
);
const output = `${result.stdout ?? ""}${result.stderr ?? ""}`.trim();
const output = redactCredentialUrls(`${result.stdout ?? ""}${result.stderr ?? ""}`.trim());
if (output) p.note(output.slice(-16 * 1024), "docker engine logs");
}

Expand Down Expand Up @@ -1414,7 +1416,7 @@ function spawnEngineBackground(
const abnormal =
(code !== null && code !== 0) || (code === null && signal !== null);
if (abnormal) {
const stderr = activeStartupStderr.text();
const stderr = redactCredentialUrls(activeStartupStderr.text());
startupFailure = {
kind: isDocker ? "docker-crashed" : "engine-crashed",
stderr:
Expand Down Expand Up @@ -1450,9 +1452,15 @@ function prepareEngineLaunch(configPath: string): {
home,
bundledConfig,
);
const stateBackendKind = getStateBackend();
try {
mkdirSync(cwd, { recursive: true });
} catch {
} catch (err) {
const failure = resolveLaunchRenderFailure(stateBackendKind, err);
if (failure.fatal) {
p.log.error(failure.message);
process.exit(1);
}
return { configPath, cwd: process.cwd() };
}
try {
Expand All @@ -1465,6 +1473,7 @@ function prepareEngineLaunch(configPath: string): {
viewerPort: getConfiguredViewerPort(),
enginePort: getEnginePort(),
},
stateBackend: { kind: stateBackendKind, redisUrl: getRedisUrl() },
};
const rewritten = bundledConfig
? rewriteBundledConfig(
Expand Down Expand Up @@ -1513,6 +1522,11 @@ function prepareEngineLaunch(configPath: string): {
cwd,
};
} catch (err) {
const failure = resolveLaunchRenderFailure(stateBackendKind, err);
if (failure.fatal) {
p.log.error(failure.message);
process.exit(1);
}
vlog(`runtime config generation failed, using bundled config verbatim: ${String(err)}`);
return { configPath, cwd };
}
Expand Down Expand Up @@ -1551,10 +1565,34 @@ function pickCompatibleIii(candidates: Array<string | null | undefined>): string
return null;
}

function warnIfDockerIgnoresStateBackend(stateBackendKind: "file" | "redis"): void {
if (stateBackendKind !== "redis") return;
p.log.warn(
"AGENTMEMORY_STATE_BACKEND=redis is not applied on the Docker path: docker-compose.yml mounts iii-config.docker.yaml read-only, so this start will not render it. " +
"Edit iii-config.docker.yaml by hand to point iii-state/iii-stream at redis (see README \"Storage backend\"), or it keeps using the file store. " +
"docker-compose.yml passes AGENTMEMORY_REDIS_URL into the engine container, so redis_url: '${AGENTMEMORY_REDIS_URL}' works in that file.",
);
}

async function startEngine(): Promise<boolean> {
let stateBackendKind: "file" | "redis";
try {
stateBackendKind = getStateBackend();
} catch (err) {
p.log.error(err instanceof Error ? err.message : String(err));
process.exit(1);
}
if (stateBackendKind === "redis" && !getRedisUrl()) {
p.log.error(
"AGENTMEMORY_STATE_BACKEND=redis requires AGENTMEMORY_REDIS_URL to be set (e.g. redis://localhost:6379). " +
"Set both in ~/.agentmemory/.env, or unset AGENTMEMORY_STATE_BACKEND to keep the default file store.",
);
process.exit(1);
}
await assertRuntimePortOwnership();
const persistedState = readEngineState();
if (persistedState?.kind === "docker") {
warnIfDockerIgnoresStateBackend(stateBackendKind);
const inspection = inspectOwnedDockerEngine(persistedState);
if (inspection.status === "unavailable" || inspection.status === "missing") {
const detail = inspection.status === "unavailable"
Expand Down Expand Up @@ -1700,6 +1738,7 @@ async function startEngine(): Promise<boolean> {
}

if (choice === "docker" && dockerBin && composeFile) {
warnIfDockerIgnoresStateBackend(stateBackendKind);
const s = p.spinner();
s.start("Starting iii-engine via Docker...");
configureDockerHostUser();
Expand Down
Loading
Loading