Long-term memory extension for the pi coding agent:
allow-listed ACP compression blocks → SQLite FTS5 → memory_search.
ACP plugins compress long conversations into summaries. This extension collects
those compression summaries from allow-listed source locations into one
searchable store and registers a memory_search tool the model can call at any
time. The compatibility target is compression blocks / compression sidecars;
the recommended upstream for pi is
billion-context-pi.
This project is independent and community-maintained. It is not affiliated with, endorsed by, or a fork of billion-context-pi.
- billion-context-pi is the recommended optional upstream for pi. It
produces the
<session>.jsonl.acp.jsoncompression sidecars that this extension indexes. Install it separately if you want pi-side compression memory:pi install npm:billion-context-pi
- This extension reads those sidecars read-only. It does not import, bundle, call, or modify billion-context-pi, and it has no code dependency on it.
- billion-context-pi is optional. Without it, pi sessions have no sidecars to index; the extension can still index opencode-acp state files and billion-context proxy session files if configured.
- This extension does not hook pi's
contextevent. It only registers thememory_searchtool and the/memorycommand, so it is safe to run alongside billion-context-pi. It does not participate in context-compression ordering or clobber compressed output. - billion-context-pi stores compression summaries verbatim and does not
redact secrets (its only summary normalizer decodes
\uXXXXescapes); the model can therefore write a credential into a summary and the sidecar keeps it. This extension redacts secrets on ingestion and re-runs the filter on everything it returns to the model, so a value that slipped in before the current filter cannot resurface. Stored rows are not rewritten; delete*.db*and rescan to scrub durably. - The sidecar format is a published downstream contract: billion-context-pi
writes
schemaVersion+produceron every save, unknown extra block fields are additive, and files are replaced atomically (billion-context-pisrc/contract.ts, schemabcp-block-v1.json). A missingschemaVersionmeans v1. This extension reads the version and skips an unknown newer version (logging once, never rewriting it), so a future format bump cannot corrupt or mis-ingest the store. billion-context (proxy/native) session envelopes ({version, payload.state.blocks}) are read at versions 1-3 the same way. - billion-context-pi is MIT-licensed. Its name and logo belong to its authors. This project's MIT license covers only this project's code.
- Allow-list first: scanning is restricted to the roots/patterns in the allow-list file. There is no global discovery across every session or message file, so scanning stays bounded even with many sessions.
- Compression blocks only: the pi adapter reads
<session>.jsonl.acp.json, the opencode adapter readsses_*.jsonstate files, and the bili adapter reads billion-context proxy sessions (<host>_<hash>.json). Raw conversation messages are never parsed during ingestion. - Incremental and durable: per-source
mtime + sizewatermarks live in a durable ledger;UNIQUE(source_file, block_id)+INSERT OR IGNOREdedupes blocks.prune()writes tombstones so a later rescan cannot resurrect pruned blocks. - Upstream lifecycle metadata: when the source records it, each block also
keeps
active,direct_block_ids(compression lineage),duration_ms,generation,survived_count, andcompress_call_id; sources keepformat_version/producer. A force rescan refreshes these in place without touching stored summaries. - Chinese-friendly search: FTS5
trigramfor queries of 3+ characters; AND-edLIKEagainstsummary OR topicfor shorter queries; a mixed mode combines both. No LLM translation or keyword generation. - Memory-line modes:
memory_searchtakes amode.blocks(default) ranks by relevance;timelinereturns the complete catalog (one line per memory: time, project, block id, tier, topic — date-grouped, never truncated);relatedcompares the sentence containing the query words in each block (configured stopwords and a data-driven IDF suppress generic-token noise; compression lineage/run/neighbor are structural evidence) and shows that evidence sentence;conceptslists the concepts you configured. Timeline bounds (from/to) accept ISO 8601 strings (date-only = that local day) or epoch ms, and are displayed in local time;limit: 0returns the full catalog, a positive limit keeps the newest N and reports the real total. Follow a catalog line withblock+detail: trueto read the stored summary. - Concept search: pass
concept— one name that expands to the concrete terms the corpus actually uses. The vocabulary is yours: define it under"concepts": {"name": ["term", …]}inpi-billion-memory.json(the extension ships no built-in vocabulary, and personal vocabulary is never committed). All terms are searched, hits are merged, ranked by how many distinct terms matched, and listMatched terms.mode: "concepts"shows what you defined. - Retired vocabulary:
"deprecated": {"old term": "replacement", "gone": ""}inpi-billion-memory.jsonrecords terms that were renamed or dropped (empty string = no replacement). A search whose query contains one ends with aVocabulary hintsline pointing at the current name, andmode: "concepts"lists the table. The extension only reads it: the vocabulary is yours to maintain, and it never belongs in this repository. - Requirement decomposition:
decompose: truesplits a conjunctive query (,,;,and, …) into clauses and searches each one separately, grouping the hits per clause instead of ANDing every token into a single query. - Read one memory:
memory_searchwithblock+detail: trueopens a catalog/search line and returns the stored (already-redacted) topic + summary; it never reads raw session messages. - Exact-duplicate merge: a memory copied across source files (fork/copy artifacts) is shown once and lists its alternate sources; blocks from the same source (multi-tier parent/child) are never collapsed.
- Lazy project resolution: the pi session header is read only when a source file actually needs (re)ingestion, and only its first line is read. An unchanged store costs no header reads.
- Optional block expansion: with
expandEnabled,memory_expandresolves a block's recorded message pointers back to the original session messages on demand. For billion-context sessions it instead reads the envelope's ownpayload.blockContents, so it never touches the host session log. It is two-step (manifest first, then an explicit selection), bounded, filtered like ingestion, and never written to the store. - No context hook: the extension never participates in pi's
contextevent; it only indexes and searches compression summaries.
- Node.js >= 22.19.0 (built-in
node:sqlite; Node 24 is recommended and tested).node:sqliteis still marked experimental in Node, so Node may print anExperimentalWarningat startup; this is expected and safe to ignore for this extension. - pi coding agent >= 0.85.1 (tested with 1.3.0; the extension uses only public extension APIs).
- At least one enabled compression source:
- billion-context-pi for pi sidecars (recommended),
- opencode-acp state files, or
- a billion-context proxy session directory (e.g. WorkBuddy/codebuddy
sessions) with
adapter: "bili-session".
- No third-party runtime dependencies. The extension uses Node built-ins and the
pi host API (
@earendil-works/pi-coding-agent, a peer dependency). Optional billion-context-pi sidecars are read from disk.
The extension is distributed through git only — there is no npm package, so
every install path is pi install git:....
# Latest main — simplest, but not reproducible
pi install git:https://github.com/tjp72/pi-billion-memory.git
# SSH instead of HTTPS — handy if you already have a key registered with GitHub
pi install git:git@github.com:tjp72/pi-billion-memory.gitAppend @<tag> to pin a release (reproducible; tags are listed on the
Releases page):
pi install git:git@github.com:tjp72/pi-billion-memory.git@<tag>
pi install git:...clones the repository and runsnpm install --omit=dev. It does not build the project, so the builtdist/directory is committed to the repository. Do not deletedist/before tagging a release.
pi update --extensions # re-reconciles git packages; pinned tags stay where they areTo move a tag-pinned install to a newer release, re-run the install line with the
new tag (pi install git:https://github.com/tjp72/pi-billion-memory.git@v0.5.2).
npm install --legacy-peer-deps
npm run build
pi install /absolute/path/to/pi-billion-memorypi listThen in a pi session:
/memory— show store stats (sources / blocks / compressed tokens)./memory sources— print the active allow-list./memory rescan— force a full allow-list rescan./memory prune <days>— durably delete blocks older than N days./memory elevation [query]— print the keyword-table work package (rows, windows, budget brake, watermark) for thedocs/ELEVATION.mdworkflow; numbers only.- Ask the model something like "what did I work on before?" so it can call
memory_search.
pi remove git:git@github.com:tjp72/pi-billion-memory.gitThe local store is not removed automatically. To delete it as well:
rm -f ~/.pi/pi-billion-memory.db ~/.pi/pi-billion-memory.db-wal ~/.pi/pi-billion-memory.db-shm
rm -f ~/.pi/pi-billion-memory.log ~/.pi/pi-billion-memory.sources.jsonl ~/.pi/pi-billion-memory.jsonNever delete pi session files (.jsonl, .jsonl.acp.json), the opencode
database, or ~/.pi/agent/settings.json to remove this extension.
pi sidecars (.jsonl.acp.json) ┐
opencode-acp ses_*.json ┤─ allow-list ─► scanSources() ─► ~/.pi/pi-billion-memory.db
bili sessions (<host>_<hash>.json)┘ (SQLite + FTS5 trigram)
│
▼
memory_search tool (on demand)
session_startruns a background allow-list scan (configurable withscanOnStartup).agent_settledandsession_shutdownincrementally scan the current pi session.- Every
memory_searchperforms a light allow-list scan first, then queries the SQLite store. - For an allow-listed pi sidecar, the session file's first line is read only
when the sidecar needs (re)ingestion, to resolve the project name from
cwd. Message lines are never read. - For opencode sources,
opencode.dbis opened read-only (orquery_only) only to map session IDs to working directories. Conversation content is never read. - For bili sources, the proxy session file is read for its compression blocks
under
payload.state.blocks; the project name comes from the file's<host>segment. Message content is never read.
- No network: the extension makes no network requests and has no telemetry.
- No raw messages at ingestion: only compression blocks/sidecars are
ingested. The pi session file is read for its first-line header (
cwd,id,timestamp,type,version) when project resolution is needed, and — only when the opt-inexpandEnabledis on andmemory_expandis called — for the specific messages a stored block points at. - Secret and URL filter: before a block is stored, its
topicandsummaryare scanned for common credentials (passwords, API keys, tokens, private keys, JWTs, auth headers, cookies, and Chinese credential labels) and for URLs. Credential values are replaced with[REDACTED], URLs with[REDACTED_URL]; a hit count is logged without the value. This is a best-effort safety net, not a guarantee. The filter applies to newly ingested/updated blocks; existing rows are not rewritten (delete~/.pi/pi-billion-memory.db*and rescan to rebuild). - No context hook: the extension never hooks pi's
contextevent. - No session writes: the extension never writes to session files.
- Local store: the SQLite database, log, config, and allow-list live under
~/.pi/by default. They stay on your machine.
~/.pi/pi-billion-memory.json is optional. Defaults:
{
"dbPath": "~/.pi/pi-billion-memory.db",
"sourcesPath": "~/.pi/pi-billion-memory.sources.jsonl",
"logPath": "~/.pi/pi-billion-memory.log",
"maxSummaryChars": 20000,
"debug": false,
"excludeDirs": [],
"excludePaths": [],
"scanOnStartup": true,
"expandEnabled": false,
"expandMaxChars": 40000,
"expandMaxMessages": 200,
"expandMaxReadBytes": 33554432,
"concepts": {},
"deprecated": {},
"stopwords": [],
"conceptsFile": ""
}dbPath: SQLite store path.sourcesPath: JSONL allow-list path.logPath: log file path (auto-truncated at 1 MB).maxSummaryChars: summaries longer than this are truncated on ingestion. Changing it does not rewrite existing rows; delete the*.db*files and restart pi to rebuild from source.debug: verbose logging.excludeDirs: directory names to skip inside pi-sidecar source roots.excludePaths: case-insensitive path substrings skipped for every source, matched against the path relative to the source root (e.g.["Temp","probe"]drops test/temp run trees).scanOnStartup: run a background allow-list scan at session start. If disabled, the current session is still force-scanned once and other sources are picked up by later scans.expandEnabled: register the opt-inmemory_expandtool (defaultfalse). See "Expand a block" below.expandMaxChars: hard cap on characters returned by onememory_expandcall (default 40000).expandMaxMessages: hard cap on messages rendered by onememory_expandcall (default 200).expandMaxReadBytes: hard cap on bytes read from a session file by onememory_expandcall (default 33554432, i.e. 32 MB). The file is opened and read in bounded chunks, so a larger session file is never slurped whole first.concepts: inline concept vocabulary ({"name": ["term", …]}); ships empty.deprecated: retired terms ({"old": "new", "gone": ""}); only produces hints.stopwords: words dropped from related-evidence tokens (function words, filler). User data: the bundle ships no list, so related mode keeps every token when this is empty.conceptsFile: optional JSON table produced by the elevation workflow (concepts,deprecated,filtered,scope,watermark). Read-only,~expanded, mtime+size cached, capped at 1 MB; inlineconceptswin on conflicts.
~/.pi/pi-billion-memory.sources.jsonl is optional. When it is missing, the
extension uses these two built-in defaults:
{"id":"pi","adapter":"pi-sidecar","root":"~/.pi/agent/sessions","pattern":"**/*.jsonl.acp.json","enabled":true}
{"id":"opencode","adapter":"opencode-acp","root":"~/.local/share/opencode/storage/plugin/acp","pattern":"ses_*.json","enabled":true,"opencodeDb":"~/.local/share/opencode/opencode.db"}- Each non-empty, non-comment line is one JSON object.
enabled: falsedisables a source without deleting its line.rootsupports~for the current user's home directory.patternsupports a recursive**/*.jsonl.acp.jsonform or a flatses_*.jsonform.opencodeDbis optional but recommended for opencode sources: it lets the adapter resolve each state file to its real working directory (project name) via a read-only lookup.projectandcwdare optional hints forpi-sidecarsources. Setting them lets the adapter skip the lazy session-header read entirely; otherwise it reads only the session file's first line (and only when the source is new or changed).projectsets the label;cwdsupports~and also derives the label from its basename.
Example: index a billion-context proxy session directory (WorkBuddy/codebuddy):
{
"id": "bili",
"adapter": "bili-session",
"root": "~/.local/share/billion-context/sessions",
"pattern": "**/*.json",
"enabled": true
}bili-sessionreads blocks underpayload.state.blocksfrom each matched<host>_<hash>.jsonfile; the project name is the envelope's ownupstreamOrigin/protocolwhen present, otherwise the<host>segment.
Example: scan only a pi sessions subtree plus one extra sidecar root:
{"id":"pi-work","adapter":"pi-sidecar","root":"~/.pi/agent/sessions","pattern":"**/*.jsonl.acp.json","enabled":true}
{"id":"pi-archive","adapter":"pi-sidecar","root":"~/archive/compressions","pattern":"**/*.jsonl.acp.json","enabled":true}Run /memory sources to confirm, then /memory rescan.
memory_searchacceptsquery, optionalproject, optionallimit(default 6, max 20), and returns matching blocks withsource,project,topic,tier,createdAt,compressedTokens, and a preview.conceptsearches every term of a concept defined in theconceptsconfig key and merges the hits;mode: "concepts"prints the configured concepts with their terms.decompose: truesearches each clause of a conjunctive query separately.mode: "timeline"acceptsfrom/toas ISO 8601 strings (a date-only value means local midnight/day end) or epoch ms; an unparsable bound is an error, not a silently ignored filter.limit: 0returns the full catalog, a positive limit keeps the newest N and addslisted/totalaccounting.preview: trueappends one bounded summary sentence per line, but only for windows of at most 80 entries.block+detail: truereads one stored block's topic+summary; passsourcewhen the id exists in several sources.mode: "concepts"also reports the vocabulary source (config,file,config+file), its scope/watermark, and the keyword and filtered-word counts.- Queries of 3+ characters use the FTS5
trigramindex, which matches Chinese substrings, mixed Chinese/English text, and code identifiers. - Queries of 1–2 characters use AND-ed
LIKEmatches against bothsummaryandtopic. - Mixed queries (long + short tokens) use FTS for the long tokens and
LIKEfor the short ones, preserving AND semantics across modes. - Pure-
LIKEqueries do not join the FTS table; they useidx_blocks_createdfor ordering. - The schema is plain JSON; the extension does not require
typeboxor any other runtime dependency.
memory_search returns summaries. When the exact original wording matters, an
opt-in tool resolves a block's recorded message pointers back to the session
messages it absorbed:
{
"expandEnabled": true
}Set it in ~/.pi/pi-billion-memory.json, then restart pi (or /reload).
memory_expand is only registered when this is on.
Expansion is deliberately two-step so that recovering text cannot silently re-spend the context that compression just saved:
memory_expand({ block: "b1" })
-> manifest only: index, role, size and ref per absorbed message, no text
memory_expand({ block: "b1", mode: "full", select: [1, 2] })
-> renders only messages 1 and 2
blockcomes from amemory_searchresult. The same id can exist in several sessions, so addsource(a substring of theSource:label, e.g. a session file name or project) when the lookup is ambiguous.limitandcharsbound a single call; they are additionally clamped byexpandMaxMessagesandexpandMaxChars.mode: "full"requires a non-emptyselect: it refuses to render a whole block, so recovering text always stays an explicit choice.- Only pi sources (
*.jsonl.acp.json) are expandable; opencode-acp state files and bili proxy sessions do not expose message references. - A
#call_...suffix on a reference renders only that one tool call, not the surrounding assistant message. - A reference whose message is gone (deleted or truncated session file) is reported as missing rather than failing the call.
- Extension-injected
custom_messagecontext is expandable as user text. Generated ids (acp_summary_*) are labelledsynthetic: they point at summaries, not at session lines, so there is nothing to recover for them. - Expanded text passes through the same secret/URL filter as ingestion, and nothing is written back to the store.
Blocks ingested before 0.5.0 have no recorded pointers. Upgrading resets the
watermark ledger once, so the next scan re-reads the existing sidecars and
backfills the pointers without duplicating blocks: INSERT OR IGNORE still
protects the stored summary text, and tombstones still block resurrection.
Run /memory rescan to trigger the backfill immediately.
Summaries are distilled knowledge whose value only the model can judge, so there is no automatic expiry. Prune manually:
/memory prune 365 # delete blocks compressed more than 365 days ago
/memory prune 0 # delete every timestamped block
/memory rescan # add new blocks; pruned blocks stay deleted
Prune is durable: every deleted block is tombstoned, and the watermark ledger
keeps pruned-but-unchanged sources skipped, so a rescan cannot resurrect them.
A pruned block can only return if the upstream source emits it with a new
blockId (or the tombstone row is removed manually).
Rough sizing: about 1 KB of body text per block; the database file runs about 5–7× the body size (SQLite + FTS index). 720 blocks is roughly 0.7 MB of body text / 4.9 MB of database; 10k blocks is roughly 15–20 MB. No intervention is needed until hundreds of thousands of blocks.
npm install --legacy-peer-deps
npm run build # tsup + tsc -> dist/
npm run typecheck # tsc --noEmit
npm run lint # oxlint
npm test # node --import tsx --test tests/*.test.ts
npm run check # format check + typecheck + lint + test + build
npm run e2e # load dist/ and verify the pi registrations
npm run verify:dist # build and fail if committed dist/ is staleRepository layout:
src/— TypeScript source modules; the pi extension entry issrc/index.ts.dist/— committed build output. Required bypi install git:..., because pi's git installer runsnpm install --omit=devand does not run a build.tests/— self-tests run with Node's test runner andtsx.scripts/— development helpers..github/workflows/— CI and release automation.
See CONTRIBUTING.md and AGENTS.md for the rules that apply to changes.
This extension is built for one person indexing their own sessions on their own machine. That is also the only configuration it is safe in.
- No accounts, no server, no sharing layer. There is no login, no sync, no
per-user permission model. The store is a local SQLite file.
projectin amemory_searchcall filters results; it is not access control. - Everyone using the same OS user shares one store. On a shared workstation,
a remote/cloud sandbox, or a container that runs as
rootfor everyone, memories and their previews become searchable by every user of that account, and — withexpandEnabledon — original session messages become readable. Use a separate OS user (and home directory) per person. - Never share the database file. SQLite is opened in WAL mode with a 5 s
busy timeout, which coordinates concurrent processes on one host only.
Putting
dbPathon NFS/SMB or any cross-host volume can corrupt the store. - Ephemeral homes rebuild from scratch. In CI or a container with a
non-persistent home, the index is rebuilt on every start; point
dbPath(and the allow-list) at persistent storage if that matters. - Redaction is best-effort, storage is plaintext. The secret/URL filter is
regex-based, applies to newly ingested blocks only, and existing rows are
never rewritten. The database is not encrypted; the store file is created
owner-only where the OS supports it (
0600on POSIX, ignored on Windows), but that is best-effort hardening rather than protection — the log file and the SQLite-wal/-shmfiles are not covered. There is no quota, retention policy, or audit log beyond/memory prune. - Cross-machine use: sync the sidecar files, not the database. Each machine should build its own index from its own source files.
If you need shared or centrally governed memory, this extension is the wrong tool — it is deliberately local-only.
- Memory entries are ACP compression summaries, not original messages.
Semantic (embedding) search is not implemented; trigram is literal substring
matching.
memory_expandrecovers the originals only for pi sources and only for blocks that recorded message pointers. - If a block is rewritten upstream, the already-ingested summary text is not
updated (
INSERT OR IGNORE); only its message pointers are refreshed./memory rescanadds new blocks. - Upstream compression formats are internal implementations that can change. The adapters are defensive (malformed/mid-write files are skipped and retried), but a format change may require an adapter update.
- Multi-tier blocks are intentionally not deduplicated: a tier-2/3 parent and
its tier-1 children can both match. Parent summaries start with a
Source: bN+bM …header that identifies what they aggregate.
MIT. See LICENSE.