Refresh mutable AGENTS.md guidance and preserve request history

README.md

@@ -202,6 +202,7 @@ Profile data lives in `~/.optchat/profiles/<name>/` (override the root with `OPT
 | `active-memory.json`, `memories/<id>/` | After an import: pointer to the active `main/` and `tree/`. Older generations are kept. |
 | `imports/pending.json` | Resumable import state |
 | `AGENTS.md` | Profile instructions |
+| `guidance/` | Deduplicated guidance contents, source metadata, ordered revisions and request associations |
 | `config.json` | Compactor and subagent models |
 | `pending-inputs.json`, `pending-reports.json` | Recovery journals |
 | `runs/` | Subagent sessions and run metadata |
@@ -229,7 +230,10 @@ Pipe names and permissions: [docs/storage.md](docs/storage.md#windows-difference
 
 - **Profiles separate memory and instructions only.** Agents keep full filesystem access and share provider credentials.
 - **Use worktrees** when parallel agents edit the same repository; they share the filesystem.
-- **Instructions**: the main agent and subagents get Pi's usual global and repository `AGENTS.md` files and your skills, followed by the profile's `AGENTS.md`, which comes last and wins. OptChat replaces only Pi's opening prompt. Prompt templates work in the main session only.
+- **Instructions**: before every model request, the main agent and subagents check applicable global, ancestor/project and profile guidance, in that order, using Pi's file priority (`AGENTS.override.md`, `AGENTS.md`, `AGENTS.MD`, `CLAUDE.md`, `CLAUDE.MD`) and worktree shadowing. The profile's `AGENTS.md` comes last; a newer file revision does not override an explicit user instruction. Skills remain available; prompt templates work in the main session only.
+- **Changing guidance**: edits apply at each agent's next request, including tool continuations, report turns and resumed work. Stable instructions precede complete current guidance, then memory and recent work. A real change adds a short notice after the recent work, with mechanical excerpts and a path to the previous copy. That notice stays at its original boundary; after refresh it is labeled as an earlier transition. Unchanged text and timestamp-only touches add nothing. No manual reread is needed.
+- **Guidance failures**: missing sources are removed at the next request; broken links, read errors, invalid UTF-8, files or combined guidance over 1 MB, and archive-write failures stop that request. Guidance is never silently truncated or replaced by stale text. Copies live under `guidance/sources/`, retain the original basename and source path, and share content objects through hard links (the profile filesystem must support them). Original bytes and BOM-stripped supplied text are distinguished. Keep this directory with your profile; no automatic archive pruning occurs.
+- **Historical guidance**: `zoom(id, 1)` and paged subagent transcripts expose references to the exact guidance associated with the original work, including after restart, source removal or an import rebuild. References appear only in the model’s recall result; they are not added to the memory view, logged tool text or summaries. Opening a historical copy explains earlier work and does not activate its instructions. Older work recorded before this feature has no invented association.
 - **Tab title**: `π personal` while waiting for you, `● π personal` while the agent works, plus `· 2 agents` while subagents run. A connected window shows `↳ personal`, `● ↳ personal`, then `↳ personal · done` or `↳ personal · disconnected`. OptChat replaces Pi's default title and puts its own back when Pi resets it (new session, reload, rename).
 - **Images** you paste, or a tool returns, are kept under the profile's `images/` (once each, shrunk to 2,048 px on the long side and about 1.5 MB with Pi's own resizer; smaller ones stay as they are). The log names each as `[image <hash>]`, and `zoom(id, 1)` returns a message's images with its text. Summaries stay text. Imported conversations keep a text placeholder.
 - **Long turns** (#146): Pi's auto-compaction is off for the main agent, since memory replaces it. Once the current turn's verbatim messages pass the Turn refresh threshold (30% of the model's window by default), they are flushed to memory, the view is rebuilt, and only the task message plus the last tool call and its result stay verbatim; the earlier steps are the view's last lines, a zoom away. In an A/B on a 200K model, turns that otherwise gave up around 100K tokens kept going and covered the whole task. `OPTCHAT_REFRESH_AT` (a fraction, e.g. `0.3`; `0` is off) overrides the setting. Subagents use Pi's own compaction instead, starting at about the same point.
@@ -272,8 +276,13 @@ Restart Pi after source changes. Use `OPTCHAT_HOME` to test against a throwaway
 npm run check      # type check
 npm test           # offline tests, no paid model calls
 npm run test:live  # paid Anthropic calls on synthetic data in a disposable profile
+npm run test:guidance:live # paid guidance/cache validation; defaults to openai/gpt-6.1-sol
 ```
 
+The guidance live check accepts `OPTCHAT_GUIDANCE_PROVIDER` and `OPTCHAT_GUIDANCE_MODEL`. It uses synthetic files, checks SQLite → PostgreSQL behavior and interpretation of earlier work, and prints the provider’s cache counters for ordinary requests and edits. Offline tests inspect the actual Anthropic, OpenAI Responses and Chat Completions adapters; those checks establish placement and stable payloads, not a guaranteed cache hit rate or cost saving.
+
+SDK hosts using a custom resource loader can pass `optchat(pi, { agentDir, discover })`. Supply its agent directory explicitly when it initially has no global guidance file; set `discover: false` to disable project/global discovery. Pi does not expose those resource-loader options to extension events. The CLI’s `--no-context-files`/`-nc` is respected for the main agent; profile instructions still apply. Subagents retain their own project discovery.
+
 ### Publishing to npm
 
 Pi's [package directory](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/packages.md) lists npm packages with the `pi-package` keyword, which the manifest already has. GitHub alone is not enough.

docs/storage.md

@@ -10,11 +10,23 @@ Each profile folder is a local Git repository, committed after each turn and on
 
 An import builds a new memory generation (`memories/<id>/`) and switches `active-memory.json` to it only when the whole tree is ready. Older generations are kept.
 
+## Guidance history
+
+`guidance/contents/<sha256>.txt` holds each distinct observed source or supplied text once. Human-readable paths under `guidance/sources/<encoded-source>-<path-hash>/<version>/<original-basename>` hard-link to those contents. Beside each original, `metadata.json` records its original path, content version, first capture time and source modification time at that capture. If injection removes a UTF-8 BOM, `supplied.txt` identifies the exact transformed text separately. Source scope and ordering belong to immutable `guidance/revisions/<id>.json` manifests. Archive paths in these manifests are relative, so moving the whole profile preserves retrieval.
+
+`guidance/requests/<uuid>.json` binds an individual agent/session request to its ordered revision. Main-memory entries carry only a request reference as metadata; import append and rebuild preserve it when entries are renumbered. Subagent transcripts keep custom request boundaries, allowing retained messages to be matched before or after Pi compaction. Associations use request identities, not timestamp proximity between parallel agents.
+
+A request record is evidence of the guidance prepared for injection, not proof that a provider accepted the request or the agent obeyed it. Inputs interrupted before preparation can have a reserved reference without a request record; recall explicitly describes that gap. Work predating the feature is left unannotated. Corrupt or missing archived contents produce an error rather than a reference presented as valid history.
+
+Guidance checks do not add conversation messages. Transition state is kept in custom Pi transcript entries, and notices are projected at their original work boundary, or labeled as an earlier transition when that boundary has left the visible context. Recall references are tool-result metadata projected into the model request; canonical tool text and both OptChat and Pi summaries omit them. Actual conversation about instructions, or an explicit tool read of an archive, remains ordinary conversation.
+
+Original source edits, moves and deletion never remove archived versions. There is no automatic archive retention limit or pruning; copy the whole profile for backup. A failed read, invalid UTF-8, source larger than 1 MB, combined guidance larger than 1 MB, or failed archive/request write aborts the next request. A missing source removes its scope; a dangling symlink is an error. This history covers injected guidance, not every filesystem edit or the rest of an execution environment.
+
 ## The `Store` seam
 
 Memory reaches disk through one seam, `Store` in `src/store.ts`: the log, summary nodes, saved view and images, which is what would travel between machines.
 
-- Runs, recovery journals, imports and their pointer, usage, settings and `AGENTS.md` stay local files.
+- Runs, recovery journals, imports and their pointer, usage, settings, `AGENTS.md` and the guidance archive stay local files. A future remote store must also transfer the archive to make its request references readable on another machine.
 - `FileStore` (the layout in the README) is the only store today. Another, such as Postgres, would implement the same few operations.
 - Its lease (take, heartbeat, release) is optional: without one, OptChat keeps its machine-local profile lock.
 - Import records (`imports/*.json`, the staging generations and the `active-memory.json` pointer) are per machine, so a second machine with the same `~/.claude` or Codex history on disk may offer those sessions for import again.

package.json

@@ -11,7 +11,8 @@
   "scripts": {
     "check": "tsc --noEmit",
     "test": "tsx --import ./test/support.ts --test test/*.test.ts",
-    "test:live": "tsx test/live.ts"
+    "test:live": "tsx test/live.ts",
+    "test:guidance:live": "tsx test/guidance-live.ts"
   },
   "devDependencies": {
     "@earendil-works/pi-agent-core": "1.0.2",

src/agents.ts

@@ -1,4 +1,6 @@
 import { randomUUID } from 'node:crypto';
+import { Guidance, guidanceState, GUIDANCE_STATE, type GuidanceState } from './guidance.ts';
+import { GuidanceRequests, GUIDANCE_REQUEST, transcriptGuidance, withGuidanceRecall } from './guidance-recall.ts';
 import { existsSync, readFileSync, statSync } from 'node:fs';
 import { join, dirname, resolve } from 'node:path';
 import { homedir } from 'node:os';
@@ -158,6 +160,14 @@ export class Children {
     const tail = live.streaming ?? live.session.messages.at(-1);
     return tail && !retained.includes(tail) ? [...retained, tail] : retained;
   }
+  /** Called only after transcript() has checked visibility. Uses retained request boundaries, including before compaction. */
+  guidanceForTranscript(id: string) {
+    const live = this.running.get(id), file = this.history.records.get(id)?.sessionFile;
+    const entries = live?.session.sessionManager.getEntries() ?? (file ? SessionManager.open(file).getEntries() : []);
+    const retained = entries.flatMap(entry => entry.type === 'message' ? [entry.message] : []);
+    const tail = live?.streaming ?? live?.session.messages.at(-1);
+    return transcriptGuidance(entries, tail && !retained.includes(tail) ? tail : undefined);
+  }
   /** Runs at work now: running, waiting for children, or stopping; not paused. */
   get working() { return [...this.running.values()].filter(live => isRunning(live.info)).map(live => live.info); }
   subscribe(listener: () => void) { this.listeners.add(listener); return () => { this.listeners.delete(listener); }; }
@@ -271,6 +281,7 @@ export class Children {
   private async open(o: { id: string; directory: string; depth: number; parentId?: string; connected: boolean; model: Model<Api>;
     thinking?: ModelChoice['thinking']; sessionManager: SessionManager }) {
     const { id, directory, depth, parentId, connected } = o;
+    const guidance = new Guidance(this.profileDirectory), requests = new GuidanceRequests(this.profileDirectory);
     const { subagentLevels, maxAgents, memorySearch } = this.settings, delegates = depth < subagentLevels;
     const instructions = [this.instructions(), delegates ? delegation(maxAgents) : NO_DELEGATION, STEERABLE, connected ? CONNECTED : NOT_CONNECTED].filter(Boolean).join('\n\n');
     // The user's settings list their installed packages; a copy in memory keeps the child from writing them back.
@@ -285,15 +296,32 @@ export class Children {
         // Same prompt as the main agent (AGENTS.md files, skills, cwd); only the OptChat preamble differs.
         pi.on('before_agent_start', event => {
           event.systemPromptOptions.customPrompt = allowSearch(`${SUBAGENT}\n\n${VIEW_DOC}${memorySearch ? SEARCH_DOC : ''}`, memorySearch);
+          event.systemPromptOptions.contextFiles = [];
           event.systemPromptOptions.sections.instructions = instructions;
         });
+        pi.on('context_with_system', (event, ctx) => {
+          try {
+            // A failed preparation has its own boundary too, with no prepared snapshot record.
+            pi.appendEntry(GUIDANCE_REQUEST, requests.input());
+            let nextGuidance: GuidanceState | undefined;
+            const prepared = guidance.prepare(event.messages, directory, guidanceState(ctx), state => { nextGuidance = state; });
+            requests.prepare(prepared.revision, ctx.sessionManager.getSessionId(), event.messages);
+            if (nextGuidance) pi.appendEntry(GUIDANCE_STATE, nextGuidance);
+            return { messages: withGuidanceRecall(prepared.messages, refs => requests.describe(refs)) };
+          }
+          catch (error) {
+            requests.resetPending();
+            ctx.abort(); this.warn(String(error));
+            return { messages: [{ role: 'system' as const, content: 'OptChat guidance unavailable. Stop.', timestamp: 0 }] };
+          }
+        });
         pi.on('before_provider_request', (event, ctx) => ctx.model?.api === 'anthropic-messages' ? cachePayload(event.payload) : event.payload);
       }],
     });
     await loader.reload();
     const { session } = await (this.options.createSession ?? createAgentSession)({ cwd: directory, resourceLoader: loader, settingsManager,
       model: o.model, thinkingLevel: o.thinking, sessionManager: o.sessionManager,
-      customTools: [...memoryTools(() => this.memory, run => this.transcript(run, id)), ...(memorySearch ? [searchTool(() => this.memory)] : []), ...(delegates ? [...this.delegationTools(id, directory, subagentLevels, maxAgents), modelsTool(() => this)] : []), this.parentTool(id, parentId, connected)],
+      customTools: [...memoryTools(() => this.memory, run => this.transcript(run, id), { run: run => this.guidanceForTranscript(run) }), ...(memorySearch ? [searchTool(() => this.memory)] : []), ...(delegates ? [...this.delegationTools(id, directory, subagentLevels, maxAgents), modelsTool(() => this)] : []), this.parentTool(id, parentId, connected)],
       excludeTools: delegates ? [] : ['spawn', 'tell', 'models'],
     });
     // Callers track the session only after this returns: clean up here if its extensions fail to start.

src/checkpoint.ts

@@ -8,7 +8,8 @@ const exec = promisify(execFile);
 const LOOSE_LIMIT_KIB = 16_384;
 
 export async function checkpoint(directory: string) {
-  const git = (args: string[]) => exec('git', ['-C', directory, ...args], { maxBuffer: 1_000_000 });
+  // Source identity plus version can exceed Windows' legacy 260-character path limit.
+  const git = (args: string[]) => exec('git', ['-c', 'core.longpaths=true', '-C', directory, ...args], { maxBuffer: 1_000_000 });
   if (!existsSync(join(directory, '.git'))) await git(['init', '-q', '-b', 'main']);
   if (!existsSync(join(directory, '.gitignore'))) atomicWrite(join(directory, '.gitignore'), '/runs/\n/memory.html\n/usage.jsonl\n*.tmp\n');
   await git(['add', '-A', '--', '.']);