From 781e7eccfe5abc2432031507f05b8ced4b137295 Mon Sep 17 00:00:00 2001 From: Beth-2709 Date: Thu, 22 Oct 2026 19:48:59 -0400 Subject: [PATCH 1/3] feat(workspace): a direct mount can be written, and a head move away from its writes is told once A mount with `backend: 'direct'` no longer has to be read-only, and one that names no `mode` is writable, as any mount with none is. Its write, edit, edit_many and delete land on disk when they answer, or don't happen, and none of its files go into the store. The writing tools keep one path for their refusals, planning and answers, and ask the mount where the bytes go: the store-backed kind's commit and push, or the direct kind's disk. An edit is planned by the same planner against disk's text as it is now, and its answer is a store-backed edit's, changed region included. A direct write (direct-write.ts, a staged write of ours, behind the mount's boundary) stages the new bytes beside the file and renames them over it, so no reader sees half a file and a process killed mid-write leaves the old file whole. The staged file is written, moded and checked through its own handle: a replaced file's copy stays the owner's alone until it takes the file's mode, after its bytes, and the rename goes ahead only while the staged name still holds that file, by its whole identity, since a file made where one was just unlinked can take its inode number. A refusal removes the staged file only when it's still that file. A hard-linked path gets a file of its own, and the answer says the other links keep the old text. Just before the rename, the path is checked synchronously: a file there must be a regular file this process may write, as a push's write would require, and an edit's file must be unchanged since its read, by its bytes, read again through a no-follow, non-blocking open and at most one byte past what the edit read, then its identity (device, inode, size, mode, owner, group, times). The bytes are compared because a kernel that stamps files from a coarse clock, as by our reading Linux before 6.13 does, can leave a same-size change within one tick unstamped. A file another writer changed meanwhile is read and planned again, three attempts in all. edit_many stages and checks every file before it renames any, refuses two targets that are one file (through a link or as two hard links), and names each file's outcome if a rename fails partway. On a mount that follows links, a write goes to the file a link leads to inside the mount, and the link stays. A deletion keeps one root for the whole call, by its identity, and its position is taken just before its unlink. Nothing is synced. After an undo or another move of the store's head, the files stay. The workspace keeps each agent's latest direct write per mount, P (the branch and the store's head when it landed), and the id of the notice of its departure from it, N. At every start of a turn, opening, continuing or resuming, the framework asks the workspace whether the history holds P or N, and where it holds neither it stores one notice before the compile and hands its id back. A write advances P only where the history holds P or N, and clears N. P is held only by the branch it was made on or one continuing it with every fork point after its head, so an undo forking at exactly that head gives the notice. N is judged by the notice's own presence in the history, through a lookup the framework hands the module when it's added; a lookup that throws decides nothing at a start and counts as not held at a write. The notice also covers a mount the agent wrote direct that now runs as 'chronicle'. P and N are saved at stop() as a global record, which no branch switch rewinds, an entry per mount and configured root, and loaded when the store is handed over, so a restart on another branch, as an offline rollback makes, still tells. A mount configured direct gets the entry saved at its own root, and every other entry, for another root, a mount not configured direct, or none, is kept for the next save; a session that can't read the record saves none, one whose record reads but isn't of the shape saved replaces it at its stop, and one that changed nothing adds no record. The framework's own files (tool-result spills, saved payloads, fetched references) pass over a mount configured direct, which getMounts() now names whether it runs direct or as 'chronicle', and materialize passes over a writable direct mount too. A host whose only writable mount is direct says so where a spill or an MCP payload couldn't be kept. Co-Authored-By: Claude Opus 5.5 --- changelog.d/writable-direct-mounts.added.md | 9 + docs/tool-result-guard.md | 8 +- src/framework.ts | 106 +- src/mcpl/tool-result-normalize.ts | 8 +- src/modules/workspace/direct-write.ts | 425 ++++++ src/modules/workspace/edit-text.ts | 29 +- src/modules/workspace/effects.ts | 2 +- src/modules/workspace/index.ts | 982 ++++++++++++- src/modules/workspace/types.ts | 45 +- test/mcp-modern-framework.test.ts | 22 + test/workspace-direct-notice.test.ts | 643 ++++++++ test/workspace-direct-write.test.ts | 1472 +++++++++++++++++++ test/workspace-direct.test.ts | 40 +- 13 files changed, 3674 insertions(+), 117 deletions(-) create mode 100644 changelog.d/writable-direct-mounts.added.md create mode 100644 src/modules/workspace/direct-write.ts create mode 100644 test/workspace-direct-notice.test.ts create mode 100644 test/workspace-direct-write.test.ts diff --git a/changelog.d/writable-direct-mounts.added.md b/changelog.d/writable-direct-mounts.added.md new file mode 100644 index 000000000..fe92682fe --- /dev/null +++ b/changelog.d/writable-direct-mounts.added.md @@ -0,0 +1,9 @@ +- **A direct workspace mount can be written:** the workspace no longer refuses a mount with `backend: 'direct'` whose `mode` isn't `'read-only'`, and one that names no `mode` is writable, as any mount with none is. Its `write`, `edit`, `edit_many` and `delete` land on disk when they answer, or don't happen, and none of its files go into the store. The settings a direct mount refuses are otherwise as before. + - **A write is whole:** the new bytes are staged beside the file, in `.workspace-staged-`, at the file's mode, special bits included as far as the kernel lets the process set them, and renamed over it, so a reader beside the agent never sees half a file, an edited `0755` script stays `0755`, and a process killed mid-write leaves the old file whole. A process that dies between staging and its rename leaves the staged file, which `ls` and `git status` show by that name and nothing sweeps; so does a refused write whose removal of its staged file fails, or whose directory another writer moved meanwhile. A new file takes `0666` less the process's umask, as a shell's would. The file's owner and group become the process's and the ones its directory gives a new file, and an ACL or extended attribute on it isn't carried over, as with any write that renames over a file. A file the process may not write is refused, as a store-backed push's write is, and so is a file in a directory it may not write, where the staged file is made. A file with other hard links gets a file of its own at its path, the other links keep the old text, and the answer's `note` says so. Nothing is synced, so a power loss or a kernel crash can lose a write its answer said had landed. + - **An edit is planned against disk's text as it is now,** by the store-backed kind's planner, and its answer is a store-backed edit's, changed region included. Just before the rename, the file is checked unchanged since the edit read it, by its bytes, read again, and by its device, inode, size, mode, owner, group and change times. A file another writer changed meanwhile is read and planned again, three attempts in all, so the edit lands on top of that change, and it's refused where its match is gone or now occurs a different number of times, where a file changed at every attempt, or where the next read refuses it as any read would, with nothing written. The answer's `note` says when a file was read again. The bytes are compared because a kernel that stamps files from a coarse clock, as Linux before 6.13 does, can leave a same-size change made within one tick unstamped. + - **`edit_many`** checks every edit in every file, stages every file and checks each unchanged before it renames any, so a full disk while staging, or another writer's change, is met before any file is replaced. If a rename then fails partway, the files before it are written and the rest are as they were, the answer names each file's outcome, and nothing puts the written ones back. A refusal says the call wrote none of its edits. Two paths that reach one file, through a link or as hard links, are refused, so the file is named one way. + - **`delete`** unlinks the file through the mount's boundary, as a store-backed push's deletion does, a followed link's entry included. One that fails after reaching the file says the file may be gone. + - **Links:** on a mount that doesn't follow links, a write, edit or deletion at a link is refused. On one that follows them, a write or edit goes to the file the link leads to, where that stays inside the mount, and the link stays. + - **After an undo or another move of the store's head,** the files stay as the departed turns left them. At the agent's next start, opening, continuing or resuming, the framework stores one notice in its history, before that start's compile, for each mount the agent wrote direct whose latest such write its history no longer holds, whether the mount runs direct now or as `'chronicle'`: "Writes to `` from turns the history no longer holds are still on disk. If it's a git checkout, `git status` and `git diff` show them; a `git checkout` of a file drops every unstaged change in it, including the ones the history still holds." Each agent is told only of its own writes. Whether the notice was given is judged by its own presence in the history: an undo that takes it away gives it again, a redo back to the latest write needs none, and going back to a branch told earlier, after another branch's notice, gives it again. The latest write's position and the notice are saved at a clean stop, in a global record the store keeps whatever branch it's on, with the mount's configured root, so a restart on another branch, such as an offline rollback's, still gives the notice, a mount left out of one session's config keeps its positions for the next, and a mount whose path changed gets none at its new path, while its old path's are kept for when it's back; a crash loses the positions of writes made since the last one, and a later move away from those writes isn't told; it can also repeat a notice given since then. A record the store can't read leaves a session with no positions to restore, and the record as it was; one that reads but isn't of the shape saved is replaced at that session's stop. A head that moves away under a running turn and comes back before the agent's next start loses the notice for the writes made while it was away. + - **Tool descriptions:** `write`, `edit`, `edit_many` and `delete` gain a sentence about the direct kind only on a host with a writable direct mount. `ls` without a path, and `status`, give such a mount's `mode` as `'read-write'` where its config names none. +- **The framework's own files pass over a mount configured direct:** tool-result spill files, saved MCP payloads and fetched references go to the first writable mount without an include that isn't direct, so none lands in a checkout, where `git status` would show it and an undo would leave it. A host whose only writable mount is direct gets no spill file and no stored reference, and the truncation note, and the stub for an MCP payload it can't show, say no mount that keeps its files in the store can hold it. `getMounts()` gives `backend: 'direct'` for a mount configured direct, whether it runs direct this session or, while the store holds work there that disk lacks, as `'chronicle'`. diff --git a/docs/tool-result-guard.md b/docs/tool-result-guard.md index cc163728a..2562d2f31 100644 --- a/docs/tool-result-guard.md +++ b/docs/tool-result-guard.md @@ -159,7 +159,9 @@ exchange, its unsubmitted payload is not admitted to memory. Each withheld result's original is written, as the result was stored (its text; for a result over the inline cap, its capped text, whose own note names the file with the whole output), to a file under the `tool-results/` -directory of the first writable mount without an include, named +directory of the first writable mount without an include that isn't +configured direct (a direct mount's directory is a checkout, whether it runs +direct or as 'chronicle'), named `-withheld---.txt` with any character outside `A-Za-z0-9._-` replaced and the name cut to 80 characters. Its stub names it: @@ -168,7 +170,7 @@ directory of the first writable mount without an include, named Otherwise the stub names the guard's audit record, for an operator: -- with no writable mount without an include, or once the framework is +- with no writable mount without an include that isn't configured direct, or once the framework is stopping: "Its result is kept in the guard's audit record (framework/tool-result-guard), for an operator."; - when the write failed: "Writing its result to workspace file … failed, so it @@ -224,7 +226,7 @@ Two audit records describe it, each with `batchId`, `messageId` and `{ toolUseId, path }` is a written file; with `error`, a failed write; with `unfinished: true`, a write still under way at the deadline. `{ toolUseId, error }` is a write that threw. `{ toolUseId, skipped }` is no write: - `no_workspace_file` (no writable mount without an include, or the framework + `no_workspace_file` (no writable mount without an include that isn't configured direct, or the framework was stopping), `workspace_unanswered` (an earlier write was still under way past its deadline), `batch_deadline` (the batch had no time left to start another write), or `branch_changed` (the workspace declined the write before its tree entry was set). diff --git a/src/framework.ts b/src/framework.ts index 9a6b73187..659309a1b 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -5418,6 +5418,16 @@ export class AgentFramework { } this.resetCompletedStop('addModule()'); await this.moduleRegistry.addModule(module); + // The workspace judges whether an agent's history holds a message as the + // framework judges its own notices, for its head-move notice + // (noticeHeadMoves) and the writes it weighs against it. A module taken + // for the workspace by its writeBinary (getWorkspaceModule) may have no + // such call, as a host's own or a test's stand-in hasn't. + const workspace = this.getWorkspaceModule(); + if (workspace === module && typeof workspace.useHistory === 'function') { + workspace.useHistory((agentName, messageId) => + this.agents.get(agentName)?.getContextManager().getMessage(messageId as MessageId) != null); + } this.emitTrace({ type: 'module:added', moduleName: module.name }); } @@ -5787,7 +5797,7 @@ export class AgentFramework { } const mount = this.firstWritableMountName(workspace); if (!mount) { - console.error(`[refs] couldn't store ${fileName}: no writable workspace mount without an include`); + console.error(`[refs] couldn't store ${fileName}: ${this.noFileMountText()}`); return null; } const path = `${mount}/refs/${fileName}`; @@ -14931,6 +14941,48 @@ export class AgentFramework { } } + /** + * Turn start, every one: where the agent's history holds neither its + * latest direct write to a mount nor the notice of its departure + * from it, as after an undo or another move of the store's head away from + * that write, the workspace's notice that those writes are still on disk + * goes into the history, privately, before the compile, as the drafts + * notice does, and the workspace keeps its id (headMoveNotice). On an + * unmoved head it finds nothing due and stores nothing. + */ + private noticeHeadMoves(agent: Agent): void { + const workspace = this.getWorkspaceModule(); + // A module taken for the workspace by its writeBinary may have no such + // check (addModule says why), and then nothing is due. + if (!workspace || typeof workspace.headMoveNotice !== 'function' || typeof workspace.headMoveNoticeStored !== 'function') return; + let due: { text: string; mounts: string[] } | null; + try { + due = workspace.headMoveNotice(agent.name); + } catch (err) { + console.error('[workspace] head-move check failed:', err); + return; + } + if (!due) return; + let mid: string; + try { + mid = agent.getContextManager().addMessage( + 'user', + [{ type: 'text', text: due.text }], + { system: true, kind: 'workspace-head-move', mounts: due.mounts } as MessageMetadata, + ); + } catch (err) { + console.error('[workspace] head-move notice failed:', err); + return; + } + // Recorded as soon as it's stored, so nothing after can leave it stored and unrecorded. + workspace.headMoveNoticeStored(agent.name, due.mounts, mid); + try { + this.emitTrace({ type: 'message:added', messageId: mid, source: 'workspace-head-move' }); + } catch (err) { + console.error('[workspace] head-move notice trace failed:', err); + } + } + /** * Logical turn end, after the turn's deferred notices are flushed: record * as noticed exactly the drafts a stored message names (the receipt, a @@ -17356,6 +17408,10 @@ export class AgentFramework { } this.midTurnInputSignals.delete(agent.name); } + // At every start, opening, continuing or resuming: a head can move under + // a running turn without ending it, so the notice that writes the history + // no longer holds are still on disk goes in before this start's compile. + this.noticeHeadMoves(agent); if (opensTurn) { // The turn's trigger channel (item-3 redux): the channel of a @@ -20476,22 +20532,47 @@ export class AgentFramework { } /** - * The first read-write mount without an include, or null. The framework - * keeps its own files there (`tool-results/`, `refs/`). A mount with an - * include holds only the files it names, so it would refuse them, and a - * setup whose only writable mount has one is treated as having none. + * The first read-write mount without an include that isn't configured + * direct, or null. The framework keeps its own files there + * (`tool-results/`, `refs/`). A mount with an include holds only the files + * it names, so it would refuse them, and a setup whose only writable mount + * has one, or is direct, is treated as having none. */ private firstWritableMountName(workspace: WorkspaceModule): string | null { const mounts = (workspace as unknown as { - getMounts?: () => Array<{ name: string; mode: string; include?: readonly string[] }>; + getMounts?: () => Array<{ name: string; mode: string; include?: readonly string[]; backend?: 'direct' }>; }).getMounts?.(); if (mounts) { - const rw = mounts.find((m) => m.mode !== 'read-only' && m.include === undefined); + // Not a mount configured direct, whether it runs direct or as + // 'chronicle' this session: it's a directory agents write directly + // when it runs direct, so these, the framework's files about the + // agent's calls, would stay there after an undo, and in a checkout, as + // a direct mount often is, `git status` would show them. + const rw = mounts.find((m) => m.mode !== 'read-only' && m.include === undefined && m.backend !== 'direct'); return rw?.name ?? null; } return null; } + /** + * Why firstWritableMountName found no mount, for the words that say so: + * none writable without an include, or only mounts configured direct, which + * it passes over. + */ + private noFileMountText(): string { + return this.directPassedOver() + ? 'no writable workspace mount without an include that keeps its files in the store' + : 'no writable workspace mount without an include'; + } + + /** Whether firstWritableMountName passed over a writable mount without an include because it's configured direct. */ + private directPassedOver(): boolean { + const mounts = (this.getWorkspaceModule() as unknown as { + getMounts?: () => Array<{ mode: string; include?: readonly string[]; backend?: 'direct' }>; + } | null)?.getMounts?.(); + return mounts?.some((m) => m.mode !== 'read-only' && m.include === undefined && m.backend === 'direct') ?? false; + } + /** * A script's word to its agent: a wake_agent() payload, or the ending of a * left script that failed, crashed or was stopped at its limit. It enters @@ -20852,7 +20933,7 @@ export class AgentFramework { } return { text: head - + '\n\n[truncated — original was ' + content.length + ' chars; no writable workspace mount without an include, full content not retained]' + + '\n\n[truncated — original was ' + content.length + ' chars; ' + this.noFileMountText() + ', full content not retained]' + kept, filePath: null, written, @@ -27405,7 +27486,12 @@ export class AgentFramework { return null; } : null; - return normalizeStandardToolResult(result, label, save, mcplToolNoWordsText(serverId, callName)); + // Where the only mount that could hold it is direct, the stub says so, + // rather than that no workspace is mounted (noFileMountText). + const noSave = workspace && !mount && this.directPassedOver() + ? 'no workspace mount that keeps its files in the store can save it' + : undefined; + return normalizeStandardToolResult(result, label, save, mcplToolNoWordsText(serverId, callName), noSave); } const blocks = (Array.isArray(result.content) ? result.content : []) as McpToolResultContent[]; // When the result contains non-text blocks (RFC-005 references, images), @@ -28456,7 +28542,7 @@ export class AgentFramework { }); let result: Awaited>; try { - result = await workspace.writeBinary(destination, bytes, image.mediaType); + result = await workspace.writeBinary(destination, bytes, image.mediaType, { agentName }); } catch (error) { // A throw leaves the file's tree entry as it was, unless its // error says that whether the entry changed isn't known diff --git a/src/mcpl/tool-result-normalize.ts b/src/mcpl/tool-result-normalize.ts index 7e2235fd2..7ee01ac6e 100644 --- a/src/mcpl/tool-result-normalize.ts +++ b/src/mcpl/tool-result-normalize.ts @@ -136,13 +136,17 @@ export interface StandardToolResult { * Content that is present but isn't a list of blocks makes it a malformed * result, never an empty success. A tool's error result with no words, or * only whitespace, reads as `noWords`, which the framework words for its - * server and tool. + * server and tool. A payload stub with no `save` says why nothing could + * keep it in `noSave`: no workspace mounted unless the caller words it + * otherwise, as the framework does for a host whose only writable mount + * is direct. */ export async function normalizeStandardToolResult( result: StandardToolResult, label: string, save: SaveToolPayload | null, noWords = 'Tool call failed', + noSave = 'no workspace is mounted to save it', ): Promise { const malformed = result.content !== undefined && !Array.isArray(result.content); const raw: unknown[] = Array.isArray(result.content) ? result.content : []; @@ -162,7 +166,7 @@ export async function normalizeStandardToolResult( const what = `[${kind}${opts.origin ? ` ${bounded(opts.origin, URI_POINTS)}` : ''}: ${bounded(mimeType)}, ${size}`; if (!save) { lost++; - return `${what}. Not retained: ${why}, and no workspace is mounted to save it]`; + return `${what}. Not retained: ${why}, and ${noSave}]`; } const fileName = `${label}-${++saved}.${extensionFor(mimeType)}`; const path = await save(fileName, Buffer.from(base64, 'base64'), mimeType).catch((error: unknown) => { diff --git a/src/modules/workspace/direct-write.ts b/src/modules/workspace/direct-write.ts new file mode 100644 index 000000000..1d4c845da --- /dev/null +++ b/src/modules/workspace/direct-write.ts @@ -0,0 +1,425 @@ +/** + * A direct mount's writes (MountConfig.backend): disk is the only copy, so a + * write lands there whole when its call answers, or doesn't happen and leaves + * disk as it was. Ported from anima-research/hands-mcp's staged write + * (src/shared.ts), behind the mount's boundary as the workspace's other + * effects hold it (effects.ts): + * + * - The file's directory is located inside the mount (checkDir), and where + * the file lands is judged as a push's write judges it (refusedLanding): + * inside the mount, in no nested mount's directory, and covered by the + * mount's include. A link at the file is refused on a mount that doesn't + * follow links; on one that does, the write goes to the file it leads to + * when the call stages it, judged the same way, and the link stays. + * - The new bytes are staged beside the file, in `.workspace-staged-`, + * at the file's mode, special bits included as far as the kernel lets this + * process set them (a setgid bit only for a member of the file's group), + * and renamed over it, so a reader beside the agent never sees half a file + * and a process killed mid-write leaves the old file whole. A file this + * process may not write is refused, as a push's write, which opens it for + * writing, is refused: the rename needs only the directory's permission. + * A process that dies between staging and its rename leaves the staged + * file, under that name, which `ls` and `git status` show; so does a + * refused write whose removal of its staged file fails or can't tell the + * file is its own, or whose directory another writer moved after the + * staging, since the removal goes by path (discard) and the path then + * leads elsewhere. Nothing sweeps it, and the answer doesn't name it. + * The removal checks the staged name holds that file and unlinks it with + * no turn of the event loop between, so only a file another writer puts + * there in that instant can be removed in its place, or one put there + * that a stat can't tell from it: on the freed inode number, with the + * same size, mode and owner, in the same tick of a coarse clock. + * - The rename gives a hard-linked path a file of its own, and the other + * links keep the old text: `otherLinks` counts them for the answer. + * - Just before the rename, an edit's file is checked unchanged since its + * read, by its bytes, read again, and its identity (checkUnchanged), + * synchronously, so the check and the renames after it run with no turn + * of the event loop between. Only a change landing in that instant isn't + * seen, which for edit_many runs from the check through its last rename: + * no rename can be made conditional. + * - No sync. The store-backed kind syncs its pushes for its agreement + * record (reconcile.ts), and a direct mount keeps none, so a power loss + * or a kernel crash can lose a direct write after its answer said it + * landed, as it can a hands-mcp write. A process dying can't: the rename is + * whole by then. + * + * Node has no openat or renameat, so the stage and the rename go through the + * checked directory's path: a directory replaced by another writer on disk + * between the directory's check and the staging, or after its last check + * (placedStill), can redirect them, as effects.ts says of a create. For + * edit_many that last check is each file's, before the other files' checks. + */ + +import { randomUUID } from 'node:crypto'; +import { accessSync, closeSync, constants as fsConstants, lstatSync, openSync, readSync, renameSync, unlinkSync } from 'node:fs'; +import type { BigIntStats } from 'node:fs'; +import { lstat, open, realpath } from 'node:fs/promises'; +import { relative, sep } from 'node:path'; +import { refusedLanding } from './effects.js'; +import { + type CheckedDir, + type MountView, + NONBLOCK, + NOT_FOLLOWED, + OUTSIDE_PARENT, + checkDir, + checkRoot, + errno, + nameOf, + parentOf, +} from './observe.js'; + +/** An open that follows no link at the path, where the platform has one. */ +const NOFOLLOW = typeof fsConstants.O_NOFOLLOW === 'number' ? fsConstants.O_NOFOLLOW : 0; + +/** The name a staged file starts with, beside the file it replaces. */ +export const STAGED_PREFIX = '.workspace-staged-'; + +/** + * Why a direct write didn't happen when its mount's root was replaced under + * it: a direct mount keeps no root to sync to (MountConfig.backend), so its + * next call works beneath whatever is there then. + */ +export const ROOT_MOVED = 'the mount root changed while this call was being applied: a new call works beneath whatever directory the mount\'s path leads to then'; + +/** + * A file's identity as a write must find it unchanged: device, inode, size, + * mode, owner, group, and modification and change times, to the nanosecond. + * The mode, owner and group are fields of their own, since a chmod or chown + * in the same tick of a coarse clock as the file's last change leaves its + * change time as it was (checkUnchanged). + */ +export interface FileIdentity { + dev: bigint; + ino: bigint; + size: bigint; + mode: bigint; + uid: bigint; + gid: bigint; + mtimeNs: bigint; + ctimeNs: bigint; +} + +export const identityOf = (info: BigIntStats): FileIdentity => ({ + dev: info.dev, + ino: info.ino, + size: info.size, + mode: info.mode, + uid: info.uid, + gid: info.gid, + mtimeNs: info.mtimeNs, + ctimeNs: info.ctimeNs, +}); + +export const sameIdentity = (a: FileIdentity, b: FileIdentity): boolean => + a.dev === b.dev && a.ino === b.ino && a.size === b.size && a.mode === b.mode && a.uid === b.uid && a.gid === b.gid + && a.mtimeNs === b.mtimeNs && a.ctimeNs === b.ctimeNs; + +/** Another writer replaced, changed or removed the file between its read and its rename; nothing was written. */ +export class FileChanged extends Error {} + +/** Why a direct write didn't happen, in words for its answer; disk is as it was. */ +export class WriteRefused extends Error {} + +/** A file's new bytes staged beside it, ready to be renamed over it (stageContained). */ +export interface Staged { + /** Where the rename lands: the file's entry in its checked directory, a followed link's file's included. */ + path: string; + /** The staged file, beside it. */ + temp: string; + /** + * Which file the staged file is, so the rename moves, and a refusal + * removes, that one and no other put at its name: its whole identity, + * since a file made where it was unlinked can take its inode number. + */ + tempIdentity: FileIdentity; + /** The mount's root and the file's directory, as checked when it was staged. */ + root: CheckedDir; + parent: CheckedDir; + /** + * The other hard links the replaced file had, counted just before the + * rename and kept once it succeeds: the rename gives this path a file of + * its own, and they keep the old text. 0 until then, and for a new file. + */ + otherLinks: number; +} + +/** + * Test seams. `afterStage` is called once a call's files are staged, before + * they're checked unchanged (another writer acting then); `beforeRename` + * just before each rename, inside its refusal, so throwing stands for the + * rename failing. Production sets neither. + */ +export const directWriteTestSeam: { + afterStage?: (paths: string[]) => void | Promise; + beforeRename?: (path: string) => void; +} = {}; + +/** The mount-relative path of a real path inside the mount's root, `/`-separated. */ +function relativeTo(rootReal: string, real: string): string { + return relative(rootReal, real).split(sep).join('/'); +} + +/** + * The checked directory a mount-relative file lies in, beneath a checked + * root, or why not. With `makeDirectory`, missing directories are made, as + * a push's write makes them; without it, a missing one refuses the write, + * since an edit replaces a file that exists. + */ +async function directoryOf(view: MountView, rootReal: string, rel: string, makeDirectory: boolean): Promise { + const checked = await checkDir(view, rootReal, parentOf(rel), { create: makeDirectory }); + if (checked.kind === 'checked') return checked.dir; + if (checked.kind === 'absent') throw new FileChanged(rel); + throw new WriteRefused(checked.outside ? OUTSIDE_PARENT : checked.reason); +} + +/** + * Stage `bytes` beside the mount-relative file `rel`, to be renamed over it: + * the first half of a direct write. The file keeps its mode, special bits + * included, as far as the kernel lets this process set them; a new file + * takes `0666` less the umask, as a shell's would. Its owner and group are + * this process's and the ones its directory gives a new file, and an ACL or + * extended attribute on the file isn't carried over, as with any write that + * renames over a file. Anything there that isn't a regular file is refused, + * and a refusal removes the staged file it made, except where the header + * says one is left. The staged file is written and moded through its own + * handle, never through its name again. + */ +export async function stageContained( + view: MountView, + rootReal: string, + rel: string, + bytes: Buffer, + { makeDirectory = false }: { makeDirectory?: boolean } = {}, +): Promise { + const root = await checkRoot(view, rootReal); + if ('reason' in root) throw new WriteRefused(root.reason); + let parent = await directoryOf(view, rootReal, rel, makeDirectory); + let path = parent.at(nameOf(rel)); + let info = await lstat(path, { bigint: true }).catch((err: unknown) => { + if (errno(err) === 'ENOENT') return null; + throw new WriteRefused(`cannot stat the file (${errno(err) ?? String(err)})`); + }); + if (info?.isSymbolicLink()) { + if (!view.followSymlinks) throw new WriteRefused(NOT_FOLLOWED); + // A followed link stands for the file it leads to, as a read takes it: + // the write goes there, judged where it lands, and the link stays. + const real = await realpath(path).catch((err: unknown) => { + throw new WriteRefused(errno(err) === 'ENOENT' ? "a symlink whose target doesn't exist" : `cannot resolve the symlink (${errno(err) ?? String(err)})`); + }); + const leaves = await refusedLanding(view, rootReal, rel, real); + if (leaves !== null) throw new WriteRefused(leaves); + const realRel = relativeTo(rootReal, real); + parent = await directoryOf(view, rootReal, realRel, false); + path = parent.at(nameOf(realRel)); + info = await lstat(path, { bigint: true }).catch((err: unknown) => { + // Gone since realpath found it: another writer's change. + if (errno(err) === 'ENOENT') throw new FileChanged(rel); + throw new WriteRefused(`cannot stat the file the symlink leads to (${errno(err) ?? String(err)})`); + }); + if (info.isSymbolicLink()) throw new FileChanged(rel); + } else { + // A file, or none yet, reached through a link among its directories + // lands where the mount may not act: in a nested mount's directory, or + // on a file its own include leaves out (refusedLanding). + const leaves = await refusedLanding(view, rootReal, rel, path); + if (leaves !== null) throw new WriteRefused(leaves); + } + if (info?.isDirectory()) throw new WriteRefused('a directory'); + if (info && !info.isFile()) throw new WriteRefused('not a regular file'); + // A fixed-length name, so a file whose own name is near the 255-byte limit can still be replaced. + const temp = parent.at(`${STAGED_PREFIX}${randomUUID()}`); + let made = false; + try { + // A new file is made at 0666 and the kernel takes the umask off as it + // makes it. A file being replaced gives the staged file its own mode, + // which only a chmod can set past the umask, and only after the bytes + // are written, since a write clears a set-user or set-group bit; until + // then the staged file is the owner's alone, so no other user can read + // a private file's new text while it's staged. + const handle = await open(temp, 'wx', info ? 0o600 : 0o666); + made = true; + let staged: Staged; + try { + await handle.writeFile(bytes); + if (info) await handle.chmod(Number(info.mode & 0o7777n)); + const own = await handle.stat({ bigint: true }); + staged = { path, temp, tempIdentity: identityOf(own), root: root.dir, parent, otherLinks: 0 }; + } catch (err) { + // Removed while its handle is still open, so its device and inode + // name it and no other file: one made at its name since can't have + // taken its inode number. One that can't be told is left. + const mine = await handle.stat({ bigint: true }).catch(() => null); + if (mine) removeIf(temp, (there) => there.dev === mine.dev && there.ino === mine.ino); + await handle.close().catch(() => undefined); + throw err; + } + try { + await handle.close(); + } catch (err) { + // Closed, or not, with its bytes in doubt: removed by its whole identity, as a refusal's. + await discard(staged); + throw err; + } + return staged; + } catch (err) { + // A staged file that couldn't be made names the directory: a push's + // write in place needs only the file, and this one needs both. + throw new WriteRefused(made + ? `cannot write the staged file beside it (${errno(err) ?? String(err)})` + : `cannot make the staged file in its directory (${errno(err) ?? String(err)}): a direct write needs to write the file's directory as well as the file`); + } +} + +/** + * Throw unless the root and the file's directory are still the ones the + * staging checked, with a push's words for the directory (effects.ts + * moved()). + */ +export async function placedStill(staged: Staged): Promise { + if (!(await staged.root.stillHere())) throw new WriteRefused(ROOT_MOVED); + if (!(await staged.parent.stillHere())) throw new WriteRefused('the directory changed since it was checked'); +} + +/** + * Throw unless the file is still what its read found: FileChanged when it + * changed or is gone, nothing written. `expected` is null for a write, which + * read nothing and replaces whatever is there. Either way a link swapped in + * at the file the rename replaces (on a mount that follows links, the file + * the link at the agent's path led to when the call staged it) is refused + * rather than replaced, and so is a file this process may not write. It reads the entry the + * rename replaces, and then the file's bytes, synchronously, as renameInto + * runs, so a check and the renames after it run with no turn of the event + * loop between. + * + * The identity, taken once the bytes are read, catches a file replaced or + * re-moded under the call, which equal bytes wouldn't show. The bytes are + * compared with the ones the edit read: a kernel that stamps files from a + * coarse clock, as Linux before 6.13 does, leaves a file's identity as it + * was after a second write of the same size within one tick of its last + * change (on Debian's Linux 6.12.113 with ext4, 1,632 to 1,752 of 2,000 + * such rewrites kept every stamp, in three runs), and the bytes show that + * change on every clock. The bytes are read through an open that follows + * no link and doesn't wait on a pipe put at the path since its lstat, and + * at most one byte past what the edit read, so the check never blocks and + * reads no more than that. + */ +export function checkUnchanged(staged: Staged, expected: { identity: FileIdentity; bytes: Buffer } | null): void { + let now: BigIntStats; + try { + now = lstatSync(staged.path, { bigint: true }); + } catch (err) { + if (errno(err) !== 'ENOENT') throw new WriteRefused(`cannot check that it hadn't changed (${errno(err) ?? String(err)})`); + if (expected !== null) throw new FileChanged(staged.path); + return; + } + if (now.isSymbolicLink()) throw new WriteRefused('it became a symlink while this call was being applied'); + if (now.isDirectory()) throw new WriteRefused('a directory'); + if (!now.isFile()) throw new WriteRefused('not a regular file'); + // As a push's write, which opens the file for writing, would be refused: + // the rename needs only the directory's permission, and would replace a + // file its owner made read-only. + try { + accessSync(staged.path, fsConstants.W_OK); + } catch (err) { + const code = errno(err); + // Removed since the lstat: a write makes the file anew; an edit's read is gone. + if (code === 'ENOENT') { + if (expected === null) return; + throw new FileChanged(staged.path); + } + if (code === 'EACCES' || code === 'EPERM') { + throw new WriteRefused(`a file the host's process may not write (${code}): its mode or owner keeps that process out, and calling again is refused until the file's owner, or whoever runs the host, changes that`); + } + throw new WriteRefused(`cannot check that the host's process may write it (${code ?? String(err)})`); + } + if (expected === null) return; + let bytes: Buffer; + let after: BigIntStats; + try { + // No link followed, and no wait on a pipe, if either was put at the path since the lstat. + const fd = openSync(staged.path, fsConstants.O_RDONLY | NOFOLLOW | NONBLOCK); + try { + // One byte past what the edit read, enough to see the file differs: + // a file another writer made larger, or a pipe a writer keeps + // filling, put at the path since, costs no more than that. + bytes = readBounded(fd, expected.bytes.length + 1); + } finally { + closeSync(fd); + } + // And the entry is still the file the edit read, after its bytes were read. + after = lstatSync(staged.path, { bigint: true }); + } catch (err) { + // ELOOP: a link put at the path since the lstat; EAGAIN: a pipe, with a + // writer and nothing in it yet. Either is another writer's change. + if (errno(err) === 'ENOENT' || errno(err) === 'ELOOP' || errno(err) === 'EAGAIN') throw new FileChanged(staged.path); + throw new WriteRefused(`cannot check that it hadn't changed (${errno(err) ?? String(err)})`); + } + if (!sameIdentity(identityOf(after), expected.identity) || !bytes.equals(expected.bytes)) throw new FileChanged(staged.path); +} + +/** Up to `limit` bytes from the start of `fd`, fewer at its end. */ +function readBounded(fd: number, limit: number): Buffer { + const buffer = Buffer.alloc(limit); + let filled = 0; + while (filled < limit) { + // From where the last read ended (a pipe has no positions to read at). + const got = readSync(fd, buffer, filled, limit - filled, null); + if (got === 0) break; + filled += got; + } + return buffer.subarray(0, filled); +} + +/** + * Rename the staged file over the file, synchronously (checkUnchanged says + * why), counting the other hard links it splits the path from just before. + * A failed rename leaves the file as it was; the caller removes the staged + * file (discard). + */ +export function renameInto(staged: Staged): void { + let otherLinks: number; + try { + directWriteTestSeam.beforeRename?.(staged.path); + // The staged file is still the one this call wrote, not another put at its name. + const temp = lstatSync(staged.temp, { bigint: true }); + if (!temp.isFile() || !sameIdentity(identityOf(temp), staged.tempIdentity)) { + throw new WriteRefused('another writer replaced the staged copy of the new text before it could be renamed over the file, which is as it was'); + } + otherLinks = otherLinksAt(staged.path); + renameSync(staged.temp, staged.path); + } catch (err) { + if (err instanceof WriteRefused) throw err; + throw new WriteRefused(`cannot rename the staged file over it (${errno(err) ?? (err instanceof Error ? err.message : String(err))})`); + } + // Set once the rename has happened, so a count never stands for a split that didn't. + staged.otherLinks = otherLinks; +} + +/** The other hard links of what the entry at `path` holds: 0 for none there, and when lstat fails, since the count only informs the answer. */ +function otherLinksAt(path: string): number { + try { + return lstatSync(path).nlink - 1; + } catch { + return 0; + } +} + +/** + * Remove a staged file that won't be renamed, best effort: only the file + * this call staged, by its whole identity, since whatever another writer + * put at its name since is theirs, as renameInto leaves it too. Checked and + * removed synchronously, as the rename is (checkUnchanged says why). + */ +export async function discard(staged: Staged): Promise { + removeIf(staged.temp, (there) => there.isFile() && sameIdentity(identityOf(there), staged.tempIdentity)); +} + +/** Unlink what's at `path` if `ours` says it's the file meant; best effort, with no turn of the event loop between the check and the unlink. */ +function removeIf(path: string, ours: (there: BigIntStats) => boolean): void { + try { + if (ours(lstatSync(path, { bigint: true }))) unlinkSync(path); + } catch { + // Gone already, or not ours to know: left, as the header says. + } +} diff --git a/src/modules/workspace/edit-text.ts b/src/modules/workspace/edit-text.ts index ad3e83422..4516c25c1 100644 --- a/src/modules/workspace/edit-text.ts +++ b/src/modules/workspace/edit-text.ts @@ -190,8 +190,13 @@ function label(problem: EditProblem): string { */ const FIRST_LINE_MAX = 400; -/** What an answer is: a refusal, whose problems the agent answers and sends again, or the answer to edits stored, which mustn't be sent again. */ -export type AnswerKind = 'refusal' | 'stored'; +/** + * What an answer is: a refusal, whose problems the agent answers and sends + * again; the answer to edits stored, which mustn't be sent again; or, on a + * direct mount, the answer to an edit_many call written partway, whose parts + * are each file's outcome. + */ +export type AnswerKind = 'refusal' | 'stored' | 'written'; /** A line cut to FIRST_LINE_MAX, as a part's first line is, and as a thrown message is in an answer's head. */ export function cutLine(line: string): string { @@ -205,18 +210,25 @@ export function cutLine(line: string): string { * shown and send the call again to see the rest: once those are answered, * their room can show the rest, details included. For edits stored, the * problems are the files disk refused, and a last line that leaves some out - * points to ls. + * points to ls. For edits written partway, they're each file's outcome, and + * one that leaves some out points to read. */ function tailText(kind: AnswerKind, maxChars: number, cut: number, problems: number, notes: number, anyShown: boolean): string { const items = [ ...(cut ? [`the details of ${cut} of these`] : []), - ...(problems ? [kind === 'refusal' ? `${problems} more ${plural(problems, 'problem')}` : `${problems} more ${plural(problems, 'file')} disk refused`] : []), + ...(problems + ? [kind === 'refusal' + ? `${problems} more ${plural(problems, 'problem')}` + : kind === 'stored' ? `${problems} more ${plural(problems, 'file')} disk refused` : `the outcomes of ${problems} more ${plural(problems, 'file')}`] + : []), ...(notes ? [`${notes} ${plural(notes, 'note')} on the files' copies on disk`] : []), ]; if (!items.length) return ''; const next = kind === 'stored' ? problems ? " ls shows each path's state." : '' - : problems && anyShown ? ' Answer the problems shown and send the call again to see the rest.' : ''; + : kind === 'written' + ? problems ? ' Reading a file shows it as disk has it now.' : '' + : problems && anyShown ? ' Answer the problems shown and send the call again to see the rest.' : ''; const list = items.length < 2 ? items[0] : `${items.slice(0, -1).join(', ')} and ${items.at(-1)}`; return `[To keep this answer within ${maxChars.toLocaleString('en-US')} characters, it leaves out ${list}.${next}]`; } @@ -271,9 +283,10 @@ export function fitted(head: string, problems: string[], notes: string[], maxCha * counts its problems and names each, in edit order, labelled with its edits. * `notes` follow the problems: which copy of a file the edits looked in, and * a conflict with disk, which the agent needs whether or not the call stored - * anything. + * anything. On a direct mount, whose edits go to disk with no store between + * (`verb` 'wrote'), edit_many says it wrote none. */ -export function refusalText(tool: EditTool, problems: EditProblem[], notes: string[], maxChars: number): string { +export function refusalText(tool: EditTool, problems: EditProblem[], notes: string[], maxChars: number, verb: 'stored' | 'wrote' = 'stored'): string { const sorted = [...problems].sort((a, b) => firstEdit(a) - firstEdit(b)); if (tool === 'edit') { const [problem] = sorted; @@ -281,7 +294,7 @@ export function refusalText(tool: EditTool, problems: EditProblem[], notes: stri const whole = problem!.kind === 'file' && problem!.whole ? text : `The edit did nothing: ${text}`; return fitted('', [whole], notes, maxChars, 'refusal'); } - const head = `The edit_many call stored none of its edits: ${sorted.length} ${plural(sorted.length, 'problem')}.`; + const head = `The edit_many call ${verb} none of its edits: ${sorted.length} ${plural(sorted.length, 'problem')}.`; return fitted(head, sorted.map((problem) => `${label(problem)}${problemText(problem, tool)}`), notes, maxChars, 'refusal'); } diff --git a/src/modules/workspace/effects.ts b/src/modules/workspace/effects.ts index c9f3f0866..a5955c929 100644 --- a/src/modules/workspace/effects.ts +++ b/src/modules/workspace/effects.ts @@ -385,7 +385,7 @@ export async function writeContained( * the link is what's refused. The opened file's check (checkOpened) asks * it, and so does a create, before anything is made. */ -async function refusedLanding(view: MountView, rootReal: string, rel: string, real: string): Promise { +export async function refusedLanding(view: MountView, rootReal: string, rel: string, real: string): Promise { const leaves = view.followSymlinks ? 'a symlink that leaves the mount' : OUTSIDE_PARENT; if (!contained(rootReal, real)) return leaves; const holder = nestedAtReal(real, view.nested); diff --git a/src/modules/workspace/index.ts b/src/modules/workspace/index.ts index bf684a69c..3ad3ea725 100644 --- a/src/modules/workspace/index.ts +++ b/src/modules/workspace/index.ts @@ -6,7 +6,7 @@ */ import { constants as fsConstants } from 'node:fs'; -import type { Stats } from 'node:fs'; +import type { BigIntStats, Stats } from 'node:fs'; import { open, readdir, readlink, stat, lstat, realpath } from 'node:fs/promises'; import { basename, dirname, isAbsolute, join, parse, resolve, relative, sep } from 'node:path'; import type { JsStore, JsTreeEntry } from '@animalabs/chronicle'; @@ -61,6 +61,23 @@ import { reconcilePass, settleAdoptionBeforeMutation, } from './reconcile.js'; +import { EffectFailed, unlinkContained } from './effects.js'; +import { RecordJournal } from '../../record-journal.js'; +import { + type FileIdentity, + type Staged, + FileChanged, + ROOT_MOVED, + WriteRefused, + checkUnchanged, + directWriteTestSeam, + discard, + identityOf, + placedStill, + renameInto, + sameIdentity, + stageContained, +} from './direct-write.js'; /** Default for how long the after-batch scan may hold the agent's next inference. */ const AGENT_ACTION_SCAN_DEADLINE_MS = 20_000; @@ -74,6 +91,8 @@ const AGENT_ACTION_SCAN_DEADLINE_MS = 20_000; const CATCH_UP_BOUND_MS = 30_000; /** Why a call is refused once the module has stopped. */ const NOT_RUNNING = 'The workspace has stopped: it takes no more work until it starts again'; +/** writeBinary's refusal when the store's branch is no longer the one its file is for. */ +const BRANCH_LEFT = 'the workspace had left the branch this file is for'; /** The workspace's own deadline must end before the framework's hook bound. */ const AGENT_ACTION_SCAN_DEADLINE_MAX_MS = TOOL_BATCH_HOOK_TIMEOUT_MS - 1_000; /** A direct mount's ignore list when its config names none (MountConfig.ignore). */ @@ -88,6 +107,8 @@ const DIRECT_MOUNT_IGNORE: readonly string[] = ['node_modules', 'dist', '.git', const DIRECT_READS_AT_ONCE = 16; /** Why sync passes over a direct mount, as its result's `skipped` says. */ const DIRECT_SYNC_SKIPPED = "a direct mount: its tools read disk as it is, so there's nothing to take in"; +/** Why materialize passes over a writable direct mount, as its result's `skipped` says. */ +const DIRECT_MATERIALIZE_SKIPPED = "a direct mount: its writes are on disk as they're made, so there's nothing to write"; export type { WorkspaceConfig, @@ -983,6 +1004,28 @@ export class WorkspaceModule implements Module { */ private notDirect = new Map>(); + /** + * Each agent's latest direct write on each mount it wrote direct, P, with + * the notice of its departure from it, N (DirectHead), by mount, then by + * agent. Saved at stop() in a global record (headsJournal), not the + * module's branch state: they describe disk, which doesn't rewind with a + * branch, so a restart on another branch, such as one an offline rollback + * made, still finds them. + */ + private directHeads = new Map>(); + + /** Where directHeads is saved and loaded (DIRECT_HEADS_RECORD_TYPE), once initStore() has read it. */ + private headsJournal: RecordJournal | null = null; + + /** The record initStore() read, as saved text, so a stop that would save the same adds none; null where there's one to replace. */ + private loadedHeads: string | null = null; + + /** Saved entries this session doesn't take up (another root, a mount not configured direct, or none), kept for the next save. */ + private carriedHeads: DirectHeadsEntry[] = []; + + /** Whether an agent's history holds a message, as the framework hands it over (useHistory). */ + private historyLookup: ((agentName: string, messageId: string) => boolean) | null = null; + constructor(config: WorkspaceConfig) { for (const mount of config.mounts) { const cap: unknown = mount.walkCap; @@ -1116,10 +1159,64 @@ export class WorkspaceModule implements Module { this.agreement.open(this.config.mounts.filter((m) => intents.has(m.name)).map((m) => ({ name: m.name, root: resolve(m.path) }))); this.chooseKinds(); + this.loadDirectHeads(store); this.applySavedState(); this.ensureRunning(); } + /** + * The positions stop() last saved (directHeads), before any call can + * write: each saved entry is a mount's positions at one root, and a mount + * this session configures direct gets the entry at its own root. Every + * other entry, a mount configured at another root, not direct, or not at + * all, is kept for the next save (carriedHeads), since its writes are + * still on disk beneath that root; a mount removed for good, and each + * root a mount has moved from, leave that small entry behind. A record + * the store can't read is logged, and this session starts with none and + * saves none, so the record stands for a session that can read it. One + * read but not of the shape saved is logged too, and this session starts + * with none and its stop replaces it, so a damaged record doesn't keep + * every later session from saving. Either way, a move away from the + * writes before it goes untold, as after a crash. + */ + private loadDirectHeads(store: JsStore): void { + const journal = new RecordJournal(store, { type: DIRECT_HEADS_RECORD_TYPE }); + const restored = new Map>(); + const carried: DirectHeadsEntry[] = []; + let loaded: unknown; + try { + loaded = journal.load().snapshot; + } catch (err) { + console.error('[workspace] could not read the saved direct-write positions:', err); + return; + } + let snapshot: DirectHeadsSnapshot; + let text: string | null; + try { + snapshot = checkedHeads(loaded === null || loaded === undefined ? [] : loaded); + text = canonicalHeads(snapshot); + } catch (err) { + console.error("[workspace] the saved direct-write positions aren't of the shape saved, so this session starts with none and its stop replaces them:", err); + snapshot = []; + text = null; + } + for (const saved of snapshot) { + // Only a direct mount's writes are the agent's on disk, and a + // store-backed one's disk follows its store. + const mount = this.mounts.get(saved.mount); + if (!mount || mount.config.backend !== 'direct' || saved.root !== resolve(mount.config.path)) { + carried.push(saved); + continue; + } + restored.set(saved.mount, new Map(Object.entries(saved.agents).map(([agent, head]) => + [agent, { branchId: head.branchId, after: head.after, notice: head.notice ?? null }]))); + } + this.headsJournal = journal; + this.loadedHeads = text; + this.carriedHeads = carried; + for (const [name, heads] of restored) this.directHeads.set(name, heads); + } + /** * Whether each mount configured direct runs direct: not while the store * holds changes there that disk lacks, on the branch selected then, so @@ -1683,18 +1780,57 @@ export class WorkspaceModule implements Module { } this.ctx.setState(state); } + this.saveDirectHeads(); this.ctx = null; } - /** Mount names + modes, and each mount's include where it sets one, for - * peer callers (framework spill/journal paths) that need a mount for files - * of their own, the first writable one without an include - * (firstWritableMountName), without reaching into private state. */ - getMounts(): Array<{ name: string; mode: 'read-write' | 'read-only'; include?: readonly string[] }> { + /** + * Save directHeads as one global record, the latest of which the next + * start loads (loadDirectHeads): each mount's positions with the root + * they were made under, and the entries this session didn't take up + * (carriedHeads), unchanged. Only when that differs from what the + * start loaded, so a session that changed nothing, or a store no direct + * write reached, adds no record, and always where the record read wasn't + * of the shape saved, which this replaces. Nothing where the store couldn't + * read the record, so it stands. A save that fails is logged, and the next + * start finds the positions of the last save that landed, as after a + * crash. + */ + private saveDirectHeads(): void { + if (!this.headsJournal) return; + const snapshot: DirectHeadsSnapshot = [ + ...this.carriedHeads, + ...[...this.directHeads].filter(([, heads]) => heads.size > 0).map(([name, heads]) => ({ + mount: name, + root: resolve(this.mounts.get(name)!.config.path), + agents: Object.fromEntries([...heads].map(([agent, { branchId, after, notice }]) => [agent, { branchId, after, ...(notice !== null ? { notice } : {}) }])), + })), + ]; + const text = canonicalHeads(snapshot); + if (text === this.loadedHeads) return; + try { + this.headsJournal.checkpoint(snapshot); + this.loadedHeads = text; + } catch (err) { + console.error('[workspace] could not save the direct-write positions:', err); + } + } + + /** Mount names + modes, each mount's include where it sets one, and + * `backend: 'direct'` for one configured direct, whether it runs direct + * this session or, while the store holds work there disk lacks, as + * 'chronicle': either way it's a directory agents write directly when it + * runs direct, whose files an undo leaves. For the + * framework's tool-result spills, saved payloads and fetched references, + * which need a mount for files of their own: the first writable one + * without an include that isn't direct (firstWritableMountName), without + * reaching into private state. */ + getMounts(): Array<{ name: string; mode: 'read-write' | 'read-only'; include?: readonly string[]; backend?: 'direct' }> { return [...this.mounts.values()].map((m) => ({ name: m.config.name, mode: m.config.mode === 'read-only' ? 'read-only' : 'read-write', ...(m.config.include ? { include: [...m.config.include] } : {}), + ...(m.config.backend === 'direct' ? { backend: 'direct' as const } : {}), })); } @@ -1716,6 +1852,23 @@ export class WorkspaceModule implements Module { : ''; const globDirect = direct ? ' On a direct mount, a match carries its size, or a note where the mount doesn\'t take it, and no state.' : ''; const syncDirect = direct ? ' A direct mount is listed under `skipped`: its tools read disk as it is, so there\'s nothing to take in.' : ''; + // Where a direct mount is writable, the four writing tools say what a + // write is there: on disk when it answers, and left there by a head move. + const writable = [...this.mounts.values()].some((m) => m.direct && m.config.mode !== 'read-only'); + const writeDirect = writable + ? ' On a direct mount, the file is on disk, whole and with its mode kept, when the call answers, and an undo or other head move leaves it there.' + : ''; + const editDirect = writable + ? " On a direct mount, the edit applies to disk's text as it is now and is on disk when the call answers; a change another writer makes " + + 'meanwhile is kept, with the edit applied again on top of it, and an undo or other head move leaves the edit on disk.' + : ''; + const editManyDirect = writable + ? " On a direct mount, every edit is checked against disk's text before any file is written, and then the files are written one by one: " + + "if a write fails partway, the answer names which files were written and which weren't, and nothing puts the written ones back. " + + 'An undo or other head move leaves them on disk.' + : ''; + const deleteDirect = writable ? ' On a direct mount, the file is removed from disk at once, and an undo or other head move doesn\'t bring it back.' : ''; + const materializeDirect = writable ? ' A direct mount writes its files to disk as they\'re made, so materialize passes over it, under `skipped` where you name it.' : ''; return [ { name: 'read', @@ -1749,7 +1902,7 @@ export class WorkspaceModule implements Module { }, { name: 'write', - description: 'Create or overwrite a file in the workspace.', + description: 'Create or overwrite a file in the workspace.' + writeDirect, inputSchema: { type: 'object' as const, properties: { @@ -1770,7 +1923,7 @@ export class WorkspaceModule implements Module { + "Only the replaced text changes, and the new text's line ends take those of the text it replaces. " + 'A change disk made to the file is taken in first, and the answer says so. ' + 'The answer shows the changed region, numbered, a long line from shortly before the change on it, as its length allows. ' - + 'For several edits at once, in one file or across files, applied all or none, use edit_many.', + + 'For several edits at once, in one file or across files, applied all or none, use edit_many.' + editDirect, inputSchema: { type: 'object' as const, properties: { @@ -1792,7 +1945,8 @@ export class WorkspaceModule implements Module { + "A refusal names the problems it finds, each by its edit's number in the list (from 1). " + 'All or none holds in the workspace: on a mount that writes to disk, disk can still refuse a file, ' + "and the answer then says every edit is stored and names each file disk refused, with its push's reason. " - + "The answer shows each edit's first changed region, numbered, a long line from shortly before each change on it, as its length allows.", + + "The answer shows each edit's first changed region, numbered, a long line from shortly before each change on it, as its length allows." + + editManyDirect, inputSchema: { type: 'object' as const, properties: { @@ -1821,7 +1975,7 @@ export class WorkspaceModule implements Module { name: 'delete', description: 'Delete a file from the workspace. On a mount without autoMaterialize the disk copy stays, ' + 'listed as workspace-deleted (no scan brings it back), until materialize with applyDeletions removes it ' - + 'or a sync of the path restores it.', + + 'or a sync of the path restores it.' + deleteDirect, inputSchema: { type: 'object' as const, properties: { @@ -1896,7 +2050,7 @@ export class WorkspaceModule implements Module { description: 'Write workspace files to the real filesystem. Use after writing/editing to push changes to disk. ' + 'Never deletes disk files unless applyDeletions is given; lists workspace deletions it left on disk. ' + 'On a mount with an include, what the include doesn\'t cover is never written, and one naming a ' - + 'directory counts what it left beneath it under `passedOver`, by mount and reason, without naming it.', + + 'directory counts what it left beneath it under `passedOver`, by mount and reason, without naming it.' + materializeDirect, inputSchema: { type: 'object' as const, properties: { @@ -2038,10 +2192,10 @@ export class WorkspaceModule implements Module { switch (call.name) { case 'read': return await this.handleRead(input as unknown as ReadInput); case 'read_image': return await this.handleReadImage(input as unknown as ReadImageInput); - case 'write': return await this.handleWrite(input as unknown as WriteInput); - case 'edit': return await this.handleEdit(input as unknown as EditInput, call.resultMaxChars); + case 'write': return await this.handleWrite(input as unknown as WriteInput, call.callerAgentName); + case 'edit': return await this.handleEdit(input as unknown as EditInput, call.resultMaxChars, call.callerAgentName); case 'edit_many': return await this.handleEditMany(input as unknown as EditManyInput, call.resultMaxChars, call.callerAgentName); - case 'delete': return await this.handleDelete(input as unknown as DeleteInput); + case 'delete': return await this.handleDelete(input as unknown as DeleteInput, call.callerAgentName); case 'ls': return await this.handleLs(input as unknown as LsInput); case 'glob': return await this.handleGlob(input as unknown as GlobInput); case 'grep': return await this.handleGrep(input as unknown as GrepInput); @@ -2182,6 +2336,12 @@ export class WorkspaceModule implements Module { * withheld original this way, for the branch its stub is on * (agent-framework #277). * + * On a mount running direct, the bytes go to disk as a direct `write`'s + * do, whole or not at all, and nothing goes into the store: `branch` is + * checked before the write starts and again just before its rename, and + * `agentName` names the agent whose write it is for the head-move notice + * (a write naming none tells no agent). + * * A write whose tree entry is set but that then fails returns success * false with `data: { inChronicle: true }`: its copy on disk was refused, * the store failed after recording the entry, which skips the push where @@ -2195,7 +2355,7 @@ export class WorkspaceModule implements Module { mountPrefixedPath: string, data: Buffer, mimeType: string, - options: { branch?: string } = {}, + options: { branch?: string; agentName?: string } = {}, ): Promise { if (!this.running()) return { success: false, isError: true, error: NOT_RUNNING }; return this.underWay(this.writeBinaryUnderWay(mountPrefixedPath, data, mimeType, options)); @@ -2205,7 +2365,7 @@ export class WorkspaceModule implements Module { mountPrefixedPath: string, data: Buffer, mimeType: string, - options: { branch?: string }, + options: { branch?: string; agentName?: string }, ): Promise { let mount: MountState; let relativePath: string; @@ -2226,12 +2386,25 @@ export class WorkspaceModule implements Module { return { success: false, error: `Content exceeds max file size (${maxSize} bytes)`, isError: true }; } const store = this.getStore(); + // A direct mount's bytes go to disk, whole, or nowhere (writeDirect), + // as the caller's agent's write, where it names one. A branch it names + // is checked just before the rename, since disk is no branch's. + if (mount.direct) { + // Checked first too, so a request already for a left branch makes no + // directory and stages nothing. + if (options.branch !== undefined && store.currentBranch().name !== options.branch) { + return { success: false, error: BRANCH_LEFT, isError: true, data: { branchLeft: true } }; + } + const written = await this.writeDirect(mount, relativePath, mountPrefixedPath, data, options.agentName, options.branch); + if ('error' in written) return { success: false, isError: true, error: written.error, ...(written.branchLeft ? { data: { branchLeft: true } } : {}) }; + return { success: true, data: { path: mountPrefixedPath, size: data.byteLength, mimeType, ...(written.note ? { note: written.note } : {}) } }; + } return this.withMount(mount, async () => { await this.settleBeforeMutation(mount, relativePath); // Checked inside the mount's turn, after anything the write waited on: // commitToolChange sets the tree entry before it first yields. if (options.branch !== undefined && store.currentBranch().name !== options.branch) { - return { success: false, error: 'the workspace had left the branch this file is for', isError: true, data: { branchLeft: true } }; + return { success: false, error: BRANCH_LEFT, isError: true, data: { branchLeft: true } }; } const blobHash = store.storeBlob(data, mimeType); // An overwrite keeps the file's mode, as the write tool's does. @@ -2490,8 +2663,10 @@ export class WorkspaceModule implements Module { * mount nested by its directories alone, since its real root holds this * one, is judged by the parse's nesting, so a root repointed between a * read's parse and its open can have it serve a path in that mount's - * directory. Each mount is placed as a start places it (placementOf), from - * where its real root and the directories above its configured path lie on + * directory, and one repointed between a direct write's parse and its + * staging can have the write land there. Each mount is placed as a start + * places it (placementOf), from where its real root and the directories + * above its configured path lie on * disk. For a mount configured at this mount's own path or beneath it, * that's where they lie now: beneath `rootReal` where they lie beneath this * mount's path, and the directories above this mount's path as resolving @@ -3765,8 +3940,7 @@ export class WorkspaceModule implements Module { } catch (err) { // A refusal that names only the path says why with its errno, as a // failed read below does. - const bare = err instanceof WorkspaceReadError && err.errno !== undefined && err.message === `Unable to read file: ${path}`; - return { error: bare ? `${err.message} (${err.errno})` : err instanceof Error ? err.message : String(err) }; + return { error: containedRefusal(err, path) }; } try { // No more than the limit is loaded, as for a file over it at its fstat. @@ -3925,7 +4099,7 @@ export class WorkspaceModule implements Module { } } - private async handleWrite(input: WriteInput): Promise { + private async handleWrite(input: WriteInput, callerAgentName?: string): Promise { if (LONE_SURROGATE.test(input.content)) { return { success: false, isError: true, error: 'The write did nothing: "content" holds a lone UTF-16 surrogate, which UTF-8 can\'t store.' }; } @@ -3943,6 +4117,12 @@ export class WorkspaceModule implements Module { const buffer = Buffer.from(input.content, 'utf-8'); const receipt = { path: input.path, size: buffer.byteLength, hash: hashContent(input.content) }; + // A direct mount's bytes go to disk, whole, or nowhere (writeDirect). + if (mount.direct) { + const written = await this.writeDirect(mount, relativePath, input.path, buffer, callerAgentName); + if ('error' in written) return { success: false, isError: true, error: written.error }; + return { success: true, data: { ...receipt, ...(written.note ? { note: written.note } : {}) } }; + } let committed = false; let refused: string | null; let modeKept: string | null; @@ -3966,13 +4146,13 @@ export class WorkspaceModule implements Module { return { success: true, data: { ...receipt, ...(modeKept ? { modeKept } : {}) } }; } - private async handleEdit(input: EditInput, maxChars = DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS): Promise { + private async handleEdit(input: EditInput, maxChars = DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS, callerAgentName?: string): Promise { return this.applyEdits('edit', [{ path: input.path, oldString: input.oldString, newString: input.newString, expected: input.replaceAll ? 'all' : 1, - }], maxChars); + }], maxChars, callerAgentName); } private async handleEditMany( @@ -4093,6 +4273,9 @@ export class WorkspaceModule implements Module { const [mount] = mounts.values(); if (tool === 'edit' && problems.length) return refuse(problems); if (!mount || mounts.size > 1) return refuse(problems); + // A direct mount's edits apply to disk's text as it is now, and land on + // disk or don't happen (applyDirectEdits). + if (mount.direct) return this.applyDirectEdits(tool, mount, [...targets.values()], problems, given.length, maxChars, callerAgentName); const store = this.getStore(); // Whether the commit set the call's entries, all of them at once. @@ -4202,48 +4385,11 @@ export class WorkspaceModule implements Module { problems.push({ kind: 'file', edits: target.edits, path: target.path, text: `Blob not found for: ${target.path}`, whole: true }); continue; } - // Decoding would turn each invalid sequence into U+FFFD, and writing - // the text back would change bytes the edits never named. - const file = decodeForEdit(blob); - if (!file) { - problems.push({ - kind: 'file', - edits: target.edits, - path: target.path, - text: `${target.path} isn't valid UTF-8, so editing it would change bytes outside the text it replaces. ` - + 'Read shows each invalid byte sequence as U+FFFD, and a write replaces the whole file as UTF-8, so writing back what read shows puts U+FFFD in their place.', - whole: false, - }); - continue; - } - if (!target.specs.length) continue; - const planned = planFile(file, blob, target.specs, hints, maxSize, counts); - if ('problems' in planned) { - // One at a time: a call of many edits can have more problems - // than a spread passes as arguments. - for (const problem of planned.problems) problems.push({ ...problem, path: target.path }); - continue; - } - // The planner refuses a file its counts take past the limit before - // building it, from the least its bytes can be; one it built past - // the limit, as its CRs or multi-byte characters can take it, is - // refused here. - if ('tooLarge' in planned || planned.plan.bytes.length > maxSize) { - const size = 'tooLarge' in planned ? `at least ${planned.tooLarge.toLocaleString('en-US')}` : planned.plan.bytes.length.toLocaleString('en-US'); - problems.push({ - kind: 'file', - edits: target.edits, - path: target.path, - text: `${target.path} would be ${size} bytes with ${tool === 'edit' ? 'this edit' : editList(target.edits)}, ` - + `past the ${maxSize.toLocaleString('en-US')}-byte limit this mount sets on a file, so the workspace couldn't keep it. ` - + "An edit that leaves the file within the limit applies. The limit is the mount's maxFileSize setting, which only whoever runs the host can change.", - whole: false, - }); - continue; - } + const plan = planTargetEdits(tool, target, blob, hints, maxSize, counts, problems); + if (plan === null) continue; plans.push({ target, - plan: planned.plan, + plan, mode: entry.mode, backOnDisk, conflictNote: pushes ? null : conflictNote(tool === 'edit' || target.edits.length === 1 ? 'is in' : 'are in'), @@ -4405,12 +4551,515 @@ export class WorkspaceModule implements Module { return { success: false, isError: true, error: fitted(head, parts, notes, maxChars, 'stored') }; } - private async handleDelete(input: DeleteInput): Promise { + // ========================================================================== + // Direct writes + // ========================================================================== + + /** + * A direct mount's write of a whole file: its bytes staged beside it and + * renamed over it (direct-write.ts), under the mount's turn, so the + * module's own calls take turns. Answers a note when the file had other + * hard links, which keep its old text, or why nothing was written. One + * that lands is the caller's latest write on the mount (recordDirectWrite). + * With `branch`, nothing is written once the store's branch is another, + * checked with no turn of the event loop before the rename (writeBinary's). + */ + private async writeDirect( + mount: MountState, + relativePath: string, + path: string, + bytes: Buffer, + agentName: string | undefined, + branch?: string, + ): Promise<{ note: string | null } | { error: string; branchLeft?: true }> { + return this.withMount(mount, async () => { + const { view, rootReal } = await this.diskView(mount); + let staged: Staged; + try { + staged = await stageContained(view, rootReal, relativePath, bytes, { makeDirectory: true }); + } catch (error) { + return { error: `The write did nothing to ${path}: ${directReason(error)}.` }; + } + try { + await directWriteTestSeam.afterStage?.([staged.path]); + await placedStill(staged); + // A write reads nothing first, so it replaces whatever is there. + checkUnchanged(staged, null); + if (branch !== undefined && this.getStore().currentBranch().name !== branch) { + await discard(staged); + return { error: BRANCH_LEFT, branchLeft: true }; + } + renameInto(staged); + } catch (error) { + await discard(staged); + return { error: `The write did nothing to ${path}: ${directReason(error)}.` }; + } + this.recordDirectWrite(mount, agentName); + return { note: linksNote(path, staged, 'write') }; + }); + } + + /** + * applyEdits on a direct mount, under the mount's turn, whose edits land + * on disk or don't happen. Each attempt reads every file from disk as it + * is now (directEditText), plans every edit against those bytes as on any + * mount (planTargetEdits), stages every file beside itself, checks every + * one unchanged since its read, and then renames each over its file in + * turn (direct-write.ts). A file another writer changed between its read + * and its check has the call read and planned again against its new + * text, three attempts in all (DIRECT_EDIT_ATTEMPTS), so edits that no + * longer apply to it, a file that changed at every attempt, or a file the + * next read refuses as any read would (now past the size limit, say) are + * refused, with nothing written. A rename that fails after others leaves + * those written and the rest as they were, and the answer names each + * file's outcome; nothing puts the written ones back. + */ + private async applyDirectEdits( + tool: EditTool, + mount: MountState, + targets: EditTarget[], + early: EditProblem[], + editCount: number, + maxChars: number, + agentName: string | undefined, + ): Promise { + const changedMeanwhile = new Set(); + return this.withMount(mount, async () => { + for (let attempt = 1; ; attempt++) { + const outcome = await this.attemptDirectEdits(tool, mount, targets, early, changedMeanwhile, editCount, maxChars, agentName); + if (outcome !== 'changed') return outcome; + if (attempt === DIRECT_EDIT_ATTEMPTS) { + const changed = targets.filter((target) => changedMeanwhile.has(target)).map((target) => target.path); + // Each attempt stops at the first file it finds changed, so these are + // the files some attempt found changed, at most one per attempt. + return { + success: false, + isError: true, + error: tool === 'edit' + ? `${changed[0]} kept changing on disk while this edit was being applied (${DIRECT_EDIT_ATTEMPTS} attempts), so nothing was written. Read it again before retrying.` + : `Another writer kept changing this call's files on disk while it was being applied: each of its ${DIRECT_EDIT_ATTEMPTS} attempts ` + + `found one changed since its read (${listed(changed)}), so nothing was written. Read ${changed.length === 1 ? 'it' : 'them'} again before retrying.`, + }; + } + } + }); + } + + /** + * One attempt of applyDirectEdits: its answer, or 'changed' when another + * writer changed a file since its read, with nothing written. `early` are + * the problems found before any file was read, which a refusal names with + * the rest. The caller holds the mount's turn. + */ + private async attemptDirectEdits( + tool: EditTool, + mount: MountState, + targets: EditTarget[], + early: EditProblem[], + changedMeanwhile: Set, + editCount: number, + maxChars: number, + agentName: string | undefined, + ): Promise { + const problems = [...early]; + const read = new Map(); + for (const target of targets) { + const got = await this.directEditText(mount, target.relativePath, target.path); + if ('changed' in got) { + changedMeanwhile.add(target); + return 'changed'; + } + if ('error' in got) problems.push({ kind: 'file', edits: target.edits, path: target.path, text: got.error, whole: true }); + else read.set(target, got); + } + // Two paths that reach one file, through a link this mount follows, or + // as two hard links (a disk that folds case has its two spellings made + // one path by the parse, short of a directory link on the way, past + // which they're refused here as one file named twice): each + // would be planned against the same text and renamed over the same file + // or split from it, so one's edits would be lost or kept apart from the + // other's. They're refused. One file read twice has one identity here + // unless a writer changed it between the reads, and the check before the + // renames then sees that and reads again. + const byIdentity = new Map(); + for (const [target, { identity }] of read) { + const key = `${identity.dev}:${identity.ino}:${identity.size}:${identity.mtimeNs}:${identity.ctimeNs}`; + const other = byIdentity.get(key); + if (!other) { + byIdentity.set(key, target); + continue; + } + problems.push({ + kind: 'file', + edits: [...other.edits, ...target.edits].sort((a, b) => a - b), + path: other.path, + text: `${other.path} and ${target.path} are one file, reached through a link or as two hard links; name it one way, so its edits are matched in one text`, + whole: true, + }); + } + const maxSize = mount.config.maxFileSize ?? DEFAULT_MAX_FILE_SIZE; + const hints = hintBudget(); + // As much as the largest file the call names holds, at least, so the + // budget is the same whatever order its files are planned in. + let largest = maxSize; + for (const { bytes } of read.values()) largest = Math.max(largest, bytes.length); + const counts = countBudget(largest); + const plans: Array<{ target: EditTarget; plan: FilePlan }> = []; + for (const target of targets) { + const got = read.get(target); + if (!got) continue; + const plan = planTargetEdits(tool, target, got.bytes, hints, maxSize, counts, problems); + if (plan !== null) plans.push({ target, plan }); + } + // A file read again for another writer's change says so, whichever way + // its edits went. + const reread = (target: EditTarget, verb: string): string | null => !changedMeanwhile.has(target) ? null + : `Disk's copy${tool === 'edit' ? '' : ` of ${target.path}`} changed while this call was being applied, so it was read again, ` + + `and ${tool === 'edit' ? 'the edit' : editList(target.edits)} ${verb} its new text.`; + const refuse = (found: EditProblem[]): ToolResult => ({ + success: false, + isError: true, + error: refusalText(tool, found, targets.map((target) => reread(target, 'looked in')).filter((note): note is string => note !== null), maxChars, 'wrote'), + }); + if (problems.length) return refuse(problems); + + const { view, rootReal } = await this.diskView(mount); + const staged: Array<{ target: EditTarget; staged: Staged }> = []; + const discardAll = async (from = 0): Promise => { + for (const { staged: file } of staged.slice(from)) await discard(file); + }; + // A refusal before any file is renamed says so where the call names several. + const nothingWritten = (target: EditTarget, error: unknown): ToolResult => refuse([{ + kind: 'file', + edits: target.edits, + path: target.path, + text: `${target.path}: ${directReason(error)}, so ${tool === 'edit' ? 'the edit' : editList(target.edits)} changed nothing.` + + (tool === 'edit_many' && plans.length > 1 ? ' No file was written.' : ''), + whole: true, + }]); + for (const { target, plan } of plans) { + try { + staged.push({ target, staged: await stageContained(view, rootReal, target.relativePath, plan.bytes) }); + } catch (error) { + await discardAll(); + if (error instanceof FileChanged) { + changedMeanwhile.add(target); + return 'changed'; + } + return nothingWritten(target, error); + } + } + try { + await directWriteTestSeam.afterStage?.(staged.map(({ staged: file }) => file.path)); + } catch (error) { + await discardAll(); + throw error; + } + for (const { target, staged: file } of staged) { + try { + await placedStill(file); + } catch (error) { + await discardAll(); + return nothingWritten(target, error); + } + } + // Every file is checked, then every file renamed, with no turn of the + // event loop between: a change to any of them before its check is seen + // before any is written, and one landing between its check and the last + // rename isn't (checkUnchanged). + for (const { target, staged: file } of staged) { + try { + checkUnchanged(file, read.get(target)!); + } catch (error) { + await discardAll(); + if (error instanceof FileChanged) { + changedMeanwhile.add(target); + return 'changed'; + } + return nothingWritten(target, error); + } + } + for (const [i, { target, staged: file }] of staged.entries()) { + try { + renameInto(file); + } catch (error) { + if (i === 0) { + await discardAll(); + return nothingWritten(target, error); + } + // Read with no turn of the event loop since the last rename. + this.recordDirectWrite(mount, agentName); + await discardAll(i); + return { success: false, isError: true, error: this.directEditsPartway(staged, i, error, maxChars) }; + } + } + this.recordDirectWrite(mount, agentName); + return { + success: true, + data: editReceipt(tool, staged.map(({ target, staged: file }) => ({ + path: target.path, + plan: plans.find((p) => p.target === target)!.plan, + notes: [reread(target, 'applied to'), linksNote(target.path, file, tool)].filter((note): note is string => note !== null), + modeKept: null, + })), editCount, maxChars), + }; + } + + /** + * The answer to an edit_many call on a direct mount whose rename of its + * `failed`th file failed after the ones before it were written: what was + * written, the failure, and that nothing puts them back, then each file's + * outcome, within `maxChars`, with the hard links a written one split + * from, as a full answer names them (linksNote). + */ + private directEditsPartway(staged: Array<{ target: EditTarget; staged: Staged }>, failed: number, error: unknown, maxChars: number): string { + const head = `The edit_many call wrote ${failed} of its ${staged.length} files before writing ${staged[failed]!.target.path} failed: ${directReason(error)}. ` + + "Nothing puts the written files back, so finish or revert them: in each written file, each of its edits replaced its oldString with its newString. Each file's outcome:"; + const parts = staged.map(({ target }, i) => `${target.path}: ${i < failed ? 'written' : i === failed ? 'not written, as it was (its write failed)' : 'not written, as it was'}`); + const notes = staged.slice(0, failed).map(({ target, staged: file }) => linksNote(target.path, file, 'edit_many')).filter((note): note is string => note !== null); + return fitted(head, parts, notes, maxChars, 'written'); + } + + /** + * A direct mount's file as an edit reads it: its bytes on disk as they are + * now, through the mount's boundary as read reaches them + * (openContainedFile), with the file's identity when the read began, which + * the write checks it against just before its rename. A file changed while + * it was read is `changed`, so the call reads it again. What the + * workspace doesn't show as text, it doesn't edit either: a file over the + * mount's size limit, and a binary one. + */ + private async directEditText( + mount: MountState, + relativePath: string, + path: string, + ): Promise<{ bytes: Buffer; identity: FileIdentity } | { error: string } | { changed: true }> { + const maxSize = mount.config.maxFileSize ?? DEFAULT_MAX_FILE_SIZE; + const over = `${path} is over the size limit (${maxSize} bytes), so the workspace doesn't edit it.`; + let opened: Awaited>; + try { + opened = await this.openContainedFile(mount, relativePath, path, (fileStat) => { + if (fileStat.size > maxSize) throw new Error(over); + }); + } catch (err) { + if (err instanceof WorkspaceReadError && err.code === 'changed') return { changed: true }; + return { error: containedRefusal(err, path) }; + } + try { + const before = identityOf(await opened.handle.stat({ bigint: true })); + const { bytes, more } = await readPrefix(opened.handle, maxSize, opened.fileStat.size); + if (more) return { error: over }; + if (looksBinary(bytes)) return { error: `${path} is binary, so the workspace doesn't edit it as text.` }; + if (!sameIdentity(before, identityOf(await opened.handle.stat({ bigint: true })))) return { changed: true }; + return { bytes, identity: before }; + } catch (err) { + const errno = errnoOf(err); + return { error: `Unable to read file: ${path}${errno ? ` (${errno})` : ''}` }; + } finally { + await opened.handle.close(); + } + } + + /** + * A direct mount's deletion: the file the read tools reach, by their + * refusals (openContainedFile), unlinked through the mount's boundary as + * a push's deletion unlinks one (unlinkContained). A followed link is the + * entry removed, as there. One that lands is the caller's latest write on + * the mount (recordDirectWrite). + */ + private async deleteDirect(mount: MountState, relativePath: string, path: string, agentName: string | undefined): Promise { + return this.withMount(mount, async (): Promise => { + // One root for the call, by its identity as the call starts: the + // open's checks and the unlink's own refuse any other directory there, + // in ROOT_MOVED's words, so a root replaced after this stat is refused + // rather than taken, but in the instant between the unlink's last + // check and the unlink, which Node can't close (effects.ts's header). + // One that can't be read now is refused here. + const disk = await this.diskView(mount); + let rootStat: BigIntStats; + try { + rootStat = await stat(disk.rootReal, { bigint: true }); + } catch (err) { + const code = errnoOf(err); + const why = code === 'ENOENT' || code === 'ENOTDIR' + ? `the mount root is gone (${code}), and calling again is refused the same way until whoever runs the host puts a directory back at the mount's path` + : code === 'EACCES' || code === 'EPERM' + ? `the mount root can't be read (${code}), and calling again is refused the same way until whoever runs the host makes it readable` + : `the mount root can't be read (${code ?? String(err)})`; + return { success: false, isError: true, error: `The delete did nothing to ${path}: ${why}.` }; + } + const view = { ...disk.view, rootIdentity: { dev: String(rootStat.dev), ino: String(rootStat.ino) } }; + const { rootReal } = disk; + try { + const opened = await this.openContainedFile(mount, relativePath, path); + await opened.handle.close(); + } catch (err) { + // The open's own words for a replaced root speak of reading. + if (err instanceof WorkspaceReadError && err.message.endsWith(DIRECT_ROOT_REPLACED)) { + return { success: false, isError: true, error: `The delete did nothing to ${path}: ${ROOT_MOVED}.` }; + } + return { success: false, isError: true, error: containedRefusal(err, path) }; + } + // The unlink is awaited, unlike a write's rename, so where the head + // was just before it says whether the head moved while it ran. + const since = this.positionNow(); + try { + await unlinkContained(view, rootReal, relativePath, { kind: 'any' }); + } catch (error) { + // A failure after the unlink reached the file's entry can't say + // whether the file is gone, so it's taken as the agent's write too. + if (error instanceof EffectFailed && error.touched) { + this.recordDirectWrite(mount, agentName, since); + return { success: false, isError: true, error: `The delete of ${path} failed after it reached the file, so the file may be gone: ${directReason(error)}. ls shows whether it is.` }; + } + return { success: false, isError: true, error: `The delete did nothing to ${path}: ${directReason(error)}.` }; + } + this.recordDirectWrite(mount, agentName, since); + return { success: true, data: { path, deleted: true }, keepWhenWithheld: ['path', 'deleted'] }; + }); + } + + /** + * After a direct write lands: the agent's latest write on the mount, P, + * moves to it, where the agent's history holds P or the notice of its + * departure from P, N (historyHoldsWrites), and N clears. Otherwise P + * stays where it is: the head moved away from it under the call's turn, + * and the next start's notice covers what it left (headMoveNotice). P is + * the branch and the store's head as it is now, read with no turn of the + * event loop since the write, so no record that lands after the write is + * taken for one before it. A deletion, whose unlink is awaited, passes + * where the head was just before its unlink (`since`): where the branch + * changed meanwhile, the head moved under the call, and P is set there, + * so the next start judges the deletion by the branch its turn was on, + * unless the history doesn't hold P already, whose notice is owed. A write no agent + * made tells no one. The store's reads here come after the write landed, + * so one that throws is logged and P stays, as for a history that can't + * be read, and the write still answers that it landed. + */ + private recordDirectWrite(mount: MountState, agentName: string | undefined, since?: DirectPosition | null): void { + if (agentName === undefined) return; + try { + const store = this.getStore(); + let heads = this.directHeads.get(mount.config.name); + if (!heads) this.directHeads.set(mount.config.name, (heads = new Map())); + const head = heads.get(agentName); + const branchId = store.currentBranch().id; + if (since && since.branchId !== branchId) { + if (!head || this.historyHoldsWrites(agentName, head) === true) heads.set(agentName, { ...since, notice: null }); + return; + } + if (head && this.historyHoldsWrites(agentName, head) !== true) return; + heads.set(agentName, { branchId, after: store.currentSequence(), notice: null }); + } catch (err) { + console.error('[workspace] could not record a direct write\'s position:', err); + } + } + + /** The store's branch and head now, or null where the store can't say. */ + private positionNow(): DirectPosition | null { + try { + const store = this.getStore(); + return { branchId: store.currentBranch().id, after: store.currentSequence() }; + } catch { + return null; + } + } + + /** + * Whether an agent's history holds its latest direct write on a mount + * (holdsPosition), or the notice of its departure from it, judged by the + * notice's own presence there, through the lookup the framework hands the + * module (useHistory); null where that lookup threw, which is logged. A + * history that can't be read decides nothing: a write takes it as not + * held, so its position stays and the notice stays owed, and a start + * gives no notice, which may not be true of a history it can't read, and + * leaves the next start to decide (headMoveNotice). + */ + private historyHoldsWrites(agentName: string, head: DirectHead): boolean | null { + if (this.holdsPosition(head)) return true; + if (head.notice === null) return false; + try { + return this.historyLookup?.(agentName, head.notice) === true; + } catch (err) { + console.error('[workspace] history lookup for the head-move notice failed:', err); + return null; + } + } + + /** + * Whether the store's current branch holds what came before a direct + * write: it's the write's branch, or continues it with every fork point on + * the way after `after`, the store's head when the write landed. A fork at + * exactly `after` doesn't: an undo forks at its turn's sequenceBefore, + * which is the head at the write when nothing was recorded in the turn + * before it. A missing branch or one without fork metadata holds nothing, + * as for the materialize guard (isLinearContinuation). + */ + private holdsPosition({ branchId, after }: DirectHead): boolean { + const store = this.getStore(); + const current = store.currentBranch(); + if (current.id === branchId) return true; + const byId = new Map(store.listBranches().map((b) => [b.id, b])); + const visited = new Set(); + let cursor = byId.get(current.id); + while (cursor && !visited.has(cursor.id)) { + visited.add(cursor.id); + if (cursor.id === branchId) return true; + if (cursor.parentId === undefined || cursor.branchPoint === undefined || cursor.branchPoint <= after) return false; + cursor = byId.get(cursor.parentId); + } + return false; + } + + /** + * The framework's check at every start of an agent's turn, opening, + * continuing or resuming: the notice due for each mount the agent wrote + * direct where its history holds neither its latest write there nor the + * notice of its departure from it (historyHoldsWrites), with those + * mounts' names; or null, as on a head that hasn't moved. The framework + * stores the text in the agent's history before the turn's compile and + * hands its message id back (headMoveNoticeStored). + */ + headMoveNotice(agentName: string): { text: string; mounts: string[] } | null { + if (!this.store) return null; + const due: string[] = []; + // Every mount where the agent wrote direct, whether it runs direct this + // session or as 'chronicle': the writes are on disk either way. + for (const name of this.mounts.keys()) { + const head = this.directHeads.get(name)?.get(agentName); + if (!head) continue; + const held = this.historyHoldsWrites(agentName, head); + // A history that can't be read decides nothing at this start. + if (held === null) return null; + if (!held) due.push(name); + } + return due.length ? { text: headMoveText(due), mounts: due } : null; + } + + /** The id of the message headMoveNotice's text was stored as, N, for each mount it named. */ + headMoveNoticeStored(agentName: string, mounts: readonly string[], messageId: string): void { + for (const name of mounts) { + const head = this.directHeads.get(name)?.get(agentName); + if (head) head.notice = messageId; + } + } + + /** + * How the module asks whether an agent's history holds a message, as the + * framework judges its own notices (`cm.getMessage(id) !== null`): handed + * over once, when the module is added. Without it, no notice is held. + */ + useHistory(lookup: (agentName: string, messageId: string) => boolean): void { + this.historyLookup = lookup; + } + + private async handleDelete(input: DeleteInput, callerAgentName?: string): Promise { const { mount, relativePath, path } = await this.parseFilePath(input.path); input = { ...input, path }; if (mount.config.mode === 'read-only') { return { success: false, error: `Mount "${mount.config.name}" is read-only`, isError: true }; } + if (mount.direct) return this.deleteDirect(mount, relativePath, input.path, callerAgentName); const store = this.getStore(); let committed = false; @@ -4458,7 +5107,9 @@ export class WorkspaceModule implements Module { const mounts = [...this.mounts.entries()].map(([name, m]) => ({ name, path: m.config.path, - mode: m.config.mode, + // A mount configured direct says its mode where its config names + // none, so a writable one shows as writable. + mode: m.config.backend === 'direct' ? m.config.mode ?? 'read-write' : m.config.mode, ...(m.config.backend === 'direct' ? { backend: m.direct ? 'direct' : 'chronicle', ignore: this.ignoreOf(m) } : {}), ...(m.config.include ? { include: [...m.config.include] } : {}), })); @@ -4922,7 +5573,7 @@ export class WorkspaceModule implements Module { if (mount.direct) { // A direct mount keeps nothing to count: its kind, and the ignore // list its walks go by, whose default no config shows. - status[name] = { path: mount.config.path, mode: mount.config.mode, backend: 'direct', ignore: this.ignoreOf(mount) }; + status[name] = { path: mount.config.path, mode: mount.config.mode ?? 'read-write', backend: 'direct', ignore: this.ignoreOf(mount) }; continue; } // What the workspace's history holds outside the mount's include isn't the mount's, @@ -4937,7 +5588,7 @@ export class WorkspaceModule implements Module { const currentBranch = store.currentBranch(); status[name] = { path: mount.config.path, - mode: mount.config.mode, + mode: mount.config.backend === 'direct' ? mount.config.mode ?? 'read-write' : mount.config.mode, // A mount configured direct runs as 'chronicle' this session // (chooseKinds), by the direct kind's ignore list where its config // names none. @@ -5109,8 +5760,9 @@ export class WorkspaceModule implements Module { } mountsToMaterialize = [{ name: input.mount, mount: m }]; } else { + // Not a mount running direct, whose writes are on disk already. mountsToMaterialize = [...this.mounts.entries()] - .filter(([, m]) => m.config.mode === 'read-write') + .filter(([, m]) => m.config.mode === 'read-write' && !m.direct) .map(([name, mount]) => ({ name, mount })); } @@ -5131,6 +5783,12 @@ export class WorkspaceModule implements Module { let proceeded = 0; for (const { name, mount } of mountsToMaterialize) { + // Only a writable direct mount the caller named gets here: it keeps no + // tree to push. A read-only one is named as any read-only mount is. + if (mount.direct && mount.config.mode !== 'read-only') { + blocked.push({ mount: name, reason: DIRECT_MATERIALIZE_SKIPPED }); + continue; + } if (mount.config.mode === 'read-only') { // Only a mount the caller named gets here read-only (a bare // materialize takes the read-write ones): say why nothing was written. @@ -5243,7 +5901,8 @@ export class WorkspaceModule implements Module { if (!this.running()) return []; const store = this.getStore(); const mount = this.mounts.get(mountName); - if (!mount || mount.config.mode === 'read-only') return []; + // A direct mount keeps no tree to push: disk is its only copy. + if (!mount || mount.config.mode === 'read-only' || mount.direct) return []; // Reset tracking — we're deliberately materializing on the new branch mount.lastMaterializedBranchId = null; @@ -5738,14 +6397,11 @@ function watchOf(config: MountConfig): 'always' | 'on-agent-action' | 'never' { /** * What a direct mount's config sets that would never act there, each with - * the way out (MountConfig.backend): a direct mount is read-only, and has no - * watcher to tune, no change event to wake on and no push to make. + * the way out (MountConfig.backend): a direct mount has no watcher to tune, + * no change event to wake on and no push to make. Either mode is its own. */ function directRefusals(mount: MountConfig): string[] { const refused: string[] = []; - if (mount.mode !== 'read-only') { - refused.push(mount.mode === undefined ? "mode must be 'read-only' (left out, it's 'read-write')" : `mode must be 'read-only', not ${shownValue(mount.mode)}`); - } if (mount.watch !== undefined && mount.watch !== 'never') refused.push(`watch ${shownValue(mount.watch)} would watch nothing (leave it out, or set 'never')`); for (const key of ['watchDebounceMs', 'watchRootPollMs'] as const) { if (mount[key] !== undefined) refused.push(`${key} would tune no watcher (leave it out)`); @@ -5756,6 +6412,124 @@ function directRefusals(mount: MountConfig): string[] { return refused; } +/** + * An agent's latest direct write on a mount, P: the store's branch, and its + * head when the write landed (`after`); and the id of the message that last + * told the agent its history no longer holds P, N, or null. + */ +interface DirectHead { + branchId: string; + after: number; + notice: string | null; +} + +/** A store position: a branch and its head. */ +type DirectPosition = Pick; + +/** The global record type directHeads is saved as: a checkpoint at a clean stop that changed it. */ +const DIRECT_HEADS_RECORD_TYPE = 'workspace/direct-heads'; + +/** One mount's positions at one root (resolved, as DiskAgreement keeps a root), by agent, with N only where it's set. */ +interface DirectHeadsEntry { + mount: string; + root: string; + agents: Record; +} + +/** directHeads as saved: an entry per mount and root its agents wrote direct under. */ +type DirectHeadsSnapshot = DirectHeadsEntry[]; + +/** + * A saved record as saveDirectHeads writes it, or a throw naming what isn't: + * a list of entries, each a mount's name, root and agents, and each agent's + * branch, head and, where set, notice. Checked whole, carried entries + * included, so a record of another shape is refused at load rather than + * carried into the next save. + */ +function checkedHeads(value: unknown): DirectHeadsSnapshot { + const isObject = (v: unknown): v is Record => typeof v === 'object' && v !== null && !Array.isArray(v); + if (!Array.isArray(value)) throw new Error("the record isn't of the shape saved"); + for (const saved of value as unknown[]) { + if (!isObject(saved) || typeof saved.mount !== 'string' || typeof saved.root !== 'string' || !isObject(saved.agents)) { + throw new Error(`an entry isn't of the shape saved: ${JSON.stringify(saved)?.slice(0, 200)}`); + } + for (const [agent, head] of Object.entries(saved.agents)) { + if (!isObject(head) || typeof head.branchId !== 'string' || typeof head.after !== 'number' + || (head.notice !== undefined && typeof head.notice !== 'string')) { + throw new Error(`agent ${JSON.stringify(agent)}'s position on mount ${JSON.stringify(saved.mount)} isn't of the shape saved`); + } + } + } + return value as DirectHeadsSnapshot; +} + +/** A saved record's content, whatever order its mounts and agents were written in, to tell whether a save would change it. */ +function canonicalHeads(snapshot: DirectHeadsSnapshot): string { + return JSON.stringify(snapshot.map(({ mount, root, agents }) => [mount, root, Object.keys(agents).sort().map((agent) => { + const { branchId, after, notice } = agents[agent]!; + return [agent, branchId, after, notice ?? null]; + })]).sort((a, b) => { + const [x, y] = [JSON.stringify(a.slice(0, 2)), JSON.stringify(b.slice(0, 2))]; + return x < y ? -1 : x > y ? 1 : 0; + })); +} + +/** + * How many times a direct edit reads and plans a file another writer keeps + * changing: enough to land on top of a writer that changes it now and then, + * such as a formatter or a build, and bounded for one that never stops. + */ +const DIRECT_EDIT_ATTEMPTS = 3; + +/** Why a direct write, edit or deletion did nothing, for its answer: a refusal's words, or any other failure's message. */ +function directReason(error: unknown): string { + if (error instanceof FileChanged) return 'it changed on disk while this call was being applied'; + // A deletion's root check (unlinkContained) has a store-backed mount's + // words, whose remedy, a sync with acceptRoot, a direct mount has no use for. + if (error instanceof Error && error.message === ROOT_REPLACED) return ROOT_MOVED; + return error instanceof Error ? error.message : String(error); +} + +/** "a", "a and b", or "a, b and c". */ +function listed(items: readonly string[]): string { + return items.length < 2 ? items.join('') : `${items.slice(0, -1).join(', ')} and ${items.at(-1)}`; +} + +/** + * The note for a direct write that gave a hard-linked path a file of its + * own: the other links keep the old text. Null for a file with none. + */ +function linksNote(path: string, staged: Staged, tool: 'write' | EditTool): string | null { + const n = staged.otherLinks; + if (n <= 0) return null; + return `${path} had ${n === 1 ? 'another hard link' : `${n} other hard links`}, which ${n === 1 ? 'keeps' : 'keep'} its old text: ` + + `the ${tool === 'write' ? 'write' : tool === 'edit' ? 'edit' : 'call'} gave this path a file of its own.`; +} + +/** + * A refusal of openContainedFile's as a direct tool gives it: its message, + * and for one that names only the path, its errno too, as a direct read's + * answer gives it (bytesOnDisk). + */ +function containedRefusal(err: unknown, path: string): string { + const bare = err instanceof WorkspaceReadError && err.errno !== undefined && err.message === `Unable to read file: ${path}`; + return bare ? `${err.message} (${err.errno})` : err instanceof Error ? err.message : String(err); +} + +/** + * The notice that writes an agent made direct remain on disk from + * turns the agent's history no longer holds. `git status` stands beside + * `git diff` because a departed turn's new file is untracked, which `git + * diff` doesn't show; and the warning says "unstaged", since `git checkout + * -- ` puts back a file's staged text, so a staged change survives it. + */ +function headMoveText(mounts: readonly string[]): string { + const one = mounts.length === 1; + return `[workspace] Writes to ${listed(mounts.map((name) => `\`${name}\``))} from turns the history no longer holds are still on disk. ` + + `If ${one ? "it's a git checkout" : "they're git checkouts"}, \`git status\` and \`git diff\` show them; ` + + 'a `git checkout` of a file drops every unstaged change in it, including the ones the history still holds.'; +} + /** A change commitToolChanges makes to one path's tree entry. */ type ToolChange = { kind: 'set'; blobHash: string; size: number; mode?: number } | { kind: 'remove'; previousHash: string }; @@ -5770,6 +6544,64 @@ interface EditTarget { specs: EditSpec[]; } +/** + * One file's edits planned against its bytes, as both kinds of mount plan + * them: its text decoded, every edit matched in it (edit-plan.ts), and the + * result held to the mount's size limit. Each problem found goes into + * `problems`, and then, as for a file named only by edits whose own input + * was refused, there's no plan (null). + */ +function planTargetEdits( + tool: EditTool, + target: EditTarget, + bytes: Buffer, + hints: ReturnType, + maxSize: number, + counts: ReturnType, + problems: EditProblem[], +): FilePlan | null { + // Decoding would turn each invalid sequence into U+FFFD, and writing + // the text back would change bytes the edits never named. + const file = decodeForEdit(bytes); + if (!file) { + problems.push({ + kind: 'file', + edits: target.edits, + path: target.path, + text: `${target.path} isn't valid UTF-8, so editing it would change bytes outside the text it replaces. ` + + 'Read shows each invalid byte sequence as U+FFFD, and a write replaces the whole file as UTF-8, so writing back what read shows puts U+FFFD in their place.', + whole: false, + }); + return null; + } + if (!target.specs.length) return null; + const planned = planFile(file, bytes, target.specs, hints, maxSize, counts); + if ('problems' in planned) { + // One at a time: a call of many edits can have more problems + // than a spread passes as arguments. + for (const problem of planned.problems) problems.push({ ...problem, path: target.path }); + return null; + } + // The planner refuses a file its counts take past the limit before + // building it, from the least its bytes can be; one it built past + // the limit, as its CRs or multi-byte characters can take it, is + // refused here. + if ('tooLarge' in planned || planned.plan.bytes.length > maxSize) { + const size = 'tooLarge' in planned ? `at least ${planned.tooLarge.toLocaleString('en-US')}` : planned.plan.bytes.length.toLocaleString('en-US'); + problems.push({ + kind: 'file', + edits: target.edits, + path: target.path, + text: `${target.path} would be ${size} bytes with ${tool === 'edit' ? 'this edit' : editList(target.edits)}, ` + + `past the ${maxSize.toLocaleString('en-US')}-byte limit this mount sets on a file, so the workspace couldn't keep it. ` + + "An edit that leaves the file within the limit applies. The limit is the mount's maxFileSize setting, which only whoever runs the host can change.", + whole: false, + }); + return null; + } + return planned.plan; +} + /** * Whether a tree entry is what a change leaves: for a set, an entry with its * blob, size and mode; for a removal, none. diff --git a/src/modules/workspace/types.ts b/src/modules/workspace/types.ts index b1c97fe53..019eddad8 100644 --- a/src/modules/workspace/types.ts +++ b/src/modules/workspace/types.ts @@ -23,21 +23,42 @@ export interface MountConfig { * three-way rule as its listings, searches, syncs and scans reach them. * A read of a path the store doesn't hold takes in a new file there; * otherwise it reads the store's copy. - * - 'direct': for a checkout an agent reads, not one it changes. Every - * read, listing and search is disk's own copy as it is when it runs, - * and none of its files go into the store. No scan or watcher + * - 'direct': for a checkout, such as a git one, whose history is its + * own. Every read, listing and search is disk's own copy as it is when + * it runs, and none of its files go into the store. No scan or watcher * runs, so no call waits behind one, and `sync`, `materialize` and the - * watch settings have nothing to act on. Only with `mode: 'read-only'`, - * and not with `watch` other than 'never', `watchDebounceMs`, - * `watchRootPollMs`, a `wakeOnChange` that would wake (true, or a - * non-empty list) or `autoMaterialize: true`: each of those would never - * act, and the workspace refuses the mount. Its `ignore` + * watch settings have nothing to act on. Not with `watch` other than + * 'never', `watchDebounceMs`, `watchRootPollMs`, a `wakeOnChange` that + * would wake (true, or a non-empty list) or `autoMaterialize: true`: + * each of those would never act, and the workspace refuses the mount. + * Writable, as any mount with no `mode` is, its `write`, `edit`, + * `edit_many` and `delete` land on disk when they answer, or don't + * happen: a file is staged beside itself in `.workspace-staged-` and + * renamed over itself, keeping its mode, so no reader sees half of it; + * an edit is applied to disk's text as it is now, and again on top of a + * change another writer makes meanwhile; and `edit_many` checks every + * edit before writing its files one by one, naming each file's outcome + * if a write fails partway. A file with other hard links gets a file of + * its own at its path, and the others keep the old text; a file this + * process may not write is refused. Nothing is synced, so a power loss + * can lose a write its answer called landed. An undo or other move of + * the store's head leaves the files as they are, and at the agent's next + * start the framework tells it once that writes from turns its history + * no longer holds are still on disk. Where each agent's latest write + * lies is saved at a clean stop, in a record no branch switch rewinds, + * so a restart on another branch still tells it, and after a crash a + * move away from the writes since then isn't told. The framework's own + * files, its + * tool-result spills, saved payloads and fetched references, never go to + * a mount configured direct. Its `ignore` * has a default (see there). Paths, links, its `include`, other mounts' * directories, size limits and the walk cap hold as on a 'chronicle' - * mount. Its tools read beneath whatever directory is at its - * root, with no check against the one a 'chronicle' run of it last - * agreed with. Where that root now lies elsewhere than the last start - * placed it, each call nests in it, as a start would now, every mount + * mount. Its tools read and write beneath whatever directory is at its + * root when each call starts, with no check against the one a + * 'chronicle' run of it last agreed with, so a root repointed through a + * link is read and written where it leads now. Where that root now lies + * elsewhere than the last start placed it, each call nests in it, as a + * start would now, every mount * configured beneath its path, where that path leads beneath its root, * and every other mount whose root, as that start placed it, lies * beneath where its root lies now, or whose configured path lies diff --git a/test/mcp-modern-framework.test.ts b/test/mcp-modern-framework.test.ts index c42997d8e..fe4de4919 100644 --- a/test/mcp-modern-framework.test.ts +++ b/test/mcp-modern-framework.test.ts @@ -147,6 +147,28 @@ test('model path: a payload the host cannot retain is reported as an incomplete }); }); +test("model path: where the only writable mount is direct, a payload's stub says no mount that keeps its files in the store can save it", async () => { + const code = mkdtempSync(join(tmpdir(), 'modern-direct-')); + const workspace = new WorkspaceModule({ mounts: [{ name: 'code', path: code, mode: 'read-write', backend: 'direct' }] }); + try { + await withModern(async ({ framework, membrane }) => { + workspace.initStore(framework.getStore()); + framework.start(); + membrane.pushResponse(createMockResponse([ + { type: 'tool_use', id: 'toolu_01MODERNMEDIA00000000001', name: 'mcpl--modern--media', input: {} }, + ] as never, 'tool_use')); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Done.' }])); + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go', metadata: {} } as never); + await waitFor('the tool round', () => (membrane.lastStream?.receivedToolResults.length ?? 0) >= 1); + const [delivered] = membrane.lastStream!.receivedToolResults[0] as Array<{ content: unknown; isError: boolean }>; + const text = JSON.stringify(delivered!.content); + assert.match(text, /\[image: image\/png, 8 B\. Not retained: a failed result is read as its text alone, and no workspace mount that keeps its files in the store can save it\]/); + }, [], [workspace]); + } finally { + rmSync(code, { recursive: true, force: true }); + } +}); + test('model path: the result reaches the model, structured-only as JSON', async () => { await withModern(async ({ framework, membrane }) => { framework.start(); diff --git a/test/workspace-direct-notice.test.ts b/test/workspace-direct-notice.test.ts new file mode 100644 index 000000000..dcb3d643e --- /dev/null +++ b/test/workspace-direct-notice.test.ts @@ -0,0 +1,643 @@ +/** + * The head-move notice on a writable direct mount: after the store's head + * moves away from an agent's latest direct write, through undoLastTurn or a + * context-manager branch switch as connectome-host's /undo makes one, the + * framework stores, at the agent's next start, one notice that those writes + * are still on disk, and that start's request carries it. Whether it was + * given is judged by the notice's own presence in the history. + */ + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { JsStore } from '@animalabs/chronicle'; +import type { NormalizedRequest, NormalizedResponse, YieldingStream } from '@animalabs/membrane'; +import { AgentFramework } from '../src/index.js'; +import { WorkspaceModule } from '../src/modules/workspace/index.js'; +import { directWriteTestSeam } from '../src/modules/workspace/direct-write.js'; +import type { MountConfig } from '../src/modules/workspace/types.js'; +import type { Module, ModuleContext } from '../src/types/module.js'; +import type { ProcessEvent, ToolCall, ToolDefinition, ToolResult } from '../src/types/events.js'; +import type { TraceEvent } from '../src/types/trace.js'; +import { MockMembrane, MockYieldingStream, createMockResponse } from './helpers/mock-membrane.js'; + +const NOTICE = 'from turns the history no longer holds are still on disk'; +const TEXT = "[workspace] Writes to `ref` from turns the history no longer holds are still on disk. If it's a git checkout, " + + '`git status` and `git diff` show them; a `git checkout` of a file drops every unstaged change in it, including the ones the history still holds.'; + +/** A turn's script: its responses, one per round, and the tool round after which its stream fails, so the framework's retry starts again. */ +type Script = { responses: NormalizedResponse[]; failAfterRound?: number }; + +/** A stream that fails once its tools have answered `failAfterRound` rounds. */ +class FailingStream extends MockYieldingStream { + constructor(responses: NormalizedResponse[], private readonly failAfterRound?: number) { + super(responses); + } + + override provideToolResults(results: unknown[], options?: Parameters[1]): void { + super.provideToolResults(results, options); + if (this.toolDepth === this.failAfterRound) { + (this as unknown as { events: unknown[] }).events.push({ type: 'error', error: new Error('socket hang up') }); + } + } +} + +/** Each agent's turns in order, by its system prompt (`zz-agent-`), wherever the request carries it. */ +class ScriptedMembrane extends MockMembrane { + readonly scripts = new Map(); + + override streamYielding(request: NormalizedRequest): YieldingStream { + this.calls.push(request); + const agent = /zz-agent-(\w+)/.exec(JSON.stringify(request))?.[1] ?? ''; + const script = this.scripts.get(agent)?.shift() ?? { responses: [createMockResponse([{ type: 'text', text: 'ok' }])] }; + const stream = new FailingStream(script.responses, script.failAfterRound); + this.lastStream = stream; + return stream as unknown as YieldingStream; + } +} + +/** A tool that moves the store's head the way the test says, called from inside a turn. */ +class Mover implements Module { + readonly name = 'mover'; + move: () => Promise = async () => {}; + async start(_ctx: ModuleContext): Promise {} + async stop(): Promise {} + getTools(): ToolDefinition[] { + return [{ name: 'move', description: 'move the head', inputSchema: { type: 'object', properties: {} } }]; + } + async handleToolCall(_call: ToolCall): Promise { + await this.move(); + return { success: true, data: { moved: true } }; + } + async onProcess(_event: ProcessEvent) { + return {}; + } +} + +const say = (text: string) => createMockResponse([{ type: 'text', text }]); +let toolIds = 0; +const use = (name: string, input: Record) => createMockResponse([{ type: 'tool_use', id: `t${++toolIds}`, name, input }], 'tool_use'); +const write = (path: string, content: string) => use('workspace--write', { path: `ref/${path}`, content }); + +type Cm = { + addMessage: (...args: unknown[]) => string; + getAllMessages(): Array<{ id: string; metadata?: Record }>; + branchAt(id: string, name: string): string; + switchBranch(name: string): Promise; +}; + +class Harness { + readonly dir = mkdtempSync(join(tmpdir(), 'af-ws-direct-notice-')); + readonly work = join(this.dir, 'work'); + readonly membrane = new ScriptedMembrane(); + readonly mover = new Mover(); + framework!: AgentFramework; + workspace!: WorkspaceModule; + traces: TraceEvent[] = []; + private users = 0; + + constructor(readonly agents: string[] = ['scout']) { + mkdirSync(this.work, { recursive: true }); + } + + async open(): Promise { + this.workspace = new WorkspaceModule({ mounts: [{ name: 'ref', path: this.work, backend: 'direct' } as MountConfig] }); + this.framework = await AgentFramework.create({ + storePath: join(this.dir, 'store'), + membrane: this.membrane.asMembrane(), + agents: this.agents.map((name) => ({ name, model: 'test-model', systemPrompt: `zz-agent-${name}` })), + modules: [this.workspace, this.mover], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + errorPolicy: { maxRetries: 1, onInferenceError: (_error: Error, _agent: string, attempt: number) => (attempt < 1 ? { retry: true, delayMs: 0 } : { retry: false }) }, + }); + // As a host gives the workspace its store (connectome-host's index.ts). + this.workspace.initStore(this.framework.getStore()); + this.traces = []; + this.framework.onTrace((event) => this.traces.push(event)); + this.framework.start(); + } + + async close(): Promise { + await this.framework.stop(); + } + + cm(agent = 'scout'): Cm { + return this.framework.getAgent(agent)!.getContextManager() as unknown as Cm; + } + + /** + * One turn of `agent`'s, with its scripts, and the requests it made; + * started by a user message unless `user` is false, for the request's + * `reason` (a resuming one, such as 'context_budget_restart', continues + * the agent's logical turn). + */ + async turn(scripts: Script[] | NormalizedResponse[], agent = 'scout', { user = true, reason = 'zz' }: { user?: boolean; reason?: string } = {}): Promise { + const list = (scripts as Array