fix(workspace): listings tell the truth about disk, drafts and conflicts

changelog.d/materialize-freshness-guard.fixed.md

@@ -1 +1 @@
-- **`materialize` no longer silently overwrites disk files changed by another writer** (#109). Before writing, each file's disk hash is compared against what the workspace layer last materialized; a mismatch means a shell/tool edit happened since, and the path is refused and listed in the result (`skipped`, with the remedy) instead of silently reverted. A disk copy that exists but cannot be read (e.g. a write-only file) is refused the same way, with the filesystem error in the reason, rather than treated as absent. `force: true` remains the deliberate override. Refused paths stay owed to every later materialize (and count in `status.pendingChanges`) without holding back the sequence the branch guard reads. `sync` re-pins the baseline when it adopts a disk copy, so the agent's next edit to that file is not refused. Baselines and refused paths persist across restarts, and a first materialize that finds every file already in place still pins the branch. Files whose disk copy already matches the target bytes are left untouched (mtime preserved). The post-undo/redo `_config` restore keeps its overwrite semantics.
+- **`materialize` no longer silently overwrites disk files changed by another writer** (#109). A file disk changed since it last agreed with the workspace is refused and listed in the result (`skipped`, with the remedy) instead of silently reverted. So is a disk copy that exists but can't be read (e.g. a write-only file), with the filesystem error in the reason, rather than treated as absent. `force: true` remains the deliberate override. A refused workspace edit stays owed: every later materialize takes it up again, and it counts in `status.pendingChanges`. A file the workspace hasn't changed, whose disk copy changed, isn't owed: a materialize naming it refuses it, and one without a path leaves it, for a `sync` of the path (take disk) or a `materialize` naming it with `force` (take the workspace). What disk last agreed with is kept as evidence that survives restarts and branch switches (see the truthful-listings entry), and a `sync` that takes a disk copy records that agreement, so the agent's next edit to that file isn't refused. A first materialize that finds every file already in place still pins the branch, and a file whose disk copy already matches the target bytes is left untouched (mtime preserved). The post-undo/redo `_config` restore keeps its overwrite semantics.

changelog.d/tool-batch-hook.added.md

@@ -0,0 +1,15 @@
+- **`Module.onToolBatchComplete(agentName)`**: called once an agent's tool batch completes,
+  when the round's last pending result arrives and before it is provided. It therefore runs
+  before anything that continues the turn: re-inference, stream resume or the end of the turn.
+  The framework awaits every module's hook in parallel, each bounded at 30 s and fail-open; a
+  hook that throws (before returning its promise or after) or times out is traced as
+  `module:batch_hook_failed` and the round goes on.
+  The agent's state is checked again afterwards, so a result whose turn was cancelled or
+  replaced during the hook is dropped (traced as `tool:result_dropped`). A hook still running
+  from an earlier batch, past its bound, is skipped rather than called again. The hook is
+  awaited in the framework's event loop, so a slow one holds every agent's event processing,
+  not only the batch's agent. Script-inner tool calls don't trigger it, nor does a batch whose
+  error results the framework fills in after its stream ended before its tools answered.
+- Workspace: `materialize` accepts `applyDeletions`, `WorkspaceConfig` accepts
+  `agentActionScanDeadlineMs` (default 20000, kept below the hook's 30 s bound), and
+  `workspace:deleted` events carry `conflicts` when a disk deletion met a newer workspace edit.

changelog.d/workspace-listing-shapes.breaking.md

@@ -0,0 +1,18 @@
+- **Workspace tool-output consumers:** `glob` returns `matches` as objects,
+  `{ path, state, size?, mimeType?, conflict?, note? }`, instead of path strings. `sync` reports
+  `conflicts` as `{ path, kind, diskCopy }`. `sync` without a path no longer replaces a workspace
+  draft with a differing disk copy: the draft stays, and where disk changed too, the path is a
+  conflict until a `sync` of that path or a forced `materialize` resolves it, or the two converge.
+  For an `unknown-provenance` or `store-origin-collision` conflict, the forced `materialize` must
+  name the path: one without a path leaves it. `ls`, `grep` and `sync` otherwise only add fields
+  (each `ls` file entry's `state`; grep's `state`, `version`, `skipped` and `incomplete`; a path
+  sync's `discarded`).
+- **Hosts and module authors reading workspace internals:** `MountState` no longer has
+  `materializedHashes` or `refusedPaths`. Disk-agreement evidence lives in `workspace/disk-agreement`
+  chronicle records, and branch-local intent in each mount's `workspace/<mount>/intent` tree state.
+  The module's saved state carries neither field, so a build from #169's branch started on a
+  store this release wrote begins with no freshness baselines.
+- **Prompt cache:** the descriptions of the workspace's `ls`, `glob`, `grep`, `sync`,
+  `materialize` and `delete` tools change, and `sync` and `materialize` take new parameters, so
+  the first request after the upgrade that carries those tools' definitions misses the cache from
+  its tool block on.

changelog.d/workspace-truthful-listings.fixed.md

@@ -0,0 +1,106 @@
+- **Workspace listings tell the truth about disk, and no scan discards a workspace edit.** `ls`,
+  `glob` and `grep` check their scope against disk before answering, under one rule over disk, the
+  workspace store and what disk last agreed with. Files created by a shell show up, including
+  binary and oversize ones, listed as `disk-only` with their size and, for images, their type.
+  A file deleted by a shell leaves the workspace on every scan path (a watcher event, `sync`, a
+  listing, the scan after a restart, and a watcher reattach) where the workspace held what disk
+  last agreed with; a workspace edit of it stays, as a `deleted-on-disk` conflict. Each file entry
+  carries a state:
+  `synced`, `workspace-draft`, `workspace-deleted`, `not-in-branch`, `disk-only`, `conflict` (with
+  its kind) or `disk-missing-provenance-unknown`. Where disk couldn't be checked, the state is
+  `unverified`, and `incomplete` names the unreadable, ignored or capped regions. A pending
+  workspace edit is never overwritten by a scan, a watcher event or a lazy read. When disk and
+  the workspace both changed, the path becomes a conflict that stays until a `sync` of that path
+  (take disk), a forced `materialize` (take the workspace) or convergence. A text disk version is
+  kept in the store; a binary or oversize one is referenced, and the listing says if it changed
+  since. `grep` labels a kept disk version `recorded-disk` once disk has changed since, and lists
+  a conflict's disk side it couldn't search as skipped. A `sync` of a path lists under
+  `discarded` each workspace change it gave up for disk's state.
+- **A mount root that isn't the directory disk last agreed with is unavailable, not empty.** The
+  workspace records the root's identity (its device and inode) with the first evidence it
+  gathers there. When the drive under a read-write mount is unmounted, the watcher recreates its
+  mountpoint as an empty directory; that root is listed as unavailable (`incomplete`, saying what
+  to do), so no scan reads its emptiness as every file deleted, and no `materialize`, forced or
+  not, writes into it. When the drive comes back at the same path, the mount is available again;
+  one that comes back elsewhere leaves it unavailable until it is back at that path. If the
+  directory was replaced on purpose, or a drive came back under a new device number, `sync` with
+  `acceptRoot: true` takes the root as it is now: what was recorded about the old one is set
+  aside, and every file is compared afresh.
+- **A workspace deletion stays deleted, and materialize never deletes from disk unasked.** Without
+  `autoMaterialize`, a deleted file's disk copy, unchanged since disk last agreed with the
+  workspace, is listed as `workspace-deleted`, and no scan or lazy read brings that copy back; a
+  copy changed on disk since is a conflict. `materialize` lists the deletions it left on disk;
+  `applyDeletions: true` removes those whose disk copy is still one the workspace held, and
+  `force` too removes those changed since. A `sync` of the path takes disk's copy back into the
+  workspace when it's one the workspace can hold (text within the size limit). Once the deletion
+  is confirmed on disk, a file that later appears at the path is a new file, adopted like any
+  other.
+  `autoMaterialize` writes and deletes follow materialize's freshness rule: a disk copy changed
+  by another writer since it last agreed is refused, not overwritten or unlinked.
+- **A `materialize` without a path takes up only what disk still owes:** workspace edits,
+  entries never checked against disk, conflicts and workspace deletions. It no longer depends on
+  the sequence the mount was last materialized at (`status.lastMaterializedSeq`), which is saved
+  only at a clean stop: after a restart that lost it, a bare `materialize` took up the whole tree.
+  A file the workspace hasn't changed is left as disk has it, with or without `force`, unless its
+  path is given. A path the evidence says nothing about is treated the same way. Where disk holds
+  a different copy (`unknown-provenance`, or `store-origin-collision` for a file the workspace
+  created), only a `materialize` that names the path with `force` overwrites it. Where an entry
+  from before the evidence has no disk copy, only one that names the path writes it, and a `sync`
+  of the path drops it from the workspace instead. A bare `materialize` trusts the evidence: if
+  disk was replaced behind the workspace, a `sync` takes disk's copies, and naming the paths with
+  `force` writes the workspace's. `status.pendingChanges` counts what disk owes by the evidence;
+  an entry never checked against disk isn't counted until a listing or a `materialize` checks it.
+- **Undo and branch switches are not mistaken for disk edits.** What disk last agreed with is kept
+  in global chronicle records rather than branch state, so selecting an earlier state keeps that
+  state as a draft over the later disk version, and a file present only in another branch's
+  history is listed `not-in-branch`, never ingested or unlinked. Each write or unlink a
+  materialize makes, and each adoption that ingests a disk change or removes a file, is recorded
+  as an intent ahead of it and completed once it is durable, so a crash or a torn record tail
+  never costs an edit. If an append of disk-agreement evidence fails inside Chronicle, the
+  workspace refuses operations that need that evidence and reports why. Those operations remain
+  unavailable until the host restarts and reloads the store's records. The append may have
+  reached the log without becoming readable in this process, so deciding a conflict from
+  possibly incomplete evidence would be unsafe. A binary or oversize disk version, for which the workspace drops its copy,
+  forgets the evidence first instead. After a crash, an interrupted write or unlink is decided by what disk shows (a copy
+  matching neither the one before nor the one intended is an `interrupted` conflict), and one
+  that shows on disk without proof it is durable is completed only by the next `materialize`,
+  which redoes it; an interrupted adoption is settled by its own branch's tree. This replaces
+  #109's stop-time baselines and refused paths,
+  which an abrupt exit or a branch switch could lose or rewind. A store that saved them doesn't
+  import them, since they name no root: its first pass records each file disk agrees with, and
+  lists any other as a conflict or of unknown provenance.
+- **`watch: 'on-agent-action'` works as documented:** after each completed tool batch, before
+  anything continues the agent's turn, the mount is scanned. A scan that outlasts
+  `agentActionScanDeadlineMs` stops holding the round and continues in the background; the miss
+  is reported on `status` (`lastAgentActionScan`) and as a `workspace:agent-action-scan-incomplete`
+  event, as are the regions a finished scan couldn't observe (`incomplete`). Batches that come
+  while a scan waits share it, and while a scan that outlasted its deadline still runs, a later
+  batch isn't held: its miss is recorded at once, and its scan comes after. The hold is in the
+  framework's event loop, so while it lasts every agent's events wait.
+- Disk text is stored as its raw bytes rather than through a UTF-8 round trip, so a file in
+  another encoding no longer differs from its own stored copy.
+- `status` with `mount` reports that mount alone, and refuses one that doesn't exist. A
+  `materialize` that names a read-only mount lists it under `skipped`, rather than answering as
+  if nothing was owed there.
+- `materialize` and `autoMaterialize` check each write and unlink where it happens, not only when
+  it is planned. A file is opened in its directory without truncating and located before any
+  byte changes: never through a symlink the mount doesn't follow, or a directory that resolves
+  outside the mount, and never into a file with other hard links, whose other names (a package
+  store's, say) a write in place would change too, even with `force`. Unless `force`, it must
+  also still hold what the materialize decided on, so a disk edit or a new file that appears
+  meanwhile is refused, not overwritten. Refused paths are listed as skipped with the reason.
+  Where the platform can sync a directory (not Windows, nor a filesystem that refuses it), every
+  directory from a written file up to the mount root is synced before the write counts as done,
+  directories it had to create included. Listings likewise accept a directory's contents only
+  while it is still the directory at its path, and a scan, write or unlink accepts what it found
+  only while the mount root is still the one it began under: a scan that finds the root replaced
+  when it looks again treats what it observed as unverified, and reports the root. Each check is made where the operation happens,
+  against what is there at that moment. Node has no `openat`, so a parent directory replaced
+  concurrently in the window between a check and a create, `mkdir`, unlink or listing can still
+  redirect that one operation: an empty file or directory can appear outside the mount, or a
+  same-named file there can be unlinked. Nor can a write or an unlink be bound to the moment of
+  its last check: an edit to the open file between the last look and the truncate can be
+  overwritten, and a file put at the path, or an edit made to it, between the last look and the
+  unlink can be removed. Content writes stay bound to the
+  descriptor verified, inside the mount and with no other hard links, before modification; a
+  hard link made to the file after that check shares the write.

src/framework.ts

@@ -6694,6 +6694,44 @@ export class AgentFramework {
           result: event.result,
         });
       }
+      // The round's last pending result: let modules catch up with what the
+      // batch's tools did (the workspace scans `on-agent-action` mounts) before
+      // anything continues the turn. Bounded and fail-open; the agent may be
+      // reset or cancelled meanwhile, so its state is checked again after.
+      if (
+        agent && agent.state.status === 'waiting_for_tools'
+        && agent.state.pending.size === 1 && agent.state.pending.has(event.callId)
+      ) {
+        // The call this result answers, as this round holds it: a new round
+        // that reuses the call's id holds a new entry for it.
+        const owed = agent.state.pending.get(event.callId);
+        const failures = await this.moduleRegistry.notifyToolBatchComplete(event.agentName);
+        for (const failure of failures) {
+          this.emitTrace({ type: 'module:batch_hook_failed', agentName: event.agentName, module: failure.module, error: failure.error });
+        }
+        // The turn may have been reset, cancelled or replaced while the hook
+        // ran, even back to waiting on a new batch, whose calls may reuse this
+        // one's id: this result is still owed only if this agent still waits
+        // on this round's call.
+        const after = agent.state as AgentState;
+        if (
+          this.agents.get(event.agentName) !== agent
+          || after.status !== 'waiting_for_tools' || after.pending.get(event.callId) !== owed
+        ) {
+          console.warn(
+            `[framework] Dropping tool-result for agent '${event.agentName}' (callId=${event.callId}): ` +
+            `its turn changed while modules caught up with the batch (status '${after.status}').`,
+          );
+          this.emitTrace({
+            type: 'tool:result_dropped',
+            agentName: event.agentName,
+            callId: event.callId,
+            agentStatus: (agent.state as AgentState).status,
+            result: event.result,
+          });
+          return;
+        }
+      }
       if (agent && agent.state.status === 'waiting_for_tools') {
         agent.provideToolResult(event.callId, event.result);