From 5522dccfcae90f1a1db95123648dd382c1a834a1 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 09:24:11 -0400 Subject: [PATCH 1/5] fix(workspace): the findings of #340's review (#272) Anarchid's review of #340 found that resuming walks moved freshness around, and seven smaller things. This answers each of them. R1-1. An agent-action scan resumed where the last walk stopped, so on a mount over its walkCap the files it used to cover after every tool batch, the first walkCap entries in the walk's order, were covered on about one batch in ceil(N / walkCap). A file a shell tool had just changed then read stale from the workspace. The scan exists so the batch's changes are in the workspace when the agent next looks, so it walks from the mount's start again and keeps no place: on a mount over its cap, every batch's scan covers the first walkCap entries in the walk's order, as every scan on 0.22 covered its first 5000. Bare syncs, directory syncs and a mount's catch-up scans still resume, and a whole mount's catch-up scan takes turns with its bare syncs. R1-2. Recording a place bound the mount's root identity, even for a mount with no evidence, so replacing the root of a mount that held only files too large to take in left it unavailable until acceptRoot. A place only says where the next walk starts, and a walk takes its turn after it in whatever directory is the root by then, so a place binds no root now. Evidence still binds it, as before. R1-3. Recording a place marked the evidence as owing a barrier, so every capped pass of an unchanged mount made a store sync. A place that's lost costs coverage, never a decision, so it owes none: a pass that moves only its place makes no store sync. R1-4. The sync tool's description said the next sync resumes where the last one stopped, but a whole mount's catch-up scan moves the same place. It now says the walk takes turns: the next sync of the same mount or directory picks up where the last turn stopped, and a mount's catch-up scan takes turns with its whole-mount syncs. The incomplete reason a capped sync gives says the same for its scope. R1-5. The changelog now says that the first passes after the upgrade catch up on a mount over 5000 entries, each pushing one created or deleted event that lists every path it took in or proved deleted. R1-6. A walk whose cap is used up by its last entry still keeps a place when directories it didn't list follow, even ones holding nothing it would take. The walk now says so where it decides; telling them apart would take another read. R1-7. walkCap's description now says it bounds what a walk observes, not the walk's own work: directories don't count against it, and each one the walk visits is read whole. R1-8. A model-based test runs the walk on random trees that change between passes, and checks each pass against the cyclic window after its place, worked out independently. Tests, in workspace-three-way: an agent-action scan takes the mount's start after every batch and leaves the syncs' place alone (replacing the test that pinned its resuming); a pass that records only a place binds no root (replacing the test that pinned the binding); a whole mount's catch-up scan takes turns with its bare syncs; a pass that moves only its place makes no store sync; and the model. --- changelog.d/272-walk-cap.fixed.md | 5 +- src/modules/workspace/disk-agreement.ts | 13 +- src/modules/workspace/index.ts | 17 ++- src/modules/workspace/observe.ts | 6 +- src/modules/workspace/reconcile.ts | 22 +-- src/modules/workspace/types.ts | 25 ++-- test/workspace-three-way.test.ts | 173 ++++++++++++++++++++++-- 7 files changed, 222 insertions(+), 39 deletions(-) diff --git a/changelog.d/272-walk-cap.fixed.md b/changelog.d/272-walk-cap.fixed.md index f2c99e57b..8cb6025fc 100644 --- a/changelog.d/272-walk-cap.fixed.md +++ b/changelog.d/272-walk-cap.fixed.md @@ -1,5 +1,6 @@ -- A workspace mount with more files than one walk takes is now covered by repeated syncs and scans (#272). Each mount sets its cap with `walkCap`, a positive integer no larger than `Number.MAX_SAFE_INTEGER` that defaults to 5000; any other value is refused when the module is created, naming the mount. A bare `sync`, a `sync` of a directory, an agent-action scan, or a watched mount's catch-up scan (at start and when its watcher reattaches) that reaches the cap lists what it didn't reach under `incomplete`, and the next one of the same scope resumes after the last entry it took, then wraps to the start. The place is kept with the mount's disk evidence, so a restart resumes there too. Before, every walk stopped at the same 5000 files, so the files past the cap were never scanned, and a change to one reached the workspace only through a `sync` of its path. A listing (`ls`, `glob`, `grep`) still takes up to the cap from its scope's start. +- A workspace mount with more files than one walk takes is now covered by repeated syncs and scans (#272). Each mount sets its cap with `walkCap`, a positive integer no larger than `Number.MAX_SAFE_INTEGER` that defaults to 5000; any other value is refused when the module is created, naming the mount. A bare `sync`, a `sync` of a directory, or a watched mount's catch-up scan (at start and when its watcher reattaches) that reaches the cap lists what it didn't reach under `incomplete`, and the next one of the same scope resumes after the last entry it took, then wraps to the start; a mount's catch-up scan takes turns with its bare syncs. The place is kept with the mount's disk evidence, so a restart resumes there too. Before, every walk stopped at the same 5000 files, so the files past the cap were never scanned, and a change to one reached the workspace only through a `sync` of its path. An agent-action scan (`watch: 'on-agent-action'`) still takes up to the cap from the mount's start after every tool batch, so the files it covered before are as current when the agent next looks as they were, and a listing (`ls`, `glob`, `grep`) still takes up to the cap from its scope's start. - A walk takes each directory's entries that aren't directories by name, then its subdirectories by name, so each walk can resume where the last stopped, whatever order the platform lists a directory in. A link, or another entry that isn't a directory, counts against the cap as a file does, since the pass reads each one it takes, through the link on a mount that follows them. - Each directory's listing is read whole, so a file deleted on disk is seen wherever the walk lists, past the cap included. Before, a directory holding more files than the cap never proved a deletion. - - The `sync` tool's description now says that a sync takes up to the mount's walk cap and that the next sync of the same mount or directory resumes where it stopped, so the first request after the upgrade that carries the workspace's tools misses the prompt cache from its tool block on. + - The `sync` tool's description now says that a sync takes up to the mount's walk cap, and that the next sync of the same mount or directory picks up where the last turn stopped, with a mount's catch-up scan taking turns with its whole-mount syncs. So the first request after the upgrade that carries the workspace's tools misses the prompt cache from its tool block on. + - On a mount over 5000 entries, the first passes after the upgrade catch up: they take in files past the old cap that were never scanned, and prove deletions in directories larger than the cap, which 0.22 never proved. Each such pass pushes one `workspace:created` or `workspace:deleted` event listing every path it took in or proved deleted, and on a `wakeOnChange` mount each of those becomes one message to the agent listing them all. - Going back to 0.22: its walks stop at the same 5000 files as before. It leaves the place alone, so coming forward again resumes there, unless 0.22 wrote a checkpoint meanwhile, which drops every place and starts each scope at its top. diff --git a/src/modules/workspace/disk-agreement.ts b/src/modules/workspace/disk-agreement.ts index 3ae6034e9..5475766aa 100644 --- a/src/modules/workspace/disk-agreement.ts +++ b/src/modules/workspace/disk-agreement.ts @@ -137,9 +137,10 @@ export class DiskAgreement { private readonly mounts = new Map(); private configured: Array<{ name: string; root: string }> = []; /** - * An append (or a rebuild) not yet made durable. Cleared only by a - * barrier that succeeds, so a failed barrier stays owed: a later pass with - * nothing new to record still settles it before deciding anything. + * An append of evidence (or a rebuild) not yet made durable. Cleared only + * by a barrier that succeeds, so a failed barrier stays owed: a later pass + * with nothing new to record still settles it before deciding anything. A + * place owes none (recordWalk). */ private unsynced = false; @@ -279,6 +280,8 @@ export class DiskAgreement { * that's lost costs coverage, never a decision: an append that didn't land * loses one, and a checkpoint written by a version that doesn't keep them * (0.22) loses them all, and each scope's walks start at its top again. + * So recording one owes no barrier: a pass that moves only its place makes + * no store sync, and the place becomes durable with the next one. */ recordWalk(mount: string, scope: string, after: string | null): void { if ((this.walkResumesAfter(mount, scope) ?? null) === after) return; @@ -319,7 +322,9 @@ export class DiskAgreement { private write(entry: JournalEntry, opts: { durable?: boolean; afterCommittedState?: boolean }): void { this.ensureReconciled(); - this.unsynced = true; // until proven durable: a failed append may still have landed + // Evidence is owed a barrier until proven durable: a failed append may + // still have landed. A place isn't (recordWalk). + if (entry.t !== 'walk') this.unsynced = true; this.journal.append(entry, opts); if (opts.durable) this.unsynced = false; // synced after this append, and so everything before it const mount = this.mounts.get((entry as { mount: string }).mount); diff --git a/src/modules/workspace/index.ts b/src/modules/workspace/index.ts index 3be26091f..fd8b579c1 100644 --- a/src/modules/workspace/index.ts +++ b/src/modules/workspace/index.ts @@ -1224,8 +1224,10 @@ export class WorkspaceModule implements Module { name: 'sync', description: 'Bring disk changes into the workspace. Without a path, every mount (or the given one) is ' + 'rechecked and nothing pending in the workspace is discarded. A sync takes up to the mount\'s walk ' - + 'cap of entries, files, links and the like (5000 unless configured); past it, what this one didn\'t reach is listed under ' - + '`incomplete`, and the next sync of the same mount or directory resumes where it stopped. With a ' + + 'cap of entries, files, links and the like (5000 unless configured). Past it, what this one didn\'t reach is listed under ' + + '`incomplete`, and the walk takes turns: the next sync of the same mount or directory picks up where ' + + 'the last turn stopped, so repeated syncs cover it all. A mount\'s own catch-up scan, at start or when ' + + 'its watcher reattaches, takes turns with its whole-mount syncs. With a ' + 'path (file or directory), ' + 'the workspace takes disk\'s state there explicitly: it restores a workspace-deleted file, resolves ' + 'conflicts toward disk, replaces a workspace draft, and removes a workspace file disk does not have. ' @@ -3004,7 +3006,10 @@ export class WorkspaceModule implements Module { * After an agent's completed tool batch, before anything continues its turn: * scan every `watch: 'on-agent-action'` mount, so what the batch's tools did * on disk (a shell command's files, a deletion) is in the workspace when the - * agent next looks. A scan that outlasts the deadline stops holding the + * agent next looks. On a mount over its walkCap, that holds for the first + * walkCap entries in the walk's order, which every batch's scan takes; files + * past them come in through the syncs, catch-up scans and listings that + * reach them. A scan that outlasts the deadline stops holding the * round: the miss is recorded on the mount's status and pushed as a * `workspace:agent-action-scan-incomplete` event, and the scan continues, * deciding on the branch selected when it applies. @@ -3073,7 +3078,11 @@ export class WorkspaceModule implements Module { const scan: AgentActionScan = { late: false, settled: Promise.resolve() }; const pass = this.withMount(mount, () => { if (mount.agentActionScanWaiting === scan) mount.agentActionScanWaiting = undefined; - return this.passUnlocked(mount, { kind: 'dir', dir: '', recursive: true }, { resume: true }); + // From the mount's start, as a listing walks, keeping no place: on a + // mount over its walkCap, every batch's scan takes the first walkCap + // entries in the walk's order, and the place that syncs and catch-up + // scans resume from stays theirs. + return this.passUnlocked(mount, { kind: 'dir', dir: '', recursive: true }); }); scan.settled = pass.then( (result) => { diff --git a/src/modules/workspace/observe.ts b/src/modules/workspace/observe.ts index 050d1bbb3..45c5559db 100644 --- a/src/modules/workspace/observe.ts +++ b/src/modules/workspace/observe.ts @@ -1059,7 +1059,11 @@ export async function walkScope( } } // The cap stopped the walk only if it left something: a walk whose last - // file used up the cap covered its scope. + // file used up the cap covered its scope. A directory the walk didn't list + // counts as left, even one that holds nothing it would take (empty, or + // only ignored names): telling them apart would take another read. Such a + // walk keeps its place and reports the directory as not visited, and the + // next one lists it on its way to the wrap. if (left === 0 && lastTaken !== undefined && (holdingUntaken.size > 0 || unlisted > 0)) walk.stoppedAfter = lastTaken; return walk; } diff --git a/src/modules/workspace/reconcile.ts b/src/modules/workspace/reconcile.ts index 639e923d6..27c2c4194 100644 --- a/src/modules/workspace/reconcile.ts +++ b/src/modules/workspace/reconcile.ts @@ -168,10 +168,11 @@ export interface PassOptions { /** Walk cap for directory scopes; the mount's `walkCap` when absent. */ cap?: number; /** - * A recursive directory pass of a sync or scan: its walk resumes after the - * last entry the previous one of this scope took, and records where it - * stopped, so repeated passes cover a scope larger than the cap. A - * listing's pass starts at its scope's start. + * A recursive directory pass of a sync or a catch-up scan: its walk resumes + * after the last entry the previous one of this scope took, and records + * where it stopped, so repeated passes cover a scope larger than the cap. + * A listing's pass, and an agent-action scan's, start at their scope's + * start and record no place. */ resume?: boolean; /** @@ -818,12 +819,12 @@ export async function reconcilePass( // Where the walk stopped, once what it observed is decided: the next pass // of this scope resumes after it, or starts afresh once this one covered // the scope. A walk that couldn't list its scope's own directory covered - // none of it, and leaves the place as it was. Like evidence, a place is - // recorded after the root it was gathered under. + // none of it, and leaves the place as it was. Unlike evidence, a place + // binds no root: it only says where the next walk starts, and that walk + // takes its turn after it in whatever directory is the root by then. let resumeAfter: string | null | undefined; if (opts.resume && walk && scope.kind === 'dir' && scope.recursive && walk.complete.has(scope.dir)) { resumeAfter = walk.stoppedAfter ?? null; - if (gathered.rootIdentity) agreement.recordRootIdentity(mount.name, gathered.rootIdentity); agreement.recordWalk(mount.name, scope.dir, resumeAfter); } @@ -851,9 +852,14 @@ export async function reconcilePass( if (walk) { const tracked = [...result.reports.values()].filter((r) => r.state === 'unverified').map((r) => r.path); if (typeof resumeAfter === 'string' && scope.kind === 'dir') { + // A whole mount's catch-up scan takes turns with its syncs; nothing + // else moves a place. + const next = scope.dir === '' + ? 'the next sync of the mount, or its catch-up scan if that comes first,' + : 'the next sync of this directory'; result.incomplete.push({ path: scope.dir, - reason: `the walk cap (${opts.cap ?? mount.walkCap}) was reached: the next sync or scan of this scope resumes after ${mount.name}/${resumeAfter}`, + reason: `the walk cap (${opts.cap ?? mount.walkCap}) was reached: ${next} resumes after ${mount.name}/${resumeAfter}`, }); } for (const region of walk.incomplete) { diff --git a/src/modules/workspace/types.ts b/src/modules/workspace/types.ts index 369ba3c40..b4f03d200 100644 --- a/src/modules/workspace/types.ts +++ b/src/modules/workspace/types.ts @@ -22,7 +22,9 @@ export interface MountConfig { * - 'on-agent-action': after each completed agent tool batch, the mount is * scanned, and the agent's next inference waits for that scan up to * `agentActionScanDeadlineMs`; a scan that takes longer finishes while - * the round goes on + * the round goes on. Each scan walks from the mount's start, so on a + * mount over its `walkCap` it covers the first `walkCap` entries every + * time, and files past them come in as `walkCap` describes * - 'never': no background reads; listings and lazy reads still observe * the paths they show */ @@ -46,13 +48,20 @@ export interface MountConfig { * The most entries one walk of this mount takes (default: 5000), a * positive integer no larger than Number.MAX_SAFE_INTEGER: files, and * links or other entries that aren't - * directories, each of which the pass observes. A sync or scan that - * reaches it reports the rest as incomplete, and the next sync or scan of - * the same scope resumes after the last entry it took, wrapping to the - * start, so repeated ones cover a larger mount; a restart keeps the place. - * A listing (`ls`, `glob`, `grep`) takes up to it from its scope's start. - * Each directory's listing is read whole, so a deletion is seen wherever - * the walk lists, whatever it took. + * directories, each of which the pass observes. A sync, or the mount's + * catch-up scan (at start, and when its watcher reattaches), that reaches + * it reports the rest as incomplete, and the next one of the same scope + * resumes after the last entry it took, wrapping to the start, so repeated + * ones cover a larger mount; a whole mount's catch-up scan takes turns + * with its syncs, and a restart keeps the place. An agent-action scan + * (`watch: 'on-agent-action'`) and a listing (`ls`, `glob`, `grep`) take + * up to it from their scope's start every time. Each directory's listing + * is read whole, so a deletion is seen wherever the walk lists, whatever + * it took. + * + * It bounds what a walk observes, not the walk's own work: directories + * don't count against it, and each one the walk visits is read whole, so + * a tree of many directories that hold nothing is walked in full. */ walkCap?: number; /** diff --git a/test/workspace-three-way.test.ts b/test/workspace-three-way.test.ts index 35d8a5ca0..f82fe5585 100644 --- a/test/workspace-three-way.test.ts +++ b/test/workspace-three-way.test.ts @@ -13,7 +13,7 @@ import { describe, test, type TestContext } from 'node:test'; import { TOOL_BATCH_HOOK_TIMEOUT_MS } from '../src/module-registry.js'; import assert from 'node:assert/strict'; import { spawnSync } from 'node:child_process'; -import { chmodSync, constants as fsConstants, existsSync, linkSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, renameSync, rmSync, statSync, symlinkSync, truncateSync, unlinkSync, utimesSync, writeFileSync } from 'node:fs'; +import { chmodSync, constants as fsConstants, existsSync, linkSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, renameSync, rmSync, statSync, symlinkSync, truncateSync, unlinkSync, utimesSync, writeFileSync } from 'node:fs'; import { open, statfs } from 'node:fs/promises'; import { createHash } from 'node:crypto'; import { createRequire, syncBuiltinESMExports } from 'node:module'; @@ -23,7 +23,7 @@ import { JsStore } from '@animalabs/chronicle'; import { WorkspaceModule } from '../src/modules/workspace/index.js'; import { DISK_AGREEMENT_RECORD_TYPE, DiskAgreement } from '../src/modules/workspace/disk-agreement.js'; import { RecordJournal } from '../src/record-journal.js'; -import { confirmAbsent, fingerprintVouches, filesystemTrustsCtime, fsTypeTrustsCtime, observePath, readPath, type MountView } from '../src/modules/workspace/observe.js'; +import { confirmAbsent, fingerprintVouches, filesystemTrustsCtime, fsTypeTrustsCtime, observePath, provenAbsent, readPath, walkScope, type MountView } from '../src/modules/workspace/observe.js'; import { EffectFailed, syncDirectories, unlinkContained, writeContained } from '../src/modules/workspace/effects.js'; import { pushPaths, settleAdoptionBeforeMutation } from '../src/modules/workspace/reconcile.js'; import type { MountConfig, WorkspaceConfig } from '../src/modules/workspace/types.js'; @@ -2234,20 +2234,34 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { const synced = await call(m, 'sync', {}); const resumes = synced.incomplete.filter((r: { reason: string }) => /resumes after/.test(r.reason)); assert.deepEqual(resumes.map((r: { mount: string; path: string; reason: string }) => [r.mount, r.path, r.reason]), - [['work', '', 'the walk cap (1) was reached: the next sync or scan of this scope resumes after work/a.txt']]); + [['work', '', 'the walk cap (1) was reached: the next sync of the mount, or its catch-up scan if that comes first, resumes after work/a.txt']]); // z/q.txt sits beneath z/, which wasn't listed: its reason is z/'s, not // that of the root, which holds an untaken b.txt. const skipped = synced.skipped?.find((x: { path: string }) => x.path === 'z/q.txt'); assert.match(skipped?.reason ?? '', /not visited: the walk cap \(1\) was reached/, JSON.stringify(synced.skipped)); }); - test('agent-action scans resume where the last stopped', async (t) => { + test("an agent-action scan takes the mount's start after every batch, and leaves the syncs' place alone", async (t) => { const env = new Env(t); for (const f of ['a.txt', 'b.txt', 'c.txt', 'd.txt']) env.writeDisk(f, f); const m = await env.open({ watch: 'on-agent-action', walkCap: 2 }); + const agreement = (m as any).agreementOrThrow(); + // What a shell tool did to the first file in the walk's order is in the + // workspace after every batch. + for (const edit of ['edit one', 'edit number two', 'the third edit, longest']) { + env.writeDisk('a.txt', edit); + await m.onToolBatchComplete('agent'); + assert.equal(await contentOf(m, 'a.txt'), edit); + } + assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt'], 'the same two files every time'); + assert.equal(agreement.walkResumesAfter('work', ''), undefined, 'and no place'); + // A bare sync keeps its place, and a batch's scan between two syncs + // doesn't move it. + await call(m, 'sync', {}); + assert.equal(agreement.walkResumesAfter('work', ''), 'b.txt'); await m.onToolBatchComplete('agent'); - assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt']); - await m.onToolBatchComplete('agent'); + assert.equal(agreement.walkResumesAfter('work', ''), 'b.txt'); + await call(m, 'sync', {}); assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt', 'c.txt', 'd.txt']); }); @@ -2675,16 +2689,151 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { assert.equal((m as any).agreementOrThrow().walkResumesAfter('work', ''), 'b.txt'); }); - test('a place is recorded after the root it was gathered under, even by a pass that records no evidence', async (t) => { + test('a pass that records only a place binds no root, so a replaced root is read as it is', async (t) => { const env = new Env(t); - // Files too large to take in: the pass records no evidence for them. - for (const f of ['a.txt', 'b.txt']) env.writeDisk(f, 'more than one byte'); + // A file too large to take in: the passes record no evidence for it. + env.writeDisk('a.txt', 'more than one byte'); const m = await env.open({ walkCap: 1, maxFileSize: 1 }); + const agreement = (m as any).agreementOrThrow(); + await call(m, 'sync', {}); + assert.equal(agreement.walkResumesAfter('work', ''), undefined, 'the walk covered the mount'); + assert.equal(agreement.rootIdentity('work'), undefined, 'and bound no root'); + env.writeDisk('b.txt', 'more than one byte'); + await call(m, 'sync', {}); + assert.equal(agreement.walkResumesAfter('work', ''), 'a.txt', 'a capped walk keeps its place'); + assert.equal(agreement.rootIdentity('work'), undefined, 'and binds no root either'); + // Another directory takes the root's place. With no evidence to guard, + // the mount reads it as it is, and the walk takes its turn after the + // place there. + renameSync(env.dir, `${env.dir}-away`); + mkdirSync(env.dir); + for (const f of ['a.txt', 'b.txt', 'c.txt']) env.writeDisk(f, 'more than one byte'); + const synced = await call(m, 'sync', {}); + assert.ok(!synced.incomplete.some((r: { reason: string }) => /disk last agreed with/.test(r.reason)), JSON.stringify(synced.incomplete)); + assert.equal(agreement.walkResumesAfter('work', ''), 'b.txt'); + }); + + test("a whole mount's catch-up scan takes turns with its bare syncs", async (t) => { + const env = new Env(t); + for (const f of ['a.txt', 'b.txt', 'c.txt', 'd.txt', 'e.txt', 'f.txt']) env.writeDisk(f, f); + const m = await env.open({ walkCap: 2 }); await call(m, 'sync', {}); + await (m as any).initialScan('work'); + assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt', 'c.txt', 'd.txt'], 'the scan took the next two'); + await call(m, 'sync', {}); + assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt', 'c.txt', 'd.txt', 'e.txt', 'f.txt'], 'and the sync the two after those'); + }); + + test('a pass that moves only its place makes no store sync', async (t) => { + const env = new Env(t); + for (const f of ['a.txt', 'b.txt', 'c.txt', 'd.txt']) env.writeDisk(f, f); + const m = await env.open({ walkCap: 2 }); + await call(m, 'sync', {}); + await call(m, 'sync', {}); + assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt', 'c.txt', 'd.txt']); const agreement = (m as any).agreementOrThrow(); - assert.equal(agreement.walkResumesAfter('work', ''), 'a.txt'); - const st = statSync(env.dir, { bigint: true }); - assert.deepEqual(agreement.rootIdentity('work'), { dev: String(st.dev), ino: String(st.ino) }); + const sync = env.store.sync.bind(env.store); + let syncs = 0; + env.store.sync = () => { + syncs++; + return sync(); + }; + // Nothing on disk changes from here, so each pass records only where it + // stopped: a place that's lost costs coverage, never a decision. + const places: Array = []; + for (let i = 0; i < 4; i++) { + await call(m, 'sync', {}); + places.push(agreement.walkResumesAfter('work', '')); + } + assert.deepEqual(places, ['b.txt', 'd.txt', 'b.txt', 'd.txt'], 'the place moved on every pass'); + assert.equal(syncs, 0, 'with no store sync'); + }); + + test('a model of the walk: on random trees that change between passes, each pass takes the cyclic window after its place', async (t) => { + const env = new Env(t); + const rootReal = realpathSync(env.dir); + const view = { root: env.dir, followSymlinks: false, maxFileSize: 1 << 20, ignore: [] } as unknown as MountView; + // A seeded generator, so a failure replays. + let seed = 272; + const rand = (n: number): number => { + seed = (Math.imul(seed, 1103515245) + 12345) >>> 0; + return (seed >>> 8) % n; + }; + // U+FF21 sorts before U+1F600 by UTF-8 byte and after it by UTF-16 code + // unit, the walk's order; U+E000 sorts between them by both. + const names = ['a', 'b', 'm', 'z', '\uFF21', '\u{1F600}', '\uE000']; + const pick = (xs: T[]): T => xs[rand(xs.length)]!; + const randomPath = (): string => { + const parts: string[] = []; + for (let depth = rand(3); depth > 0; depth--) parts.push(pick(names)); + return [...parts, `${pick(names)}.${rand(4) === 0 ? 'lnk' : 'txt'}`].join('/'); + }; + const add = (path: string): void => { + const full = env.disk(path); + mkdirSync(dirname(full), { recursive: true }); + if (lstatSync(full, { throwIfNoEntry: false })) return; + if (path.endsWith('.lnk')) symlinkSync('nowhere', full); // dangling: an entry that isn't a file + else writeFileSync(full, path); + }; + /** Every entry that isn't a directory, as readdir shows it now. */ + const entries = (dir = ''): string[] => { + const out: string[] = []; + for (const d of readdirSync(env.disk(dir), { withFileTypes: true })) { + const path = dir ? `${dir}/${d.name}` : d.name; + if (d.isDirectory()) out.push(...entries(path)); + else out.push(path); + } + return out; + }; + // The walk's order, worked out independently: in each directory, its + // entries that aren't directories by name, then its subdirectories by + // name, each with everything beneath it. + const order = (a: string, b: string): number => { + const pa = a.split('/'); + const pb = b.split('/'); + for (let i = 0; ; i++) { + const leafA = i === pa.length - 1; + const leafB = i === pb.length - 1; + if (leafA !== leafB) return leafA ? -1 : 1; + if (pa[i] !== pb[i]) return pa[i]! < pb[i]! ? -1 : 1; + if (leafA) return 0; + } + }; + + for (let tree = 0; tree < 25; tree++) { + rmSync(env.dir, { recursive: true, force: true }); + mkdirSync(env.dir); + for (let i = 4 + rand(16); i > 0; i--) add(randomPath()); + const cap = 1 + rand(8); + let place: string | undefined; + for (let pass = 0; pass < 12; pass++) { + // Changes between passes: entries added, removed, the place itself + // removed, and whole directories removed. + for (let change = rand(4); change > 0; change--) { + const now = entries(); + const kind = rand(4); + if (kind === 0 || now.length === 0) add(randomPath()); + else if (kind === 1) rmSync(env.disk(pick(now))); + else if (kind === 2 && place !== undefined && now.includes(place)) rmSync(env.disk(place)); + else { + const dirs = now.filter((p) => p.includes('/')).map((p) => p.slice(0, p.lastIndexOf('/'))); + if (dirs.length > 0) rmSync(env.disk(pick(dirs)), { recursive: true, force: true }); + } + } + const all = entries().sort(order); + const after = place === undefined ? [] : all.filter((p) => order(p, place!) > 0); + const upto = place === undefined ? all : all.filter((p) => order(p, place!) <= 0); + const window = [...after, ...upto].slice(0, cap); + const walk = await walkScope(view, rootReal, '', true, cap, place === undefined ? {} : { resumeAfter: place }); + const taken = [...walk.files, ...walk.othersTaken]; + const where = `tree ${tree}, pass ${pass}, cap ${cap}, after ${place}`; + assert.deepEqual(taken.sort(order), [...window].sort(order), `${where}: took the cyclic window`); + for (const p of all) assert.equal(provenAbsent(walk, p), false, `${where}: ${p} is on disk`); + if (walk.stoppedAfter !== undefined) assert.equal(walk.stoppedAfter, window.at(-1), `${where}: the place is the last entry taken`); + else assert.ok(all.length <= cap, `${where}: no place only once the walk took everything`); + place = walk.stoppedAfter; + } + } }); test('places go through a checkpoint, and a scope a walk covered stays cleared', (t) => { From 5430e1014348639d841115d09c7ffa8a7c643ab7 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 10:02:32 -0400 Subject: [PATCH 2/5] fix(workspace): the internal review's round on #340's follow-up (#272) Our internal review of the follow-up found these, each fixed here. A capped sync's reason named the mount's catch-up scan as the other turn for every mount, though only a watched mount (watch: 'always') runs one. The pass's runtime now says whether the mount does, and only then does the reason name it. The sync tool's description says the catch-up scan is a watched mount's, and the agent-action scan's comment names syncs and the listings and reads that reach a file past the cap. Text about the rotation now says what it costs and what it promises: - a watched mount's catch-up scan takes its turn in the rotation, so on a mount over the cap, a change made while the process was down waits for the rotation to reach it (the changelog, walkCap's description, and the comment where the catch-up starts); - only syncs list what they didn't reach under incomplete; - a restart resumes at the place unless the machine went down before the store made the latest place durable; - no walk reached the files past the old 5000, where a path sync or a watched mount's watcher did; - a catching-up pass pushes a created event and a deleted event; - the agent-action scan and listings take entries in the walk's name order, so on a mount over the cap they can reach different files than 0.22's first 5000, which followed the platform's listing order; - the disk-agreement header says a place binds no root, and applyRoot's reset gives its own reason for clearing places. Tests: the directory's and the watched mount's capped reasons; the agent-action test reads the stored file rather than the read tool, checks that each capped scan reports its cap, and that a covered file a shell tool deleted is gone after the batch; the model test's comment on U+E000 is corrected. --- changelog.d/272-walk-cap.fixed.md | 7 +++-- src/modules/workspace/disk-agreement.ts | 8 ++++-- src/modules/workspace/index.ts | 18 +++++++----- src/modules/workspace/reconcile.ts | 18 ++++++++---- src/modules/workspace/types.ts | 21 ++++++++------ test/workspace-three-way.test.ts | 38 +++++++++++++++++++++---- 6 files changed, 78 insertions(+), 32 deletions(-) diff --git a/changelog.d/272-walk-cap.fixed.md b/changelog.d/272-walk-cap.fixed.md index 8cb6025fc..d0254665c 100644 --- a/changelog.d/272-walk-cap.fixed.md +++ b/changelog.d/272-walk-cap.fixed.md @@ -1,6 +1,7 @@ -- A workspace mount with more files than one walk takes is now covered by repeated syncs and scans (#272). Each mount sets its cap with `walkCap`, a positive integer no larger than `Number.MAX_SAFE_INTEGER` that defaults to 5000; any other value is refused when the module is created, naming the mount. A bare `sync`, a `sync` of a directory, or a watched mount's catch-up scan (at start and when its watcher reattaches) that reaches the cap lists what it didn't reach under `incomplete`, and the next one of the same scope resumes after the last entry it took, then wraps to the start; a mount's catch-up scan takes turns with its bare syncs. The place is kept with the mount's disk evidence, so a restart resumes there too. Before, every walk stopped at the same 5000 files, so the files past the cap were never scanned, and a change to one reached the workspace only through a `sync` of its path. An agent-action scan (`watch: 'on-agent-action'`) still takes up to the cap from the mount's start after every tool batch, so the files it covered before are as current when the agent next looks as they were, and a listing (`ls`, `glob`, `grep`) still takes up to the cap from its scope's start. +- A workspace mount with more files than one walk takes is now covered by repeated syncs (#272). Each mount sets its cap with `walkCap`, a positive integer no larger than `Number.MAX_SAFE_INTEGER` that defaults to 5000; any other value is refused when the module is created, naming the mount. A bare `sync` or a `sync` of a directory that reaches the cap lists what it didn't reach under `incomplete`, and the next one of the same scope resumes after the last entry it took, then wraps to the start. A watched mount's catch-up scan (at start and when its watcher reattaches) takes its turn in the same rotation as the mount's bare syncs. The place is kept with the mount's disk evidence, so a restart resumes there too, unless the machine went down before the store made the latest place durable, which only starts the next walk at an earlier place. Before, every walk stopped at the same 5000 files, so no walk ever reached the files past them. An agent-action scan (`watch: 'on-agent-action'`) still takes up to the cap from the mount's start after every tool batch, and a listing (`ls`, `glob`, `grep`) still takes up to the cap from its scope's start. Both now take entries in the walk's name order, so on a mount over the cap they can reach different files than 0.22's first 5000, which followed the order the platform lists a directory in. - A walk takes each directory's entries that aren't directories by name, then its subdirectories by name, so each walk can resume where the last stopped, whatever order the platform lists a directory in. A link, or another entry that isn't a directory, counts against the cap as a file does, since the pass reads each one it takes, through the link on a mount that follows them. - Each directory's listing is read whole, so a file deleted on disk is seen wherever the walk lists, past the cap included. Before, a directory holding more files than the cap never proved a deletion. - - The `sync` tool's description now says that a sync takes up to the mount's walk cap, and that the next sync of the same mount or directory picks up where the last turn stopped, with a mount's catch-up scan taking turns with its whole-mount syncs. So the first request after the upgrade that carries the workspace's tools misses the prompt cache from its tool block on. - - On a mount over 5000 entries, the first passes after the upgrade catch up: they take in files past the old cap that were never scanned, and prove deletions in directories larger than the cap, which 0.22 never proved. Each such pass pushes one `workspace:created` or `workspace:deleted` event listing every path it took in or proved deleted, and on a `wakeOnChange` mount each of those becomes one message to the agent listing them all. + - Because a watched mount's catch-up scan takes its turn in the rotation, on a mount over the cap a change made while the process was down reaches the workspace when the rotation comes to it, or when a sync of its path takes it. Before, the catch-up scan at start took a change among the first 5000 files every time, and never one past them. + - The `sync` tool's description now says that a sync takes up to the mount's walk cap, and that the next sync of the same mount or directory picks up where the last turn stopped, with a watched mount's catch-up scan taking turns with its whole-mount syncs. So the first request after the upgrade that carries the workspace's tools misses the prompt cache from its tool block on. + - On a mount over 5000 entries, the first syncs and catch-up scans after the upgrade catch up: they take in files past the old cap that were never scanned, and prove deletions in directories larger than the cap, which 0.22 never proved. Each such pass pushes a `workspace:created` event listing every file it took in and a `workspace:deleted` event listing every deletion it proved, and a `wakeOnChange` mount turns each event into one message to the agent that lists them all. - Going back to 0.22: its walks stop at the same 5000 files as before. It leaves the place alone, so coming forward again resumes there, unless 0.22 wrote a checkpoint meanwhile, which drops every place and starts each scope at its top. diff --git a/src/modules/workspace/disk-agreement.ts b/src/modules/workspace/disk-agreement.ts index 5475766aa..76b909d9c 100644 --- a/src/modules/workspace/disk-agreement.ts +++ b/src/modules/workspace/disk-agreement.ts @@ -15,7 +15,9 @@ * that isn't that directory any more (an unmounted drive whose mountpoint is * now empty, a replaced directory) is an unavailable mount, never a disk on * which every file was deleted; only an explicit acceptance takes it, and - * that starts its evidence afresh. + * that starts its evidence afresh. A place, where a walk stopped (`walks`), + * binds no root: it only says where the next walk starts, in whatever + * directory the root is by then. * * Every transition is write-ahead (see reconcile.ts): an `intent` entry * (`pending`, with the prior value and the expected one) is durable before the @@ -371,8 +373,8 @@ function apply(mount: MountEvidence, entry: JournalEntry): void { function applyRoot(mount: MountEvidence, entry: Extract): void { mount.identity = entry.identity; if (entry.reset) { - // Evidence about another directory, and where its walks stopped, say - // nothing about this one. + // Evidence about another directory says nothing about this one, and the + // accepted root's walks start at its top. mount.paths.clear(); mount.walks.clear(); } diff --git a/src/modules/workspace/index.ts b/src/modules/workspace/index.ts index fd8b579c1..79d74be46 100644 --- a/src/modules/workspace/index.ts +++ b/src/modules/workspace/index.ts @@ -791,6 +791,7 @@ export class WorkspaceModule implements Module { intents: this.intents.get(mount.config.name)!, readOnly: mount.config.mode === 'read-only', walkCap: mount.config.walkCap ?? WALK_CAP, + catchUpScan: (mount.config.watch ?? 'always') === 'always', }; } @@ -994,9 +995,12 @@ export class WorkspaceModule implements Module { this.watchers.set(name, watcher); // Chokidar is started with ignoreInitial:true, so changes made while - // the session was down would be invisible. One full pass catches up - // under the three-way rule: only real differences produce tree - // changes and events. + // the session was down would be invisible. A catch-up pass takes them + // in under the three-way rule: only real differences produce tree + // changes and events. It takes its turn with the mount's bare syncs, + // so on a mount over its walkCap it covers the walkCap entries after + // where the last turn stopped, and a change elsewhere waits for the + // rotation to reach it, or for a sync of its path. void this.initialScan(name); } @@ -1226,8 +1230,8 @@ export class WorkspaceModule implements Module { + 'rechecked and nothing pending in the workspace is discarded. A sync takes up to the mount\'s walk ' + 'cap of entries, files, links and the like (5000 unless configured). Past it, what this one didn\'t reach is listed under ' + '`incomplete`, and the walk takes turns: the next sync of the same mount or directory picks up where ' - + 'the last turn stopped, so repeated syncs cover it all. A mount\'s own catch-up scan, at start or when ' - + 'its watcher reattaches, takes turns with its whole-mount syncs. With a ' + + 'the last turn stopped, so repeated syncs cover it all. On a mount watched continuously, its own catch-up ' + + 'scan, at start and when its watcher reattaches, takes turns with its whole-mount syncs. With a ' + 'path (file or directory), ' + 'the workspace takes disk\'s state there explicitly: it restores a workspace-deleted file, resolves ' + 'conflicts toward disk, replaces a workspace draft, and removes a workspace file disk does not have. ' @@ -3008,8 +3012,8 @@ export class WorkspaceModule implements Module { * on disk (a shell command's files, a deletion) is in the workspace when the * agent next looks. On a mount over its walkCap, that holds for the first * walkCap entries in the walk's order, which every batch's scan takes; files - * past them come in through the syncs, catch-up scans and listings that - * reach them. A scan that outlasts the deadline stops holding the + * past them come in through syncs, and the listings and reads that reach + * them. A scan that outlasts the deadline stops holding the * round: the miss is recorded on the mount's status and pushed as a * `workspace:agent-action-scan-incomplete` event, and the scan continues, * deciding on the branch selected when it applies. diff --git a/src/modules/workspace/reconcile.ts b/src/modules/workspace/reconcile.ts index 27c2c4194..d2b0a4b13 100644 --- a/src/modules/workspace/reconcile.ts +++ b/src/modules/workspace/reconcile.ts @@ -154,6 +154,12 @@ export interface MountRuntime { readOnly: boolean; /** The most entries one walk of the mount takes, files and the other entries that aren't directories (`MountConfig.walkCap`). */ walkCap: number; + /** + * Whether the mount runs a catch-up scan (`watch: 'always'`, at start and + * when its watcher reattaches), which takes turns with its whole-mount + * syncs. + */ + catchUpScan: boolean; } export type Scope = @@ -852,11 +858,13 @@ export async function reconcilePass( if (walk) { const tracked = [...result.reports.values()].filter((r) => r.state === 'unverified').map((r) => r.path); if (typeof resumeAfter === 'string' && scope.kind === 'dir') { - // A whole mount's catch-up scan takes turns with its syncs; nothing - // else moves a place. - const next = scope.dir === '' - ? 'the next sync of the mount, or its catch-up scan if that comes first,' - : 'the next sync of this directory'; + // A watched mount's catch-up scan takes turns with its whole-mount + // syncs; nothing else moves a place. + const next = scope.dir !== '' + ? 'the next sync of this directory' + : mount.catchUpScan + ? 'the next sync of the mount, or its catch-up scan if that comes first,' + : 'the next sync of the mount'; result.incomplete.push({ path: scope.dir, reason: `the walk cap (${opts.cap ?? mount.walkCap}) was reached: ${next} resumes after ${mount.name}/${resumeAfter}`, diff --git a/src/modules/workspace/types.ts b/src/modules/workspace/types.ts index b4f03d200..0ee943f53 100644 --- a/src/modules/workspace/types.ts +++ b/src/modules/workspace/types.ts @@ -48,16 +48,19 @@ export interface MountConfig { * The most entries one walk of this mount takes (default: 5000), a * positive integer no larger than Number.MAX_SAFE_INTEGER: files, and * links or other entries that aren't - * directories, each of which the pass observes. A sync, or the mount's - * catch-up scan (at start, and when its watcher reattaches), that reaches - * it reports the rest as incomplete, and the next one of the same scope + * directories, each of which the pass observes. A sync that reaches it + * reports the rest as incomplete, and the next sync of the same scope * resumes after the last entry it took, wrapping to the start, so repeated - * ones cover a larger mount; a whole mount's catch-up scan takes turns - * with its syncs, and a restart keeps the place. An agent-action scan - * (`watch: 'on-agent-action'`) and a listing (`ls`, `glob`, `grep`) take - * up to it from their scope's start every time. Each directory's listing - * is read whole, so a deletion is seen wherever the walk lists, whatever - * it took. + * syncs cover a larger mount. A watched mount's catch-up scan (at start, + * and when its watcher reattaches) takes its turn in the same rotation as + * the mount's bare syncs, so on a mount over the cap, a change made while + * the process was down waits for the rotation to reach it. The place is + * kept with the mount's disk evidence, so a restart resumes there, unless + * the machine went down before the store made the latest place durable. + * An agent-action scan (`watch: 'on-agent-action'`) and a listing (`ls`, + * `glob`, `grep`) take up to it from their scope's start every time. Each + * directory's listing is read whole, so a deletion is seen wherever the + * walk lists, whatever it took. * * It bounds what a walk observes, not the walk's own work: directories * don't count against it, and each one the walk visits is read whole, so diff --git a/test/workspace-three-way.test.ts b/test/workspace-three-way.test.ts index f82fe5585..83b6f4b95 100644 --- a/test/workspace-three-way.test.ts +++ b/test/workspace-three-way.test.ts @@ -2141,8 +2141,11 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { const env = new Env(t); for (const f of ['d/a.txt', 'd/b.txt', 'd/c.txt', 'top.txt']) env.writeDisk(f, f); const m = await env.open({ walkCap: 2 }); - await call(m, 'sync', { path: 'work/d' }); + const first = await call(m, 'sync', { path: 'work/d' }); assert.deepEqual(syncedNames(env), ['d/a.txt', 'd/b.txt']); + assert.ok(first.incomplete.some((r: { path: string; reason: string }) => + r.path === 'd' && r.reason === 'the walk cap (2) was reached: the next sync of this directory resumes after work/d/b.txt'), + JSON.stringify(first.incomplete)); await call(m, 'sync', { path: 'work/d' }); assert.deepEqual(syncedNames(env), ['d/a.txt', 'd/b.txt', 'd/c.txt']); assert.equal((m as any).agreementOrThrow().walkResumesAfter('work', ''), undefined, 'the mount root keeps its own place'); @@ -2234,24 +2237,44 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { const synced = await call(m, 'sync', {}); const resumes = synced.incomplete.filter((r: { reason: string }) => /resumes after/.test(r.reason)); assert.deepEqual(resumes.map((r: { mount: string; path: string; reason: string }) => [r.mount, r.path, r.reason]), - [['work', '', 'the walk cap (1) was reached: the next sync of the mount, or its catch-up scan if that comes first, resumes after work/a.txt']]); + [['work', '', 'the walk cap (1) was reached: the next sync of the mount resumes after work/a.txt']]); // z/q.txt sits beneath z/, which wasn't listed: its reason is z/'s, not // that of the root, which holds an untaken b.txt. const skipped = synced.skipped?.find((x: { path: string }) => x.path === 'z/q.txt'); assert.match(skipped?.reason ?? '', /not visited: the walk cap \(1\) was reached/, JSON.stringify(synced.skipped)); }); + test("a watched mount's capped sync names its catch-up scan as the other turn", async (t) => { + const env = new Env(t); + for (const f of ['a.txt', 'b.txt']) env.writeDisk(f, f); + const m = await env.open({ walkCap: 1 }); + // Read as a watched mount from here; opened unwatched, so no watcher runs. + (m as any).mounts.get('work').config.watch = 'always'; + const synced = await call(m, 'sync', {}); + assert.ok(synced.incomplete.some((r: { path: string; reason: string }) => r.path === '' + && r.reason === 'the walk cap (1) was reached: the next sync of the mount, or its catch-up scan if that comes first, resumes after work/a.txt'), + JSON.stringify(synced.incomplete)); + }); + test("an agent-action scan takes the mount's start after every batch, and leaves the syncs' place alone", async (t) => { const env = new Env(t); for (const f of ['a.txt', 'b.txt', 'c.txt', 'd.txt']) env.writeDisk(f, f); const m = await env.open({ watch: 'on-agent-action', walkCap: 2 }); const agreement = (m as any).agreementOrThrow(); + const stored = (p: string) => { + const e = env.store.treeGet(TREE, p); + return e ? env.store.getBlob(e.blobHash)?.toString('utf8') : null; + }; + const capReports = () => env.events.filter((e) => e.type === 'workspace:agent-action-scan-incomplete').length; // What a shell tool did to the first file in the walk's order is in the - // workspace after every batch. + // workspace after every batch, and each capped scan says where the cap + // stopped it. for (const edit of ['edit one', 'edit number two', 'the third edit, longest']) { env.writeDisk('a.txt', edit); + const reported = capReports(); await m.onToolBatchComplete('agent'); - assert.equal(await contentOf(m, 'a.txt'), edit); + assert.equal(stored('a.txt'), edit); + assert.equal(capReports(), reported + 1, 'the batch\'s scan reported its cap'); } assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt'], 'the same two files every time'); assert.equal(agreement.walkResumesAfter('work', ''), undefined, 'and no place'); @@ -2263,6 +2286,10 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { assert.equal(agreement.walkResumesAfter('work', ''), 'b.txt'); await call(m, 'sync', {}); assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt', 'c.txt', 'd.txt']); + // A file a shell tool deleted among them is gone after the batch. + env.rmDisk('b.txt'); + await m.onToolBatchComplete('agent'); + assert.equal(stored('b.txt'), null); }); test("a watched mount's catch-up scan resumes where the last stopped", async (t) => { @@ -2760,7 +2787,8 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { return (seed >>> 8) % n; }; // U+FF21 sorts before U+1F600 by UTF-8 byte and after it by UTF-16 code - // unit, the walk's order; U+E000 sorts between them by both. + // unit, the walk's order; U+E000 sorts before both by UTF-8 byte and + // between them by UTF-16 code unit. const names = ['a', 'b', 'm', 'z', '\uFF21', '\u{1F600}', '\uE000']; const pick = (xs: T[]): T => xs[rand(xs.length)]!; const randomPath = (): string => { From 236db0b029959fd31e310a1b3e2211aab0b2bca6 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 17:39:26 -0400 Subject: [PATCH 3/5] test(workspace): count a place-only pass's store syncs with ctime trust off Greptile found that "a pass that moves only its place makes no store sync" depends on timing. On a Linux filesystem that maintains ctime, the setup's syncs hash the files within the racy window of their writes, so their fingerprints don't vouch. Once the files are older than that window, the next pass that rehashes one records a fresh fingerprint that vouches, and that evidence calls store.sync(). A runner slow enough for a counted pass to cross the window fails the count: with a 2.1 s pause before counting, the four passes made four syncs. The test now turns ctime trust off for its mount, so no fingerprint vouches and none is re-recorded, however slowly it runs: what's left for a pass to record is its place. The same pause now passes. --- test/workspace-three-way.test.ts | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/workspace-three-way.test.ts b/test/workspace-three-way.test.ts index 83b6f4b95..034b6edf3 100644 --- a/test/workspace-three-way.test.ts +++ b/test/workspace-three-way.test.ts @@ -2755,6 +2755,10 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { const env = new Env(t); for (const f of ['a.txt', 'b.txt', 'c.txt', 'd.txt']) env.writeDisk(f, f); const m = await env.open({ walkCap: 2 }); + // Without ctime trust no fingerprint vouches, so no pass re-records one + // once the files are older than the racy window, however slowly this + // runs: what's left to record is the place alone. + (m as any).ctimeTrust.set(realpathSync(env.dir), null); await call(m, 'sync', {}); await call(m, 'sync', {}); assert.deepEqual(syncedNames(env), ['a.txt', 'b.txt', 'c.txt', 'd.txt']); From 7e31e2a92263e5fb80af859171880bfebab58ee7 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 22:59:48 -0400 Subject: [PATCH 4/5] docs(workspace): the minor points of Anarchid's review of #376 (#272) R1-1. The changelog's upgrade note said each catching-up pass pushes a created event and a deleted event, and that a wakeOnChange mount turns each into a message. A pass pushes one event for each kind of change it finds, created, modified or deleted, and a mount turns into a message only the kinds its wakeOnChange admits. The note now says so, names the edits and deletions of tracked files past the old cap that the first passes find, and says that any pass listing a directory larger than the cap, a listing or an agent-action scan included, proves deletions there that 0.22 never proved. R1-2. The agent-action scan's comment said files past the cap come in through the reads that reach them, but a read takes in only a file the workspace doesn't hold yet (ensureSynced). It now names syncs and the listings that reach them. R1-3. Nothing read the sync tool's description, so #340's wording could come back unnoticed. A test now matches its two sentences about turns: the next sync of the same mount or directory picks up where the last turn stopped, and a watched mount's catch-up scan takes turns with its whole-mount syncs. With #340's sentence put back, it fails. --- changelog.d/272-walk-cap.fixed.md | 2 +- src/modules/workspace/index.ts | 11 ++++++----- test/workspace-three-way.test.ts | 9 +++++++++ 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/changelog.d/272-walk-cap.fixed.md b/changelog.d/272-walk-cap.fixed.md index d0254665c..728a4dd9f 100644 --- a/changelog.d/272-walk-cap.fixed.md +++ b/changelog.d/272-walk-cap.fixed.md @@ -3,5 +3,5 @@ - Each directory's listing is read whole, so a file deleted on disk is seen wherever the walk lists, past the cap included. Before, a directory holding more files than the cap never proved a deletion. - Because a watched mount's catch-up scan takes its turn in the rotation, on a mount over the cap a change made while the process was down reaches the workspace when the rotation comes to it, or when a sync of its path takes it. Before, the catch-up scan at start took a change among the first 5000 files every time, and never one past them. - The `sync` tool's description now says that a sync takes up to the mount's walk cap, and that the next sync of the same mount or directory picks up where the last turn stopped, with a watched mount's catch-up scan taking turns with its whole-mount syncs. So the first request after the upgrade that carries the workspace's tools misses the prompt cache from its tool block on. - - On a mount over 5000 entries, the first syncs and catch-up scans after the upgrade catch up: they take in files past the old cap that were never scanned, and prove deletions in directories larger than the cap, which 0.22 never proved. Each such pass pushes a `workspace:created` event listing every file it took in and a `workspace:deleted` event listing every deletion it proved, and a `wakeOnChange` mount turns each event into one message to the agent that lists them all. + - On a mount over 5000 entries, the first syncs and catch-up scans after the upgrade catch up on what 0.22's walks never reached: files past the old cap, never scanned, and edits and deletions of tracked files there. Each such pass pushes one event for each kind of change it finds (`workspace:created`, `workspace:modified` or `workspace:deleted`), each listing every path, and a mount whose `wakeOnChange` admits that kind turns each into one message to the agent. Any pass that lists a directory larger than the cap, a listing or an agent-action scan included, also proves deletions there that 0.22 never proved. - Going back to 0.22: its walks stop at the same 5000 files as before. It leaves the place alone, so coming forward again resumes there, unless 0.22 wrote a checkpoint meanwhile, which drops every place and starts each scope at its top. diff --git a/src/modules/workspace/index.ts b/src/modules/workspace/index.ts index 79d74be46..479000375 100644 --- a/src/modules/workspace/index.ts +++ b/src/modules/workspace/index.ts @@ -3012,11 +3012,12 @@ export class WorkspaceModule implements Module { * on disk (a shell command's files, a deletion) is in the workspace when the * agent next looks. On a mount over its walkCap, that holds for the first * walkCap entries in the walk's order, which every batch's scan takes; files - * past them come in through syncs, and the listings and reads that reach - * them. A scan that outlasts the deadline stops holding the - * round: the miss is recorded on the mount's status and pushed as a - * `workspace:agent-action-scan-incomplete` event, and the scan continues, - * deciding on the branch selected when it applies. + * past them come in through syncs and the listings that reach them, while a + * read takes in only a file the workspace doesn't hold yet. A scan that + * outlasts the deadline stops holding the round: the miss is recorded on + * the mount's status and pushed as a `workspace:agent-action-scan-incomplete` + * event, and the scan continues, deciding on the branch selected when it + * applies. */ async onToolBatchComplete(_agentName?: string): Promise { // Every agent's workspace is the store's current branch, so the agent diff --git a/test/workspace-three-way.test.ts b/test/workspace-three-way.test.ts index 034b6edf3..308eee651 100644 --- a/test/workspace-three-way.test.ts +++ b/test/workspace-three-way.test.ts @@ -2244,6 +2244,15 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { assert.match(skipped?.reason ?? '', /not visited: the walk cap \(1\) was reached/, JSON.stringify(synced.skipped)); }); + test("the sync tool's description says how the walk takes turns", async (t) => { + const env = new Env(t); + const m = await env.open({}); + const sync = m.getTools().find((tool: { name: string }) => tool.name === 'sync'); + assert.ok(sync); + assert.match(sync.description, /the next sync of the same mount or directory picks up where the last turn stopped/); + assert.match(sync.description, /its own catch-up scan, at start and when its watcher reattaches, takes turns with its whole-mount syncs/); + }); + test("a watched mount's capped sync names its catch-up scan as the other turn", async (t) => { const env = new Env(t); for (const f of ['a.txt', 'b.txt']) env.writeDisk(f, f); From 16f3e026f92983ec4050e91d7fb9b23ead93e0f3 Mon Sep 17 00:00:00 2001 From: Gladys-2413 Date: Thu, 15 Oct 2026 23:09:50 -0400 Subject: [PATCH 5/5] test(workspace): a listing and an agent-action scan prove a deletion in a large directory The changelog's upgrade note says that any pass listing a directory larger than the cap, a listing or an agent-action scan included, proves deletions there that 0.22 never proved. A test pinned it for a bare sync only; it now covers an ls and an agent-action scan the same way. --- test/workspace-three-way.test.ts | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/test/workspace-three-way.test.ts b/test/workspace-three-way.test.ts index 308eee651..250e929d8 100644 --- a/test/workspace-three-way.test.ts +++ b/test/workspace-three-way.test.ts @@ -2127,6 +2127,20 @@ describe('a mount larger than its walk cap (agent-framework #272)', () => { assert.notEqual(env.store.treeGet(TREE, 'big/d.txt'), null); }); + test('a listing, and an agent-action scan, prove a deletion in a directory larger than the cap too', async (t) => { + for (const via of ['ls', 'agent-action'] as const) { + const env = new Env(t); + const m = await env.open({ walkCap: 2, ...(via === 'agent-action' ? { watch: 'on-agent-action' } : {}) }); + for (const f of ['big/a.txt', 'big/b.txt', 'big/c.txt', 'big/d.txt', 'big/e.txt']) await seedSynced(env, m, f, f); + env.rmDisk('big/e.txt'); + // Each takes a.txt and b.txt; big's listing, read whole, lacks e.txt. + if (via === 'ls') await call(m, 'ls', { path: 'work/big' }); + else await m.onToolBatchComplete('agent'); + assert.equal(env.store.treeGet(TREE, 'big/e.txt'), null, via); + assert.notEqual(env.store.treeGet(TREE, 'big/d.txt'), null, via); + } + }); + test('a listing starts at its scope every time, and leaves no place behind', async (t) => { const env = new Env(t); for (const f of ['a.txt', 'b.txt', 'c.txt']) env.writeDisk(f, f);