[nissa-seru]

glob now starts its walk at the directory its pattern names before its first wildcard, and grep at the one its glob filter names. On a workspace mount larger than its walk cap, a search for one directory's files then finds them. Before, a walk of the whole mount could reach the cap before it got there, and the search answered with no matches.

Problem

One walk of a mount takes at most walkCap entries, 5,000 by default. glob walked its path, or each mount's root, and matched its pattern against what the walk took. grep walked the same way and applied its glob filter to what it took. On a mount past the cap, the walk could stop before it reached the directory the pattern names. The search then found nothing there, and listed the directories it never reached under incomplete.

A probe mounted six repositories and fourteen worktrees of agent-framework side by side, read-only, under one mount: about 6,000 files after a typical ignore list. It looked for membrane's sources with glob (membrane/src/**/*.ts) and with grep's filter. On main (c5d857b1), each found none of the 56 files on disk that match. The first four regions each listed under incomplete were:

code/connectome-host/changelog.d: the walk cap (5000) was reached before everything here was taken
code/context-manager: not visited: the walk cap (5000) was reached
code/discord-mcpl: not visited: the walk cap (5000) was reached
code/membrane: not visited: the walk cap (5000) was reached

Changes

In src/modules/workspace/index.ts:

• literalDirOf(pattern) gives the directory a pattern names before its first wildcard, made of its leading segments that hold none of globToRegex's wildcards (*, ?, {), short of the last segment.
• walkStart(mount, base, beneath) gives where a search's walk starts. It takes the search's path as given, then each segment of the pattern's directory while disk lists that segment, in the directory reached so far, as a directory with exactly that spelling. It stops before a segment spelled any other way, not there, or not a directory, a link included. It lists only directories it has checked lie inside the mount, and none the mount passes over by design.
• glob starts its walk at walkStart of its path and its pattern's directory.
• grep starts at the deeper of its path and its filter's directory, the same way. The filter matches paths relative to the mount, so its directory is relative to the mount too. When neither directory holds the other, no file beneath the path can pass the filter, and grep searches nothing in that mount.
• deeperOf(a, b) gives the deeper of two directories, or null when neither holds the other.

In src/modules/workspace/types.ts, the doc of walkCap says where each search's walk starts, and the doc of initialSyncDone says the flag records that a pass over the whole mount has run.

The changelog fragment is changelog.d/search-starts-at-pattern-dir.fixed.md.

With #401. #401 lets a pattern start with the mount's name, so project/src/**/*.ts in mount project also matches src/a.ts there. The two pull requests conflict in glob and grep. Merged by taking both sides of each conflict, #401's seven tests and this pull request's pass. A probe of that merge, beside #401 merged onto main alone, shows two gaps (its source is under Tests):

• A mount-prefixed pattern doesn't get this pull request's start. Its directory as written isn't one disk lists in the mount, so its walk starts at the mount's root. Past a walk cap of 2, in mount notes, notes/diary/*.md finds nothing, where diary/*.md finds notes/diary/x.md.
• In a mount holding a directory named like itself, the walk starts in that directory. In mount project, which holds project/README.md and src/main.ts, project/src/*.ts finds nothing, where fix(workspace): glob and grep's glob filter take a mount-prefixed pattern #401 alone finds project/src/main.ts.

Closing both gaps needs each reading of a pattern to start its own walk.

With #468. #468 refuses a path through a symbolic link to a directory, on a mount that doesn't follow symlinks. The two conflict only in two import lines. Merged, with both sides of those lines kept, this pull request's tests pass, and so do the workspace-* test files, 514 tests.

What each part is for, and what was cut

• Starting at the pattern's literal directory. Every path the pattern matches lies beneath that directory, since globToRegex anchors its regex and reads these segments literally. So the walk misses no match, and the cap counts only entries beneath it.
• Leaving out the last segment. It names the files, so the directory it lies in is where the walk starts. walkStart would stop at a file anyway, so leaving the segment in fails no test. Leaving it out spares walkStart listing that directory, which the walk lists next anyway.
• Stopping at a wildcard even where a directory has the segment's name. A directory can be named l?b or {a,b}. l?b/x.ts also matches lib/x.ts, and a walk started at l?b would miss it.
• Starting only where disk lists each segment with that spelling. The walk names what it finds from its start's spelling. On a case-insensitive disk, as macOS's is by default, a walk started at SRC for a directory listed as src would, by our reading, name everything SRC/…, so SRC/*.ts would take src/a.ts in a second time, as SRC/a.ts, and announce it as created. From the deepest directory disk lists, the walk names files as disk lists them, and the case-sensitive pattern matches none, as before. A segment that's empty, . or .., which no path holds, stops it the same way.
• Checking each directory before listing it, and stopping before a link or a directory passed over by design. To choose a start, it lists only directories it has checked lie inside the mount, and none the mount passes over by design, such as one on its ignore list or another mount's. A walk of the whole mount doesn't go through a link to a directory, so a pattern reaches nothing through one that it didn't reach before.
• grep's deeper directory. The path and the filter each narrow the search, and every file both allow lies beneath the deeper of the two.
• Searching nothing when they don't overlap. No file could pass the filter, and a walk there would take in disk changes that nothing asked about.
• Not taken: telling a missing directory from one spelled another way. A missing directory falls back to its parent too, which walks more than it needs to. Telling them apart would take a second check per segment, for a pattern that names nothing.
• Not taken: letting a search walk past the cap. The cap bounds what one call observes. A search past it would have to answer in pages or walk the whole mount. A pattern that starts with a wildcard, such as **/*.ts, still walks its whole scope and can still stop at the cap, and its answer names what it didn't reach, as before.
• Not taken: pruning the walk by the rest of the pattern, such as visiting only src directories one level down for */src/*.ts. That would change the walk every pass shares, for patterns that don't start with their directory.

What callers will notice

• Past the cap, a pattern or filter that names a directory finds the files there, as long as that directory's own walk fits the cap.
• A search's incomplete names only what lies beneath the directory its walk starts at. When that's the pattern's directory, regions elsewhere in the mount, where nothing could match, are no longer named.
• Within the cap, a search's answer is otherwise the same, with one exception. A pattern into a directory the mount passes over by design, such as one on its ignore list or another mount's directory, now names that directory under incomplete, as a search given that directory as its path does. Before, a walk of the whole mount passed it over, and named it only when it held files the store tracked, and never when it was another mount's.
• A pattern whose directory disk doesn't list, under that spelling or at all, walks from the deepest directory disk does list, as a search walked from its path or the mount's root before.
• A grep whose path and filter don't overlap searches nothing and takes nothing in from disk. Before, it walked the path, and named under incomplete what that walk couldn't reach.
• A search's walk takes in, and announces as workspace events, only what changed on disk beneath its start. Changes elsewhere wait for the next pass that covers them. A search that starts below the mount's root no longer marks the mount's first whole sync done in status (initialSyncDone).

Tests

Ten new tests in test/workspace-three-way.test.ts, in "a search starts at the directory its pattern names":

• glob past the cap: at a walk cap of 2, z/deep/*.ts finds z/deep's two files, and so do deep/*.ts with path: work/z and *.ts with path: work/z/deep. **/*.ts stops in a and finds none.
• grep past the cap: the filter z/deep/*.ts finds the match, and so do a filter deeper than the path, a path deeper than the filter, and a filter naming the path's own directory.
• within the cap: eleven patterns, as glob patterns and as grep filters, find what a walk of the whole mount finds, and none leaves anything under incomplete. Among them are src/{lib,x}/*.js, src/l?b/*, ./src/*.ts, ../work/src/*.ts, src//a.ts and README.md/*.ts.
• wildcard names: with directories lib, l?b, a, b, {a,b} and *, the patterns l?b/x.ts, {a,b}/x.ts and */x.ts each find every directory they match.
• no overlap: grep with path: work/test finds nothing with the filter src/**, nor with testing/**, a sibling whose name starts with the path's, and takes none of the three directories in.
• another spelling: beside src/a.ts, SRC/*.ts finds nothing, takes nothing in as SRC/a.ts, and walks from the root, which takes src/a.ts in under its own spelling. Its first two checks can fail only on a case-insensitive disk.
• a directory disk doesn't list: at a walk cap of 2, z/nowhere/*.ts walks from z, and names z/deep, which that walk didn't reach, under incomplete.
• what a walk takes in: src/*.ts takes in src/a.ts and announces it, and leaves docs/b.md to a later pass. initialSyncDone stays false through it and a grep whose filter names src, until **/*.md walks the whole mount.
• by design: vendor/lib/*.js, with vendor on the ignore list, and a/b/sub/*.js, with a/b another mount's directory, each name that directory under incomplete, as glob and grep, with the region a path naming it gets.
• linked directories: link/*.ts, through a link inside the mount, and out/sub/*.ts, through one leading out of it, find nothing and take nothing in, as glob and grep. sub/*.ts with path: work/out names work/out under incomplete, as before.

At b3026f64, on main c5d857b1, on Linux with Node 24.22.0:

• tsc --noEmit and the build are clean.

• The ten new tests pass, and so do the eleven workspace-* test files, 462 tests in all.

• Red-first: with main's src/ in place, six of the ten fail. Four pass there too, as controls that pin what doesn't change: the tests within the cap, of wildcard names, of linked directories, and of another spelling, which on Linux passes either way.

• Removals: sixteen variants of the change, each run alone, fail at least one of the ten:

  • glob or grep starting at its path, grep starting at the deeper directory without walkStart, and grep walking its path when it doesn't overlap the filter;
  • walkStart taking the pattern's directory unchecked, starting at the mount's root when the pattern has no directory, listing a directory it hasn't checked, going down through a link, or going down into a directory passed over by design;
  • literalDirOf reading *, ? or { as literal;
  • deeperOf always giving the filter's directory, or always the path, dropping its equal-directory case, or matching a prefix without its /.

  A seventeenth, literalDirOf keeping the last segment, fails none, as "What each part is for" says.

• The probe above, at this head: glob found 56 files and grep's filter 56, the 56 on disk, with nothing under incomplete. Its source is below. Its agent-framework and fourteen worktrees are at 3adbdb86, and the other five repositories are clones of their default branch, taken for each run. The run at this head counted 6,011 files, and the run on main, minutes later, 6,015.

The probe (run with npx tsx from the repository's root, with dependencies installed)

// A mount past its walk cap: six repositories and fourteen worktrees of
// agent-framework side by side under one read-only mount, with a typical
// ignore list. Prints what glob and grep's filter find for membrane's
// sources, beside the count on disk.
import { spawnSync } from 'node:child_process';
import { mkdirSync, rmSync } from 'node:fs';
import { join } from 'node:path';
import { JsStore } from '@animalabs/chronicle';
import { WorkspaceModule } from './src/modules/workspace/index.js';

const ROOT = join(process.env.TMPDIR ?? '/tmp', 'breadth-glob');
const IGNORE = ['node_modules', 'dist', '.git', 'target', '__pycache__', '.venv', 'build', 'coverage'];
const AF_COMMIT = '3adbdb869233ee0d3bd5b63e78ec81a196104291';
const sh = (cmd: string, cwd: string) => {
  const r = spawnSync('bash', ['-c', cmd], { cwd, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });
  if (r.status !== 0) throw new Error(`${cmd}: ${r.stderr}`);
  return r.stdout;
};

rmSync(ROOT, { recursive: true, force: true });
const tree = join(ROOT, 'tree');
mkdirSync(tree, { recursive: true });
for (const [dir, url] of [
  ['agent-framework', 'https://github.com/anima-research/agent-framework.git'],
  ['membrane', 'https://github.com/antra-tess/membrane.git'],
  ['context-manager', 'https://github.com/anima-research/context-manager.git'],
  ['connectome-host', 'https://github.com/anima-research/connectome-host.git'],
  ['chronicle', 'https://github.com/anima-research/chronicle.git'],
  ['discord-mcpl', 'https://github.com/anima-research/discord-mcpl.git'],
]) sh(`git clone --quiet ${url} ${dir}`, tree);
sh(`git checkout -q ${AF_COMMIT}`, join(tree, 'agent-framework'));
for (let i = 1; i <= 14; i++) {
  const n = String(i).padStart(2, '0');
  sh(`git worktree add -q ../af-wt-${n} -b wt-${n} ${AF_COMMIT}`, join(tree, 'agent-framework'));
}
const prune = `\\( ${IGNORE.map((n) => `-name ${JSON.stringify(n)}`).join(' -o ')} \\) -prune -o`;
const filesInMount = Number(sh(`find . ${prune} ! -type d -print | wc -l`, tree).trim());
// What 'membrane/src/**/*.ts' matches: a .ts file at least one directory below membrane/src.
const onDisk = Number(sh(`find membrane/src -mindepth 2 -name '*.ts' -type f | wc -l`, tree).trim());
const membraneFiles = Number(sh(`find membrane/src -name '*.ts' -type f | wc -l`, tree).trim());

const store = JsStore.openOrCreate({ path: join(ROOT, 'ws.chronicle') });
const ws = new WorkspaceModule({ mounts: [{ name: 'code', path: tree, mode: 'read-only', watch: 'never', ignore: IGNORE }] });
ws.initStore(store);
await ws.start({ isRestart: false, getState: () => null, setState: () => {}, pushEvent: () => {} } as never);
const call = async (name: string, input: Record<string, unknown>) =>
  (await ws.handleToolCall({ id: name, name, input } as never)) as { success: boolean; data?: any; error?: string };

const glob = await call('glob', { pattern: 'membrane/src/**/*.ts' });
const grep = await call('grep', { pattern: 'export', glob: 'membrane/src/**/*.ts' });
const incompleteOf = (d: any) => (d?.incomplete ?? []).map((r: { path: string; reason: string }) => `${r.path}: ${r.reason}`).slice(0, 4);
console.log(JSON.stringify({
  filesInMount,
  walkCap: 5000,
  onDiskMatching: onDisk,
  membraneTsFiles: membraneFiles,
  glob: { ok: glob.success, count: glob.data?.count ?? null, incomplete: incompleteOf(glob.data), error: glob.error ?? null },
  grepFilter: { ok: grep.success, files: grep.data?.results?.length ?? null, incomplete: incompleteOf(grep.data), error: grep.error ?? null },
}, null, 1));
await ws.stop();
store.close();

The probe of #401 merged with this pull request

// What a mount-prefixed pattern (#401) finds, with and without this pull
// request's start. Prints JSON.
//   notes: a mount past its walk cap (2): a/1..3.txt, then diary/x.md.
//   project: a mount holding a directory named like itself, beside src/.
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { JsStore } from '@animalabs/chronicle';
import { WorkspaceModule } from './src/modules/workspace/index.js';

const root = mkdtempSync(join(tmpdir(), 'probe-401-'));
const files: Record<string, string> = {
  'notes/a/1.txt': 'f', 'notes/a/2.txt': 'f', 'notes/a/3.txt': 'f', 'notes/diary/x.md': 'entry',
  'project/project/README.md': 'inner', 'project/src/main.ts': 'export const x = 1;',
};
for (const [path, content] of Object.entries(files)) {
  mkdirSync(dirname(join(root, path)), { recursive: true });
  writeFileSync(join(root, path), content);
}
const results: Record<string, unknown> = {};
const cases: Array<[string, string, Record<string, unknown>]> = [
  ['notes', 'glob', { pattern: 'notes/diary/*.md' }],
  ['notes', 'glob', { pattern: 'diary/*.md' }],
  ['project', 'glob', { pattern: 'project/src/*.ts' }],
  ['project', 'glob', { pattern: 'src/*.ts' }],
];
for (const [mountName, tool, input] of cases) {
  // A fresh store and module per case: no search takes in what another walked.
  const storeDir = mkdtempSync(join(tmpdir(), 'probe-401-store-'));
  const store = JsStore.openOrCreate({ path: join(storeDir, 'ws') });
  const ws = new WorkspaceModule({ mounts: [{ name: mountName, path: join(root, mountName), mode: 'read-only', watch: 'never', walkCap: 2 }] });
  ws.initStore(store);
  const r = (await ws.handleToolCall({ id: 'p', name: tool, input } as never)) as { success: boolean; data?: any; error?: string };
  results[`${mountName} ${tool} ${JSON.stringify(input)}`] = r.success
    ? { matches: (r.data.matches ?? []).map((m: { path: string }) => m.path), incomplete: (r.data.incomplete ?? []).map((x: { path: string; reason: string }) => `${x.path}: ${x.reason}`) }
    : { error: r.error };
  store.close();
  rmSync(storeDir, { recursive: true, force: true });
}
rmSync(root, { recursive: true, force: true });
console.log(JSON.stringify(results, null, 1));

Not verified

• A case-insensitive disk, as on macOS by default. The first two checks of the test of another spelling can fail only there, and on Linux that test passes before this change too. CI's macOS runner runs it.
• Node 20, and macOS, for these tests: CI runs them.

Found, not changed here

• A path spelled in another case. parsePath takes a path's spelling as it stands. On a case-insensitive disk, by our reading of the code, a path spelled in another case reaches the directory disk lists under its own spelling, and what a tool finds or writes there is named, and stored, under the path's spelling. We haven't run a case-insensitive disk. glob, grep, ls, read and write all take their path this way, so it needs one fix where paths are parsed, not one in each search.

---

• Changelog fragment added: changelog.d/search-starts-at-pattern-dir.fixed.md

Written by Beth-2709 (Claude Opus 5.5), on Ada-1017's calls, from Gladys-2592's design. Phoebe-2419 found the defect, measuring the workspace at breadth. Esther-2584 (Claude Opus 5.5) reviewed the change, and two rounds of internal review read it before it opened.

Description checked and posted by Fiona-2757 (Claude Opus 5.5).

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's Pi tooling, and posted from the shared nissa-seru account.

[greptile-apps]

Required label not found on this PR.