From f09fe482ef4a3830f58168fd78851cec2e60e98e Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sat, 14 Nov 2026 23:28:56 -0500 Subject: [PATCH 01/25] feat: discover ~/.codex/worktrees and a weekly home folder scan as reduce-only checkouts --- llp/0004-discovered-checkouts.decision.md | 59 +++++- packages/core/state/discovery-scan.ts | 56 ++++++ packages/core/state/index.ts | 1 + packages/core/state/paths.ts | 4 + .../__tests__/discovered-reduce-only.test.ts | 138 +++++++++++++ .../src/__tests__/discovery-protected.test.ts | 16 +- .../src/__tests__/discovery-scan.test.ts | 151 ++++++++++++++ .../stim-cli/src/__tests__/discovery.test.ts | 35 ++++ .../stim-cli/src/commands/gc/worktrees.ts | 13 +- packages/stim-cli/src/guide/cleanup.ts | 22 ++- packages/stim-cli/src/stimd/passes/run.ts | 2 + .../src/workspace/discovered-removal.ts | 5 +- .../stim-cli/src/workspace/discovery-scan.ts | 184 ++++++++++++++++++ packages/stim-cli/src/workspace/discovery.ts | 66 ++++++- website/docs/worktrees.md | 6 +- 15 files changed, 739 insertions(+), 19 deletions(-) create mode 100644 packages/core/state/discovery-scan.ts create mode 100644 packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts create mode 100644 packages/stim-cli/src/__tests__/discovery-scan.test.ts create mode 100644 packages/stim-cli/src/workspace/discovery-scan.ts diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index a6f3541f84..aec0503851 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -57,6 +57,62 @@ the maintainer's words. Everything else in this section was chosen by the coordinator and the review of #5519, not by the maintainer. +## Tool folders and the full scan + +Discovery also reads two more sources, and what they find is reduce-only: Stim +reduces its build outputs and, in a linked worktree, `node_modules` and Pods by +the discovered-checkout rules, and never removes the checkout or worktree. The +hold that keeps a worktree inside a dot-folder of its main checkout +(`.claude/worktrees`) applies to it. + +- Common tool folders: linked worktrees of known repositories in + `~/.codex/worktrees`. The list is one constant with that one path. +- The full scan: a pruned walk of the home folder for checkouts that Stim's + project detection recognises and discovery does not reach. The daemon runs it + weekly and, while the disk level is above normal, at most hourly; a command + never runs it. It reads `~/Documents`, skips `~/Desktop`, `~/Downloads`, + `~/Library`, other media folders, dot-folders, symlinks, other volumes and + dependency, build and cache folders, stops at a recognised project and goes + seven levels deep. A folder the system refuses is skipped and recorded, never + an error. The result is stored in `maintenance/discovery-scan.json` under the + Stim home (when it ran, what it found, what it could not read) and discovery + reads that file, so a maintenance pass or `stim gc` never walks the disk. +- At most 40 reduce-only checkouts are handled per discovery, in a window over + them that moves every 15 minutes. + +A folder named in `maintenance.discoverRoots` keeps the full rules. A scoped +`STIM_HOME` discovers nothing from either source. + +Reason: on one MacBook `~/.codex/worktrees` held 53 worktrees and 12 GB that +Stim listed once as outside the folders it manages, with no size and no way +forward. + +Maintainer decision, 2026-11-14 (#5762), verbatim: + +> ok so lets do: +> discovery - current logic + common paths + full scan once in a while (also triggered more often on low disk), full scan can try to be fast, ie like not scan places we know theres gonna be no projects and lots of files so can keep it fast ish, prob will trigger permission promts for Documents folder prob the only one it makes sense to scan but I guess its fine. +> +> discovered workspaces - cleaned using our safer rules we have already, doctor lists them and suggest adding as managed path or delete if idle for long time + +Earlier the same evening, on including common tool folders by default: "Could +these common paths just be included by default in Discovered checkout". + +The coordinator's reading, not the maintainer's words, for the points that +decision leaves open: + +1. Checkouts found only through a common path or the full scan are reduced and + never removed by Stim. Removal is what doctor suggests to the person. +2. Common paths start with `~/.codex/worktrees`. Others are added only with + evidence that a tool uses that path. +3. "Once in a while" is weekly; on low disk the scan runs when a volume's level + rises above normal, at most once an hour. +4. Doctor reads the last scan's result and does not run the scan. +5. `~/Desktop`, `~/Downloads` and other volumes are not scanned. +6. The bound of 40 per discovery, the 15 minute window, the depth of seven and + the folders pruned are implementation choices, not decisions. + +Doctor's listing and the line printed on low disk are not part of this change. + ## Removing a discovered worktree needs more evidence A discovered linked worktree follows the registered-workspace rules for build @@ -91,8 +147,9 @@ Chosen by the coordinator and the reviews of #5519, not by the maintainer. ## Links -- Issues: #5512, #5515, #5757. Pull request: #5519. +- Issues: #5512, #5515, #5757, #5762. Pull request: #5519. - Code: `packages/stim-cli/src/workspace/discovery.ts`, + `packages/stim-cli/src/workspace/discovery-scan.ts`, `packages/stim-cli/src/workspace/discovered-removal.ts`, `packages/stim-cli/src/workspace/protected-locations.ts`, `packages/stim-cli/src/workspace/hidden-edits.ts`, diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts new file mode 100644 index 0000000000..c455404598 --- /dev/null +++ b/packages/core/state/discovery-scan.ts @@ -0,0 +1,56 @@ +import { isJsonObject, readJsonObject } from './json-file.ts'; +import { discoveryScanFile } from './paths.ts'; + +/** + * What the last full scan of the home folder found: the checkouts Stim's project detection recognises that + * `maintenance.discoverRoots`, registered projects and the common tool folders do not reach, and what it could not read. + */ +export interface DiscoveryScan { + version: 1; + /** When the scan started, epoch ms. A scan that failed or ran out of time is stamped too. */ + ranAt: number; + durationMs: number; + trigger: 'weekly' | 'low-disk'; + /** Folders the scan read. */ + visited: number; + /** Canonical paths of the recognised checkouts. */ + found: string[]; + /** Folders the system refused or that did not answer, skipped without error. */ + unreadable: string[]; + /** The scan stopped at its time or folder limit before it finished. */ + truncated: boolean; +} + +const strings = (value: unknown): value is string[] => + Array.isArray(value) && value.every((entry) => typeof entry === 'string'); +const finite = (value: unknown): value is number => typeof value === 'number' && Number.isFinite(value); + +/** Rejects missing, corrupt or unsupported payloads without throwing. */ +export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { + if ( + !isJsonObject(value) || + value.version !== 1 || + !finite(value.ranAt) || + !finite(value.durationMs) || + (value.trigger !== 'weekly' && value.trigger !== 'low-disk') || + !finite(value.visited) || + !strings(value.found) || + !strings(value.unreadable) || + typeof value.truncated !== 'boolean' + ) + return null; + return { + version: 1, + ranAt: value.ranAt, + durationMs: value.durationMs, + trigger: value.trigger, + visited: value.visited, + found: value.found, + unreadable: value.unreadable, + truncated: value.truncated, + }; +} + +export function readDiscoveryScan(): DiscoveryScan | null { + return parseDiscoveryScan(readJsonObject(discoveryScanFile())); +} diff --git a/packages/core/state/index.ts b/packages/core/state/index.ts index 3a26000e9e..a710b757f3 100644 --- a/packages/core/state/index.ts +++ b/packages/core/state/index.ts @@ -52,6 +52,7 @@ export * from './tailnet.ts'; export * from './server-service.ts'; export * from './hosted-logs.ts'; export * from './maintenance.ts'; +export * from './discovery-scan.ts'; export * from './gc-log.ts'; export * from './build-ledger.ts'; export * from './agent-device.ts'; diff --git a/packages/core/state/paths.ts b/packages/core/state/paths.ts index 7293dee392..b2f70936e4 100644 --- a/packages/core/state/paths.ts +++ b/packages/core/state/paths.ts @@ -240,6 +240,10 @@ export function maintenanceStateFile(): string { return join(maintenanceDir(), 'state.json'); } +export function discoveryScanFile(): string { + return join(maintenanceDir(), 'discovery-scan.json'); +} + export function gcLogDir(): string { return join(configDir(), 'gc'); } diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts new file mode 100644 index 0000000000..78a31b6b9f --- /dev/null +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -0,0 +1,138 @@ +import { execFileSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + realpathSync, + rmSync, + utimesSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { saveConfig } from '../workspace/config.ts'; +import { collectWorktreeSweep, removeWorktrees } from '../commands/gc/worktrees.ts'; +import { + boundReduceOnly, + discoverCheckouts, + discoveredOutside, + type DiscoveredCheckout, +} from '../workspace/discovery.ts'; + +vi.mock('../workspace/protected-locations.ts', async (original) => { + const real = await original(); + return { ...real, isProtectedLocation: () => false }; +}); + +const day = 86_400_000; + +describe('checkouts found through a tool folder or the full scan', () => { + let base: string; + let main: string; + const old = new Date(Date.now() - 3 * day); + const git = (cwd: string, ...args: string[]) => + execFileSync('git', ['-C', cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', ...args], { + encoding: 'utf8', + env: { ...process.env, GIT_AUTHOR_DATE: old.toISOString(), GIT_COMMITTER_DATE: old.toISOString() }, + }); + const age = (dir: string) => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const path = join(dir, entry.name); + if (entry.isDirectory() && entry.name !== '.git') age(path); + utimesSync(path, old, old); + } + utimesSync(dir, old, old); + }; + const worktree = (path: string, name: string) => { + mkdirSync(join(path, '..'), { recursive: true }); + git(main, 'worktree', 'add', '-q', '-b', name, path); + const gitdir = git(path, 'rev-parse', '--absolute-git-dir').trim(); + age(path); + for (const file of ['index', 'HEAD', join('logs', 'HEAD')]) utimesSync(join(gitdir, file), old, old); + return realpathSync(path); + }; + + beforeEach(() => { + base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-reduce-only-'))); + vi.stubEnv('HOME', join(base, 'home')); + vi.stubEnv('STIM_HOME', ''); + main = join(base, 'dev', 'main'); + mkdirSync(main, { recursive: true }); + mkdirSync(join(base, 'home')); + writeFileSync(join(main, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + git(main, 'init', '-q', '-b', 'main'); + git(base, 'init', '-q', '--bare', '-b', 'main', join(base, 'remote.git')); + git(main, 'remote', 'add', 'origin', join(base, 'remote.git')); + git(main, 'add', '-A'); + git(main, 'commit', '-q', '-m', 'init'); + git(main, 'push', '-q', '-u', 'origin', 'main'); + git(main, 'remote', 'set-head', 'origin', 'main'); + saveConfig({ version: 2, projects: { [main]: {} }, repos: {} }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + rmSync(base, { recursive: true, force: true }); + }); + + it('discovers a ~/.codex/worktrees worktree of a known repository as reduce-only, and no other home dot-folder worktree', async () => { + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); + const other = worktree(join(base, 'home', '.other', 'wt'), 'other'); + const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); + const found = await discoverCheckouts({ projects: { [main]: {} }, repos: {} }, [main]); + expect(found.map(({ path, reduceOnly }) => [path, reduceOnly])).toEqual([ + [sibling, undefined], + [codex, true], + ]); + expect((await discoveredOutside({ projects: { [main]: {} }, repos: {} }, [main])).map(({ path }) => path)).toEqual([ + other, + ]); + }); + + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); + const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); + const planned = await collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false }); + const byPath = Object.fromEntries(planned.worktrees.map((entry) => [entry.path, entry])); + expect(byPath[sibling]?.skipped).toBeNull(); + expect(byPath[codex]).toMatchObject({ + discovered: true, + reduceOnly: true, + skipped: expect.stringContaining('never removes'), + }); + const worktrees = planned.worktrees.filter((entry) => entry.path === codex); + for (const entry of worktrees) Object.assign(entry, { skipped: null, skipCode: null }); + vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + await removeWorktrees({ ...planned, worktrees }, { now: Date.now() }); + expect(existsSync(codex)).toBe(true); + }); +}); + +describe('boundReduceOnly', () => { + const entry = (n: number, reduceOnly = true): DiscoveredCheckout => ({ + path: `/p/${String(n).padStart(3, '0')}`, + origin: 'scan', + from: 'full scan', + ...(reduceOnly ? { reduceOnly: true } : {}), + }); + const window = 15 * 60_000; + + it('keeps every full-rule checkout, and a moving window of at most the limit of the reduce-only ones', () => { + const all = [entry(1, false), ...Array.from({ length: 10 }, (_, i) => entry(i + 10))]; + const now = 100 * window; + const first = boundReduceOnly(all, now, 4); + expect(first.filter((e) => e.reduceOnly)).toHaveLength(4); + expect(first[0]).toBe(all[0]); + const seen = new Set(); + for (let i = 0; i < 10; i++) for (const e of boundReduceOnly(all, now + i * window, 4)) seen.add(e.path); + expect(seen.size).toBe(all.length); + }); + + it('returns everything when the reduce-only ones fit the limit', () => { + const all = [entry(1), entry(2)]; + expect(boundReduceOnly(all, 0, 4)).toEqual(all); + }); +}); diff --git a/packages/stim-cli/src/__tests__/discovery-protected.test.ts b/packages/stim-cli/src/__tests__/discovery-protected.test.ts index e60f2d2aed..ec81f4456b 100644 --- a/packages/stim-cli/src/__tests__/discovery-protected.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-protected.test.ts @@ -1,7 +1,8 @@ import { mkdirSync, mkdtempSync, realpathSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { worktreeInScope } from '../workspace/discovery.ts'; +import { inToolFolder, worktreeInScope } from '../workspace/discovery.ts'; +import { isProtectedLocation } from '../workspace/protected-locations.ts'; describe('a protected location and maintenance.discoverRoots, with the real protection rule', () => { let base: string; @@ -21,4 +22,17 @@ describe('a protected location and maintenance.discoverRoots, with the real prot expect(worktreeInScope(join(roots, 'wt'), { ...where, roots: [] })).toBe(false); expect(worktreeInScope(join(base, 'other', 'wt'), { ...where, roots: [roots] })).toBe(false); }); + + it('protects every home dot-folder, yet names only ~/.codex/worktrees as a tool folder', () => { + const home = join(base, 'home'); + for (const dir of ['.codex/worktrees/a/app', '.codex/sessions/a', '.claude/worktrees/a', 'dev/a']) + mkdirSync(join(home, dir), { recursive: true }); + const roots = { home, temps: [] }; + expect(isProtectedLocation(join(home, '.codex', 'worktrees', 'a', 'app'), roots)).toBe(true); + expect(isProtectedLocation(join(home, 'dev', 'a'), roots)).toBe(false); + expect(inToolFolder(join(home, '.codex', 'worktrees', 'a', 'app'), home)).toBe(true); + expect(inToolFolder(join(home, '.codex', 'sessions', 'a'), home)).toBe(false); + expect(inToolFolder(join(home, '.claude', 'worktrees', 'a'), home)).toBe(false); + expect(inToolFolder(join(home, 'dev', 'a'), home)).toBe(false); + }); }); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts new file mode 100644 index 0000000000..fa0fb21d7d --- /dev/null +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -0,0 +1,151 @@ +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, + symlinkSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { testContext } from './_context.ts'; +import { discoveryScanFile, readDiscoveryScan } from '@stim-cli/core/state'; +import { + discoveryScanDue, + runDiscoveryScanIfDue, + scanDescends, + scanForCheckouts, +} from '../workspace/discovery-scan.ts'; + +const marked = (path: string) => existsSync(join(path, 'package.json')); + +describe('scanDescends', () => { + it('skips dot-folders, dependency and build folders everywhere, and the excluded folders of the home folder only', () => { + for (const name of ['.git', '.cache', 'node_modules', 'Pods', 'build', 'target', 'Library']) + expect(scanDescends(name, 3)).toBe(false); + for (const name of ['Desktop', 'Downloads', 'Movies']) { + expect(scanDescends(name, 1)).toBe(false); + expect(scanDescends(name, 2)).toBe(true); + } + expect(scanDescends('Documents', 1)).toBe(true); + expect(scanDescends('dev', 1)).toBe(true); + }); +}); + +describe('scanForCheckouts', () => { + let home: string; + const project = (...parts: string[]) => { + mkdirSync(join(home, ...parts), { recursive: true }); + writeFileSync(join(home, ...parts, 'package.json'), '{}'); + }; + + beforeEach(() => { + home = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-'))); + }); + + afterEach(() => rmSync(home, { recursive: true, force: true })); + + it('finds projects at depth, stops at one, and never reads pruned, dot, excluded or linked folders', async () => { + project('Documents', 'a'); + project('dev', 'deep', 'er', 'b'); + project('dev', 'c'); + mkdirSync(join(home, 'dev', 'c', 'inner')); + project('dev', 'c', 'inner'); + for (const skipped of [ + ['Desktop', 'x'], + ['Downloads', 'x'], + ['dev', 'node_modules', 'x'], + ['dev', '.hidden', 'x'], + ['dev', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'too-deep'], + ]) + project(...skipped); + symlinkSync(join(home, 'Documents', 'a'), join(home, 'dev', 'linked')); + const scan = await scanForCheckouts(home, marked); + expect(scan.found.toSorted()).toEqual([ + join(home, 'Documents', 'a'), + join(home, 'dev', 'c'), + join(home, 'dev', 'deep', 'er', 'b'), + ]); + expect(scan.truncated).toBe(false); + expect(scan.unreadable).toEqual([]); + }); + + it('records a folder the system refuses as unreadable and goes on', async () => { + project('dev', 'ok'); + mkdirSync(join(home, 'Documents')); + mkdirSync(join(home, 'Documents', 'locked'), { mode: 0o000 }); + try { + const scan = await scanForCheckouts(home, marked); + expect(scan.found).toEqual([join(home, 'dev', 'ok')]); + const refused = process.platform !== 'win32' && process.getuid?.() !== 0; + expect(scan.unreadable).toEqual(refused ? [join(home, 'Documents', 'locked')] : []); + } finally { + chmodSync(join(home, 'Documents', 'locked'), 0o700); + } + }); + + it('stops at its time budget and says so', async () => { + project('dev', 'one'); + const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); + expect(scan).toMatchObject({ found: [], truncated: true }); + }); +}); + +describe('discoveryScanDue', () => { + const now = Date.UTC(2026, 10, 14); + const ago = (ms: number) => ({ ranAt: now - ms }); + it('runs weekly, hourly at low disk, and when no scan or a future one is recorded', () => { + expect(discoveryScanDue(null, now, false)).toBe(true); + expect(discoveryScanDue(ago(-1000), now, false)).toBe(true); + expect(discoveryScanDue(ago(6 * 86_400_000), now, false)).toBe(false); + expect(discoveryScanDue(ago(7 * 86_400_000), now, false)).toBe(true); + expect(discoveryScanDue(ago(30 * 60_000), now, true)).toBe(false); + expect(discoveryScanDue(ago(61 * 60_000), now, true)).toBe(true); + }); +}); + +describe('runDiscoveryScanIfDue', () => { + let base: string; + const tick = (env: NodeJS.ProcessEnv) => ({ operation: testContext({ env }) }); + + beforeEach(() => { + base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-run-'))); + mkdirSync(join(base, 'home', 'dev', 'app'), { recursive: true }); + writeFileSync(join(base, 'home', 'dev', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + }); + + afterEach(() => { + vi.unstubAllEnvs(); + rmSync(base, { recursive: true, force: true }); + }); + + it('never scans for a command, only for a daemon tick', async () => { + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', join(base, 'machine')); + mkdirSync(join(base, 'machine')); + await runDiscoveryScanIfDue(undefined, null, join(base, 'home')); + expect(readDiscoveryScan()).toBeNull(); + }); + + it('never scans in a scoped home and writes nothing', async () => { + vi.stubEnv('STIM_HOME', join(base, 'state')); + await runDiscoveryScanIfDue(tick({ STIM_HOME: join(base, 'state') }), null, join(base, 'home')); + expect(existsSync(join(base, 'state'))).toBe(false); + }); + + it('stamps the result under the Stim home and does not scan again until it is due', async () => { + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', join(base, 'machine')); + mkdirSync(join(base, 'machine')); + await runDiscoveryScanIfDue(tick({}), null, join(base, 'home')); + const first = readDiscoveryScan(); + expect(first).toMatchObject({ trigger: 'weekly', found: [join(base, 'home', 'dev', 'app')], truncated: false }); + expect(discoveryScanFile().startsWith(join(base, 'machine'))).toBe(true); + const before = readFileSync(discoveryScanFile(), 'utf8'); + await runDiscoveryScanIfDue(tick({}), null, join(base, 'home')); + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + }); +}); diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index cf2a68afaf..800daf6e1d 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -12,6 +12,7 @@ import { import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { type Config } from '@stim-cli/core/state'; +import { testContext } from './_context.ts'; import { scanSourceWrites } from '../workspace/discovered-activity.ts'; import { discoverCheckouts, @@ -150,6 +151,40 @@ describe('discoverCheckouts', () => { expect((await discoveredOutside(config, [repo])).map(({ path }) => path)).toEqual([elsewhere]); }); + it('takes the checkouts of the last full scan as reduce-only in an unscoped home, and none in a scoped one', async () => { + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', join(base, 'home')); + const docs = join(base, 'home', 'Documents', 'old-app'); + app(docs); + mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); + const scan = { + version: 1, + ranAt: 1, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [docs, join(base, 'gone')], + unreadable: [], + truncated: false, + }; + writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); + expect( + (await discoverCheckouts({ projects: {}, repos: {} }, [])).map(({ path, origin, reduceOnly }) => [ + path, + origin, + reduceOnly, + ]), + ).toEqual([[docs, 'scan', true]]); + vi.stubEnv('STIM_HOME', join(base, 'scoped')); + expect( + await discoverCheckouts( + { projects: {}, repos: {} }, + [], + testContext({ env: { STIM_HOME: join(base, 'scoped') } }), + ), + ).toEqual([]); + }); + it('never takes the home directory as a folder that holds known projects', async () => { vi.stubEnv('STIM_HOME', ''); vi.stubEnv('HOME', join(base, 'home')); diff --git a/packages/stim-cli/src/commands/gc/worktrees.ts b/packages/stim-cli/src/commands/gc/worktrees.ts index f933e2b761..e3aa3b81c8 100644 --- a/packages/stim-cli/src/commands/gc/worktrees.ts +++ b/packages/stim-cli/src/commands/gc/worktrees.ts @@ -173,6 +173,8 @@ interface WorktreeCandidate { activity: WorktreeActivity | null; /** Discovery found it; see `WorktreeFacts.discovered`. */ discovered?: boolean; + /** Discovered through a tool folder or the full scan: never removed, only reduced. */ + reduceOnly?: boolean; /** The main checkout of its repository, for a discovered worktree. */ source?: string; /** How many uncommitted or untracked entries removal archives first; absent or 0 for a clean worktree. */ @@ -600,6 +602,7 @@ async function discoveredHold(candidate: WorktreeCandidate, operation?: StimCont main: candidate.source ?? null, idleMs: dormantMs(operation), reads: discoveredReads(operation?.signal), + reduceOnly: candidate.reduceOnly === true, operation, }); return hold ? [hold] : []; @@ -831,6 +834,7 @@ export async function collectWorktreeSweep({ const outside: WorktreeCandidate[] = []; const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); const discoveredPaths = new Set(discovered.map(({ path }) => path)); + const reduceOnlyPaths = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); const candidates = await candidateRootsAsync(operation, discovered); if (candidates.length) phase('worktrees', `checking ${plural(candidates.length, 'project')} for merged or idle worktrees`, operation); @@ -914,7 +918,13 @@ export async function collectWorktreeSweep({ skipped: null, eligibleAt: null, activity, - ...(discoveredOnly ? { discovered: true, ...('refusal' in source ? {} : { source: source.path }) } : {}), + ...(discoveredOnly + ? { + discovered: true, + ...(roots.some((root) => reduceOnlyPaths.has(root)) ? { reduceOnly: true } : {}), + ...('refusal' in source ? {} : { source: source.path }), + } + : {}), ...(archiveDirty ? { dirty } : {}), }, facts, @@ -977,6 +987,7 @@ export async function collectWorktreeSweep({ main: candidate.source ?? null, idleMs: dormantMs(operation), reads, + reduceOnly: candidate.reduceOnly === true, operation, }); if (hold) Object.assign(candidate, { skipCode: 'recent-activity', skipped: `kept: ${hold}`, eligibleAt: null }); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 072f408061..0ef2175502 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -28,7 +28,8 @@ STIM_HOME home runs it like any other. The stim-daemon runs a pass in its own process on each tick when a check is due: a disk and memory reading for status (every minute), hosted idle stops (every 5 minutes), directory sizes (hourly), finished worktrees (every 15 -minutes) and the age sweep (daily). The reading plans nothing, the idle check +minutes) and the age sweep (daily); the daemon also rescans the home folder for +checkouts weekly, or hourly at low disk (DISCOVERED CHECKOUTS). The reading plans nothing, the idle check plans only idle stops, and the cap trims follow the size, worktree and sweep checks. Pressure itself is the daemon's: each tick classifies the machine normal, high or critical and runs the pressure ladder @@ -1206,6 +1207,21 @@ directory and / are never a folder that holds known projects. A scoped STIM_HOME discovers only from its own maintenance.discoverRoots. Discovery registers nothing, writes nothing and recomputes each run; no git fetch runs in a discovered repository and pull requests come from the cache. +Two more sources find checkouts that Stim only reduces and never removes +(reduce-only): linked worktrees of known repositories in ~/.codex/worktrees, +and the checkouts of the full scan. The scan is a pruned walk of the home +folder, run by the daemon weekly and, while the disk level is above normal, at +most hourly; no command runs it. It reads ~/Documents (macOS may +ask once), skips ~/Desktop, ~/Downloads, ~/Library, the media folders, other +volumes, dot-folders, symlinks and dependency or build folders, stops at a +recognised project and goes seven levels deep. A folder the system refuses is +skipped and recorded. The result is kept in maintenance/discovery-scan.json +under the Stim home (when it ran, what it found, what it could not read) and +discovery reads that file, so a pass or gc never walks the disk. Reduce-only +checkouts have their build outputs and, in a linked worktree, node_modules and +Pods reduced by the rules below; at most 40 of them are handled at a time, in a +window that moves every 15 minutes. A folder named in maintenance.discoverRoots +keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine Idle, with no Stim activity record, means the newest of its source file writes (git-ignored files included; node_modules, Pods and nested worktrees skipped, @@ -1231,8 +1247,8 @@ Any change to a tracked file, even Podfile.lock or project.pbxproj, is dirty work and keeps it, as does any tracked file marked assume-unchanged or skip-worktree, which hides edits from git status. A worktree inside a dot-folder of its main checkout other than .worktrees (.claude/worktrees) is -managed by another tool and is never removed, though its outputs are reduced. -Dirty work is never archived. +managed by another tool and is never removed, though its outputs are reduced; +so is a reduce-only checkout (above). Dirty work is never archived. stim gc --dry-run lists the discovered checkouts and what each would lose, or why it is kept; one over the scan bound is listed under Needs a person. diff --git a/packages/stim-cli/src/stimd/passes/run.ts b/packages/stim-cli/src/stimd/passes/run.ts index 7b6ac078c5..a8b102872a 100644 --- a/packages/stim-cli/src/stimd/passes/run.ts +++ b/packages/stim-cli/src/stimd/passes/run.ts @@ -36,6 +36,7 @@ import { executeAction, type ActionOutcome } from './act.ts'; import { plannedMaintenance, triggeringRoot, type PlannedMaintenance } from './preview.ts'; import { maintenanceLogger } from './log.ts'; import { executedActionName } from '../../maintenance-kind.ts'; +import { runDiscoveryScanIfDue } from '../../workspace/discovery-scan.ts'; import { maintenanceItem, openGcLog, type GcLog } from '../../commands/gc/log.ts'; function writeState(state: MaintenanceState): void { @@ -416,6 +417,7 @@ export async function runMaintenance( recent: [], plan: [], }; + await runDiscoveryScanIfDue(tick, state.pressure); const checks = due(state, settings, startedAt); if (checks.length === 0) return; const background = BACKGROUND_TRIGGERS.has(trigger); diff --git a/packages/stim-cli/src/workspace/discovered-removal.ts b/packages/stim-cli/src/workspace/discovered-removal.ts index d00aef7f45..1a0eca555c 100644 --- a/packages/stim-cli/src/workspace/discovered-removal.ts +++ b/packages/stim-cli/src/workspace/discovered-removal.ts @@ -111,9 +111,12 @@ export async function discoveredRemovalHold( main, idleMs, reads, + reduceOnly = false, operation, - }: { main: string | null; idleMs: number; reads: DiscoveredReads; operation?: StimContext }, + }: { main: string | null; idleMs: number; reads: DiscoveredReads; reduceOnly?: boolean; operation?: StimContext }, ): Promise { + if (reduceOnly) + return 'it was found in a tool folder or by the full scan, so Stim reduces its outputs and never removes it'; const folder = main ? relative(main, path).split('/')[0] : undefined; if (folder?.startsWith('.') && folder !== '.worktrees' && !folder.startsWith('..')) return `it is managed by another tool (inside ${folder} of its main checkout)`; diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts new file mode 100644 index 0000000000..243c28a704 --- /dev/null +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -0,0 +1,184 @@ +import { lstat, readdir } from 'node:fs/promises'; +import { mkdirSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; +import { + discoveryScanFile, + maintenanceDir, + readDiscoveryScan, + type DiscoveryScan, + type MaintenancePressure, +} from '@stim-cli/core/state'; +import { canonicalPath } from '../commands/gc/paths.ts'; +import { recognises } from './discovery.ts'; + +const HOUR_MS = 60 * 60_000; +const WEEK_MS = 7 * 24 * HOUR_MS; +const MAX_DEPTH = 7; +const MAX_FOLDERS = 200_000; +const BUDGET_MS = 2 * 60_000; +const READ_TIMEOUT_MS = 3000; + +/** Folders directly in the home folder the scan never reads: system, media and the two the maintainer excluded. */ +const HOME_SKIPPED = new Set([ + 'Library', + 'Applications', + 'Desktop', + 'Downloads', + 'Movies', + 'Music', + 'Pictures', + 'Public', +]); + +/** Folders at any depth that hold many files and no projects: dependencies, build output and tool caches. */ +const PRUNED = new Set([ + 'node_modules', + 'Pods', + 'Carthage', + 'build', + 'dist', + 'out', + 'target', + 'vendor', + 'DerivedData', + 'venv', + '__pycache__', + 'site-packages', + 'coverage', + 'Library', +]); + +// @ref LLP 0004#tool-folders-and-the-full-scan -- what the scan prunes +/** Whether the scan descends into a folder named `name`, `depth` levels below the home folder (1 is a child of it). */ +export function scanDescends(name: string, depth: number): boolean { + if (name.startsWith('.') || PRUNED.has(name)) return false; + return depth !== 1 || !HOME_SKIPPED.has(name); +} + +interface ScanOutcome { + found: string[]; + unreadable: string[]; + visited: number; + truncated: boolean; +} + +const errorCode = (error: unknown) => (error instanceof Error && 'code' in error ? String(error.code) : ''); + +/** + * A pruned walk below `home`: folders for which `isCheckout` holds are results and are not entered; dependency, build + * output, tool-cache and dot-folders, symlinks and other volumes are skipped; a folder the system refuses or that does + * not answer is recorded as unreadable, never an error. It stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or + * `budgetMs`, and says so in `truncated`. + */ +export async function scanForCheckouts( + home: string, + isCheckout: (path: string) => boolean, + { budgetMs = BUDGET_MS, signal }: { budgetMs?: number; signal?: AbortSignal } = {}, +): Promise { + const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false }; + const deadline = Date.now() + budgetMs; + const volume = (await lstat(home)).dev; + const walk = async (folder: string, depth: number): Promise => { + signal?.throwIfAborted(); + if (outcome.truncated) return; + if (Date.now() > deadline || outcome.visited >= MAX_FOLDERS) { + outcome.truncated = true; + return; + } + outcome.visited++; + let timer: NodeJS.Timeout | undefined; + let entries; + try { + entries = await Promise.race([ + readdir(folder, { withFileTypes: true }), + new Promise((_, fail) => { + timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), READ_TIMEOUT_MS); + }), + ]); + } catch (error) { + if (['EACCES', 'EPERM', 'ETIMEDOUT'].includes(errorCode(error))) outcome.unreadable.push(folder); + return; + } finally { + clearTimeout(timer); + } + if (depth > 0 && isCheckout(folder)) { + outcome.found.push(folder); + return; + } + if (depth >= MAX_DEPTH) return; + for (const entry of entries) { + if (!entry.isDirectory() || entry.isSymbolicLink() || !scanDescends(entry.name, depth + 1)) continue; + const path = join(folder, entry.name); + if ((await lstat(path).catch(() => null))?.dev !== volume) continue; + await walk(path, depth + 1); + } + }; + await walk(home, 0); + return outcome; +} + +/** Whether a scan is due: weekly, and hourly while a volume is below its normal free-disk level. */ +export function discoveryScanDue(last: Pick | null, now: number, lowDisk: boolean): boolean { + if (!last || last.ranAt > now) return true; + return now - last.ranAt >= (lowDisk ? HOUR_MS : WEEK_MS); +} + +function writeScan(scan: DiscoveryScan): void { + mkdirSync(maintenanceDir(), { recursive: true }); + withDirLock(join(maintenanceDir(), 'discovery-scan.lock'), () => { + const file = discoveryScanFile(); + const tmp = `${file}.${process.pid}.tmp`; + writeFileSync(tmp, `${JSON.stringify(scan)}\n`); + try { + renameSync(tmp, file); + } finally { + try { + unlinkSync(tmp); + } catch {} + } + }); +} + +/** + * Scans the home folder when the weekly or low-disk schedule says so and stores the result under the Stim home. A scan + * that fails or runs out of time is stored too, with the checkouts the last scan found, so it is not retried every + * tick. Only a daemon tick scans, never a command, and a scoped STIM_HOME never does. + */ +export async function runDiscoveryScanIfDue( + tick: { operation: StimContext } | undefined, + pressure: Pick | null, + home: string = homedir(), +): Promise { + if (!tick || isScopedHome(tick.operation.env)) return; + const { operation } = tick; + const lowDisk = (pressure?.diskLevel ?? 0) > 0; + const last = readDiscoveryScan(); + const now = Date.now(); + if (!discoveryScanDue(last, now, lowDisk)) return; + const trigger = lowDisk ? 'low-disk' : 'weekly'; + try { + const outcome = await scanForCheckouts(canonicalPath(home), recognises, { signal: operation.signal }); + writeScan({ + version: 1, + ranAt: now, + durationMs: Date.now() - now, + trigger, + ...outcome, + found: outcome.found.toSorted(), + }); + } catch { + operation.signal.throwIfAborted(); + writeScan({ + version: 1, + ranAt: now, + durationMs: Date.now() - now, + trigger, + visited: 0, + found: last?.found ?? [], + unreadable: last?.unreadable ?? [], + truncated: true, + }); + } +} diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index 0af4fb2f70..aa6d2eda6a 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -3,7 +3,7 @@ import { existsSync, readdirSync, statSync } from 'node:fs'; import { homedir } from 'node:os'; import { dirname, isAbsolute, join, parse, resolve, sep } from 'node:path'; import { isScopedHome, type StimContext } from '@stim-cli/core'; -import { machineSettingValue, type Config } from '@stim-cli/core/state'; +import { machineSettingValue, readDiscoveryScan, type Config } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; import { projectRegistry } from '../integrations/projects.ts'; import { isProtectedLocation } from './protected-locations.ts'; @@ -11,7 +11,18 @@ import { matchWorktreeEntry, resolveSourceCheckoutAsync } from './worktree.ts'; const DISCOVER_ROOTS_SETTING = 'maintenance.discoverRoots'; -type DiscoveryOrigin = 'worktree' | 'sibling' | 'root'; +type DiscoveryOrigin = 'worktree' | 'sibling' | 'root' | 'scan'; + +// @ref LLP 0004#tool-folders-and-the-full-scan -- reduce-only tool folders +/** + * Folders under the home folder where another tool puts linked worktrees of the user's repositories. Stim discovers + * those worktrees of known repositories and reduces their outputs, and never removes them. + */ +const TOOL_FOLDERS = ['.codex/worktrees']; + +/** At most this many checkouts found only through a tool folder or the full scan are handled per discovery. */ +const MAX_REDUCE_ONLY = 40; +const REDUCE_ONLY_WINDOW_MS = 15 * 60_000; export interface DiscoveredCheckout { /** The canonical path of the checkout. */ @@ -19,6 +30,8 @@ export interface DiscoveredCheckout { origin: DiscoveryOrigin; /** The repository, folder or `maintenance.discoverRoots` entry that led to it. */ from: string; + /** Found through a tool folder or the full scan: Stim reduces its outputs and never removes it. */ + reduceOnly?: boolean; } export interface DiscoveryChild { @@ -40,15 +53,17 @@ export function selectDiscovered({ children, }: { known: readonly string[]; - worktrees: readonly { path: string; from: string }[]; + worktrees: readonly { path: string; from: string; reduceOnly?: boolean }[]; children: readonly DiscoveryChild[]; }): DiscoveredCheckout[] { const found = new Map(); const covered = (path: string) => known.some((root) => path === root || path.startsWith(`${root}${sep}`) || root.startsWith(`${path}${sep}`)); - for (const { path, from } of worktrees) if (!covered(path)) found.set(path, { path, origin: 'worktree', from }); + for (const { path, from, reduceOnly } of worktrees) + if (!covered(path)) found.set(path, { path, origin: 'worktree', from, ...(reduceOnly ? { reduceOnly } : {}) }); for (const { path, origin, from, recognised } of children) - if (recognised && !found.has(path) && !covered(path)) found.set(path, { path, origin, from }); + if (recognised && !found.has(path) && !covered(path)) + found.set(path, { path, origin, from, ...(origin === 'scan' ? { reduceOnly: true } : {}) }); return [...found.values()].toSorted((a, b) => a.path.localeCompare(b.path)); } @@ -68,7 +83,7 @@ function isProject(path: string): boolean { * Whether Stim's project detection recognises the folder: it is a project itself, or a direct child of its `apps/`, * `examples/` or `packages/` is, as in a monorepo whose root holds no app. */ -function recognises(path: string): boolean { +export function recognises(path: string): boolean { if (isProject(path)) return true; return PROJECT_FOLDERS.some((folder) => { try { @@ -154,6 +169,28 @@ export function broadDiscoverRoots(config: Config | null, env: NodeJS.ProcessEnv const inside = (path: string, folder: string) => path === folder || path.startsWith(`${folder}${sep}`); +/** Whether `path` lies inside a tool folder (see `TOOL_FOLDERS`) of the home folder `home`. */ +export function inToolFolder(path: string, home: string): boolean { + return TOOL_FOLDERS.some((folder) => inside(path, join(home, folder))); +} + +/** + * At most `limit` of the checkouts found only through a tool folder or the full scan, a window over them that moves + * with `now`, so every one is handled within a few windows and one pass pays for a bounded number. The rest are left + * out of this discovery and stay as they are. + */ +export function boundReduceOnly( + found: readonly DiscoveredCheckout[], + now: number, + limit: number = MAX_REDUCE_ONLY, +): DiscoveredCheckout[] { + const only = found.filter((entry) => entry.reduceOnly); + if (only.length <= limit) return [...found]; + const start = (Math.floor(now / REDUCE_ONLY_WINDOW_MS) * limit) % only.length; + const kept = new Set(Array.from({ length: limit }, (_, i) => only[(start + i) % only.length])); + return found.filter((entry) => !entry.reduceOnly || kept.has(entry)); +} + /** * Whether Stim acts on a discovered linked worktree: it lies in its repository's own main checkout folder, in a folder * that holds a known project, or in a `maintenance.discoverRoots` folder, and not in a protected location. A path inside @@ -265,7 +302,7 @@ async function computeDiscovered( repositories.push(repository); } } - const worktrees: { path: string; from: string }[] = []; + const worktrees: { path: string; from: string; reduceOnly?: boolean }[] = []; const outside: Discovery['outside'] = []; for (const { entries, main } of repositories) { for (const entry of entries) { @@ -273,15 +310,24 @@ async function computeDiscovered( if (path === canonicalPath(main)) continue; if (worktreeInScope(path, { main: canonicalPath(main), folders: [...folders.keys()], roots: canonicalRoots })) worktrees.push({ path, from: main }); + else if (inToolFolder(path, canonicalPath(homedir()))) worktrees.push({ path, from: main, reduceOnly: true }); else outside.push({ path, from: main }); } } - const allowed = children.filter( + const scanned: DiscoveryChild[] = scoped + ? [] + : (readDiscoveryScan()?.found ?? []) + .filter((path) => isAbsolute(path) && existsSync(path)) + .map(canonicalPath) + .filter((path) => !isBroadFolder(path)) + .map((path) => ({ path, origin: 'scan', from: 'full scan', recognised: true })); + const allowed = [...children, ...scanned].filter( (child) => canonicalRoots.some((root) => inside(child.path, root)) || !isProtectedLocation(child.path), ); const knownRoots = known.map(canonicalPath); - const selected = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); - const listed = new Set(selected.map(({ path }) => path)); + const everything = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); + const selected = boundReduceOnly(everything, Date.now()); + const listed = new Set(everything.map(({ path }) => path)); const coveredByKnown = (path: string) => knownRoots.some((root) => inside(path, root) || inside(root, path)); return { found: selected, diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 27001477c6..288db6c039 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -84,11 +84,13 @@ A finished worktree with uncommitted changes can be removed after Stim saves a r Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. Discovery finds direct children of the folders that hold the checkouts of your registered projects, direct children of the folders in `maintenance.discoverRoots`, and the linked worktrees of their repositories. Use absolute paths for discovery roots. A leading `~` expands to your home folder. Stim refuses the home folder, its ancestors, and filesystem roots. `stim doctor` reports a configured path that is missing or is not a folder. A child counts only when Stim's project detection recognises it as a project it handles, itself or through a direct child of its `apps/`, `examples/` or `packages/` folder. -Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`). A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`). A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. + +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan weekly and, while the disk level is above normal, at most once an hour; no command runs it. The scan reads `~/Documents`, so macOS may ask for permission once. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, dot-folders, symlinks and dependency or build folders, and it stops at a recognised project. A folder it cannot read is skipped. Discovery uses the last scan's saved result, so `stim gc` never walks your disk itself. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. -A clean linked worktree whose branch is merged or whose pull request ended is removed only when it is idle by that same evidence at every pressure level, with no grace shortcut, and every git-ignored file also exists, byte for byte, in the main checkout. The exception is regenerable folders: directories git reports as ignored that are named `node_modules` or like build output (`build`, `dist`, `target`, `.next`, `.turbo`, `.expo` and similar), whose files are deleted with the worktree. An ignored file elsewhere that the main checkout lacks, even inside a tracked `dist/` folder, keeps the worktree. Any change to a tracked file, even `Podfile.lock` or `project.pbxproj`, is dirty work and keeps it, and so does a tracked file marked assume-unchanged or skip-worktree, which hides edits from `git status`. A worktree inside a dot-folder of its main checkout other than `.worktrees` (for example `.claude/worktrees`) belongs to another tool and is never removed, although its build output and dependencies follow the normal idle rules. Dirty work is never archived, where a registered finished worktree's dirty work is archived first. +A clean linked worktree whose branch is merged or whose pull request ended is removed only when it is idle by that same evidence at every pressure level, with no grace shortcut, and every git-ignored file also exists, byte for byte, in the main checkout. The exception is regenerable folders: directories git reports as ignored that are named `node_modules` or like build output (`build`, `dist`, `target`, `.next`, `.turbo`, `.expo` and similar), whose files are deleted with the worktree. An ignored file elsewhere that the main checkout lacks, even inside a tracked `dist/` folder, keeps the worktree. Any change to a tracked file, even `Podfile.lock` or `project.pbxproj`, is dirty work and keeps it, and so does a tracked file marked assume-unchanged or skip-worktree, which hides edits from `git status`. A worktree inside a dot-folder of its main checkout other than `.worktrees` (for example `.claude/worktrees`) belongs to another tool and is never removed, although its build output and dependencies follow the normal idle rules. The same holds for a checkout found through `~/.codex/worktrees` or the full scan. Dirty work is never archived, where a registered finished worktree's dirty work is archived first. Date: Sat, 14 Nov 2026 23:30:34 -0500 Subject: [PATCH 02/25] chore: record the discovered checkout reduceOnly field in the API report --- packages/stim-cli/api.api.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/packages/stim-cli/api.api.txt b/packages/stim-cli/api.api.txt index 7f66107a72..10df1fb611 100644 --- a/packages/stim-cli/api.api.txt +++ b/packages/stim-cli/api.api.txt @@ -360,6 +360,7 @@ [*] interface DiscoveredCheckout { @@1 from: string; [*] interface DiscoveredCheckout { @@1 origin: DiscoveryOrigin; [*] interface DiscoveredCheckout { @@1 path: string; +[*] interface DiscoveredCheckout { @@1 reduceOnly?: boolean; [*] interface DoctorReport { [*] interface DoctorReport { @@1 @internal daemon: DaemonStatus & { [*] interface DoctorReport { @@1 @internal maintenance: ReturnType & { @@ -1499,7 +1500,7 @@ [*] type DeviceToolRequest = { @branch4 @@1 platform: "android"; [*] type DeviceToolRequest = { @branch4 @@1 state: "on" | "off"; [*] type DeviceToolRequest = { @branch4 @@1 tool: "network"; -[*] type DiscoveryOrigin = "worktree" | "sibling" | "root"; +[*] type DiscoveryOrigin = "worktree" | "sibling" | "root" | "scan"; [*] type DiskPressureLevel = 0 | 1 | 2 | 3; [*] type DoctorPlatform = "ios" | "android"; [*] type EngineOptions = Omit & { From 1153971a75003c716a29e1f89496211ca8128580 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sat, 14 Nov 2026 23:37:17 -0500 Subject: [PATCH 03/25] fix: the discovery scan trigger reads the tick disk level and a scan can never end a maintenance pass --- llp/0004-discovered-checkouts.decision.md | 50 +++++++++---- .../__tests__/discovered-reduce-only.test.ts | 1 + .../src/__tests__/discovery-scan.test.ts | 65 +++++++++++++---- .../stim-cli/src/__tests__/discovery.test.ts | 1 + packages/stim-cli/src/guide/cleanup.ts | 5 +- packages/stim-cli/src/stimd/passes/run.ts | 2 +- .../stim-cli/src/workspace/discovery-scan.ts | 73 ++++++++++--------- packages/stim-cli/src/workspace/discovery.ts | 5 +- .../src/workspace/protected-locations.ts | 2 +- website/docs/worktrees.md | 4 +- 10 files changed, 139 insertions(+), 69 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index aec0503851..2ad05a2afc 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -16,7 +16,9 @@ registered checkouts, and direct children of the folders in `maintenance.discoverRoots`. A child counts only when Stim's project detection recognises it, itself or through a direct child of its `apps`, `examples` or `packages` folder. Discovery registers nothing, writes nothing, recomputes each -run and never runs `git fetch`. +run and never runs `git fetch`. Two more sources, the common tool folders and the +full scan (see "Tool folders and the full scan"), add checkouts that are only +reduced. Reason: checkouts no `stim` command ran in can hold most of a disk; an audit of one MacBook on 2026-11-11 found 119 of 151 checkouts, 248 GB, unknown to Stim, @@ -39,14 +41,17 @@ checkout folder, a folder that holds a known project, or a `maintenance.discoverRoots` folder. Nothing under the system temporary directory or a dot-folder directly inside the home directory is ever touched, unless it lies inside a `maintenance.discoverRoots` folder: a path inside such a folder -is never a protected location, matched after symlinks resolve. The home +is never a protected location, matched after symlinks resolve. The one other +exception is `~/.codex/worktrees`, whose worktrees are reduce-only (see "Tool +folders and the full scan"). The home directory and `/` are never a folder that holds known projects. A scoped `STIM_HOME` discovers only from its own `maintenance.discoverRoots`. Reason: `git worktree list` also returns worktrees other tools own, such as `~/.codex/worktrees`, pin checkouts under a home dot-folder and agent scratch folders under the temporary directory. Stim cannot tell what those tools still -need. A scratch home must not reach the repositories of the real one. +need, so it removes none of them; it reduces only the outputs of the +`~/.codex/worktrees` ones. A scratch home must not reach the repositories of the real one. Maintainer decision: "Def need to fix it if its the case, paths added to discover root need to always be considered", 2026-11-14 (#5757), about a folder @@ -97,21 +102,38 @@ Maintainer decision, 2026-11-14 (#5762), verbatim: Earlier the same evening, on including common tool folders by default: "Could these common paths just be included by default in Discovered checkout". -The coordinator's reading, not the maintainer's words, for the points that +The coordinator's reading, not the maintainer's words, for the points the decision leaves open: 1. Checkouts found only through a common path or the full scan are reduced and - never removed by Stim. Removal is what doctor suggests to the person. -2. Common paths start with `~/.codex/worktrees`. Others are added only with - evidence that a tool uses that path. -3. "Once in a while" is weekly; on low disk the scan runs when a volume's level + never removed by Stim: build outputs, tagged caches, and a linked worktree's + `node_modules` and Pods go by the discovered-checkout rules, and the checkout + or worktree itself stays. This is the rule that already applies to a worktree + in a dot-folder of its main checkout (`.claude/worktrees`). Removal is what + doctor suggests to the person. A folder named in `maintenance.discoverRoots` + keeps the full discovered rules, removal of finished worktrees included. +2. Common paths start with `~/.codex/worktrees`. Other tools' folders are added + only with evidence that the tool uses that path. +3. "Once in a while" is weekly. On low disk the scan runs when a volume's level rises above normal, at most once an hour. -4. Doctor reads the last scan's result and does not run the scan. -5. `~/Desktop`, `~/Downloads` and other volumes are not scanned. -6. The bound of 40 per discovery, the 15 minute window, the depth of seven and - the folders pruned are implementation choices, not decisions. - -Doctor's listing and the line printed on low disk are not part of this change. +4. Doctor reads the last scan's result and does not run the scan, so it stays + fast. +5. The line Stim commands print on low disk also names what the scan found, its + size and the command that adds it. +6. `~/Desktop` and `~/Downloads` are not scanned. Other volumes are not scanned. +7. "Idle for a long time" has no number yet. + +Chosen by the implementing lane, not by the maintainer or the coordinator: the +bound of 40 reduce-only checkouts per discovery in a window that moves every 15 +minutes (not "least recently handled first"), the depth of seven, the folders +pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and +`~/Applications` left out, the low-disk scan running once per rise of the disk +level (the daemon's high or critical disk level, `disk.minFreeGb`) at most +hourly rather than hourly for as long as the disk stays low, and the scan +running only in the daemon. + +Not part of this change: points 4, 5 and 7, which belong to the doctor listing +and the low-disk line. ## Removing a discovered worktree needs more evidence diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 78a31b6b9f..7d79cda078 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -56,6 +56,7 @@ describe('checkouts found through a tool folder or the full scan', () => { beforeEach(() => { base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-reduce-only-'))); vi.stubEnv('HOME', join(base, 'home')); + vi.stubEnv('USERPROFILE', join(base, 'home')); vi.stubEnv('STIM_HOME', ''); main = join(base, 'dev', 'main'); mkdirSync(main, { recursive: true }); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index fa0fb21d7d..e8848c78b5 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -97,7 +97,7 @@ describe('scanForCheckouts', () => { describe('discoveryScanDue', () => { const now = Date.UTC(2026, 10, 14); const ago = (ms: number) => ({ ranAt: now - ms }); - it('runs weekly, hourly at low disk, and when no scan or a future one is recorded', () => { + it('runs weekly, an hour after a rise in the disk level, and when no scan or a future one is recorded', () => { expect(discoveryScanDue(null, now, false)).toBe(true); expect(discoveryScanDue(ago(-1000), now, false)).toBe(true); expect(discoveryScanDue(ago(6 * 86_400_000), now, false)).toBe(false); @@ -109,12 +109,23 @@ describe('discoveryScanDue', () => { describe('runDiscoveryScanIfDue', () => { let base: string; - const tick = (env: NodeJS.ProcessEnv) => ({ operation: testContext({ env }) }); + const normal = [{ volume: '/', freeMb: 500_000 }]; + const low = [{ volume: '/', freeMb: 10_000 }]; + const tick = (disk = normal, env: NodeJS.ProcessEnv = {}) => ({ + operation: testContext({ env }), + measured: { measurement: { disk } }, + }); + const memory = () => ({ attemptedAt: 0, wasLow: false }); + const scanned = () => readDiscoveryScan()?.trigger; beforeEach(() => { base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-run-'))); mkdirSync(join(base, 'home', 'dev', 'app'), { recursive: true }); writeFileSync(join(base, 'home', 'dev', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + mkdirSync(join(base, 'machine')); + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', join(base, 'machine')); + vi.stubEnv('USERPROFILE', join(base, 'machine')); }); afterEach(() => { @@ -123,29 +134,55 @@ describe('runDiscoveryScanIfDue', () => { }); it('never scans for a command, only for a daemon tick', async () => { - vi.stubEnv('STIM_HOME', ''); - vi.stubEnv('HOME', join(base, 'machine')); - mkdirSync(join(base, 'machine')); - await runDiscoveryScanIfDue(undefined, null, join(base, 'home')); + await runDiscoveryScanIfDue(undefined, { home: join(base, 'home'), memory: memory() }); expect(readDiscoveryScan()).toBeNull(); }); it('never scans in a scoped home and writes nothing', async () => { vi.stubEnv('STIM_HOME', join(base, 'state')); - await runDiscoveryScanIfDue(tick({ STIM_HOME: join(base, 'state') }), null, join(base, 'home')); + await runDiscoveryScanIfDue(tick(normal, { STIM_HOME: join(base, 'state') }), { + home: join(base, 'home'), + memory: memory(), + }); expect(existsSync(join(base, 'state'))).toBe(false); }); it('stamps the result under the Stim home and does not scan again until it is due', async () => { - vi.stubEnv('STIM_HOME', ''); - vi.stubEnv('HOME', join(base, 'machine')); - mkdirSync(join(base, 'machine')); - await runDiscoveryScanIfDue(tick({}), null, join(base, 'home')); - const first = readDiscoveryScan(); - expect(first).toMatchObject({ trigger: 'weekly', found: [join(base, 'home', 'dev', 'app')], truncated: false }); + const options = { home: join(base, 'home'), memory: memory() }; + await runDiscoveryScanIfDue(tick(), options); + expect(readDiscoveryScan()).toMatchObject({ + trigger: 'weekly', + found: [join(base, 'home', 'dev', 'app')], + truncated: false, + }); expect(discoveryScanFile().startsWith(join(base, 'machine'))).toBe(true); const before = readFileSync(discoveryScanFile(), 'utf8'); - await runDiscoveryScanIfDue(tick({}), null, join(base, 'home')); + await runDiscoveryScanIfDue(tick(), options); expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); }); + + it('scans once the disk level rises above normal an hour after the last scan, not again while it stays low', async () => { + const options = { home: join(base, 'home'), memory: memory() }; + const backdate = () => { + writeFileSync(discoveryScanFile(), JSON.stringify({ ...readDiscoveryScan(), ranAt: Date.now() - 2 * 3_600_000 })); + options.memory.attemptedAt = 0; + }; + await runDiscoveryScanIfDue(tick(), options); + backdate(); + await runDiscoveryScanIfDue(tick(low), options); + expect(scanned()).toBe('low-disk'); + backdate(); + await runDiscoveryScanIfDue(tick(low), options); + expect(Date.now() - (readDiscoveryScan()?.ranAt ?? 0)).toBeGreaterThan(3_600_000); + }); + + it('does not walk again within the interval when the result cannot be stored, and does not throw', async () => { + writeFileSync(join(base, 'machine', '.stim'), 'a file where the Stim home folder belongs'); + const options = { home: join(base, 'home'), memory: memory() }; + await expect(runDiscoveryScanIfDue(tick(), options)).resolves.toBeUndefined(); + expect(options.memory.attemptedAt).toBeGreaterThan(0); + const attempted = options.memory.attemptedAt; + await runDiscoveryScanIfDue(tick(), options); + expect(options.memory.attemptedAt).toBe(attempted); + }); }); diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index 800daf6e1d..daa924228d 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -154,6 +154,7 @@ describe('discoverCheckouts', () => { it('takes the checkouts of the last full scan as reduce-only in an unscoped home, and none in a scoped one', async () => { vi.stubEnv('STIM_HOME', ''); vi.stubEnv('HOME', join(base, 'home')); + vi.stubEnv('USERPROFILE', join(base, 'home')); const docs = join(base, 'home', 'Documents', 'old-app'); app(docs); mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 0ef2175502..9c2a0482fe 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1201,7 +1201,8 @@ or a maintenance.discoverRoots folder; every other worktree (another tool's home, a pin checkout) is listed by stim gc --dry-run as outside the folders Stim manages and gets no action of any kind. Nothing under the system temporary directory or a dot-folder directly inside the home directory is ever touched, -except inside a maintenance.discoverRoots folder, which is always considered. +except inside a maintenance.discoverRoots folder, which is always considered, +and the ~/.codex/worktrees worktrees below, which are only reduced. A worktree nested in a registered root is left to that workspace. The home directory and / are never a folder that holds known projects. A scoped STIM_HOME discovers only from its own maintenance.discoverRoots. Discovery @@ -1210,7 +1211,7 @@ a discovered repository and pull requests come from the cache. Two more sources find checkouts that Stim only reduces and never removes (reduce-only): linked worktrees of known repositories in ~/.codex/worktrees, and the checkouts of the full scan. The scan is a pruned walk of the home -folder, run by the daemon weekly and, while the disk level is above normal, at +folder, run by the daemon weekly and, when the disk level rises above normal, at most hourly; no command runs it. It reads ~/Documents (macOS may ask once), skips ~/Desktop, ~/Downloads, ~/Library, the media folders, other volumes, dot-folders, symlinks and dependency or build folders, stops at a diff --git a/packages/stim-cli/src/stimd/passes/run.ts b/packages/stim-cli/src/stimd/passes/run.ts index a8b102872a..c9986edf5c 100644 --- a/packages/stim-cli/src/stimd/passes/run.ts +++ b/packages/stim-cli/src/stimd/passes/run.ts @@ -417,7 +417,7 @@ export async function runMaintenance( recent: [], plan: [], }; - await runDiscoveryScanIfDue(tick, state.pressure); + await runDiscoveryScanIfDue(tick); const checks = due(state, settings, startedAt); if (checks.length === 0) return; const background = BACKGROUND_TRIGGERS.has(trigger); diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 243c28a704..909a6dfcff 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -8,9 +8,11 @@ import { maintenanceDir, readDiscoveryScan, type DiscoveryScan, - type MaintenancePressure, + type DaemonMeasurement, } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; +import { diskPressure, diskThresholds } from '../daemon/pressure.ts'; +import { loadConfig } from './config.ts'; import { recognises } from './discovery.ts'; const HOUR_MS = 60 * 60_000; @@ -119,12 +121,20 @@ export async function scanForCheckouts( return outcome; } -/** Whether a scan is due: weekly, and hourly while a volume is below its normal free-disk level. */ -export function discoveryScanDue(last: Pick | null, now: number, lowDisk: boolean): boolean { +/** Whether a scan is due: weekly, and once an hour after the disk level rose above normal. */ +export function discoveryScanDue(last: Pick | null, now: number, rose: boolean): boolean { if (!last || last.ranAt > now) return true; - return now - last.ranAt >= (lowDisk ? HOUR_MS : WEEK_MS); + return now - last.ranAt >= (rose ? HOUR_MS : WEEK_MS); } +/** What this process remembers between ticks: when it last tried a scan, and whether the disk level was above normal. */ +interface ScanMemory { + attemptedAt: number; + wasLow: boolean; +} + +const processMemory: ScanMemory = { attemptedAt: 0, wasLow: false }; + function writeScan(scan: DiscoveryScan): void { mkdirSync(maintenanceDir(), { recursive: true }); withDirLock(join(maintenanceDir(), 'discovery-scan.lock'), () => { @@ -142,43 +152,40 @@ function writeScan(scan: DiscoveryScan): void { } /** - * Scans the home folder when the weekly or low-disk schedule says so and stores the result under the Stim home. A scan - * that fails or runs out of time is stored too, with the checkouts the last scan found, so it is not retried every - * tick. Only a daemon tick scans, never a command, and a scoped STIM_HOME never does. + * Scans the home folder when the weekly schedule says so, or an hour after the last scan when the disk level has just + * risen above normal, and stores the result under the Stim home. A scan that fails or runs out of time is stored too, + * with the checkouts the last scan found, and this process does not retry within the interval even when nothing could + * be stored. Only a daemon tick scans, never a command, and a scoped STIM_HOME never does. Nothing but an abort + * escapes, so a scan can never stop the maintenance pass it runs in. */ export async function runDiscoveryScanIfDue( - tick: { operation: StimContext } | undefined, - pressure: Pick | null, - home: string = homedir(), + tick: { operation: StimContext; measured: { measurement: Pick } } | undefined, + { home = homedir(), memory = processMemory }: { home?: string; memory?: ScanMemory } = {}, ): Promise { if (!tick || isScopedHome(tick.operation.env)) return; const { operation } = tick; - const lowDisk = (pressure?.diskLevel ?? 0) > 0; - const last = readDiscoveryScan(); const now = Date.now(); - if (!discoveryScanDue(last, now, lowDisk)) return; - const trigger = lowDisk ? 'low-disk' : 'weekly'; + const thresholds = diskThresholds(loadConfig(), operation.env).thresholds; + const low = diskPressure(tick.measured.measurement.disk, thresholds).level !== 'normal'; + const rose = low && !memory.wasLow; + memory.wasLow = low; + const last = readDiscoveryScan(); + const ranAt = Math.max(last?.ranAt ?? 0, memory.attemptedAt); + if (!discoveryScanDue(ranAt > 0 ? { ranAt } : null, now, rose)) return; + memory.attemptedAt = now; + const trigger = rose ? 'low-disk' : 'weekly'; + const store = (scan: Omit) => + writeScan({ version: 1, ranAt: now, durationMs: Date.now() - now, trigger, ...scan }); try { const outcome = await scanForCheckouts(canonicalPath(home), recognises, { signal: operation.signal }); - writeScan({ - version: 1, - ranAt: now, - durationMs: Date.now() - now, - trigger, - ...outcome, - found: outcome.found.toSorted(), - }); + store({ ...outcome, found: outcome.found.toSorted() }); } catch { - operation.signal.throwIfAborted(); - writeScan({ - version: 1, - ranAt: now, - durationMs: Date.now() - now, - trigger, - visited: 0, - found: last?.found ?? [], - unreadable: last?.unreadable ?? [], - truncated: true, - }); + if (operation.signal.aborted) { + memory.attemptedAt = 0; + return; + } + try { + store({ visited: 0, found: last?.found ?? [], unreadable: last?.unreadable ?? [], truncated: true }); + } catch {} } } diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index aa6d2eda6a..a4c38a9917 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -218,8 +218,9 @@ interface Discovery { * projects (`git worktree list`) inside the folders Stim manages, recognised direct children of the folders holding * those checkouts, and recognised direct children of `maintenance.discoverRoots`. Nothing under the system temporary * directory or a dot-folder of the home directory is included unless it lies inside a `maintenance.discoverRoots` - * folder. A scoped STIM_HOME discovers only from that setting, so a scratch home never reaches the repositories of - * another. Nothing is registered, written or persisted. + * folder, or is a worktree in a tool folder or a checkout of the last full scan, which are reduce-only. A scoped + * STIM_HOME discovers only from that setting, so a scratch home never reaches the repositories of another. Discovery + * registers and writes nothing; the full scan's result is read from the file the daemon wrote. */ export async function discoverCheckouts( config: Config | null, diff --git a/packages/stim-cli/src/workspace/protected-locations.ts b/packages/stim-cli/src/workspace/protected-locations.ts index 6a4c903f37..76d2ae185f 100644 --- a/packages/stim-cli/src/workspace/protected-locations.ts +++ b/packages/stim-cli/src/workspace/protected-locations.ts @@ -25,7 +25,7 @@ function real(path: string): string | null { /** * Whether a path lies under the system temporary directory, or under a dot-folder directly inside the home directory * (`~/.codex`, `~/.config`, `~/.exact`): other tools own what is there, so Stim never acts on it. A path that does not - * resolve counts as protected. + * resolve counts as protected. Discovery lists the worktrees in `~/.codex/worktrees` anyway, as reduce-only. */ export function isProtectedLocation(path: string, roots: Roots = defaultRoots()): boolean { const target = real(path); diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 288db6c039..1bc96076d4 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -84,9 +84,9 @@ A finished worktree with uncommitted changes can be removed after Stim saves a r Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. Discovery finds direct children of the folders that hold the checkouts of your registered projects, direct children of the folders in `maintenance.discoverRoots`, and the linked worktrees of their repositories. Use absolute paths for discovery roots. A leading `~` expands to your home folder. Stim refuses the home folder, its ancestors, and filesystem roots. `stim doctor` reports a configured path that is missing or is not a folder. A child counts only when Stim's project detection recognises it as a project it handles, itself or through a direct child of its `apps/`, `examples/` or `packages/` folder. -Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`). A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. +Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan weekly and, while the disk level is above normal, at most once an hour; no command runs it. The scan reads `~/Documents`, so macOS may ask for permission once. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, dot-folders, symlinks and dependency or build folders, and it stops at a recognised project. A folder it cannot read is skipped. Discovery uses the last scan's saved result, so `stim gc` never walks your disk itself. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan weekly and, when the disk level rises above normal, at most once an hour; no command runs it. The scan reads `~/Documents`, so macOS may ask for permission once. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, dot-folders, symlinks and dependency or build folders, and it stops at a recognised project. A folder it cannot read is skipped. Discovery uses the last scan's saved result, so `stim gc` never walks your disk itself. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 9a0630fb389c875eb08ab427b4f7dad2c0199112 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 00:00:52 -0500 Subject: [PATCH 04/25] feat: the home folder scan enters home dot-folders except named tool homes and caches --- llp/0004-discovered-checkouts.decision.md | 23 +++++- .../__tests__/discovered-reduce-only.test.ts | 31 +++++++- .../src/__tests__/discovered-stages.test.ts | 5 +- .../__tests__/discovered-worktrees.test.ts | 5 +- .../src/__tests__/discovery-scan.test.ts | 26 +++++-- .../stim-cli/src/__tests__/discovery.test.ts | 5 +- .../gc-workspaces.integration.test.ts | 5 +- packages/stim-cli/src/guide/cleanup.ts | 4 +- .../stim-cli/src/workspace/discovery-scan.ts | 74 +++++++++++++++++-- packages/stim-cli/src/workspace/discovery.ts | 8 +- .../src/workspace/protected-locations.ts | 22 +++++- website/docs/worktrees.md | 2 +- 12 files changed, 181 insertions(+), 29 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 2ad05a2afc..ef42c3e33e 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -76,8 +76,9 @@ hold that keeps a worktree inside a dot-folder of its main checkout project detection recognises and discovery does not reach. The daemon runs it weekly and, while the disk level is above normal, at most hourly; a command never runs it. It reads `~/Documents`, skips `~/Desktop`, `~/Downloads`, - `~/Library`, other media folders, dot-folders, symlinks, other volumes and - dependency, build and cache folders, stops at a recognised project and goes + `~/Library`, other media folders, symlinks, other volumes, dependency, build + and cache folders, dot-folders below the first level and the named tool homes + among the home folder's dot-folders (see below), and the Stim home, stops at a recognised project and goes seven levels deep. A folder the system refuses is skipped and recorded, never an error. The result is stored in `maintenance/discovery-scan.json` under the Stim home (when it ran, what it found, what it could not read) and discovery @@ -88,6 +89,24 @@ hold that keeps a worktree inside a dot-folder of its main checkout A folder named in `maintenance.discoverRoots` keeps the full rules. A scoped `STIM_HOME` discovers nothing from either source. +Reason for entering home dot-folders: checkouts live there too, for example +about 18 checkouts and 80 GB of Cargo `target/` under `~/.exact/exact2` (#5757), +so a blanket skip of dot-folders would miss the layout that started this work. +The scan enters a dot-folder directly in the home folder unless it is in a +named prune list of tool homes, package and build caches and runtimes +(`.git`, `.npm`, `.yarn`, `.bun`, `.gradle`, `.cargo`, `.rustup`, `.nvm`, +`.rvm`, `.cache`, `.android`, `.vscode`, `.cursor`, `.agent-device`, `.claude`, +`.codex` and others in `HOME_DOT_PRUNED`) or named like a cache (`*-cache`). +The list comes from entry counts to depth four on one machine, led by `.gradle` +(67 thousand), `.agent-device` (45 thousand), the two Metro cache folders +(25 and 17 thousand), `.npm`, `.bun`, `.rvm` and `.yarn` (15 thousand each). +`.codex` is reached through its worktrees folder, and the Stim home is never +scanned. Dot-folders deeper down (`.worktrees`, `.claude/worktrees`) are +skipped because git reaches their worktrees. The scan's finds under a home +dot-folder pass the protection rule for the system temporary directory only: +the scan itself decides which dot-folders to enter, so the dot-folder part of +`isProtectedLocation` does not apply to them, and they stay reduce-only. + Reason: on one MacBook `~/.codex/worktrees` held 53 worktrees and 12 GB that Stim listed once as outside the folders it manages, with no size and no way forward. diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 7d79cda078..033e76361d 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -9,7 +9,7 @@ import { utimesSync, writeFileSync, } from 'node:fs'; -import { tmpdir } from 'node:os'; +import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; import { saveConfig } from '../workspace/config.ts'; import { collectWorktreeSweep, removeWorktrees } from '../commands/gc/worktrees.ts'; @@ -22,7 +22,11 @@ import { vi.mock('../workspace/protected-locations.ts', async (original) => { const real = await original(); - return { ...real, isProtectedLocation: () => false }; + return { + ...real, + isProtectedLocation: (path: string) => real.isProtectedLocation(path, { home: homedir(), temps: [] }), + isTemporaryLocation: () => false, + }; }); const day = 86_400_000; @@ -92,6 +96,29 @@ describe('checkouts found through a tool folder or the full scan', () => { ]); }); + it('discovers a checkout under a home dot-folder only when the scan result names it, as reduce-only', async () => { + const exact = join(base, 'home', '.exact', 'exact2'); + mkdirSync(exact, { recursive: true }); + writeFileSync(join(exact, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + const config = { projects: { [main]: {} }, repos: {} }; + expect(await discoverCheckouts(config, [main])).toEqual([]); + mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); + const scan = { + version: 1, + ranAt: 1, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [exact], + unreadable: [], + truncated: false, + }; + writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); + expect( + (await discoverCheckouts(config, [main])).map(({ path, origin, reduceOnly }) => [path, origin, reduceOnly]), + ).toEqual([[exact, 'scan', true]]); + }); + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); diff --git a/packages/stim-cli/src/__tests__/discovered-stages.test.ts b/packages/stim-cli/src/__tests__/discovered-stages.test.ts index 30885f8a54..eb1b00fe82 100644 --- a/packages/stim-cli/src/__tests__/discovered-stages.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-stages.test.ts @@ -21,7 +21,10 @@ const fail = (): never => { throw new Error('no candidate'); }; -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, + isTemporaryLocation: () => false, +})); describe('a discovered checkout in the idle stages', () => { let base: string; diff --git a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts index f8b5df4da3..9a580c5930 100644 --- a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts @@ -17,7 +17,10 @@ import * as reads from '../workspace/discovered-reads.ts'; import { discoveredRemovalHold } from '../workspace/discovered-removal.ts'; import { worktreeInScope } from '../workspace/discovery.ts'; -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, + isTemporaryLocation: () => false, +})); const day = 86_400_000; diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index e8848c78b5..387684ebc6 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -23,8 +23,8 @@ import { const marked = (path: string) => existsSync(join(path, 'package.json')); describe('scanDescends', () => { - it('skips dot-folders, dependency and build folders everywhere, and the excluded folders of the home folder only', () => { - for (const name of ['.git', '.cache', 'node_modules', 'Pods', 'build', 'target', 'Library']) + it('skips dependency and build folders everywhere, and the excluded folders of the home folder only', () => { + for (const name of ['node_modules', 'Pods', 'build', 'target', 'Library']) expect(scanDescends(name, 3)).toBe(false); for (const name of ['Desktop', 'Downloads', 'Movies']) { expect(scanDescends(name, 1)).toBe(false); @@ -33,6 +33,13 @@ describe('scanDescends', () => { expect(scanDescends('Documents', 1)).toBe(true); expect(scanDescends('dev', 1)).toBe(true); }); + + it('enters a dot-folder of the home folder unless it is a tool home or cache, and no deeper dot-folder', () => { + expect(scanDescends('.exact', 1)).toBe(true); + for (const name of ['.git', '.npm', '.gradle', '.cache', '.stim', '.codex', '.member-app-metro-cache']) + expect(scanDescends(name, 1)).toBe(false); + for (const name of ['.worktrees', '.exact', '.claude']) expect(scanDescends(name, 2)).toBe(false); + }); }); describe('scanForCheckouts', () => { @@ -52,6 +59,7 @@ describe('scanForCheckouts', () => { project('Documents', 'a'); project('dev', 'deep', 'er', 'b'); project('dev', 'c'); + project('.exact', 'exact2'); mkdirSync(join(home, 'dev', 'c', 'inner')); project('dev', 'c', 'inner'); for (const skipped of [ @@ -59,16 +67,20 @@ describe('scanForCheckouts', () => { ['Downloads', 'x'], ['dev', 'node_modules', 'x'], ['dev', '.hidden', 'x'], + ['.npm', 'x'], ['dev', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'too-deep'], ]) project(...skipped); symlinkSync(join(home, 'Documents', 'a'), join(home, 'dev', 'linked')); const scan = await scanForCheckouts(home, marked); - expect(scan.found.toSorted()).toEqual([ - join(home, 'Documents', 'a'), - join(home, 'dev', 'c'), - join(home, 'dev', 'deep', 'er', 'b'), - ]); + expect(scan.found.toSorted()).toEqual( + [ + join(home, 'Documents', 'a'), + join(home, 'dev', 'c'), + join(home, 'dev', 'deep', 'er', 'b'), + join(home, '.exact', 'exact2'), + ].toSorted(), + ); expect(scan.truncated).toBe(false); expect(scan.unreadable).toEqual([]); }); diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index daa924228d..2ddf3fe80a 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -22,7 +22,10 @@ import { type DiscoveryChild, } from '../workspace/discovery.ts'; -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, + isTemporaryLocation: () => false, +})); describe('isBroadFolder', () => { it('refuses the home folder, a folder holding it and the filesystem root, and accepts a folder below it', () => { diff --git a/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts b/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts index 8af12be743..60db0b8e01 100644 --- a/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts +++ b/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts @@ -423,7 +423,10 @@ describe('workspace directories without workspace.json', () => { }); }); -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, + isTemporaryLocation: () => false, +})); test('listWorkspaceDirs resolves each directory through its workspace.json', () => { const { root } = goneWorkspace('app'); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 9c2a0482fe..32515e06f9 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1214,7 +1214,9 @@ and the checkouts of the full scan. The scan is a pruned walk of the home folder, run by the daemon weekly and, when the disk level rises above normal, at most hourly; no command runs it. It reads ~/Documents (macOS may ask once), skips ~/Desktop, ~/Downloads, ~/Library, the media folders, other -volumes, dot-folders, symlinks and dependency or build folders, stops at a +volumes, symlinks, dependency or build folders, deeper dot-folders and the tool +homes and caches among the home folder's dot-folders (.npm, .gradle, .cache and +the like; .exact is entered), stops at a recognised project and goes seven levels deep. A folder the system refuses is skipped and recorded. The result is kept in maintenance/discovery-scan.json under the Stim home (when it ran, what it found, what it could not read) and diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 909a6dfcff..04ec40881e 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -2,7 +2,7 @@ import { lstat, readdir } from 'node:fs/promises'; import { mkdirSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; import { homedir } from 'node:os'; import { join } from 'node:path'; -import { isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; +import { configDir, isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; import { discoveryScanFile, maintenanceDir, @@ -13,7 +13,8 @@ import { import { canonicalPath } from '../commands/gc/paths.ts'; import { diskPressure, diskThresholds } from '../daemon/pressure.ts'; import { loadConfig } from './config.ts'; -import { recognises } from './discovery.ts'; +import { inside, recognises } from './discovery.ts'; +import { isTemporaryLocation } from './protected-locations.ts'; const HOUR_MS = 60 * 60_000; const WEEK_MS = 7 * 24 * HOUR_MS; @@ -34,6 +35,53 @@ const HOME_SKIPPED = new Set([ 'Public', ]); +// @ref LLP 0004#tool-folders-and-the-full-scan -- the home dot-folders the scan does not enter +/** + * Dot-folders directly in the home folder that hold many files and no projects: tool homes, package and build caches, + * runtimes, editor state and other tools' own state. Counted on one machine (entries to depth four): .gradle 67k, + * .agent-device 45k, .member-app-metro-cache 25k, .dartwords-metro-cache 17k, .npm 16k, .bun 16k, .rvm 15k, .yarn 15k, + * .android 7k, .vscode 6k, .cursor 5k, .claude 3k, .cache 2k, .ivy2 2k, .cargo 2k. `.codex` is reached through its + * worktrees folder, and `.stim` is the Stim home. Every other home dot-folder is entered, such as `.exact`. + */ +const HOME_DOT_PRUNED = new Set([ + '.git', + '.Trash', + '.stim', + '.stim-npm', + '.stim-audits', + '.codex', + '.claude', + '.gradle', + '.agent-device', + '.npm', + '.pnpm-store', + '.yarn', + '.bun', + '.cargo', + '.rustup', + '.nvm', + '.rvm', + '.cache', + '.android', + '.cocoapods', + '.vscode', + '.cursor', + '.antigravity', + '.local', + '.config', + '.docker', + '.m2', + '.ivy2', + '.gem', + '.expo', + '.zplug', + '.virtualenvs', + '.wrangler', + '.foundry', + '.pw-profiles', + '.swiftpm', +]); + /** Folders at any depth that hold many files and no projects: dependencies, build output and tool caches. */ const PRUNED = new Set([ 'node_modules', @@ -53,9 +101,14 @@ const PRUNED = new Set([ ]); // @ref LLP 0004#tool-folders-and-the-full-scan -- what the scan prunes -/** Whether the scan descends into a folder named `name`, `depth` levels below the home folder (1 is a child of it). */ +/** + * Whether the scan descends into a folder named `name`, `depth` levels below the home folder (1 is a child of it). + * Dot-folders are entered only directly in the home folder and not when they are in `HOME_DOT_PRUNED` or named like a + * cache; deeper ones, such as `.worktrees`, are skipped because git reaches their worktrees. + */ export function scanDescends(name: string, depth: number): boolean { - if (name.startsWith('.') || PRUNED.has(name)) return false; + if (name.startsWith('.')) return depth === 1 && !HOME_DOT_PRUNED.has(name) && !name.endsWith('-cache'); + if (PRUNED.has(name)) return false; return depth !== 1 || !HOME_SKIPPED.has(name); } @@ -77,7 +130,11 @@ const errorCode = (error: unknown) => (error instanceof Error && 'code' in error export async function scanForCheckouts( home: string, isCheckout: (path: string) => boolean, - { budgetMs = BUDGET_MS, signal }: { budgetMs?: number; signal?: AbortSignal } = {}, + { + budgetMs = BUDGET_MS, + signal, + skip = () => false, + }: { budgetMs?: number; signal?: AbortSignal; skip?: (path: string) => boolean } = {}, ): Promise { const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false }; const deadline = Date.now() + budgetMs; @@ -113,6 +170,7 @@ export async function scanForCheckouts( for (const entry of entries) { if (!entry.isDirectory() || entry.isSymbolicLink() || !scanDescends(entry.name, depth + 1)) continue; const path = join(folder, entry.name); + if (skip(path)) continue; if ((await lstat(path).catch(() => null))?.dev !== volume) continue; await walk(path, depth + 1); } @@ -177,7 +235,11 @@ export async function runDiscoveryScanIfDue( const store = (scan: Omit) => writeScan({ version: 1, ranAt: now, durationMs: Date.now() - now, trigger, ...scan }); try { - const outcome = await scanForCheckouts(canonicalPath(home), recognises, { signal: operation.signal }); + const stimHome = canonicalPath(configDir()); + const homeIsTemporary = isTemporaryLocation(home); + const skip = (path: string) => + inside(canonicalPath(path), stimHome) || (!homeIsTemporary && isTemporaryLocation(path)); + const outcome = await scanForCheckouts(canonicalPath(home), recognises, { signal: operation.signal, skip }); store({ ...outcome, found: outcome.found.toSorted() }); } catch { if (operation.signal.aborted) { diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index a4c38a9917..85088fe11f 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -6,7 +6,7 @@ import { isScopedHome, type StimContext } from '@stim-cli/core'; import { machineSettingValue, readDiscoveryScan, type Config } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; import { projectRegistry } from '../integrations/projects.ts'; -import { isProtectedLocation } from './protected-locations.ts'; +import { isProtectedLocation, isTemporaryLocation } from './protected-locations.ts'; import { matchWorktreeEntry, resolveSourceCheckoutAsync } from './worktree.ts'; const DISCOVER_ROOTS_SETTING = 'maintenance.discoverRoots'; @@ -167,7 +167,7 @@ export function broadDiscoverRoots(config: Config | null, env: NodeJS.ProcessEnv return configuredRoots(config, env).roots.filter((root) => existsSync(root) && isBroadFolder(canonicalPath(root))); } -const inside = (path: string, folder: string) => path === folder || path.startsWith(`${folder}${sep}`); +export const inside = (path: string, folder: string): boolean => path === folder || path.startsWith(`${folder}${sep}`); /** Whether `path` lies inside a tool folder (see `TOOL_FOLDERS`) of the home folder `home`. */ export function inToolFolder(path: string, home: string): boolean { @@ -323,7 +323,9 @@ async function computeDiscovered( .filter((path) => !isBroadFolder(path)) .map((path) => ({ path, origin: 'scan', from: 'full scan', recognised: true })); const allowed = [...children, ...scanned].filter( - (child) => canonicalRoots.some((root) => inside(child.path, root)) || !isProtectedLocation(child.path), + (child) => + canonicalRoots.some((root) => inside(child.path, root)) || + (child.origin === 'scan' ? !isTemporaryLocation(child.path) : !isProtectedLocation(child.path)), ); const knownRoots = known.map(canonicalPath); const everything = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); diff --git a/packages/stim-cli/src/workspace/protected-locations.ts b/packages/stim-cli/src/workspace/protected-locations.ts index 76d2ae185f..41c4ec8743 100644 --- a/packages/stim-cli/src/workspace/protected-locations.ts +++ b/packages/stim-cli/src/workspace/protected-locations.ts @@ -28,10 +28,26 @@ function real(path: string): string | null { * resolve counts as protected. Discovery lists the worktrees in `~/.codex/worktrees` anyway, as reduce-only. */ export function isProtectedLocation(path: string, roots: Roots = defaultRoots()): boolean { + return isTemporaryLocation(path, roots) || isHomeDotFolder(path, roots); +} + +/** Whether a path lies under the system temporary directory; a path that does not resolve counts as temporary. */ +export function isTemporaryLocation(path: string, roots: Roots = defaultRoots()): boolean { + const target = real(path); + if (target === null) return true; + return roots.temps.some((folder) => { + const temp = real(folder) ?? folder; + return target === temp || target.startsWith(`${temp}${sep}`); + }); +} + +/** Whether a path lies under a dot-folder directly inside the home directory; a path that does not resolve counts. */ +function isHomeDotFolder(path: string, roots: Roots = defaultRoots()): boolean { const target = real(path); if (target === null) return true; - const inside = (folder: string) => target === folder || target.startsWith(`${folder}${sep}`); - if (roots.temps.some((folder) => inside(real(folder) ?? folder))) return true; const home = real(roots.home) ?? roots.home; - return inside(home) && relative(home, target).split(sep)[0]?.startsWith('.') === true; + return ( + (target === home || target.startsWith(`${home}${sep}`)) && + relative(home, target).split(sep)[0]?.startsWith('.') === true + ); } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 1bc96076d4..eda1d40dbe 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan weekly and, when the disk level rises above normal, at most once an hour; no command runs it. The scan reads `~/Documents`, so macOS may ask for permission once. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, dot-folders, symlinks and dependency or build folders, and it stops at a recognised project. A folder it cannot read is skipped. Discovery uses the last scan's saved result, so `stim gc` never walks your disk itself. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan weekly and, when the disk level rises above normal, at most once an hour; no command runs it. The scan reads `~/Documents`, so macOS may ask for permission once. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, and the tool homes and caches among your home folder's dot-folders (`.npm`, `.gradle`, `.cache` and similar), and it stops at a recognised project. It does enter other dot-folders of your home folder, such as `~/.exact`, and never the Stim home. A folder it cannot read is skipped. Discovery uses the last scan's saved result, so `stim gc` never walks your disk itself. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 0f086523d0d3601480bf490da827961db42b3354 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 00:09:45 -0500 Subject: [PATCH 05/25] fix: revalidate cached scan finds, persist the scan schedule, run the scan after the pass and window reduce-only checkouts without hiding them --- llp/0004-discovered-checkouts.decision.md | 28 ++- packages/core/state/discovery-scan.ts | 21 +- packages/stim-cli/api.api.txt | 1 + .../__tests__/discovered-reduce-only.test.ts | 80 ++++-- .../src/__tests__/discovery-scan.test.ts | 168 +++++++++---- .../stim-cli/src/__tests__/discovery.test.ts | 4 +- packages/stim-cli/src/daemon/stages.ts | 7 +- packages/stim-cli/src/guide/cleanup.ts | 38 +-- .../stim-cli/src/stimd/passes/artifacts.ts | 4 +- .../stim-cli/src/stimd/passes/cargo-target.ts | 4 +- .../stim-cli/src/stimd/passes/level-run.ts | 5 +- packages/stim-cli/src/stimd/passes/run.ts | 4 +- .../stim-cli/src/workspace/discovery-scan.ts | 238 +++++++----------- packages/stim-cli/src/workspace/discovery.ts | 52 ++-- packages/stim-cli/src/workspace/scan-rules.ts | 135 ++++++++++ website/docs/worktrees.md | 2 +- 16 files changed, 521 insertions(+), 270 deletions(-) create mode 100644 packages/stim-cli/src/workspace/scan-rules.ts diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index ef42c3e33e..d35fb83599 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -74,8 +74,8 @@ hold that keeps a worktree inside a dot-folder of its main checkout `~/.codex/worktrees`. The list is one constant with that one path. - The full scan: a pruned walk of the home folder for checkouts that Stim's project detection recognises and discovery does not reach. The daemon runs it - weekly and, while the disk level is above normal, at most hourly; a command - never runs it. It reads `~/Documents`, skips `~/Desktop`, `~/Downloads`, + weekly and, while the disk level is above normal, when no scan has run for an + hour; a command never runs it. It reads `~/Documents`, skips `~/Desktop`, `~/Downloads`, `~/Library`, other media folders, symlinks, other volumes, dependency, build and cache folders, dot-folders below the first level and the named tool homes among the home folder's dot-folders (see below), and the Stim home, stops at a recognised project and goes @@ -83,8 +83,16 @@ hold that keeps a worktree inside a dot-folder of its main checkout an error. The result is stored in `maintenance/discovery-scan.json` under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a maintenance pass or `stim gc` never walks the disk. -- At most 40 reduce-only checkouts are handled per discovery, in a window over - them that moves every 15 minutes. +- Passes evaluate at most 40 reduce-only checkouts at a time, in a window over + them that moves every 15 minutes. Discovery still lists every one, and the + report says which are left for a later pass. +- The stored result keeps the schedule (last attempt, last attempt that did not + fail, last attempt at low disk) and the finds. A failed or truncated scan + keeps the previous finds, and a finished one keeps those under a folder it + could not read, so a worse result never replaces a better one. Discovery + revalidates each find before using it: a real folder, not a symlink, on the + home volume, below no skipped folder, still recognised. +- The scan runs after a maintenance pass has acted, so it never delays cleanup. A folder named in `maintenance.discoverRoots` keeps the full rules. A scoped `STIM_HOME` discovers nothing from either source. @@ -96,7 +104,9 @@ The scan enters a dot-folder directly in the home folder unless it is in a named prune list of tool homes, package and build caches and runtimes (`.git`, `.npm`, `.yarn`, `.bun`, `.gradle`, `.cargo`, `.rustup`, `.nvm`, `.rvm`, `.cache`, `.android`, `.vscode`, `.cursor`, `.agent-device`, `.claude`, -`.codex` and others in `HOME_DOT_PRUNED`) or named like a cache (`*-cache`). +`.codex` and others in `HOME_DOT_PRUNED`, and by prefix every `.stim*` folder: the Stim +home and other Stim state, where the build worker's client checkouts must never become +reduce-only) or named like a cache (`*-cache`). The list comes from entry counts to depth four on one machine, led by `.gradle` (67 thousand), `.agent-device` (45 thousand), the two Metro cache folders (25 and 17 thousand), `.npm`, `.bun`, `.rvm` and `.yarn` (15 thousand each). @@ -146,10 +156,10 @@ Chosen by the implementing lane, not by the maintainer or the coordinator: the bound of 40 reduce-only checkouts per discovery in a window that moves every 15 minutes (not "least recently handled first"), the depth of seven, the folders pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and -`~/Applications` left out, the low-disk scan running once per rise of the disk -level (the daemon's high or critical disk level, `disk.minFreeGb`) at most -hourly rather than hourly for as long as the disk stays low, and the scan -running only in the daemon. +`~/Applications` left out, the low-disk scan running hourly while the daemon's disk level is above +normal (high or critical, `disk.minFreeGb`) and no scan ran for an hour, the +retry of a failed scan after an hour, and the scan running only in the daemon +after the pass. Not part of this change: points 4, 5 and 7, which belong to the doctor listing and the low-disk line. diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts index c455404598..1c84961e94 100644 --- a/packages/core/state/discovery-scan.ts +++ b/packages/core/state/discovery-scan.ts @@ -6,14 +6,21 @@ import { discoveryScanFile } from './paths.ts'; * `maintenance.discoverRoots`, registered projects and the common tool folders do not reach, and what it could not read. */ export interface DiscoveryScan { - version: 1; - /** When the scan started, epoch ms. A scan that failed or ran out of time is stamped too. */ + version: 2; + /** When the last attempt started, epoch ms. A failed or truncated attempt is stamped too. */ ranAt: number; + /** When the last attempt that did not fail started; 0 when none did. The weekly schedule counts from it. */ + completedAt: number; + /** When the last attempt that ran while a volume's level was above normal started; 0 when none did. */ + lowScanAt: number; durationMs: number; trigger: 'weekly' | 'low-disk'; /** Folders the scan read. */ visited: number; - /** Canonical paths of the recognised checkouts. */ + /** + * Canonical paths of the recognised checkouts. A failed or truncated scan keeps the previous scan's finds that it did + * not reach, and a scan keeps those under a folder it could not read; discovery revalidates each before using it. + */ found: string[]; /** Folders the system refused or that did not answer, skipped without error. */ unreadable: string[]; @@ -29,8 +36,10 @@ const finite = (value: unknown): value is number => typeof value === 'number' && export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { if ( !isJsonObject(value) || - value.version !== 1 || + value.version !== 2 || !finite(value.ranAt) || + !finite(value.completedAt) || + !finite(value.lowScanAt) || !finite(value.durationMs) || (value.trigger !== 'weekly' && value.trigger !== 'low-disk') || !finite(value.visited) || @@ -40,8 +49,10 @@ export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { ) return null; return { - version: 1, + version: 2, ranAt: value.ranAt, + completedAt: value.completedAt, + lowScanAt: value.lowScanAt, durationMs: value.durationMs, trigger: value.trigger, visited: value.visited, diff --git a/packages/stim-cli/api.api.txt b/packages/stim-cli/api.api.txt index 10df1fb611..03f5d1fd7d 100644 --- a/packages/stim-cli/api.api.txt +++ b/packages/stim-cli/api.api.txt @@ -357,6 +357,7 @@ [*] interface DevicePlacement { @@1 machine?: string; [*] interface DevicePlacement { @@1 reason: string; [*] interface DiscoveredCheckout { +[*] interface DiscoveredCheckout { @@1 deferred?: boolean; [*] interface DiscoveredCheckout { @@1 from: string; [*] interface DiscoveredCheckout { @@1 origin: DiscoveryOrigin; [*] interface DiscoveredCheckout { @@1 path: string; diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 033e76361d..02cb26a347 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -6,15 +6,18 @@ import { readdirSync, realpathSync, rmSync, + symlinkSync, utimesSync, writeFileSync, } from 'node:fs'; import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; +import { artifactCheckoutsAsync } from '../stimd/passes/artifacts.ts'; +import { stoppedOutputCandidates } from '../daemon/stages.ts'; import { saveConfig } from '../workspace/config.ts'; import { collectWorktreeSweep, removeWorktrees } from '../commands/gc/worktrees.ts'; import { - boundReduceOnly, + windowReduceOnly, discoverCheckouts, discoveredOutside, type DiscoveredCheckout, @@ -96,29 +99,70 @@ describe('checkouts found through a tool folder or the full scan', () => { ]); }); - it('discovers a checkout under a home dot-folder only when the scan result names it, as reduce-only', async () => { - const exact = join(base, 'home', '.exact', 'exact2'); - mkdirSync(exact, { recursive: true }); - writeFileSync(join(exact, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); - const config = { projects: { [main]: {} }, repos: {} }; - expect(await discoverCheckouts(config, [main])).toEqual([]); + const writeScan = (found: string[]) => { mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); const scan = { - version: 1, + version: 2, ranAt: 1, + completedAt: 1, + lowScanAt: 0, durationMs: 1, trigger: 'weekly', visited: 1, - found: [exact], + found, unreadable: [], truncated: false, }; writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); + }; + + it('discovers a checkout under a home dot-folder only when the scan result names it, as reduce-only', async () => { + const exact = join(base, 'home', '.exact', 'exact2'); + mkdirSync(exact, { recursive: true }); + writeFileSync(join(exact, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + const config = { projects: { [main]: {} }, repos: {} }; + expect(await discoverCheckouts(config, [main])).toEqual([]); + writeScan([exact]); expect( (await discoverCheckouts(config, [main])).map(({ path, origin, reduceOnly }) => [path, origin, reduceOnly]), ).toEqual([[exact, 'scan', true]]); }); + it('drops a cached checkout that was replaced by a symlink to a folder the scan never enters, and plans nothing under it', async () => { + const moved = join(base, 'elsewhere', 'app'); + mkdirSync(moved, { recursive: true }); + writeFileSync(join(moved, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + mkdirSync(join(base, 'home', 'Documents'), { recursive: true }); + const stale = join(base, 'home', 'Documents', 'app'); + symlinkSync(moved, stale); + writeScan([stale, moved]); + const config = { projects: { [main]: {} }, repos: {} }; + expect(await discoverCheckouts(config, [main])).toEqual([]); + const planned = await artifactCheckoutsAsync(new AbortController().signal); + expect(planned.filter((entry) => entry.path.includes('elsewhere') || entry.path.includes('Documents'))).toEqual([]); + }); + + it('holds a scan-found linked worktree at the removal plan', async () => { + const found = worktree(join(base, 'home', 'Documents', 'feature-wt'), 'feature'); + writeScan([found]); + const planned = await collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false }); + expect(planned.worktrees.find((entry) => entry.path === found)).toMatchObject({ + discovered: true, + reduceOnly: true, + skipped: expect.stringContaining('never removes'), + }); + }); + + it('clears the build output of an idle reduce-only worktree in the dormant stage and leaves the worktree', async () => { + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); + mkdirSync(join(codex, 'android', 'app', 'build'), { recursive: true }); + age(codex); + const config = { projects: { [main]: {} }, repos: {} }; + const stage = await stoppedOutputCandidates(config, { devices: [], simsFailed: false }, day); + expect(stage.map((entry) => entry.path)).toEqual([codex]); + expect(existsSync(codex)).toBe(true); + }); + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); @@ -139,7 +183,7 @@ describe('checkouts found through a tool folder or the full scan', () => { }); }); -describe('boundReduceOnly', () => { +describe('windowReduceOnly', () => { const entry = (n: number, reduceOnly = true): DiscoveredCheckout => ({ path: `/p/${String(n).padStart(3, '0')}`, origin: 'scan', @@ -147,20 +191,22 @@ describe('boundReduceOnly', () => { ...(reduceOnly ? { reduceOnly: true } : {}), }); const window = 15 * 60_000; + const handled = (found: DiscoveredCheckout[]) => found.filter((e) => !e.deferred).map((e) => e.path); - it('keeps every full-rule checkout, and a moving window of at most the limit of the reduce-only ones', () => { + it('lists every checkout, defers all but a moving window of the reduce-only ones, and takes each in turn', () => { const all = [entry(1, false), ...Array.from({ length: 10 }, (_, i) => entry(i + 10))]; const now = 100 * window; - const first = boundReduceOnly(all, now, 4); - expect(first.filter((e) => e.reduceOnly)).toHaveLength(4); - expect(first[0]).toBe(all[0]); + const first = windowReduceOnly(all, now, 4); + expect(first.map((e) => e.path)).toEqual(all.map((e) => e.path)); + expect(first.filter((e) => e.reduceOnly && !e.deferred)).toHaveLength(4); + expect(first[0]?.deferred).toBeUndefined(); const seen = new Set(); - for (let i = 0; i < 10; i++) for (const e of boundReduceOnly(all, now + i * window, 4)) seen.add(e.path); + for (let i = 0; i < 10; i++) for (const path of handled(windowReduceOnly(all, now + i * window, 4))) seen.add(path); expect(seen.size).toBe(all.length); }); - it('returns everything when the reduce-only ones fit the limit', () => { + it('defers nothing when the reduce-only ones fit the limit', () => { const all = [entry(1), entry(2)]; - expect(boundReduceOnly(all, 0, 4)).toEqual(all); + expect(windowReduceOnly(all, 0, 4)).toEqual(all); }); }); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index 387684ebc6..81915b1c92 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -12,13 +12,9 @@ import { import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { testContext } from './_context.ts'; -import { discoveryScanFile, readDiscoveryScan } from '@stim-cli/core/state'; -import { - discoveryScanDue, - runDiscoveryScanIfDue, - scanDescends, - scanForCheckouts, -} from '../workspace/discovery-scan.ts'; +import { discoveryScanFile, parseDiscoveryScan, readDiscoveryScan } from '@stim-cli/core/state'; +import { discoveryScanDue, mergeFound, runDiscoveryScanIfDue, scanForCheckouts } from '../workspace/discovery-scan.ts'; +import { scanDescends } from '../workspace/scan-rules.ts'; const marked = (path: string) => existsSync(join(path, 'package.json')); @@ -36,7 +32,17 @@ describe('scanDescends', () => { it('enters a dot-folder of the home folder unless it is a tool home or cache, and no deeper dot-folder', () => { expect(scanDescends('.exact', 1)).toBe(true); - for (const name of ['.git', '.npm', '.gradle', '.cache', '.stim', '.codex', '.member-app-metro-cache']) + for (const name of [ + '.git', + '.npm', + '.gradle', + '.cache', + '.stim', + '.stim-agent', + '.stim-release', + '.codex', + '.member-app-metro-cache', + ]) expect(scanDescends(name, 1)).toBe(false); for (const name of ['.worktrees', '.exact', '.claude']) expect(scanDescends(name, 2)).toBe(false); }); @@ -108,14 +114,67 @@ describe('scanForCheckouts', () => { describe('discoveryScanDue', () => { const now = Date.UTC(2026, 10, 14); - const ago = (ms: number) => ({ ranAt: now - ms }); - it('runs weekly, an hour after a rise in the disk level, and when no scan or a future one is recorded', () => { + const at = (ranAgo: number, completedAgo = ranAgo, lowAgo = 1e12) => ({ + ranAt: now - ranAgo, + completedAt: now - completedAgo, + lowScanAt: now - lowAgo, + }); + const hour = 3_600_000; + const day = 24 * hour; + + it('runs weekly from the last scan that did not fail, and when none or a future one is recorded', () => { expect(discoveryScanDue(null, now, false)).toBe(true); - expect(discoveryScanDue(ago(-1000), now, false)).toBe(true); - expect(discoveryScanDue(ago(6 * 86_400_000), now, false)).toBe(false); - expect(discoveryScanDue(ago(7 * 86_400_000), now, false)).toBe(true); - expect(discoveryScanDue(ago(30 * 60_000), now, true)).toBe(false); - expect(discoveryScanDue(ago(61 * 60_000), now, true)).toBe(true); + expect(discoveryScanDue(at(-1000), now, false)).toBe(true); + expect(discoveryScanDue(at(6 * day), now, false)).toBe(false); + expect(discoveryScanDue(at(7 * day), now, false)).toBe(true); + }); + + it('retries a failed scan after an hour, not after a week, and never twice within an hour', () => { + expect(discoveryScanDue(at(30 * 60_000, 8 * day), now, false)).toBe(false); + expect(discoveryScanDue(at(2 * hour, 8 * day), now, false)).toBe(true); + }); + + it('scans while the disk level is above normal when no scan ran during it for an hour, even right after the weekly one', () => { + expect(discoveryScanDue(at(2 * hour), now, true)).toBe(true); + expect(discoveryScanDue(at(2 * hour, 2 * hour, 3 * hour), now, true)).toBe(true); + expect(discoveryScanDue(at(2 * hour, 2 * hour, 90 * 60_000), now, true)).toBe(true); + expect(discoveryScanDue(at(30 * 60_000, 30 * 60_000, 30 * 60_000), now, true)).toBe(false); + }); +}); + +describe('mergeFound', () => { + const outcome = (found: string[], unreadable: string[], truncated: boolean) => ({ + found, + unreadable, + truncated, + visited: 1, + }); + it('keeps the previous finds under an unreadable folder, drops the others, and keeps all of them when truncated', () => { + const previous = ['/h/Documents/a', '/h/dev/b']; + expect(mergeFound(previous, outcome(['/h/dev/c'], ['/h/Documents'], false))).toEqual([ + '/h/Documents/a', + '/h/dev/c', + ]); + expect(mergeFound(previous, outcome(['/h/dev/c'], [], true))).toEqual(['/h/Documents/a', '/h/dev/b', '/h/dev/c']); + }); +}); + +describe('a scan file that is corrupt or in an older format', () => { + it('is no result', () => { + expect( + parseDiscoveryScan({ + version: 1, + ranAt: 1, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [], + unreadable: [], + truncated: false, + }), + ).toBeNull(); + expect(parseDiscoveryScan({ version: 2, ranAt: 1 })).toBeNull(); + expect(parseDiscoveryScan('nope')).toBeNull(); }); }); @@ -123,12 +182,13 @@ describe('runDiscoveryScanIfDue', () => { let base: string; const normal = [{ volume: '/', freeMb: 500_000 }]; const low = [{ volume: '/', freeMb: 10_000 }]; - const tick = (disk = normal, env: NodeJS.ProcessEnv = {}) => ({ - operation: testContext({ env }), + const tick = (disk = normal, env: NodeJS.ProcessEnv = {}, operation = testContext({ env })) => ({ + operation, measured: { measurement: { disk } }, }); - const memory = () => ({ attemptedAt: 0, wasLow: false }); - const scanned = () => readDiscoveryScan()?.trigger; + const hour = 3_600_000; + const t0 = Date.UTC(2026, 10, 14); + const options = (now: number) => ({ home: join(base, 'home'), now }); beforeEach(() => { base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-run-'))); @@ -146,55 +206,61 @@ describe('runDiscoveryScanIfDue', () => { }); it('never scans for a command, only for a daemon tick', async () => { - await runDiscoveryScanIfDue(undefined, { home: join(base, 'home'), memory: memory() }); + await runDiscoveryScanIfDue(undefined, options(t0)); expect(readDiscoveryScan()).toBeNull(); }); it('never scans in a scoped home and writes nothing', async () => { vi.stubEnv('STIM_HOME', join(base, 'state')); - await runDiscoveryScanIfDue(tick(normal, { STIM_HOME: join(base, 'state') }), { - home: join(base, 'home'), - memory: memory(), - }); + await runDiscoveryScanIfDue(tick(normal, { STIM_HOME: join(base, 'state') }), options(t0)); expect(existsSync(join(base, 'state'))).toBe(false); }); - it('stamps the result under the Stim home and does not scan again until it is due', async () => { - const options = { home: join(base, 'home'), memory: memory() }; - await runDiscoveryScanIfDue(tick(), options); + it('stores the result and its schedule under the Stim home, and not again until due', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); expect(readDiscoveryScan()).toMatchObject({ trigger: 'weekly', + ranAt: t0, + completedAt: t0, + lowScanAt: 0, found: [join(base, 'home', 'dev', 'app')], truncated: false, }); expect(discoveryScanFile().startsWith(join(base, 'machine'))).toBe(true); const before = readFileSync(discoveryScanFile(), 'utf8'); - await runDiscoveryScanIfDue(tick(), options); + await runDiscoveryScanIfDue(tick(), options(t0 + 2 * hour)); expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); }); - it('scans once the disk level rises above normal an hour after the last scan, not again while it stays low', async () => { - const options = { home: join(base, 'home'), memory: memory() }; - const backdate = () => { - writeFileSync(discoveryScanFile(), JSON.stringify({ ...readDiscoveryScan(), ranAt: Date.now() - 2 * 3_600_000 })); - options.memory.attemptedAt = 0; - }; - await runDiscoveryScanIfDue(tick(), options); - backdate(); - await runDiscoveryScanIfDue(tick(low), options); - expect(scanned()).toBe('low-disk'); - backdate(); - await runDiscoveryScanIfDue(tick(low), options); - expect(Date.now() - (readDiscoveryScan()?.ranAt ?? 0)).toBeGreaterThan(3_600_000); - }); - - it('does not walk again within the interval when the result cannot be stored, and does not throw', async () => { - writeFileSync(join(base, 'machine', '.stim'), 'a file where the Stim home folder belongs'); - const options = { home: join(base, 'home'), memory: memory() }; - await expect(runDiscoveryScanIfDue(tick(), options)).resolves.toBeUndefined(); - expect(options.memory.attemptedAt).toBeGreaterThan(0); - const attempted = options.memory.attemptedAt; - await runDiscoveryScanIfDue(tick(), options); - expect(options.memory.attemptedAt).toBe(attempted); + it('scans again an hour or more after a scan when the disk level is above normal, whatever a worker remembers', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + await runDiscoveryScanIfDue(tick(low), options(t0 + 2 * hour)); + expect(readDiscoveryScan()).toMatchObject({ + trigger: 'low-disk', + ranAt: t0 + 2 * hour, + lowScanAt: t0 + 2 * hour, + completedAt: t0 + 2 * hour, + }); + }); + + it('keeps the previous finds when a scan fails and records the attempt', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + rmSync(join(base, 'home'), { recursive: true }); + await runDiscoveryScanIfDue(tick(low), options(t0 + 2 * hour)); + expect(readDiscoveryScan()).toMatchObject({ + ranAt: t0 + 2 * hour, + completedAt: t0, + truncated: true, + found: [join(base, 'home', 'dev', 'app')], + }); + }); + + it('lets an abort escape instead of returning normally, and stores nothing', async () => { + const controller = new AbortController(); + controller.abort(); + await expect( + runDiscoveryScanIfDue(tick(normal, {}, testContext({ signal: controller.signal })), options(t0)), + ).rejects.toThrow(/abort/i); + expect(readDiscoveryScan()).toBeNull(); }); }); diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index 2ddf3fe80a..72ce4b1316 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -162,7 +162,9 @@ describe('discoverCheckouts', () => { app(docs); mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); const scan = { - version: 1, + version: 2, + completedAt: 1, + lowScanAt: 0, ranAt: 1, durationMs: 1, trigger: 'weekly', diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index f52d5b6f69..eb20c5bd3d 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -232,10 +232,9 @@ async function idleWorkspaces( const readings: (Omit & { idleMs: number | null })[] = []; const reads = discoveredReads(reading.operation?.signal); const discovered = discover - ? (await discoverCheckouts(config, listedRoots(), reading.operation)).map(({ path }): [string, ProjectRecord] => [ - path, - {}, - ]) + ? (await discoverCheckouts(config, listedRoots(), reading.operation)) + .filter(({ deferred }) => !deferred) + .map(({ path }): [string, ProjectRecord] => [path, {}]) : []; for (const [path, project] of [...Object.entries(config.projects), ...discovered]) { if (sameRoot(path, reading.protectedRoot) || !(await worthIt(path, project))) continue; diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 32515e06f9..a4db5be937 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -29,7 +29,7 @@ The stim-daemon runs a pass in its own process on each tick when a check is due: a disk and memory reading for status (every minute), hosted idle stops (every 5 minutes), directory sizes (hourly), finished worktrees (every 15 minutes) and the age sweep (daily); the daemon also rescans the home folder for -checkouts weekly, or hourly at low disk (DISCOVERED CHECKOUTS). The reading plans nothing, the idle check +checkouts after the pass, weekly and also at low disk (DISCOVERED CHECKOUTS). The reading plans nothing, the idle check plans only idle stops, and the cap trims follow the size, worktree and sweep checks. Pressure itself is the daemon's: each tick classifies the machine normal, high or critical and runs the pressure ladder @@ -1211,19 +1211,29 @@ a discovered repository and pull requests come from the cache. Two more sources find checkouts that Stim only reduces and never removes (reduce-only): linked worktrees of known repositories in ~/.codex/worktrees, and the checkouts of the full scan. The scan is a pruned walk of the home -folder, run by the daemon weekly and, when the disk level rises above normal, at -most hourly; no command runs it. It reads ~/Documents (macOS may -ask once), skips ~/Desktop, ~/Downloads, ~/Library, the media folders, other -volumes, symlinks, dependency or build folders, deeper dot-folders and the tool -homes and caches among the home folder's dot-folders (.npm, .gradle, .cache and -the like; .exact is entered), stops at a -recognised project and goes seven levels deep. A folder the system refuses is -skipped and recorded. The result is kept in maintenance/discovery-scan.json -under the Stim home (when it ran, what it found, what it could not read) and -discovery reads that file, so a pass or gc never walks the disk. Reduce-only -checkouts have their build outputs and, in a linked worktree, node_modules and -Pods reduced by the rules below; at most 40 of them are handled at a time, in a -window that moves every 15 minutes. A folder named in maintenance.discoverRoots +folder, run by the daemon after a maintenance pass has acted (never before its +cleanup, never by a command): weekly, and also while the disk level is above +normal when no scan has run for an hour; a failed scan is retried after an hour. +The schedule is kept with the result, so a restarted daemon forgets nothing. It +reads ~/Documents (macOS may ask once; a folder that does not answer within 3 +seconds, such as one waiting for that answer, is recorded as unreadable and is +read on the next scan), skips ~/Desktop, ~/Downloads, ~/Library, the media +folders, other volumes, symlinks, dependency or build folders, deeper +dot-folders, the Stim home and every dot-folder starting with .stim, the build +worker root, and the tool homes and caches among the home folder's other +dot-folders (.npm, .gradle, .cache and the like; .exact is entered), stops at a +recognised project and goes seven levels deep within 45 seconds. The result is +kept in maintenance/discovery-scan.json under the Stim home (when it ran, what +it found, what it could not read) and discovery reads that file, so a pass or gc +never walks the disk. A scan that fails or stops early keeps the previous finds, +and a finished one keeps those under a folder it could not read. Discovery checks +each recorded checkout again before using it: still a real folder (not a +symlink), on the home volume, below no skipped folder and still a project; +anything else is dropped. Reduce-only checkouts have their build outputs and, in +a linked worktree, node_modules and Pods reduced by the rules below. Passes +evaluate at most 40 of them at a time, in a window that moves every 15 minutes; +stim gc --dry-run lists all of them and says which are left for a later pass. A +folder named in maintenance.discoverRoots keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine Idle, with no Stim activity record, means the newest of its source file writes diff --git a/packages/stim-cli/src/stimd/passes/artifacts.ts b/packages/stim-cli/src/stimd/passes/artifacts.ts index f60f86c4e4..4b8d94051d 100644 --- a/packages/stim-cli/src/stimd/passes/artifacts.ts +++ b/packages/stim-cli/src/stimd/passes/artifacts.ts @@ -139,7 +139,9 @@ export async function artifactCheckoutsAsync( operation?: StimContext, ): Promise { const found = new Map(); - const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); + const discovered = (await discoverCheckouts(loadConfig(), listedRoots(), operation)).filter( + (entry) => !entry.deferred, + ); const known = new Set([...listedRoots(), ...discovered.map(({ path }) => path)]); for (const root of (await candidateRootsAsync(operation, discovered)).filter((entry) => known.has(entry))) { signal.throwIfAborted(); diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index c77ddc43af..a78c2086f1 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -107,7 +107,9 @@ async function cargoTargetCandidates( }; for (const root of registered) await add(root, false); const discovered = new Set( - (await discoverCheckouts(loadConfig(), listedRoots(), operation)).map(({ path }) => canonicalPath(path)), + (await discoverCheckouts(loadConfig(), listedRoots(), operation)) + .filter(({ deferred }) => !deferred) + .map(({ path }) => canonicalPath(path)), ); for (const checkout of await artifactCheckoutsAsync(signal, operation)) for (const base of appRoots(checkout, 'build')) await add(base, discovered.has(canonicalPath(checkout.path))); diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index 6a92975290..a50d7458ed 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -214,8 +214,9 @@ export async function runMaintenanceAtLevel({ for (const checkout of found) { const mine = actions.filter((action) => ownerOf(action.target) === checkout.path); const taken = mine.some((action) => action.outcome === 'would' || action.outcome === 'done'); - const kept = - taken || idleMs <= 0 || !existsSync(checkout.path) + const kept = checkout.deferred + ? 'left for a later pass: at most 40 checkouts found through a tool folder or the full scan are handled at a time' + : taken || idleMs <= 0 || !existsSync(checkout.path) ? null : await discoveredKeepReason(checkout.path, idleMs, operation, reads); if (kept && /source entries to check|cannot be read/.test(kept)) diff --git a/packages/stim-cli/src/stimd/passes/run.ts b/packages/stim-cli/src/stimd/passes/run.ts index c9986edf5c..1ca151f931 100644 --- a/packages/stim-cli/src/stimd/passes/run.ts +++ b/packages/stim-cli/src/stimd/passes/run.ts @@ -402,12 +402,14 @@ export async function runMaintenance( crashLine(error); return; } + let scanAfter = false; let logger: ReturnType | undefined; let gcLog: GcLog | undefined; try { const resolved = resolveMaintenanceSettings(); const settings = report && resolved.mode === 'on' ? { ...resolved, mode: 'report' as const } : resolved; if (settings.mode === 'off') return; + scanAfter = true; const state: MaintenanceState = readMaintenanceState() ?? { version: 1, lastAt: {}, @@ -417,7 +419,6 @@ export async function runMaintenance( recent: [], plan: [], }; - await runDiscoveryScanIfDue(tick); const checks = due(state, settings, startedAt); if (checks.length === 0) return; const background = BACKGROUND_TRIGGERS.has(trigger); @@ -738,6 +739,7 @@ export async function runMaintenance( logger?.close(); keepReserve(); releaseClaim(claim); + if (scanAfter) await runDiscoveryScanIfDue(tick); } } diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 04ec40881e..a550592b04 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -1,117 +1,27 @@ import { lstat, readdir } from 'node:fs/promises'; import { mkdirSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; import { homedir } from 'node:os'; -import { join } from 'node:path'; -import { configDir, isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; +import { join, sep } from 'node:path'; +import { isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; import { discoveryScanFile, maintenanceDir, readDiscoveryScan, - type DiscoveryScan, type DaemonMeasurement, + type DiscoveryScan, } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; import { diskPressure, diskThresholds } from '../daemon/pressure.ts'; import { loadConfig } from './config.ts'; -import { inside, recognises } from './discovery.ts'; -import { isTemporaryLocation } from './protected-locations.ts'; +import { recognises } from './discovery.ts'; +import { MAX_DEPTH, scanDescends, scanSkipper } from './scan-rules.ts'; const HOUR_MS = 60 * 60_000; const WEEK_MS = 7 * 24 * HOUR_MS; -const MAX_DEPTH = 7; const MAX_FOLDERS = 200_000; -const BUDGET_MS = 2 * 60_000; +const BUDGET_MS = 45_000; const READ_TIMEOUT_MS = 3000; -/** Folders directly in the home folder the scan never reads: system, media and the two the maintainer excluded. */ -const HOME_SKIPPED = new Set([ - 'Library', - 'Applications', - 'Desktop', - 'Downloads', - 'Movies', - 'Music', - 'Pictures', - 'Public', -]); - -// @ref LLP 0004#tool-folders-and-the-full-scan -- the home dot-folders the scan does not enter -/** - * Dot-folders directly in the home folder that hold many files and no projects: tool homes, package and build caches, - * runtimes, editor state and other tools' own state. Counted on one machine (entries to depth four): .gradle 67k, - * .agent-device 45k, .member-app-metro-cache 25k, .dartwords-metro-cache 17k, .npm 16k, .bun 16k, .rvm 15k, .yarn 15k, - * .android 7k, .vscode 6k, .cursor 5k, .claude 3k, .cache 2k, .ivy2 2k, .cargo 2k. `.codex` is reached through its - * worktrees folder, and `.stim` is the Stim home. Every other home dot-folder is entered, such as `.exact`. - */ -const HOME_DOT_PRUNED = new Set([ - '.git', - '.Trash', - '.stim', - '.stim-npm', - '.stim-audits', - '.codex', - '.claude', - '.gradle', - '.agent-device', - '.npm', - '.pnpm-store', - '.yarn', - '.bun', - '.cargo', - '.rustup', - '.nvm', - '.rvm', - '.cache', - '.android', - '.cocoapods', - '.vscode', - '.cursor', - '.antigravity', - '.local', - '.config', - '.docker', - '.m2', - '.ivy2', - '.gem', - '.expo', - '.zplug', - '.virtualenvs', - '.wrangler', - '.foundry', - '.pw-profiles', - '.swiftpm', -]); - -/** Folders at any depth that hold many files and no projects: dependencies, build output and tool caches. */ -const PRUNED = new Set([ - 'node_modules', - 'Pods', - 'Carthage', - 'build', - 'dist', - 'out', - 'target', - 'vendor', - 'DerivedData', - 'venv', - '__pycache__', - 'site-packages', - 'coverage', - 'Library', -]); - -// @ref LLP 0004#tool-folders-and-the-full-scan -- what the scan prunes -/** - * Whether the scan descends into a folder named `name`, `depth` levels below the home folder (1 is a child of it). - * Dot-folders are entered only directly in the home folder and not when they are in `HOME_DOT_PRUNED` or named like a - * cache; deeper ones, such as `.worktrees`, are skipped because git reaches their worktrees. - */ -export function scanDescends(name: string, depth: number): boolean { - if (name.startsWith('.')) return depth === 1 && !HOME_DOT_PRUNED.has(name) && !name.endsWith('-cache'); - if (PRUNED.has(name)) return false; - return depth !== 1 || !HOME_SKIPPED.has(name); -} - interface ScanOutcome { found: string[]; unreadable: string[]; @@ -121,11 +31,28 @@ interface ScanOutcome { const errorCode = (error: unknown) => (error instanceof Error && 'code' in error ? String(error.code) : ''); +/** `read`'s result, or an ETIMEDOUT rejection when it has not answered in time, as a hung mount or a permission prompt would not. */ +async function answered(read: Promise): Promise { + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + read, + new Promise((_, fail) => { + timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), READ_TIMEOUT_MS); + }), + ]); + } finally { + clearTimeout(timer); + } +} + +const unanswered = (error: unknown) => ['EACCES', 'EPERM', 'ETIMEDOUT'].includes(errorCode(error)); + /** - * A pruned walk below `home`: folders for which `isCheckout` holds are results and are not entered; dependency, build - * output, tool-cache and dot-folders, symlinks and other volumes are skipped; a folder the system refuses or that does - * not answer is recorded as unreadable, never an error. It stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or - * `budgetMs`, and says so in `truncated`. + * A pruned walk below `home` (`scanDescends`, `skip`): folders for which `isCheckout` holds are results and are not + * entered; symlinks and other volumes are skipped; a folder the system refuses or that does not answer within 3 seconds + * is recorded as unreadable, never an error. It stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or `budgetMs`, and + * says so in `truncated`. An abort rejects. */ export async function scanForCheckouts( home: string, @@ -147,20 +74,13 @@ export async function scanForCheckouts( return; } outcome.visited++; - let timer: NodeJS.Timeout | undefined; let entries; try { - entries = await Promise.race([ - readdir(folder, { withFileTypes: true }), - new Promise((_, fail) => { - timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), READ_TIMEOUT_MS); - }), - ]); + entries = await answered(readdir(folder, { withFileTypes: true })); } catch (error) { - if (['EACCES', 'EPERM', 'ETIMEDOUT'].includes(errorCode(error))) outcome.unreadable.push(folder); + signal?.throwIfAborted(); + if (unanswered(error)) outcome.unreadable.push(folder); return; - } finally { - clearTimeout(timer); } if (depth > 0 && isCheckout(folder)) { outcome.found.push(folder); @@ -171,27 +91,49 @@ export async function scanForCheckouts( if (!entry.isDirectory() || entry.isSymbolicLink() || !scanDescends(entry.name, depth + 1)) continue; const path = join(folder, entry.name); if (skip(path)) continue; - if ((await lstat(path).catch(() => null))?.dev !== volume) continue; - await walk(path, depth + 1); + let stat; + try { + stat = await answered(lstat(path)); + } catch (error) { + signal?.throwIfAborted(); + if (unanswered(error)) outcome.unreadable.push(path); + continue; + } + if (stat.dev === volume) await walk(path, depth + 1); } }; await walk(home, 0); return outcome; } -/** Whether a scan is due: weekly, and once an hour after the disk level rose above normal. */ -export function discoveryScanDue(last: Pick | null, now: number, rose: boolean): boolean { - if (!last || last.ranAt > now) return true; - return now - last.ranAt >= (rose ? HOUR_MS : WEEK_MS); +/** + * The finds to store: a complete scan's, plus the previous ones under a folder it could not read; a truncated scan's, + * plus every previous one. A worse result never replaces a better one, and a stale entry is dropped at read by + * `scanEntryValid`, not here. + */ +export function mergeFound(previous: readonly string[], outcome: ScanOutcome): string[] { + const under = (path: string) => outcome.unreadable.some((folder) => path.startsWith(`${folder}${sep}`)); + const kept = outcome.truncated ? previous : previous.filter(under); + return [...new Set([...outcome.found, ...kept])].toSorted(); } -/** What this process remembers between ticks: when it last tried a scan, and whether the disk level was above normal. */ -interface ScanMemory { - attemptedAt: number; - wasLow: boolean; +/** + * Whether a scan is due. Never twice within an hour, so a failing or truncated scan is retried hourly and a rise and + * fall of the disk level cannot start a run of scans; then weekly from the last attempt that did not fail, or hourly + * while the level is above normal and no scan has run during it for an hour. + */ +export function discoveryScanDue( + last: Pick | null, + now: number, + low: boolean, +): boolean { + if (!last || last.ranAt > now) return true; + if (now - last.ranAt < HOUR_MS) return false; + return now - last.completedAt >= WEEK_MS || (low && now - last.lowScanAt >= HOUR_MS); } -const processMemory: ScanMemory = { attemptedAt: 0, wasLow: false }; +/** When this process last tried a scan, so a result that cannot be stored does not start a walk on every tick. */ +let attemptedHere = 0; function writeScan(scan: DiscoveryScan): void { mkdirSync(maintenanceDir(), { recursive: true }); @@ -210,44 +152,50 @@ function writeScan(scan: DiscoveryScan): void { } /** - * Scans the home folder when the weekly schedule says so, or an hour after the last scan when the disk level has just - * risen above normal, and stores the result under the Stim home. A scan that fails or runs out of time is stored too, - * with the checkouts the last scan found, and this process does not retry within the interval even when nothing could - * be stored. Only a daemon tick scans, never a command, and a scoped STIM_HOME never does. Nothing but an abort - * escapes, so a scan can never stop the maintenance pass it runs in. + * Scans the home folder when the schedule says so (`discoveryScanDue`) and stores the result under the Stim home, with + * the schedule's own times, so a retired worker or a restarted daemon forgets nothing. A failed scan stores nothing but + * its attempt time, keeps the previous finds and is retried after an hour. Only a daemon tick scans, never a command, + * and a scoped STIM_HOME never does. Only an abort escapes, so a scan can never end the maintenance pass it follows. */ export async function runDiscoveryScanIfDue( tick: { operation: StimContext; measured: { measurement: Pick } } | undefined, - { home = homedir(), memory = processMemory }: { home?: string; memory?: ScanMemory } = {}, + { home = homedir(), now = Date.now() }: { home?: string; now?: number } = {}, ): Promise { if (!tick || isScopedHome(tick.operation.env)) return; const { operation } = tick; - const now = Date.now(); const thresholds = diskThresholds(loadConfig(), operation.env).thresholds; const low = diskPressure(tick.measured.measurement.disk, thresholds).level !== 'normal'; - const rose = low && !memory.wasLow; - memory.wasLow = low; const last = readDiscoveryScan(); - const ranAt = Math.max(last?.ranAt ?? 0, memory.attemptedAt); - if (!discoveryScanDue(ranAt > 0 ? { ranAt } : null, now, rose)) return; - memory.attemptedAt = now; - const trigger = rose ? 'low-disk' : 'weekly'; - const store = (scan: Omit) => - writeScan({ version: 1, ranAt: now, durationMs: Date.now() - now, trigger, ...scan }); + if (!discoveryScanDue(last, now, low) || (now >= attemptedHere && now - attemptedHere < HOUR_MS)) return; + attemptedHere = now; + const attempt = { + version: 2, + ranAt: now, + lowScanAt: low ? now : (last?.lowScanAt ?? 0), + trigger: low ? 'low-disk' : 'weekly', + } as const; + const failed = { ...attempt, completedAt: last?.completedAt ?? 0, durationMs: 0, visited: 0, truncated: true }; try { - const stimHome = canonicalPath(configDir()); - const homeIsTemporary = isTemporaryLocation(home); - const skip = (path: string) => - inside(canonicalPath(path), stimHome) || (!homeIsTemporary && isTemporaryLocation(path)); - const outcome = await scanForCheckouts(canonicalPath(home), recognises, { signal: operation.signal, skip }); - store({ ...outcome, found: outcome.found.toSorted() }); + const realHome = canonicalPath(home); + const outcome = await scanForCheckouts(realHome, recognises, { + signal: operation.signal, + skip: scanSkipper(realHome), + }); + writeScan({ + ...attempt, + completedAt: now, + durationMs: Date.now() - now, + ...outcome, + found: mergeFound(last?.found ?? [], outcome), + unreadable: outcome.unreadable, + }); } catch { if (operation.signal.aborted) { - memory.attemptedAt = 0; - return; + attemptedHere = 0; + operation.signal.throwIfAborted(); } try { - store({ visited: 0, found: last?.found ?? [], unreadable: last?.unreadable ?? [], truncated: true }); + writeScan({ ...failed, found: last?.found ?? [], unreadable: last?.unreadable ?? [] }); } catch {} } } diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index 85088fe11f..50b0a3d6f0 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -7,6 +7,7 @@ import { machineSettingValue, readDiscoveryScan, type Config } from '@stim-cli/c import { canonicalPath } from '../commands/gc/paths.ts'; import { projectRegistry } from '../integrations/projects.ts'; import { isProtectedLocation, isTemporaryLocation } from './protected-locations.ts'; +import { scanEntryValid, scanSkipper } from './scan-rules.ts'; import { matchWorktreeEntry, resolveSourceCheckoutAsync } from './worktree.ts'; const DISCOVER_ROOTS_SETTING = 'maintenance.discoverRoots'; @@ -32,6 +33,11 @@ export interface DiscoveredCheckout { from: string; /** Found through a tool folder or the full scan: Stim reduces its outputs and never removes it. */ reduceOnly?: boolean; + /** + * A reduce-only checkout outside the window of at most `MAX_REDUCE_ONLY` that this discovery hands to the passes that + * evaluate each checkout; a later window takes it. Reports still list it. + */ + deferred?: boolean; } export interface DiscoveryChild { @@ -167,7 +173,7 @@ export function broadDiscoverRoots(config: Config | null, env: NodeJS.ProcessEnv return configuredRoots(config, env).roots.filter((root) => existsSync(root) && isBroadFolder(canonicalPath(root))); } -export const inside = (path: string, folder: string): boolean => path === folder || path.startsWith(`${folder}${sep}`); +const inside = (path: string, folder: string): boolean => path === folder || path.startsWith(`${folder}${sep}`); /** Whether `path` lies inside a tool folder (see `TOOL_FOLDERS`) of the home folder `home`. */ export function inToolFolder(path: string, home: string): boolean { @@ -175,11 +181,11 @@ export function inToolFolder(path: string, home: string): boolean { } /** - * At most `limit` of the checkouts found only through a tool folder or the full scan, a window over them that moves - * with `now`, so every one is handled within a few windows and one pass pays for a bounded number. The rest are left - * out of this discovery and stay as they are. + * Marks all but `limit` of the checkouts found only through a tool folder or the full scan as `deferred`: a window over + * them that moves with `now`, so every one is handled within a few windows and one pass evaluates a bounded number. + * Every checkout stays in the list, so no report or classification loses one. */ -export function boundReduceOnly( +export function windowReduceOnly( found: readonly DiscoveredCheckout[], now: number, limit: number = MAX_REDUCE_ONLY, @@ -188,7 +194,7 @@ export function boundReduceOnly( if (only.length <= limit) return [...found]; const start = (Math.floor(now / REDUCE_ONLY_WINDOW_MS) * limit) % only.length; const kept = new Set(Array.from({ length: limit }, (_, i) => only[(start + i) % only.length])); - return found.filter((entry) => !entry.reduceOnly || kept.has(entry)); + return found.map((entry) => (entry.reduceOnly && !kept.has(entry) ? { ...entry, deferred: true } : entry)); } /** @@ -226,8 +232,9 @@ export async function discoverCheckouts( config: Config | null, known: readonly string[], operation?: StimContext, + now: number = Date.now(), ): Promise { - return (await discovery(config, known, operation)).found; + return (await discovery(config, known, operation, now)).found; } /** What discovery reached outside the folders Stim manages. */ @@ -241,17 +248,32 @@ export async function discoveredOutside( const memo = new WeakMap>(); -function discovery(config: Config | null, known: readonly string[], operation?: StimContext): Promise { - if (!operation) return computeDiscovered(config, known, operation); +function discovery( + config: Config | null, + known: readonly string[], + operation?: StimContext, + now: number = Date.now(), +): Promise { + if (!operation) return computeDiscovered(config, known, operation, now); let found = memo.get(operation); - if (!found) memo.set(operation, (found = computeDiscovered(config, known, operation))); + if (!found) memo.set(operation, (found = computeDiscovered(config, known, operation, now))); return found; } +/** The checkouts of the last full scan that are still what the scan found, see `scanEntryValid`. */ +function revalidatedScan(): DiscoveryChild[] { + const home = canonicalPath(homedir()); + const rules = { home, isCheckout: recognises, skip: scanSkipper(home) }; + return (readDiscoveryScan()?.found ?? []) + .filter((path) => isAbsolute(path) && scanEntryValid(path, rules)) + .map((path) => ({ path, origin: 'scan', from: 'full scan', recognised: true })); +} + async function computeDiscovered( config: Config | null, known: readonly string[], operation?: StimContext, + now: number = Date.now(), ): Promise { const env = operation?.env ?? {}; const scoped = isScopedHome(operation?.env); @@ -315,13 +337,7 @@ async function computeDiscovered( else outside.push({ path, from: main }); } } - const scanned: DiscoveryChild[] = scoped - ? [] - : (readDiscoveryScan()?.found ?? []) - .filter((path) => isAbsolute(path) && existsSync(path)) - .map(canonicalPath) - .filter((path) => !isBroadFolder(path)) - .map((path) => ({ path, origin: 'scan', from: 'full scan', recognised: true })); + const scanned: DiscoveryChild[] = scoped ? [] : revalidatedScan(); const allowed = [...children, ...scanned].filter( (child) => canonicalRoots.some((root) => inside(child.path, root)) || @@ -329,7 +345,7 @@ async function computeDiscovered( ); const knownRoots = known.map(canonicalPath); const everything = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); - const selected = boundReduceOnly(everything, Date.now()); + const selected = windowReduceOnly(everything, now); const listed = new Set(everything.map(({ path }) => path)); const coveredByKnown = (path: string) => knownRoots.some((root) => inside(path, root) || inside(root, path)); return { diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts new file mode 100644 index 0000000000..e6eabd8e65 --- /dev/null +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -0,0 +1,135 @@ +import { lstatSync, realpathSync, statSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { relative, sep } from 'node:path'; +import { configDir } from '@stim-cli/core'; +import { buildWorkerRoot } from '@stim-cli/core/state'; +import { canonicalPath } from '../commands/gc/paths.ts'; +import { loadConfig } from './config.ts'; + +export const MAX_DEPTH = 7; + +const within = (path: string, folder: string): boolean => path === folder || path.startsWith(`${folder}${sep}`); + +/** Folders directly in the home folder the scan never reads: system, media and the two the maintainer excluded. */ +const HOME_SKIPPED = new Set([ + 'Library', + 'Applications', + 'Desktop', + 'Downloads', + 'Movies', + 'Music', + 'Pictures', + 'Public', +]); + +// @ref LLP 0004#tool-folders-and-the-full-scan -- the home dot-folders the scan does not enter +/** + * Dot-folders directly in the home folder that hold many files and no projects: tool homes, package and build caches, + * runtimes, editor state and other tools' own state. Counted on one machine (entries to depth four): .gradle 67k, + * .agent-device 45k, .member-app-metro-cache 25k, .dartwords-metro-cache 17k, .npm 16k, .bun 16k, .rvm 15k, .yarn 15k, + * .android 7k, .vscode 6k, .cursor 5k, .claude 3k, .cache 2k, .ivy2 2k, .cargo 2k. `.codex` is reached through its + * worktrees folder. Every other home dot-folder is entered, such as `.exact`, except those starting with `.stim`: the + * Stim home and every other Stim state folder (`.stim-agent`, `.stim-release`, `.stim-npm`) hold no checkouts the user owns, + * and the build worker's client checkouts must never become reduce-only. + */ +const HOME_DOT_PRUNED = new Set([ + '.git', + '.Trash', + '.codex', + '.claude', + '.gradle', + '.agent-device', + '.npm', + '.pnpm-store', + '.yarn', + '.bun', + '.cargo', + '.rustup', + '.nvm', + '.rvm', + '.cache', + '.android', + '.cocoapods', + '.vscode', + '.cursor', + '.antigravity', + '.local', + '.config', + '.docker', + '.m2', + '.ivy2', + '.gem', + '.expo', + '.zplug', + '.virtualenvs', + '.wrangler', + '.foundry', + '.pw-profiles', + '.swiftpm', +]); + +/** Folders at any depth that hold many files and no projects: dependencies, build output and tool caches. */ +const PRUNED = new Set([ + 'node_modules', + 'Pods', + 'Carthage', + 'build', + 'dist', + 'out', + 'target', + 'vendor', + 'DerivedData', + 'venv', + '__pycache__', + 'site-packages', + 'coverage', + 'Library', +]); + +// @ref LLP 0004#tool-folders-and-the-full-scan -- what the scan prunes +/** + * Whether the scan descends into a folder named `name`, `depth` levels below the home folder (1 is a child of it). + * Dot-folders are entered only directly in the home folder and not when they are in `HOME_DOT_PRUNED` or named like a + * cache; deeper ones, such as `.worktrees`, are skipped because git reaches their worktrees. + */ +export function scanDescends(name: string, depth: number): boolean { + if (name.startsWith('.')) + return depth === 1 && !name.startsWith('.stim') && !HOME_DOT_PRUNED.has(name) && !name.endsWith('-cache'); + if (PRUNED.has(name)) return false; + return depth !== 1 || !HOME_SKIPPED.has(name); +} + +/** + * The paths the scan never enters, by rule: the Stim home, the build worker root, and the system temporary directory + * unless the home folder itself is under it. Computed once, with no per-folder realpath: the walk never follows a + * symlink, so a path below the canonical home is already canonical. + */ +export function scanSkipper(home: string): (path: string) => boolean { + const stim = [canonicalPath(configDir()), canonicalPath(buildWorkerRoot(loadConfig()))]; + const temps = [tmpdir(), '/tmp', '/var/tmp', '/var/folders'].map(canonicalPath); + const homeIsTemporary = temps.some((temp) => within(home, temp)); + return (path) => + stim.some((folder) => within(path, folder)) || (!homeIsTemporary && temps.some((temp) => within(path, temp))); +} + +/** + * Whether a checkout a scan recorded may still be used: it is a real directory (not a symlink), its canonical path is + * the recorded one, it lies below the home folder on the home volume, no folder between the two is one the scan skips + * (`scanDescends`, `skip`), and `isCheckout` still recognises it. Anything else, such as a checkout moved away with a + * symlink left behind, is dropped. + */ +export function scanEntryValid( + path: string, + { home, isCheckout, skip }: { home: string; isCheckout: (path: string) => boolean; skip: (path: string) => boolean }, +): boolean { + try { + if (!within(path, home) || path === home || lstatSync(path).isSymbolicLink() || !statSync(path).isDirectory()) + return false; + if (realpathSync(path) !== path || statSync(path).dev !== statSync(home).dev || skip(path)) return false; + const names = relative(home, path).split(sep); + if (names.length > MAX_DEPTH) return false; + return names.every((name, i) => scanDescends(name, i + 1)) && isCheckout(path); + } catch { + return false; + } +} diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index eda1d40dbe..5f462cb982 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan weekly and, when the disk level rises above normal, at most once an hour; no command runs it. The scan reads `~/Documents`, so macOS may ask for permission once. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, and the tool homes and caches among your home folder's dot-folders (`.npm`, `.gradle`, `.cache` and similar), and it stops at a recognised project. It does enter other dot-folders of your home folder, such as `~/.exact`, and never the Stim home. A folder it cannot read is skipped. Discovery uses the last scan's saved result, so `stim gc` never walks your disk itself. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It does enter other dot-folders of your home folder, such as `~/.exact`. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From e2e8a86fabab6c9dca8eaa96ec6dd682d221cd5e Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 00:50:18 -0500 Subject: [PATCH 06/25] fix: scan finds under home dot-folders are recorded and never acted on, keep checkouts that PATH launchers run from, and keep a scan failure from masking a pass failure --- llp/0004-discovered-checkouts.decision.md | 33 ++++++-- packages/core/state/discovery-scan.ts | 7 ++ .../__tests__/discovered-reduce-only.test.ts | 39 +++++++-- .../src/__tests__/discovered-stages.test.ts | 1 - .../__tests__/discovered-worktrees.test.ts | 2 +- .../src/__tests__/discovery-scan.test.ts | 55 +++++++++---- .../stim-cli/src/__tests__/discovery.test.ts | 2 +- .../gc-workspaces.integration.test.ts | 1 - .../src/__tests__/maintenance-run.test.ts | 37 ++++++++- .../stim-cli/src/__tests__/scan-rules.test.ts | 57 +++++++++++++ .../stim-cli/src/commands/gc/worktrees.ts | 4 +- packages/stim-cli/src/daemon/stages.ts | 4 +- packages/stim-cli/src/guide/cleanup.ts | 7 +- .../stim-cli/src/stimd/passes/level-run.ts | 2 +- packages/stim-cli/src/stimd/passes/run.ts | 15 +++- .../src/workspace/discovered-activity.ts | 4 +- .../src/workspace/discovered-reads.ts | 7 +- .../stim-cli/src/workspace/discovery-scan.ts | 79 ++++++++++++------- packages/stim-cli/src/workspace/discovery.ts | 6 +- .../src/workspace/protected-locations.ts | 22 +----- .../stim-cli/src/workspace/running-from.ts | 44 ++++++++++- packages/stim-cli/src/workspace/scan-rules.ts | 15 +++- website/docs/worktrees.md | 2 +- 23 files changed, 350 insertions(+), 95 deletions(-) create mode 100644 packages/stim-cli/src/__tests__/scan-rules.test.ts diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index d35fb83599..0589cc57ea 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -111,11 +111,34 @@ The list comes from entry counts to depth four on one machine, led by `.gradle` (67 thousand), `.agent-device` (45 thousand), the two Metro cache folders (25 and 17 thousand), `.npm`, `.bun`, `.rvm` and `.yarn` (15 thousand each). `.codex` is reached through its worktrees folder, and the Stim home is never -scanned. Dot-folders deeper down (`.worktrees`, `.claude/worktrees`) are -skipped because git reaches their worktrees. The scan's finds under a home -dot-folder pass the protection rule for the system temporary directory only: -the scan itself decides which dot-folders to enter, so the dot-folder part of -`isProtectedLocation` does not apply to them, and they stay reduce-only. +scanned. The version managers and package caches `.volta`, `.asdf`, `.mise`, +`.fnm`, `.pyenv`, `.rbenv`, `.sdkman`, `.deno`, `.pub-cache` and `.fvm` are +pruned too: their installs hold thousands of files and no checkouts, and one +large one could truncate the walk on every run. Dot-folders deeper down +(`.worktrees`, `.claude/worktrees`) are skipped because git reaches their +worktrees. + +The coordinator's decision following the review of #5764, not the maintainer's +words: what the scan finds under a home dot-folder is recorded in the result +(`dotFolderFound`) and never selected by discovery, so no pass acts on it. The +protection rule for home dot-folders applies to scan finds in full; the only +ways into a home dot-folder stay `maintenance.discoverRoots` and the tool folder +list. Reason: Stim cannot tell an installed tool from a checkout there, for +example a Swift CLI cloned under `~/.mytool` whose launcher on PATH points into +its ignored `.build/release`; clearing that output once idle would break the +installed command. The record lets doctor report what the folder holds and the +command that adds it to `maintenance.discoverRoots`; that listing is not part of +this change. + +A scan find outside a home dot-folder can also be an installed tool. Discovery +keeps a discovered checkout, and so its outputs, while an executable reachable +through a PATH directory (and `~/.local/bin`, `~/bin`, `/usr/local/bin`, +`/opt/homebrew/bin`) is a symlink that resolves into it. A wrapper script that +only names a path is not followed. + +A scan-found source checkout follows the existing discovered rule and loses +`node_modules` and Pods after `idle.sourceDormantDays` (7); the maintainer has +not confirmed that consequence. Reason: on one MacBook `~/.codex/worktrees` held 53 worktrees and 12 GB that Stim listed once as outside the folders it manages, with no size and no way diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts index 1c84961e94..4220039d79 100644 --- a/packages/core/state/discovery-scan.ts +++ b/packages/core/state/discovery-scan.ts @@ -22,6 +22,11 @@ export interface DiscoveryScan { * not reach, and a scan keeps those under a folder it could not read; discovery revalidates each before using it. */ found: string[]; + /** + * Canonical paths of recognised checkouts under a dot-folder of the home folder, recorded for the doctor listing and + * never used by discovery: Stim cannot tell an installed tool from a checkout there. + */ + dotFolderFound: string[]; /** Folders the system refused or that did not answer, skipped without error. */ unreadable: string[]; /** The scan stopped at its time or folder limit before it finished. */ @@ -44,6 +49,7 @@ export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { (value.trigger !== 'weekly' && value.trigger !== 'low-disk') || !finite(value.visited) || !strings(value.found) || + !strings(value.dotFolderFound) || !strings(value.unreadable) || typeof value.truncated !== 'boolean' ) @@ -57,6 +63,7 @@ export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { trigger: value.trigger, visited: value.visited, found: value.found, + dotFolderFound: value.dotFolderFound, unreadable: value.unreadable, truncated: value.truncated, }; diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 02cb26a347..df03702966 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -28,7 +28,6 @@ vi.mock('../workspace/protected-locations.ts', async (original) => { return { ...real, isProtectedLocation: (path: string) => real.isProtectedLocation(path, { home: homedir(), temps: [] }), - isTemporaryLocation: () => false, }; }); @@ -99,7 +98,7 @@ describe('checkouts found through a tool folder or the full scan', () => { ]); }); - const writeScan = (found: string[]) => { + const writeScan = (found: string[], dotFolderFound: string[] = []) => { mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); const scan = { version: 2, @@ -110,22 +109,50 @@ describe('checkouts found through a tool folder or the full scan', () => { trigger: 'weekly', visited: 1, found, + dotFolderFound, unreadable: [], truncated: false, }; writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); }; - it('discovers a checkout under a home dot-folder only when the scan result names it, as reduce-only', async () => { + it('never selects a checkout under a home dot-folder from the scan result, only through maintenance.discoverRoots', async () => { const exact = join(base, 'home', '.exact', 'exact2'); mkdirSync(exact, { recursive: true }); writeFileSync(join(exact, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); const config = { projects: { [main]: {} }, repos: {} }; + writeScan([], [exact]); expect(await discoverCheckouts(config, [main])).toEqual([]); - writeScan([exact]); + writeScan([exact], [exact]); + expect(await discoverCheckouts(config, [main])).toEqual([]); + const rooted = { ...config, maintenance: { discoverRoots: [join(base, 'home', '.exact')] } }; expect( - (await discoverCheckouts(config, [main])).map(({ path, origin, reduceOnly }) => [path, origin, reduceOnly]), - ).toEqual([[exact, 'scan', true]]); + (await discoverCheckouts(rooted, [main])).map(({ path, origin, reduceOnly }) => [path, origin, reduceOnly]), + ).toEqual([[exact, 'root', undefined]]); + }); + + it('keeps the output of a scan-found checkout that an executable on PATH runs from', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(join(tool, 'android', 'app', 'build', 'bin'), { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(tool, 'android', 'app', 'build', 'bin', 'mytool'), '#!/bin/sh\n'); + execFileSync('git', ['-C', tool, 'init', '-q']); + writeFileSync(join(tool, '.gitignore'), 'build/\n'); + execFileSync('git', ['-C', tool, 'add', '-A']); + execFileSync('git', ['-C', tool, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x'], { + env: { ...process.env, GIT_AUTHOR_DATE: old.toISOString(), GIT_COMMITTER_DATE: old.toISOString() }, + }); + age(tool); + const gitdir = join(tool, '.git'); + for (const file of ['index', 'HEAD', join('logs', 'HEAD')]) utimesSync(join(gitdir, file), old, old); + writeScan([tool]); + const config = { projects: { [main]: {} }, repos: {} }; + const reading = { devices: [], simsFailed: false }; + expect((await stoppedOutputCandidates(config, reading, day)).map((entry) => entry.path)).toEqual([tool]); + const bin = join(base, 'home', 'bin'); + mkdirSync(bin); + symlinkSync(join(tool, 'android', 'app', 'build', 'bin', 'mytool'), join(bin, 'mytool')); + expect(await stoppedOutputCandidates(config, reading, day)).toEqual([]); }); it('drops a cached checkout that was replaced by a symlink to a folder the scan never enters, and plans nothing under it', async () => { diff --git a/packages/stim-cli/src/__tests__/discovered-stages.test.ts b/packages/stim-cli/src/__tests__/discovered-stages.test.ts index eb1b00fe82..45b75ba935 100644 --- a/packages/stim-cli/src/__tests__/discovered-stages.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-stages.test.ts @@ -23,7 +23,6 @@ const fail = (): never => { vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false, - isTemporaryLocation: () => false, })); describe('a discovered checkout in the idle stages', () => { diff --git a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts index 9a580c5930..72583d4101 100644 --- a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts @@ -19,7 +19,6 @@ import { worktreeInScope } from '../workspace/discovery.ts'; vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false, - isTemporaryLocation: () => false, })); const day = 86_400_000; @@ -185,6 +184,7 @@ describe('a discovered worktree on the removal path', () => { it('keeps a worktree a process runs from or an agent session works in', async () => { worktree('held'); vi.spyOn(reads, 'discoveredReads').mockImplementation(() => ({ + launchers: async () => [], processes: async () => ({ commands: [{ pid: 1, line: `node ${join(base, 'roots', 'held')}/server.js` }], cwds: [], diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index 81915b1c92..9636764d8e 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -9,6 +9,7 @@ import { symlinkSync, writeFileSync, } from 'node:fs'; +import { lstat } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { testContext } from './_context.ts'; @@ -30,21 +31,12 @@ describe('scanDescends', () => { expect(scanDescends('dev', 1)).toBe(true); }); - it('enters a dot-folder of the home folder unless it is a tool home or cache, and no deeper dot-folder', () => { + it('enters a dot-folder directly in the home folder, but not a Stim one, a cache, or a deeper one', () => { expect(scanDescends('.exact', 1)).toBe(true); - for (const name of [ - '.git', - '.npm', - '.gradle', - '.cache', - '.stim', - '.stim-agent', - '.stim-release', - '.codex', - '.member-app-metro-cache', - ]) - expect(scanDescends(name, 1)).toBe(false); - for (const name of ['.worktrees', '.exact', '.claude']) expect(scanDescends(name, 2)).toBe(false); + expect(scanDescends('.stim-agent', 1)).toBe(false); + expect(scanDescends('.npm', 1)).toBe(false); + expect(scanDescends('.some-tool-cache', 1)).toBe(false); + expect(scanDescends('.exact', 2)).toBe(false); }); }); @@ -105,6 +97,26 @@ describe('scanForCheckouts', () => { } }); + it('stops reading siblings at once when its budget is spent, as repeated timed-out stats spend it', async () => { + for (let i = 0; i < 100; i++) project(`p${String(i).padStart(3, '0')}`); + let stats = 0; + const hung = async (path: string) => { + if (path === home) return lstat(path); + stats++; + return new Promise(() => {}); + }; + const pending = scanForCheckouts(home, marked, { budgetMs: 40, readTimeoutMs: 10, stat: hung }); + const state = { done: false }; + void pending.then(() => (state.done = true)); + while (!state.done) { + if (stats > 0) await vi.advanceTimersByTimeAsync(10); + await new Promise((resolve) => setImmediate(resolve)); + } + const scan = await pending; + expect(scan.truncated).toBe(true); + expect(stats).toBeLessThan(10); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); @@ -188,7 +200,7 @@ describe('runDiscoveryScanIfDue', () => { }); const hour = 3_600_000; const t0 = Date.UTC(2026, 10, 14); - const options = (now: number) => ({ home: join(base, 'home'), now }); + const options = (now: number) => ({ home: join(base, 'home'), now, attempted: { at: 0 } }); beforeEach(() => { base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-run-'))); @@ -216,6 +228,19 @@ describe('runDiscoveryScanIfDue', () => { expect(existsSync(join(base, 'state'))).toBe(false); }); + it('records finds under a home dot-folder apart from the others', async () => { + mkdirSync(join(base, 'home', '.exact', 'exact2'), { recursive: true }); + writeFileSync( + join(base, 'home', '.exact', 'exact2', 'package.json'), + JSON.stringify({ dependencies: { expo: '*' } }), + ); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ + found: [join(base, 'home', 'dev', 'app')], + dotFolderFound: [join(base, 'home', '.exact', 'exact2')], + }); + }); + it('stores the result and its schedule under the Stim home, and not again until due', async () => { await runDiscoveryScanIfDue(tick(), options(t0)); expect(readDiscoveryScan()).toMatchObject({ diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index 72ce4b1316..4aac6eeae0 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -24,7 +24,6 @@ import { vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false, - isTemporaryLocation: () => false, })); describe('isBroadFolder', () => { @@ -170,6 +169,7 @@ describe('discoverCheckouts', () => { trigger: 'weekly', visited: 1, found: [docs, join(base, 'gone')], + dotFolderFound: [], unreadable: [], truncated: false, }; diff --git a/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts b/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts index 60db0b8e01..2012804281 100644 --- a/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts +++ b/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts @@ -425,7 +425,6 @@ describe('workspace directories without workspace.json', () => { vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false, - isTemporaryLocation: () => false, })); test('listWorkspaceDirs resolves each directory through its workspace.json', () => { diff --git a/packages/stim-cli/src/__tests__/maintenance-run.test.ts b/packages/stim-cli/src/__tests__/maintenance-run.test.ts index 1a4e513d54..2e013ebe05 100644 --- a/packages/stim-cli/src/__tests__/maintenance-run.test.ts +++ b/packages/stim-cli/src/__tests__/maintenance-run.test.ts @@ -12,7 +12,7 @@ import { workspaceLogsDir, sharedCompilationCache, } from '@stim-cli/core/state'; -import { runMaintenance } from '../stimd/passes/run.ts'; +import { runMaintenance, type MaintenanceTick } from '../stimd/passes/run.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; import * as measurements from '../stimd/passes/measure.ts'; import * as preview from '../stimd/passes/preview.ts'; @@ -25,6 +25,7 @@ import { runCli } from '../cli/define-op.ts'; import { warnCriticalDisk } from '../stimd/passes/trigger.ts'; import { resetExecutor, setExecutor } from '../exec.ts'; import { makeChildProcess } from './_factories.ts'; +import { testContext } from './_context.ts'; const disk = vi.hoisted(() => ({ full: false, reserve: '', code: 'ENOSPC', partialReserveWrite: false })); vi.mock('@stim-cli/core/ownership-claim', async (importOriginal) => { @@ -502,3 +503,37 @@ test('an invalid cache cap falls back only that cap while report mode remains en delete process.env.STIM_CACHES_BUILD_CACHE_MAX_GB; } }); + +test('an unverified child cleanup in a pass surfaces as that failure, not as the abort of a due discovery scan', async () => { + const failure = Object.assign(new Error('unverified cleanup'), { + code: 'STIM_CLAIM_UNAVAILABLE', + reportCleanup: true, + }); + const controller = new AbortController(); + vi.spyOn(measurements, 'pressureFromMeasurement').mockImplementation(() => { + controller.abort(); + throw failure; + }); + const scan = vi + .spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue') + .mockImplementation(async () => controller.signal.throwIfAborted()); + const tick: MaintenanceTick = { + operation: testContext({ signal: controller.signal }), + measured: { + measurement: { + disk: [{ volume: '/', freeMb: 500_000 }], + memory: { level: 'normal', availableBytes: null, totalBytes: 0 }, + load1: 0, + cores: 1, + bootedSimulators: null, + runningEmulators: null, + interaction: null, + }, + sims: null, + simsError: null, + devices: [], + }, + }; + await expect(runMaintenance('daemon', { tick })).rejects.toBe(failure); + expect(scan).not.toHaveBeenCalled(); +}); diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts new file mode 100644 index 0000000000..44264a51ed --- /dev/null +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -0,0 +1,57 @@ +import { existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { scanEntryValid } from '../workspace/scan-rules.ts'; + +describe('scanEntryValid with a hand-edited scan result', () => { + let home: string; + const isCheckout = (path: string) => existsSync(join(path, 'package.json')); + const rules = () => ({ home, isCheckout, skip: (path: string) => path.startsWith(join(home, '.stim')) }); + const project = (...parts: string[]) => { + mkdirSync(join(home, ...parts), { recursive: true }); + writeFileSync(join(home, ...parts, 'package.json'), '{}'); + return join(home, ...parts); + }; + + beforeEach(() => { + home = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-rules-'))); + }); + + afterEach(() => rmSync(home, { recursive: true, force: true })); + + it('accepts a real checkout the scan could have found', () => { + expect(scanEntryValid(project('dev', 'app'), rules())).toBe(true); + }); + + it('refuses entries the scan never finds: the filesystem root, another home, the Stim home, pruned and skipped folders', () => { + const other = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-other-'))); + try { + writeFileSync(join(other, 'package.json'), '{}'); + for (const path of [ + '/', + home, + other, + project('.stim', 'x'), + project('.npm', 'x'), + project('dev', 'node_modules', 'x'), + project('Desktop', 'x'), + ]) + expect(scanEntryValid(path, rules())).toBe(false); + } finally { + rmSync(other, { recursive: true, force: true }); + } + }); + + it('refuses a path through a symlinked ancestor, a symlinked checkout and a different-case spelling of a pruned folder', () => { + const real = project('elsewhere', 'app'); + mkdirSync(join(home, 'dev')); + symlinkSync(join(home, 'elsewhere'), join(home, 'dev', 'link')); + symlinkSync(real, join(home, 'dev', 'app-link')); + expect(scanEntryValid(join(home, 'dev', 'link', 'app'), rules())).toBe(false); + expect(scanEntryValid(join(home, 'dev', 'app-link'), rules())).toBe(false); + const cached = project('.claude', 'x'); + expect(scanEntryValid(cached, rules())).toBe(false); + const upper = join(home, '.CLAUDE', 'x'); + expect(scanEntryValid(upper, rules())).toBe(false); + }); +}); diff --git a/packages/stim-cli/src/commands/gc/worktrees.ts b/packages/stim-cli/src/commands/gc/worktrees.ts index e3aa3b81c8..398c8cadba 100644 --- a/packages/stim-cli/src/commands/gc/worktrees.ts +++ b/packages/stim-cli/src/commands/gc/worktrees.ts @@ -601,7 +601,7 @@ async function discoveredHold(candidate: WorktreeCandidate, operation?: StimCont const hold = await discoveredRemovalHold(candidate.path, { main: candidate.source ?? null, idleMs: dormantMs(operation), - reads: discoveredReads(operation?.signal), + reads: discoveredReads(operation?.signal, operation?.env), reduceOnly: candidate.reduceOnly === true, operation, }); @@ -981,7 +981,7 @@ export async function collectWorktreeSweep({ const handed = worktrees.filter((candidate) => candidate.discovered && candidate.skipped === null); if (handed.length) { const { discoveredRemovalHold } = await import('../../workspace/discovered-removal.ts'); - const reads = discoveredReads(operation?.signal); + const reads = discoveredReads(operation?.signal, operation?.env); for (const candidate of handed) { const hold = await discoveredRemovalHold(candidate.path, { main: candidate.source ?? null, diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index eb20c5bd3d..de4918abef 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -145,7 +145,7 @@ async function idleVerdict( { locked = false, operation, - reads = discoveredReads(operation?.signal), + reads = discoveredReads(operation?.signal, operation?.env), full = false, }: { locked?: boolean; operation?: StimContext; reads?: DiscoveredReads; full?: boolean } = {}, ): Promise<{ idleMs: number } | { kept: string }> { @@ -230,7 +230,7 @@ async function idleWorkspaces( { discover = false, full = false, onKept }: Omit & { discover?: boolean } = {}, ): Promise { const readings: (Omit & { idleMs: number | null })[] = []; - const reads = discoveredReads(reading.operation?.signal); + const reads = discoveredReads(reading.operation?.signal, reading.operation?.env); const discovered = discover ? (await discoverCheckouts(config, listedRoots(), reading.operation)) .filter(({ deferred }) => !deferred) diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index a4db5be937..62249bef2c 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1229,7 +1229,12 @@ never walks the disk. A scan that fails or stops early keeps the previous finds, and a finished one keeps those under a folder it could not read. Discovery checks each recorded checkout again before using it: still a real folder (not a symlink), on the home volume, below no skipped folder and still a project; -anything else is dropped. Reduce-only checkouts have their build outputs and, in +anything else is dropped. Checkouts the scan finds under a dot-folder of the +home folder are recorded in the result and never acted on, because an installed +tool cannot be told from a checkout there; maintenance.discoverRoots is the way +to manage one. A discovered checkout that an executable on PATH (a symlink in a +PATH directory, ~/.local/bin, ~/bin, /usr/local/bin or /opt/homebrew/bin) +resolves into is kept whole. Reduce-only checkouts have their build outputs and, in a linked worktree, node_modules and Pods reduced by the rules below. Passes evaluate at most 40 of them at a time, in a window that moves every 15 minutes; stim gc --dry-run lists all of them and says which are left for a later pass. A diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index a50d7458ed..cfee84d953 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -205,7 +205,7 @@ export async function runMaintenanceAtLevel({ for (const action of ladder?.levels.flatMap((entry) => entry.actions) ?? []) take(fromLadder(action)); } const idleMs = level === 'normal' ? dormantAfterMs(loadConfig(), env) : PRESSURE_IDLE_MS; - const reads = discoveredReads(operation?.signal); + const reads = discoveredReads(operation?.signal, operation?.env); const discovered: LevelRun['discovered'] = []; const ownerOf = (path: string) => found diff --git a/packages/stim-cli/src/stimd/passes/run.ts b/packages/stim-cli/src/stimd/passes/run.ts index 1ca151f931..4435734373 100644 --- a/packages/stim-cli/src/stimd/passes/run.ts +++ b/packages/stim-cli/src/stimd/passes/run.ts @@ -403,6 +403,7 @@ export async function runMaintenance( return; } let scanAfter = false; + let passFailed = false; let logger: ReturnType | undefined; let gcLog: GcLog | undefined; try { @@ -726,6 +727,7 @@ export async function runMaintenance( } writeState(state); } catch (error) { + passFailed = true; throwUnverifiedCleanup(error); const message = error instanceof Error ? error.message : String(error); try { @@ -739,10 +741,21 @@ export async function runMaintenance( logger?.close(); keepReserve(); releaseClaim(claim); - if (scanAfter) await runDiscoveryScanIfDue(tick); + if (scanAfter) await scanAfterPass(tick, passFailed); } } +/** + * The discovery scan, after the pass has released its claim. Not after a failed pass or a cancellation, and nothing it + * throws can reach the caller: the pass's own error, such as an unverified child cleanup, is the one that must surface. + */ +async function scanAfterPass(tick: MaintenanceTick | undefined, passFailed: boolean): Promise { + if (passFailed || tick?.operation.signal.aborted) return; + try { + await runDiscoveryScanIfDue(tick); + } catch {} +} + function crashLine(error: unknown): void { try { mkdirSync(maintenanceDir(), { recursive: true }); diff --git a/packages/stim-cli/src/workspace/discovered-activity.ts b/packages/stim-cli/src/workspace/discovered-activity.ts index 42c984cee3..8073fe4a7d 100644 --- a/packages/stim-cli/src/workspace/discovered-activity.ts +++ b/packages/stim-cli/src/workspace/discovered-activity.ts @@ -5,7 +5,7 @@ import { worktreeActivityOfAsync } from '../commands/gc/worktrees.ts'; import type { IdleStopEvidence } from '../policy/idle.ts'; import { gitQuietAsync } from './git.ts'; import type { DiscoveredReads } from './discovered-reads.ts'; -import { runningFromReason } from './running-from.ts'; +import { launcherReason, runningFromReason } from './running-from.ts'; /** Folders holding dependencies or other checkouts, skipped wherever they are. */ const ALWAYS_SKIPPED = new Set(['.git', '.worktrees', 'node_modules', 'Pods']); @@ -138,5 +138,7 @@ export async function discoveredEvidence( keep.push(...(await reads.sessions(path, now))); const running = runningFromReason(path, await reads.processes()); if (running) keep.push(running); + const launcher = launcherReason(path, await reads.launchers()); + if (launcher) keep.push(launcher); return { activity, keep }; } diff --git a/packages/stim-cli/src/workspace/discovered-reads.ts b/packages/stim-cli/src/workspace/discovered-reads.ts index d8d54fef3d..b52f4a4c79 100644 --- a/packages/stim-cli/src/workspace/discovered-reads.ts +++ b/packages/stim-cli/src/workspace/discovered-reads.ts @@ -1,19 +1,22 @@ import type { AgentSession } from '@stim-cli/core/state'; import { agentKey, attributeAgentSessions, discoverAgentSessions } from '../agent-sessions.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; -import { processReader, type RunningProcesses } from './running-from.ts'; +import { launcherReader, processReader, type RunningProcesses } from './running-from.ts'; /** What one pass reads once for every checkout: the process table and the agent sessions. */ export interface DiscoveredReads { processes: () => Promise; + /** Where the executables on PATH that are symlinks resolve to. */ + launchers: () => Promise; /** Why an agent session working in the path keeps it. */ sessions: (path: string, now: number) => Promise; } -export function discoveredReads(signal?: AbortSignal): DiscoveredReads { +export function discoveredReads(signal?: AbortSignal, env?: NodeJS.ProcessEnv): DiscoveredReads { let found: Promise | undefined; return { processes: processReader(signal), + launchers: launcherReader(signal, env), sessions: async (path, now) => { found ??= discoverAgentSessions({ now }); const sessions = await found; diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index a550592b04..7b56dd4777 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -1,7 +1,7 @@ import { lstat, readdir } from 'node:fs/promises'; import { mkdirSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; import { homedir } from 'node:os'; -import { join, sep } from 'node:path'; +import { join, relative, sep } from 'node:path'; import { isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; import { discoveryScanFile, @@ -32,13 +32,13 @@ interface ScanOutcome { const errorCode = (error: unknown) => (error instanceof Error && 'code' in error ? String(error.code) : ''); /** `read`'s result, or an ETIMEDOUT rejection when it has not answered in time, as a hung mount or a permission prompt would not. */ -async function answered(read: Promise): Promise { +async function answered(read: Promise, timeoutMs: number): Promise { let timer: NodeJS.Timeout | undefined; try { return await Promise.race([ read, new Promise((_, fail) => { - timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), READ_TIMEOUT_MS); + timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), timeoutMs); }), ]); } finally { @@ -61,22 +61,30 @@ export async function scanForCheckouts( budgetMs = BUDGET_MS, signal, skip = () => false, - }: { budgetMs?: number; signal?: AbortSignal; skip?: (path: string) => boolean } = {}, + readTimeoutMs = READ_TIMEOUT_MS, + stat: statOf = (path) => lstat(path), + }: { + budgetMs?: number; + signal?: AbortSignal; + skip?: (path: string) => boolean; + readTimeoutMs?: number; + stat?: (path: string) => Promise<{ dev: number }>; + } = {}, ): Promise { const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false }; const deadline = Date.now() + budgetMs; - const volume = (await lstat(home)).dev; - const walk = async (folder: string, depth: number): Promise => { + const volume = (await statOf(home)).dev; + const exhausted = (): boolean => { signal?.throwIfAborted(); - if (outcome.truncated) return; - if (Date.now() > deadline || outcome.visited >= MAX_FOLDERS) { - outcome.truncated = true; - return; - } + if (!outcome.truncated && (Date.now() > deadline || outcome.visited >= MAX_FOLDERS)) outcome.truncated = true; + return outcome.truncated; + }; + const walk = async (folder: string, depth: number): Promise => { + if (exhausted()) return; outcome.visited++; let entries; try { - entries = await answered(readdir(folder, { withFileTypes: true })); + entries = await answered(readdir(folder, { withFileTypes: true }), readTimeoutMs); } catch (error) { signal?.throwIfAborted(); if (unanswered(error)) outcome.unreadable.push(folder); @@ -88,18 +96,19 @@ export async function scanForCheckouts( } if (depth >= MAX_DEPTH) return; for (const entry of entries) { + if (exhausted()) return; if (!entry.isDirectory() || entry.isSymbolicLink() || !scanDescends(entry.name, depth + 1)) continue; const path = join(folder, entry.name); if (skip(path)) continue; - let stat; + let info; try { - stat = await answered(lstat(path)); + info = await answered(statOf(path), readTimeoutMs); } catch (error) { signal?.throwIfAborted(); if (unanswered(error)) outcome.unreadable.push(path); continue; } - if (stat.dev === volume) await walk(path, depth + 1); + if (info.dev === volume) await walk(path, depth + 1); } }; await walk(home, 0); @@ -133,7 +142,7 @@ export function discoveryScanDue( } /** When this process last tried a scan, so a result that cannot be stored does not start a walk on every tick. */ -let attemptedHere = 0; +const attempts = { at: 0 }; function writeScan(scan: DiscoveryScan): void { mkdirSync(maintenanceDir(), { recursive: true }); @@ -159,15 +168,25 @@ function writeScan(scan: DiscoveryScan): void { */ export async function runDiscoveryScanIfDue( tick: { operation: StimContext; measured: { measurement: Pick } } | undefined, - { home = homedir(), now = Date.now() }: { home?: string; now?: number } = {}, + { + home = homedir(), + now = Date.now(), + attempted = attempts, + }: { home?: string; now?: number; attempted?: { at: number } } = {}, ): Promise { if (!tick || isScopedHome(tick.operation.env)) return; const { operation } = tick; - const thresholds = diskThresholds(loadConfig(), operation.env).thresholds; - const low = diskPressure(tick.measured.measurement.disk, thresholds).level !== 'normal'; - const last = readDiscoveryScan(); - if (!discoveryScanDue(last, now, low) || (now >= attemptedHere && now - attemptedHere < HOUR_MS)) return; - attemptedHere = now; + let last: DiscoveryScan | null; + let low: boolean; + try { + const thresholds = diskThresholds(loadConfig(), operation.env).thresholds; + low = diskPressure(tick.measured.measurement.disk, thresholds).level !== 'normal'; + last = readDiscoveryScan(); + } catch { + return; + } + if (!discoveryScanDue(last, now, low) || (now >= attempted.at && now - attempted.at < HOUR_MS)) return; + attempted.at = now; const attempt = { version: 2, ranAt: now, @@ -175,8 +194,9 @@ export async function runDiscoveryScanIfDue( trigger: low ? 'low-disk' : 'weekly', } as const; const failed = { ...attempt, completedAt: last?.completedAt ?? 0, durationMs: 0, visited: 0, truncated: true }; + const realHome = canonicalPath(home); + const dotFolder = (path: string) => relative(realHome, path).startsWith('.'); try { - const realHome = canonicalPath(home); const outcome = await scanForCheckouts(realHome, recognises, { signal: operation.signal, skip: scanSkipper(realHome), @@ -186,16 +206,21 @@ export async function runDiscoveryScanIfDue( completedAt: now, durationMs: Date.now() - now, ...outcome, - found: mergeFound(last?.found ?? [], outcome), - unreadable: outcome.unreadable, + found: mergeFound(last?.found ?? [], { ...outcome, found: outcome.found.filter((p) => !dotFolder(p)) }), + dotFolderFound: mergeFound(last?.dotFolderFound ?? [], { ...outcome, found: outcome.found.filter(dotFolder) }), }); } catch { if (operation.signal.aborted) { - attemptedHere = 0; + attempted.at = 0; operation.signal.throwIfAborted(); } try { - writeScan({ ...failed, found: last?.found ?? [], unreadable: last?.unreadable ?? [] }); + writeScan({ + ...failed, + found: last?.found ?? [], + dotFolderFound: last?.dotFolderFound ?? [], + unreadable: last?.unreadable ?? [], + }); } catch {} } } diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index 50b0a3d6f0..9e79d2be1e 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -6,7 +6,7 @@ import { isScopedHome, type StimContext } from '@stim-cli/core'; import { machineSettingValue, readDiscoveryScan, type Config } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; import { projectRegistry } from '../integrations/projects.ts'; -import { isProtectedLocation, isTemporaryLocation } from './protected-locations.ts'; +import { isProtectedLocation } from './protected-locations.ts'; import { scanEntryValid, scanSkipper } from './scan-rules.ts'; import { matchWorktreeEntry, resolveSourceCheckoutAsync } from './worktree.ts'; @@ -339,9 +339,7 @@ async function computeDiscovered( } const scanned: DiscoveryChild[] = scoped ? [] : revalidatedScan(); const allowed = [...children, ...scanned].filter( - (child) => - canonicalRoots.some((root) => inside(child.path, root)) || - (child.origin === 'scan' ? !isTemporaryLocation(child.path) : !isProtectedLocation(child.path)), + (child) => canonicalRoots.some((root) => inside(child.path, root)) || !isProtectedLocation(child.path), ); const knownRoots = known.map(canonicalPath); const everything = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); diff --git a/packages/stim-cli/src/workspace/protected-locations.ts b/packages/stim-cli/src/workspace/protected-locations.ts index 41c4ec8743..76d2ae185f 100644 --- a/packages/stim-cli/src/workspace/protected-locations.ts +++ b/packages/stim-cli/src/workspace/protected-locations.ts @@ -28,26 +28,10 @@ function real(path: string): string | null { * resolve counts as protected. Discovery lists the worktrees in `~/.codex/worktrees` anyway, as reduce-only. */ export function isProtectedLocation(path: string, roots: Roots = defaultRoots()): boolean { - return isTemporaryLocation(path, roots) || isHomeDotFolder(path, roots); -} - -/** Whether a path lies under the system temporary directory; a path that does not resolve counts as temporary. */ -export function isTemporaryLocation(path: string, roots: Roots = defaultRoots()): boolean { - const target = real(path); - if (target === null) return true; - return roots.temps.some((folder) => { - const temp = real(folder) ?? folder; - return target === temp || target.startsWith(`${temp}${sep}`); - }); -} - -/** Whether a path lies under a dot-folder directly inside the home directory; a path that does not resolve counts. */ -function isHomeDotFolder(path: string, roots: Roots = defaultRoots()): boolean { const target = real(path); if (target === null) return true; + const inside = (folder: string) => target === folder || target.startsWith(`${folder}${sep}`); + if (roots.temps.some((folder) => inside(real(folder) ?? folder))) return true; const home = real(roots.home) ?? roots.home; - return ( - (target === home || target.startsWith(`${home}${sep}`)) && - relative(home, target).split(sep)[0]?.startsWith('.') === true - ); + return inside(home) && relative(home, target).split(sep)[0]?.startsWith('.') === true; } diff --git a/packages/stim-cli/src/workspace/running-from.ts b/packages/stim-cli/src/workspace/running-from.ts index 7cc9bb76f1..2d7e9697e1 100644 --- a/packages/stim-cli/src/workspace/running-from.ts +++ b/packages/stim-cli/src/workspace/running-from.ts @@ -1,5 +1,7 @@ import { readdirSync, readlinkSync, realpathSync } from 'node:fs'; -import { join, sep } from 'node:path'; +import { lstat, readdir, realpath } from 'node:fs/promises'; +import { homedir } from 'node:os'; +import { delimiter, join, sep } from 'node:path'; import { canonicalPath } from '../commands/gc/paths.ts'; import { getExecutor } from '../exec.ts'; import { serviceRunningFrom } from './service-guard.ts'; @@ -131,6 +133,46 @@ export function processReader(signal?: AbortSignal): () => Promise (table ??= readRunningProcessesAsync(signal).then((running) => othersRunningAsync(running, signal))); } +/** + * Where the executables a person installs by linking them live: the PATH directories of this process, and the usual + * user bin folders a daemon's minimal PATH leaves out. + */ +function launcherDirectories(env: NodeJS.ProcessEnv): string[] { + const home = homedir(); + const extra = [join(home, '.local', 'bin'), join(home, 'bin'), '/usr/local/bin', '/opt/homebrew/bin']; + return [...new Set([...(env.PATH ?? '').split(delimiter).filter(Boolean), ...extra])]; +} + +/** + * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader: where each + * executable reachable through PATH really lives. A wrapper script that merely names a path is not followed. + */ +export function launcherReader(signal?: AbortSignal, env: NodeJS.ProcessEnv = {}): () => Promise { + let targets: Promise | undefined; + const read = async (): Promise => { + const found: string[] = []; + for (const directory of launcherDirectories(env)) { + signal?.throwIfAborted(); + const names = await readdir(directory).catch(() => []); + for (const name of names) { + const entry = join(directory, name); + if (!(await lstat(entry).catch(() => null))?.isSymbolicLink()) continue; + const target = await realpath(entry).catch(() => null); + if (target) found.push(target); + } + } + return found; + }; + return () => (targets ??= read()); +} + +/** Why an executable on PATH, as `targets` lists them, keeps `folder`: one resolves to a path inside it, or null. */ +export function launcherReason(folder: string, targets: readonly string[]): string | null { + const root = canonicalPath(folder); + const target = targets.find((path) => path.startsWith(`${root}${sep}`)); + return target ? `an executable on PATH runs from it (${target})` : null; +} + function stimOnPath(): string | null { const found = getExecutor().findExecutable('stim'); if (!found) return null; diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index e6eabd8e65..e5d905fcc1 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -28,7 +28,8 @@ const HOME_SKIPPED = new Set([ * runtimes, editor state and other tools' own state. Counted on one machine (entries to depth four): .gradle 67k, * .agent-device 45k, .member-app-metro-cache 25k, .dartwords-metro-cache 17k, .npm 16k, .bun 16k, .rvm 15k, .yarn 15k, * .android 7k, .vscode 6k, .cursor 5k, .claude 3k, .cache 2k, .ivy2 2k, .cargo 2k. `.codex` is reached through its - * worktrees folder. Every other home dot-folder is entered, such as `.exact`, except those starting with `.stim`: the + * worktrees folder. `.volta`, `.asdf`, `.mise`, `.fnm`, `.pyenv`, `.rbenv`, `.sdkman`, `.deno`, `.pub-cache` and `.fvm` are + * version managers and package caches whose installs hold thousands of files and no checkouts of the user's. Every other home dot-folder is entered, such as `.exact`, except those starting with `.stim`: the * Stim home and every other Stim state folder (`.stim-agent`, `.stim-release`, `.stim-npm`) hold no checkouts the user owns, * and the build worker's client checkouts must never become reduce-only. */ @@ -66,6 +67,16 @@ const HOME_DOT_PRUNED = new Set([ '.foundry', '.pw-profiles', '.swiftpm', + '.volta', + '.asdf', + '.mise', + '.fnm', + '.pyenv', + '.rbenv', + '.sdkman', + '.deno', + '.pub-cache', + '.fvm', ]); /** Folders at any depth that hold many files and no projects: dependencies, build output and tool caches. */ @@ -125,7 +136,7 @@ export function scanEntryValid( try { if (!within(path, home) || path === home || lstatSync(path).isSymbolicLink() || !statSync(path).isDirectory()) return false; - if (realpathSync(path) !== path || statSync(path).dev !== statSync(home).dev || skip(path)) return false; + if (realpathSync.native(path) !== path || statSync(path).dev !== statSync(home).dev || skip(path)) return false; const names = relative(home, path).split(sep); if (names.length > MAX_DEPTH) return false; return names.every((name, i) => scanDescends(name, i + 1)) && isCheckout(path); diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 5f462cb982..db94ee4dc0 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It does enter other dot-folders of your home folder, such as `~/.exact`. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout that an executable on your `PATH` links into is kept whole. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From b6de5cf3f0d3bc1ca0efce23b6d7bf5048f24a3a Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 00:58:41 -0500 Subject: [PATCH 07/25] fix: consult PATH launchers only for reduce-only checkouts --- llp/0004-discovered-checkouts.decision.md | 2 +- packages/stim-cli/src/daemon/stages.ts | 28 ++++++++++++++----- packages/stim-cli/src/guide/cleanup.ts | 2 +- .../stim-cli/src/stimd/passes/level-run.ts | 2 +- .../src/workspace/discovered-activity.ts | 5 +++- website/docs/worktrees.md | 2 +- 6 files changed, 29 insertions(+), 12 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 0589cc57ea..c7d7fa0bfb 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -131,7 +131,7 @@ command that adds it to `maintenance.discoverRoots`; that listing is not part of this change. A scan find outside a home dot-folder can also be an installed tool. Discovery -keeps a discovered checkout, and so its outputs, while an executable reachable +keeps a reduce-only checkout (a tool folder or scan find), and so its outputs, while an executable reachable through a PATH directory (and `~/.local/bin`, `~/bin`, `/usr/local/bin`, `/opt/homebrew/bin`) is a symlink that resolves into it. A wrapper script that only names a path is not followed. diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index de4918abef..164fc409c1 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -147,7 +147,14 @@ async function idleVerdict( operation, reads = discoveredReads(operation?.signal, operation?.env), full = false, - }: { locked?: boolean; operation?: StimContext; reads?: DiscoveredReads; full?: boolean } = {}, + reduceOnly = false, + }: { + locked?: boolean; + operation?: StimContext; + reads?: DiscoveredReads; + full?: boolean; + reduceOnly?: boolean; + } = {}, ): Promise<{ idleMs: number } | { kept: string }> { const now = Date.now(); const pressured = setting === PRESSURE_THRESHOLD; @@ -155,7 +162,7 @@ async function idleVerdict( const evidence = !full && (getProject(path) || existsSync(workspaceMetadataFile(path))) ? stim - : mergeEvidence(stim, await discoveredEvidence(path, { now, idleMs: thresholdMs, reads, operation })); + : mergeEvidence(stim, await discoveredEvidence(path, { now, idleMs: thresholdMs, reads, operation, reduceOnly })); const decision = decideIdleStop(evidence, { idleMs: thresholdMs, now, setting, pressured }); if (!decision.stop) return { kept: decision.reason }; try { @@ -201,9 +208,11 @@ export async function discoveredKeepReason( thresholdMs: number, operation?: StimContext, reads?: DiscoveredReads, + reduceOnly = false, ): Promise { const verdict = await idleVerdict(path, thresholdMs, 'idle.dormantDays', { operation, + reduceOnly, ...(reads ? { reads } : {}), }); return 'kept' in verdict ? verdict.kept : null; @@ -231,11 +240,11 @@ async function idleWorkspaces( ): Promise { const readings: (Omit & { idleMs: number | null })[] = []; const reads = discoveredReads(reading.operation?.signal, reading.operation?.env); - const discovered = discover - ? (await discoverCheckouts(config, listedRoots(), reading.operation)) - .filter(({ deferred }) => !deferred) - .map(({ path }): [string, ProjectRecord] => [path, {}]) + const found = discover + ? (await discoverCheckouts(config, listedRoots(), reading.operation)).filter(({ deferred }) => !deferred) : []; + const discovered = found.map(({ path }): [string, ProjectRecord] => [path, {}]); + const reduceOnly = new Set(found.filter((entry) => entry.reduceOnly).map(({ path }) => path)); for (const [path, project] of [...Object.entries(config.projects), ...discovered]) { if (sameRoot(path, reading.protectedRoot) || !(await worthIt(path, project))) continue; const used = Date.parse(String(readWorkspaceState(path)?.lastUsedAt ?? '')); @@ -247,7 +256,12 @@ async function idleWorkspaces( let idleMs: number | null = null; if (await runsSomething(path, project, reading)) onKept?.(path, 'something of it runs'); else { - const verdict = await idleVerdict(path, thresholdMs, setting, { operation: reading.operation, reads, full }); + const verdict = await idleVerdict(path, thresholdMs, setting, { + operation: reading.operation, + reads, + full, + reduceOnly: reduceOnly.has(path), + }); if ('idleMs' in verdict) idleMs = verdict.idleMs; else onKept?.(path, verdict.kept); } diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 62249bef2c..4b0276adeb 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1232,7 +1232,7 @@ symlink), on the home volume, below no skipped folder and still a project; anything else is dropped. Checkouts the scan finds under a dot-folder of the home folder are recorded in the result and never acted on, because an installed tool cannot be told from a checkout there; maintenance.discoverRoots is the way -to manage one. A discovered checkout that an executable on PATH (a symlink in a +to manage one. A reduce-only checkout that an executable on PATH (a symlink in a PATH directory, ~/.local/bin, ~/bin, /usr/local/bin or /opt/homebrew/bin) resolves into is kept whole. Reduce-only checkouts have their build outputs and, in a linked worktree, node_modules and Pods reduced by the rules below. Passes diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index cfee84d953..f9a2045c8e 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -218,7 +218,7 @@ export async function runMaintenanceAtLevel({ ? 'left for a later pass: at most 40 checkouts found through a tool folder or the full scan are handled at a time' : taken || idleMs <= 0 || !existsSync(checkout.path) ? null - : await discoveredKeepReason(checkout.path, idleMs, operation, reads); + : await discoveredKeepReason(checkout.path, idleMs, operation, reads, checkout.reduceOnly === true); if (kept && /source entries to check|cannot be read/.test(kept)) blocked.push(`Discovered checkout ${checkout.path}: ${kept}, so it is never reduced`); discovered.push({ ...checkout, actions: mine, ...(kept ? { kept } : {}) }); diff --git a/packages/stim-cli/src/workspace/discovered-activity.ts b/packages/stim-cli/src/workspace/discovered-activity.ts index 8073fe4a7d..9c0bdafd2a 100644 --- a/packages/stim-cli/src/workspace/discovered-activity.ts +++ b/packages/stim-cli/src/workspace/discovered-activity.ts @@ -110,11 +110,14 @@ export async function discoveredEvidence( idleMs, reads, operation, + reduceOnly = false, }: { now: number; idleMs: number; reads: DiscoveredReads; operation?: StimContext; + /** Found through a tool folder or the full scan, where an installed tool cannot be told from a checkout. */ + reduceOnly?: boolean; }, ): Promise { const signal = operation?.signal; @@ -138,7 +141,7 @@ export async function discoveredEvidence( keep.push(...(await reads.sessions(path, now))); const running = runningFromReason(path, await reads.processes()); if (running) keep.push(running); - const launcher = launcherReason(path, await reads.launchers()); + const launcher = reduceOnly ? launcherReason(path, await reads.launchers()) : null; if (launcher) keep.push(launcher); return { activity, keep }; } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index db94ee4dc0..4bc5e5a867 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout that an executable on your `PATH` links into is kept whole. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` links into is kept whole. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 79ebfb72522745de0298d6283c9fc15a60aebdec Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 01:25:32 -0500 Subject: [PATCH 08/25] fix: every output, dependency and Cargo route keeps a reduce-only checkout a PATH launcher runs from, and an unreadable launcher directory keeps them all --- llp/0004-discovered-checkouts.decision.md | 29 ++++-- .../__tests__/discovered-reduce-only.test.ts | 52 ++++++++++ .../__tests__/discovered-worktrees.test.ts | 2 +- .../stim-cli/src/__tests__/launchers.test.ts | 50 ++++++++++ .../src/__tests__/maintenance-run.test.ts | 44 +++++---- .../stim-cli/src/commands/gc/worktrees.ts | 4 +- packages/stim-cli/src/daemon/stages.ts | 16 +++- packages/stim-cli/src/guide/cleanup.ts | 8 +- .../stim-cli/src/stimd/passes/artifacts.ts | 20 +++- .../stim-cli/src/stimd/passes/cargo-target.ts | 49 +++++++--- .../stim-cli/src/stimd/passes/level-run.ts | 2 +- .../src/workspace/discovered-activity.ts | 3 +- .../src/workspace/discovered-reads.ts | 12 ++- .../src/workspace/launcher-directories.ts | 24 +++++ packages/stim-cli/src/workspace/launchers.ts | 94 +++++++++++++++++++ .../src/workspace/regenerable-dirs.ts | 7 +- .../stim-cli/src/workspace/running-from.ts | 44 +-------- website/docs/worktrees.md | 2 +- 18 files changed, 360 insertions(+), 102 deletions(-) create mode 100644 packages/stim-cli/src/__tests__/launchers.test.ts create mode 100644 packages/stim-cli/src/workspace/launcher-directories.ts create mode 100644 packages/stim-cli/src/workspace/launchers.ts diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index c7d7fa0bfb..b36472d8bf 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -99,7 +99,8 @@ A folder named in `maintenance.discoverRoots` keeps the full rules. A scoped Reason for entering home dot-folders: checkouts live there too, for example about 18 checkouts and 80 GB of Cargo `target/` under `~/.exact/exact2` (#5757), -so a blanket skip of dot-folders would miss the layout that started this work. +so a blanket skip of dot-folders would never even report the layout that started +this work. The scan records them; it does not act on them (see below). The scan enters a dot-folder directly in the home folder unless it is in a named prune list of tool homes, package and build caches and runtimes (`.git`, `.npm`, `.yarn`, `.bun`, `.gradle`, `.cargo`, `.rustup`, `.nvm`, @@ -130,11 +131,22 @@ installed command. The record lets doctor report what the folder holds and the command that adds it to `maintenance.discoverRoots`; that listing is not part of this change. -A scan find outside a home dot-folder can also be an installed tool. Discovery -keeps a reduce-only checkout (a tool folder or scan find), and so its outputs, while an executable reachable -through a PATH directory (and `~/.local/bin`, `~/bin`, `/usr/local/bin`, -`/opt/homebrew/bin`) is a symlink that resolves into it. A wrapper script that -only names a path is not followed. +A scan find outside a home dot-folder can also be an installed tool. Every route +that deletes outputs, dependencies or Cargo data of a reduce-only checkout keeps +the checkout while an executable reachable through a PATH directory is a symlink +that resolves into it: the idle stage (`discoveredEvidence`, and again under the +locks in `clearStoppedOutputs`), the artifact and dependency passes +(`artifactReads` into `keptReason`, which `collectArtifactsAsync`, +`clearArtifact` and the source dependency trees all pass through), and the Cargo +target measurement, the pressure plan and the acting trim +(`regenerableDirProblem` with the launcher reader). The directories read are +those on PATH and the per-user bin folders a daemon started by launchd does not +list: `~/.local/bin`, `~/bin`, `~/.cargo/bin`, `~/.bun/bin`, `~/.volta/bin`, +`~/Library/pnpm`, the active nvm bin, `/usr/local/bin` and `/opt/homebrew/bin`. +A directory or entry that does not exist, and a dangling symlink, is no +launcher. Any other failure to inspect a directory (permission, I/O, a timeout) +keeps every reduce-only checkout for that operation, with a reason naming the +directory. A wrapper script that only names a path is not followed. A scan-found source checkout follows the existing discovered rule and loses `node_modules` and Pods after `idle.sourceDormantDays` (7); the maintainer has @@ -181,8 +193,9 @@ minutes (not "least recently handled first"), the depth of seven, the folders pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and `~/Applications` left out, the low-disk scan running hourly while the daemon's disk level is above normal (high or critical, `disk.minFreeGb`) and no scan ran for an hour, the -retry of a failed scan after an hour, and the scan running only in the daemon -after the pass. +retry of a failed scan after an hour, the scan running only in the daemon +after the pass, and the PATH launcher keep. The scan is not deferred under host +load; it takes about a second and a half on one MacBook. Not part of this change: points 4, 5 and 7, which belong to the doctor listing and the low-disk line. diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index df03702966..c3a17b2d0a 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -12,6 +12,9 @@ import { } from 'node:fs'; import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; +import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; +import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; +import { testContext } from './_context.ts'; import { artifactCheckoutsAsync } from '../stimd/passes/artifacts.ts'; import { stoppedOutputCandidates } from '../daemon/stages.ts'; import { saveConfig } from '../workspace/config.ts'; @@ -31,6 +34,9 @@ vi.mock('../workspace/protected-locations.ts', async (original) => { }; }); +const launcherDirs = vi.hoisted(() => ({ list: new Array() })); +vi.mock('../workspace/launcher-directories.ts', () => ({ launcherDirectories: () => launcherDirs.list })); + const day = 86_400_000; describe('checkouts found through a tool folder or the full scan', () => { @@ -61,6 +67,7 @@ describe('checkouts found through a tool folder or the full scan', () => { beforeEach(() => { base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-reduce-only-'))); + launcherDirs.list = [join(base, 'home', 'bin')]; vi.stubEnv('HOME', join(base, 'home')); vi.stubEnv('USERPROFILE', join(base, 'home')); vi.stubEnv('STIM_HOME', ''); @@ -155,6 +162,51 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(await stoppedOutputCandidates(config, reading, day)).toEqual([]); }); + it('keeps the Cargo target of a scan-found checkout at high disk, in the plan and at the trim, while a PATH executable links into it', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(join(tool, 'ios', 'App.xcodeproj'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', 'incremental'), { recursive: true }); + mkdirSync(join(tool, 'target', 'release'), { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(tool, '.gitignore'), 'target/\n'); + writeFileSync(join(tool, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(tool, 'target', 'debug', 'incremental', 'data'), 'x'.repeat(10_000)); + writeFileSync(join(tool, 'target', 'release', 'mytool'), '#!/bin/sh\n'); + execFileSync('git', ['-C', tool, 'init', '-q']); + execFileSync('git', ['-C', tool, 'add', '-A']); + execFileSync('git', ['-C', tool, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x']); + writeScan([tool]); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + const operation = testContext({ env: {} }); + const plan = () => + planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation, + }); + const planned = await plan(); + expect(planned.actions).toMatchObject([{ kind: 'would-clear-build-dirs', target: join(tool, 'target') }]); + + mkdirSync(join(base, 'home', 'bin')); + symlinkSync(join(tool, 'target', 'release', 'mytool'), join(base, 'home', 'bin', 'mytool')); + const kept = await planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + expect(kept.actions).toEqual([]); + expect(kept.skips).toMatchObject([{ target: join(tool, 'target'), reason: expect.stringContaining('on PATH') }]); + const [action] = planned.actions; + if (!action) throw new Error('no trim was planned'); + const trim = await trimCargoTargetAction(action, testContext({ env: {} })); + expect(trim).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); + }); + it('drops a cached checkout that was replaced by a symlink to a folder the scan never enters, and plans nothing under it', async () => { const moved = join(base, 'elsewhere', 'app'); mkdirSync(moved, { recursive: true }); diff --git a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts index 72583d4101..71b4ed45bf 100644 --- a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts @@ -184,7 +184,7 @@ describe('a discovered worktree on the removal path', () => { it('keeps a worktree a process runs from or an agent session works in', async () => { worktree('held'); vi.spyOn(reads, 'discoveredReads').mockImplementation(() => ({ - launchers: async () => [], + launchers: async () => ({ targets: [], incomplete: [] }), processes: async () => ({ commands: [{ pid: 1, line: `node ${join(base, 'roots', 'held')}/server.js` }], cwds: [], diff --git a/packages/stim-cli/src/__tests__/launchers.test.ts b/packages/stim-cli/src/__tests__/launchers.test.ts new file mode 100644 index 0000000000..4742cd32eb --- /dev/null +++ b/packages/stim-cli/src/__tests__/launchers.test.ts @@ -0,0 +1,50 @@ +import { launcherReader, launcherReason } from '../workspace/launchers.ts'; + +const missing = (code: string) => Object.assign(new Error(code), { code }); + +/** A fake of the directories on PATH: `listing` maps a directory to its entries, an Error, or nothing for absent. */ +const io = (listing: Record, links: Record = {}) => ({ + readdir: async (path: string) => { + const entry = listing[path]; + if (entry instanceof Error) throw entry; + if (!entry) throw missing('ENOENT'); + return entry; + }, + isSymlink: async (path: string) => path in links, + realpath: async (path: string) => { + const target = links[path]; + if (target instanceof Error) throw target; + if (!target) throw missing('ENOENT'); + return target; + }, +}); + +const scan = (directories: string[], fake: ReturnType) => + launcherReader(undefined, { directories, io: fake })(); + +describe('the PATH launcher scan', () => { + it('finds a symlink that resolves into a checkout, and nothing for a checkout none points into', async () => { + const result = await scan(['/bin1'], io({ '/bin1': ['tool'] }, { '/bin1/tool': '/work/app/target/release/tool' })); + expect(result).toEqual({ targets: ['/work/app/target/release/tool'], incomplete: [] }); + expect(launcherReason('/work/app', result)).toContain('/work/app/target/release/tool'); + expect(launcherReason('/work/other', result)).toBeNull(); + }); + + it('treats a missing directory, a dangling symlink and a file entry as no launcher, not as unknown', async () => { + const result = await scan( + ['/absent', '/bin1'], + io({ '/bin1': ['dangling', 'plain'] }, { '/bin1/dangling': missing('ENOENT') }), + ); + expect(result).toEqual({ targets: [], incomplete: [] }); + expect(launcherReason('/work/app', result)).toBeNull(); + }); + + it('keeps every checkout when a directory or an entry could not be inspected, naming the directory', async () => { + const unreadable = await scan(['/locked', '/bin1'], io({ '/locked': missing('EACCES'), '/bin1': [] })); + expect(unreadable.incomplete).toEqual(['/locked']); + expect(launcherReason('/work/app', unreadable)).toContain('/locked'); + const unresolved = await scan(['/bin1'], io({ '/bin1': ['tool'] }, { '/bin1/tool': missing('EIO') })); + expect(unresolved.incomplete).toEqual(['/bin1']); + expect(launcherReason('/work/app', unresolved)).toContain('cannot be ruled out'); + }); +}); diff --git a/packages/stim-cli/src/__tests__/maintenance-run.test.ts b/packages/stim-cli/src/__tests__/maintenance-run.test.ts index 2e013ebe05..9e29beea58 100644 --- a/packages/stim-cli/src/__tests__/maintenance-run.test.ts +++ b/packages/stim-cli/src/__tests__/maintenance-run.test.ts @@ -504,6 +504,32 @@ test('an invalid cache cap falls back only that cap while report mode remains en } }); +const quietTick = (signal?: AbortSignal): MaintenanceTick => ({ + operation: testContext(signal ? { signal } : {}), + measured: { + measurement: { + disk: [{ volume: '/', freeMb: 500_000 }], + memory: { level: 'normal', availableBytes: null, totalBytes: 0 }, + load1: 0, + cores: 1, + bootedSimulators: null, + runningEmulators: null, + interaction: null, + }, + sims: null, + simsError: null, + devices: [], + }, +}); + +test('the discovery scan runs once after a successful daemon pass, with that pass tick', async () => { + const scan = vi.spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue').mockResolvedValue(); + const tick = quietTick(); + await runMaintenance('daemon', { tick }); + expect(scan).toHaveBeenCalledTimes(1); + expect(scan).toHaveBeenCalledWith(tick); +}); + test('an unverified child cleanup in a pass surfaces as that failure, not as the abort of a due discovery scan', async () => { const failure = Object.assign(new Error('unverified cleanup'), { code: 'STIM_CLAIM_UNAVAILABLE', @@ -517,23 +543,7 @@ test('an unverified child cleanup in a pass surfaces as that failure, not as the const scan = vi .spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue') .mockImplementation(async () => controller.signal.throwIfAborted()); - const tick: MaintenanceTick = { - operation: testContext({ signal: controller.signal }), - measured: { - measurement: { - disk: [{ volume: '/', freeMb: 500_000 }], - memory: { level: 'normal', availableBytes: null, totalBytes: 0 }, - load1: 0, - cores: 1, - bootedSimulators: null, - runningEmulators: null, - interaction: null, - }, - sims: null, - simsError: null, - devices: [], - }, - }; + const tick = quietTick(controller.signal); await expect(runMaintenance('daemon', { tick })).rejects.toBe(failure); expect(scan).not.toHaveBeenCalled(); }); diff --git a/packages/stim-cli/src/commands/gc/worktrees.ts b/packages/stim-cli/src/commands/gc/worktrees.ts index 398c8cadba..1c13b488b5 100644 --- a/packages/stim-cli/src/commands/gc/worktrees.ts +++ b/packages/stim-cli/src/commands/gc/worktrees.ts @@ -601,7 +601,7 @@ async function discoveredHold(candidate: WorktreeCandidate, operation?: StimCont const hold = await discoveredRemovalHold(candidate.path, { main: candidate.source ?? null, idleMs: dormantMs(operation), - reads: discoveredReads(operation?.signal, operation?.env), + reads: discoveredReads(operation?.signal, operation), reduceOnly: candidate.reduceOnly === true, operation, }); @@ -981,7 +981,7 @@ export async function collectWorktreeSweep({ const handed = worktrees.filter((candidate) => candidate.discovered && candidate.skipped === null); if (handed.length) { const { discoveredRemovalHold } = await import('../../workspace/discovered-removal.ts'); - const reads = discoveredReads(operation?.signal, operation?.env); + const reads = discoveredReads(operation?.signal, operation); for (const candidate of handed) { const hold = await discoveredRemovalHold(candidate.path, { main: candidate.source ?? null, diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index 164fc409c1..2c6591ecf6 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -64,6 +64,8 @@ export interface StageCandidate { idleMs: number; /** A source checkout is taken after every linked worktree. */ role: CheckoutRole; + /** Found through a tool folder or the full scan; a PATH launcher into it keeps its outputs. */ + reduceOnly?: boolean; } /** `idle.dormantDays` in milliseconds; 0 when the dormant stage is off. */ @@ -145,7 +147,7 @@ async function idleVerdict( { locked = false, operation, - reads = discoveredReads(operation?.signal, operation?.env), + reads = discoveredReads(operation?.signal, operation), full = false, reduceOnly = false, }: { @@ -239,7 +241,7 @@ async function idleWorkspaces( { discover = false, full = false, onKept }: Omit & { discover?: boolean } = {}, ): Promise { const readings: (Omit & { idleMs: number | null })[] = []; - const reads = discoveredReads(reading.operation?.signal, reading.operation?.env); + const reads = discoveredReads(reading.operation?.signal, reading.operation); const found = discover ? (await discoverCheckouts(config, listedRoots(), reading.operation)).filter(({ deferred }) => !deferred) : []; @@ -265,7 +267,7 @@ async function idleWorkspaces( if ('idleMs' in verdict) idleMs = verdict.idleMs; else onKept?.(path, verdict.kept); } - readings.push({ path, project, idleMs }); + readings.push({ path, project, idleMs, ...(reduceOnly.has(path) ? { reduceOnly: true } : {}) }); } catch (error) { if (isClaimUnavailable(error) && 'reportCleanup' in error) throw error; reading.operation?.signal.throwIfAborted(); @@ -339,7 +341,13 @@ export async function clearStoppedOutputs( workspaceDir(path), 'metro-start', async (): Promise => { - if ((await idleFor(path, thresholdMs, PRESSURE_THRESHOLD, { locked: true, operation })) === null) + if ( + (await idleFor(path, thresholdMs, PRESSURE_THRESHOLD, { + locked: true, + operation, + ...(candidate.reduceOnly ? { reduceOnly: true } : {}), + })) === null + ) return { outcome: 'kept', bytes: 0, detail: 'it is no longer idle' }; const outputs = await idleOutputs(path, config); if (!outputs.paths.length) return null; diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index f882fbfc97..a51016fb1f 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1241,8 +1241,12 @@ anything else is dropped. Checkouts the scan finds under a dot-folder of the home folder are recorded in the result and never acted on, because an installed tool cannot be told from a checkout there; maintenance.discoverRoots is the way to manage one. A reduce-only checkout that an executable on PATH (a symlink in a -PATH directory, ~/.local/bin, ~/bin, /usr/local/bin or /opt/homebrew/bin) -resolves into is kept whole. Reduce-only checkouts have their build outputs and, in +PATH directory, ~/.local/bin, ~/bin, ~/.cargo/bin, ~/.bun/bin, ~/.volta/bin, +~/Library/pnpm, the active nvm bin, /usr/local/bin or /opt/homebrew/bin) +resolves into is kept whole, by every pass that deletes its outputs, +dependencies or Cargo target. A directory that cannot be read (not one that is +missing) keeps every reduce-only checkout for that pass, and the reason names +it. Reduce-only checkouts have their build outputs and, in a linked worktree, node_modules and Pods reduced by the rules below. Passes evaluate at most 40 of them at a time, in a window that moves every 15 minutes; stim gc --dry-run lists all of them and says which are left for a later pass. A diff --git a/packages/stim-cli/src/stimd/passes/artifacts.ts b/packages/stim-cli/src/stimd/passes/artifacts.ts index 4b8d94051d..ad0ffa99c6 100644 --- a/packages/stim-cli/src/stimd/passes/artifacts.ts +++ b/packages/stim-cli/src/stimd/passes/artifacts.ts @@ -20,6 +20,7 @@ import { import { reclaimKeys } from '../../commands/worktree.ts'; import { loadConfig } from '../../workspace/config.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; +import { launcherReason, launchersOf } from '../../workspace/launchers.ts'; import { gitQuiet, gitQuietAsync } from '../../workspace/git.ts'; import { childEnvironment } from '../../workspace/git-read.ts'; import { withIdleWorkspace, workspaceInUse, workspaceInUseAsync } from '../../workspace/in-use.ts'; @@ -67,6 +68,8 @@ export interface ArtifactWorktree { keys: string[]; /** Why a running agent session keeps the worktree, read when the worktrees were listed. */ agents?: string[]; + /** Found through a tool folder or the full scan, where an installed tool cannot be told from a checkout. */ + reduceOnly?: boolean; } export interface ArtifactCandidate { @@ -77,6 +80,7 @@ export interface ArtifactCandidate { agents?: string[]; /** In a source checkout, whose dependencies no worktree warm restores. */ source?: boolean; + reduceOnly?: boolean; } function isDirectory(path: string): boolean { @@ -143,6 +147,7 @@ export async function artifactCheckoutsAsync( (entry) => !entry.deferred, ); const known = new Set([...listedRoots(), ...discovered.map(({ path }) => path)]); + const reduceOnly = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); for (const root of (await candidateRootsAsync(operation, discovered)).filter((entry) => known.has(entry))) { signal.throwIfAborted(); const { entries, ...source } = await resolveSourceCheckoutAsync(root, { @@ -151,7 +156,11 @@ export async function artifactCheckoutsAsync( declareFor: operation, }); const entry = artifactWorktree(root, entries, source); - if (entry) found.set(entry.path, entry); + if (entry) + found.set( + entry.path, + reduceOnly.has(root) || found.get(entry.path)?.reduceOnly ? { ...entry, reduceOnly: true } : entry, + ); } return [...found.values()]; } @@ -254,6 +263,8 @@ type ArtifactReads = { tracked: string | null; activity: ReturnType; written: number | null; + /** Why an executable on PATH keeps a reduce-only checkout, or null. */ + launcher: string | null; }; async function artifactReads( @@ -273,7 +284,8 @@ async function artifactReads( const windowed = tracked !== null && !tracked.trim() && kind === 'dependencies'; const activity = windowed ? await worktreeActivityOfAsync(worktree, keys, operation) : null; const written = windowed ? await newestWriteAsync(path, SCAN_DEPTH[kind], signal) : null; - return { tracked, activity, written }; + const launcher = candidate.reduceOnly ? launcherReason(worktree, await launchersOf(operation, signal)()) : null; + return { tracked, activity, written, launcher }; } async function pinsOf( @@ -342,6 +354,7 @@ function keptReason( ): string | null { const guarded = artifactGuard(candidate, kind, pinned, locked, running, inUse); if (guarded) return guarded; + if (reads?.launcher) return reads.launcher; const { worktree, path, keys } = candidate; const tracked = reads ? reads.tracked : gitQuiet(worktree, ['ls-files', '--', relative(worktree, path)]); if (tracked === null) return 'git cannot tell whether it holds tracked files'; @@ -450,6 +463,7 @@ export async function collectArtifactsAsync( keys: worktree.keys, locked: worktree.locked, agents: worktree.agents, + ...(worktree.reduceOnly ? { reduceOnly: true } : {}), ...(options.source ? { source: true } : {}), }; const inUse = protectedHere ? null : await inUseOf(candidate, kind, signal, operation); @@ -610,6 +624,7 @@ export async function clearArtifact( keys: worktree.keys, locked: worktree.locked, agents, + ...(worktree.reduceOnly ? { reduceOnly: true } : {}), }; const running = kind === 'dependencies' ? await readRunningProcessesAsync() : null; const pinned = await pinsOf(candidate.keys, options.pinned); @@ -735,6 +750,7 @@ export async function clearSourceTrees( locked: worktree.locked, agents: (await agentSessionUse([worktree.path], Date.now(), signal))[0] ?? [], source: true, + ...(worktree.reduceOnly ? { reduceOnly: true } : {}), }; const checkTree = async (path: string, held?: readonly string[]): Promise => { const candidate = { ...base, path, ...(held ? { source: false } : {}) }; diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index f1c15227ea..fc643554d9 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -9,6 +9,7 @@ import { measuredDirectorySizeAsync } from '../../fs-util.ts'; import { canonicalPath } from '../../commands/gc/paths.ts'; import { listedRoots } from '../../commands/gc/worktrees.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; +import { launchersOf } from '../../workspace/launchers.ts'; import { cargoTargetDirsAsync, isBuildDirPath, regenerableDirProblem } from '../../workspace/regenerable-dirs.ts'; import { appRoots, artifactCheckoutsAsync } from './artifacts.ts'; import { readVolumeSpaceAsync } from './budget.ts'; @@ -32,6 +33,8 @@ export interface CargoTargetCandidate { dir: string; identity: string; discovered: boolean; + /** Found through a tool folder or the full scan: a PATH launcher into the checkout keeps its target. */ + reduceOnly?: boolean; measuredAt: number | null; } @@ -94,14 +97,21 @@ async function cargoTargetCandidates( ): Promise { const signal = operation?.signal ?? new AbortController().signal; const found: CargoTargetCandidate[] = []; - const add = async (base: string, discovered: boolean) => { + const add = async (base: string, discovered: boolean, reduceOnly = false) => { const root = canonicalPath(base); for (const path of await cargoTargetDirsAsync(root)) { try { await lstat(path); // Windows 8.3 aliases need native realpath. If it fails, keep the path for the existing ownership checks. const identity = process.platform === 'win32' ? await realpath(path).catch(() => path) : path; - found.push({ root, dir: path, identity, discovered, measuredAt: lastMeasured.get(path) ?? null }); + found.push({ + root, + dir: path, + identity, + discovered, + ...(reduceOnly ? { reduceOnly } : {}), + measuredAt: lastMeasured.get(path) ?? null, + }); } catch {} } }; @@ -112,7 +122,8 @@ async function cargoTargetCandidates( .map(({ path }) => canonicalPath(path)), ); for (const checkout of await artifactCheckoutsAsync(signal, operation)) - for (const base of appRoots(checkout, 'build')) await add(base, discovered.has(canonicalPath(checkout.path))); + for (const base of appRoots(checkout, 'build')) + await add(base, discovered.has(canonicalPath(checkout.path)), checkout.reduceOnly === true); return found; } @@ -139,9 +150,12 @@ export async function measureCargoTargets( now, ); scan.unlisted = unlisted.map(({ dir }) => dir); - for (const { root, dir } of measured) { + for (const { root, dir, reduceOnly } of measured) { lastMeasured.set(dir, now); - const problem = await regenerableDirProblem(root, dir, operation?.signal, { processes: false }); + const problem = await regenerableDirProblem(root, dir, operation?.signal, { + processes: false, + ...(reduceOnly ? { launchers: launchersOf(operation) } : {}), + }); if (problem) { scan.refused.set(dir, { workspace: root, reason: problem }); continue; @@ -345,11 +359,14 @@ export async function planCargoTargetsAtPressure({ if (settings.cargoTargetMaxGb === 0) return plan; const registered = Object.keys(loadConfig()?.projects ?? {}); const { measured } = selectCargoTargets(await cargoTargetCandidates(registered, operation), Infinity, now); - for (const { root: workspace, dir } of measured) { + for (const { root: workspace, dir, reduceOnly } of measured) { signal?.throwIfAborted(); const tightened = level ?? (await diskLevel(dir, signal)); if (tightened === 'normal') continue; - const problem = await regenerableDirProblem(workspace, dir, signal, { processes: false }); + const problem = await regenerableDirProblem(workspace, dir, signal, { + processes: false, + ...(reduceOnly ? { launchers: launchersOf(operation, signal) } : {}), + }); if (problem) { plan.skips.push({ target: dir, reason: problem, workspace }); continue; @@ -385,13 +402,14 @@ export async function planCargoTargetsAtPressure({ return plan; } -async function stillManaged(root: string, operation?: StimContext): Promise { +async function managedAs(root: string, operation?: StimContext): Promise<{ reduceOnly: boolean } | null> { const canonical = canonicalPath(root); - if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) return true; + if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) return { reduceOnly: false }; const signal = operation?.signal ?? new AbortController().signal; - return (await artifactCheckoutsAsync(signal, operation)).some((checkout) => - appRoots(checkout, 'build').some((base) => canonicalPath(base) === canonical), + const checkout = (await artifactCheckoutsAsync(signal, operation)).find((entry) => + appRoots(entry, 'build').some((base) => canonicalPath(base) === canonical), ); + return checkout ? { reduceOnly: checkout.reduceOnly === true } : null; } async function targetVolume(target: string, signal?: AbortSignal): Promise<{ freeMb: number; device: number } | null> { @@ -409,9 +427,12 @@ export async function trimCargoTargetAction( ): Promise { const root = action.workspace; if (!root || !isBuildDirPath(action.target)) return kept('it is no longer a Cargo target/ Stim recognises'); - if (!(await stillManaged(root, operation))) - return kept('its checkout is no longer in a workspace or discovery scope'); - const problem = await regenerableDirProblem(root, action.target, operation?.signal, { processes: false }); + const managed = await managedAs(root, operation); + if (!managed) return kept('its checkout is no longer in a workspace or discovery scope'); + const problem = await regenerableDirProblem(root, action.target, operation?.signal, { + processes: false, + ...(managed.reduceOnly ? { launchers: launchersOf(operation) } : {}), + }); if (problem) return kept(problem); const measure = action.level === undefined; const before = measure ? null : await targetVolume(action.target, operation?.signal); diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index e0c4f42f93..17ba0137d7 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -240,7 +240,7 @@ export async function runMaintenanceAtLevel({ for (const action of ladder?.levels.flatMap((entry) => entry.actions) ?? []) take(fromLadder(action)); } const idleMs = level === 'normal' ? dormantAfterMs(loadConfig(), env) : PRESSURE_IDLE_MS; - const reads = discoveredReads(operation?.signal, operation?.env); + const reads = discoveredReads(operation?.signal, operation); const discovered: LevelRun['discovered'] = []; const ownerOf = (path: string) => found diff --git a/packages/stim-cli/src/workspace/discovered-activity.ts b/packages/stim-cli/src/workspace/discovered-activity.ts index 9c0bdafd2a..deea1f6615 100644 --- a/packages/stim-cli/src/workspace/discovered-activity.ts +++ b/packages/stim-cli/src/workspace/discovered-activity.ts @@ -5,7 +5,8 @@ import { worktreeActivityOfAsync } from '../commands/gc/worktrees.ts'; import type { IdleStopEvidence } from '../policy/idle.ts'; import { gitQuietAsync } from './git.ts'; import type { DiscoveredReads } from './discovered-reads.ts'; -import { launcherReason, runningFromReason } from './running-from.ts'; +import { launcherReason } from './launchers.ts'; +import { runningFromReason } from './running-from.ts'; /** Folders holding dependencies or other checkouts, skipped wherever they are. */ const ALWAYS_SKIPPED = new Set(['.git', '.worktrees', 'node_modules', 'Pods']); diff --git a/packages/stim-cli/src/workspace/discovered-reads.ts b/packages/stim-cli/src/workspace/discovered-reads.ts index b52f4a4c79..a997666ccb 100644 --- a/packages/stim-cli/src/workspace/discovered-reads.ts +++ b/packages/stim-cli/src/workspace/discovered-reads.ts @@ -1,22 +1,24 @@ import type { AgentSession } from '@stim-cli/core/state'; import { agentKey, attributeAgentSessions, discoverAgentSessions } from '../agent-sessions.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; -import { launcherReader, processReader, type RunningProcesses } from './running-from.ts'; +import { launchersOf, type LauncherScan } from './launchers.ts'; +import { processReader, type RunningProcesses } from './running-from.ts'; +import type { StimContext } from '@stim-cli/core'; /** What one pass reads once for every checkout: the process table and the agent sessions. */ export interface DiscoveredReads { processes: () => Promise; - /** Where the executables on PATH that are symlinks resolve to. */ - launchers: () => Promise; + /** Where the executables on PATH that are symlinks resolve to, and what could not be read. */ + launchers: () => Promise; /** Why an agent session working in the path keeps it. */ sessions: (path: string, now: number) => Promise; } -export function discoveredReads(signal?: AbortSignal, env?: NodeJS.ProcessEnv): DiscoveredReads { +export function discoveredReads(signal?: AbortSignal, operation?: StimContext): DiscoveredReads { let found: Promise | undefined; return { processes: processReader(signal), - launchers: launcherReader(signal, env), + launchers: launchersOf(operation, signal), sessions: async (path, now) => { found ??= discoverAgentSessions({ now }); const sessions = await found; diff --git a/packages/stim-cli/src/workspace/launcher-directories.ts b/packages/stim-cli/src/workspace/launcher-directories.ts new file mode 100644 index 0000000000..e758c8a3f4 --- /dev/null +++ b/packages/stim-cli/src/workspace/launcher-directories.ts @@ -0,0 +1,24 @@ +import { homedir } from 'node:os'; +import { delimiter, join } from 'node:path'; + +/** + * Where the executables a person installs by linking them live: the PATH directories of the process, and the per-user + * bin folders a daemon started by launchd, with its minimal PATH, does not list: `~/.local/bin` and `~/bin` (user + * links), `~/.cargo/bin` (cargo install), `~/.bun/bin` (bun), `~/.volta/bin` (Volta), `~/Library/pnpm` (pnpm's global + * bin), the active nvm bin (`NVM_BIN`), and `/usr/local/bin` and `/opt/homebrew/bin` (Homebrew). + */ +export function launcherDirectories(env: NodeJS.ProcessEnv): string[] { + const home = homedir(); + const extra = [ + join(home, '.local', 'bin'), + join(home, 'bin'), + join(home, '.cargo', 'bin'), + join(home, '.bun', 'bin'), + join(home, '.volta', 'bin'), + join(home, 'Library', 'pnpm'), + ...(env.NVM_BIN ? [env.NVM_BIN] : []), + '/usr/local/bin', + '/opt/homebrew/bin', + ]; + return [...new Set([...(env.PATH ?? '').split(delimiter).filter(Boolean), ...extra])]; +} diff --git a/packages/stim-cli/src/workspace/launchers.ts b/packages/stim-cli/src/workspace/launchers.ts new file mode 100644 index 0000000000..03165e6522 --- /dev/null +++ b/packages/stim-cli/src/workspace/launchers.ts @@ -0,0 +1,94 @@ +import { lstat, readdir, realpath } from 'node:fs/promises'; +import { join, sep } from 'node:path'; +import type { StimContext } from '@stim-cli/core'; +import { canonicalPath } from '../commands/gc/paths.ts'; +import { launcherDirectories } from './launcher-directories.ts'; + +/** + * Where the executables on PATH that are symlinks resolve to, and the directories whose inspection could not be + * completed. A directory or entry that does not exist, and a dangling symlink, is no launcher. Anything else that + * fails (permission, I/O, a timeout) is incomplete, and nothing is known about that directory. + */ +export interface LauncherScan { + targets: string[]; + incomplete: string[]; +} + +interface LauncherIo { + readdir: (path: string) => Promise; + isSymlink: (path: string) => Promise; + realpath: (path: string) => Promise; +} + +const REAL_IO: LauncherIo = { + readdir: (path) => readdir(path), + isSymlink: async (path) => (await lstat(path)).isSymbolicLink(), + realpath: (path) => realpath(path), +}; + +const ABSENT = ['ENOENT', 'ENOTDIR']; +const absent = (error: unknown) => error instanceof Error && 'code' in error && ABSENT.includes(String(error.code)); + +/** + * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader. A wrapper + * script that merely names a path is not followed. + */ +export function launcherReader( + signal?: AbortSignal, + { directories, io = REAL_IO }: { directories: readonly string[]; io?: LauncherIo } = { directories: [] }, +): () => Promise { + let scan: Promise | undefined; + const read = async (): Promise => { + const found: LauncherScan = { targets: [], incomplete: [] }; + for (const directory of directories) { + signal?.throwIfAborted(); + try { + for (const name of await io.readdir(directory)) { + const entry = join(directory, name); + try { + if (!(await io.isSymlink(entry))) continue; + found.targets.push(await io.realpath(entry)); + } catch (error) { + if (!absent(error)) throw error; + } + } + } catch (error) { + signal?.throwIfAborted(); + if (!absent(error)) found.incomplete.push(directory); + } + } + return found; + }; + return () => (scan ??= read()); +} + +const readers = new WeakMap Promise>(); + +/** The launcher reader of an operation, so every route of one pass reads the directories once. */ +export function launchersOf( + operation: StimContext | undefined, + signal?: AbortSignal, + directories?: readonly string[], +): () => Promise { + const make = () => + launcherReader(signal ?? operation?.signal, { + directories: directories ?? launcherDirectories(operation?.env ?? {}), + }); + if (!operation) return make(); + let reader = readers.get(operation); + if (!reader) readers.set(operation, (reader = make())); + return reader; +} + +/** + * Why `folder` is kept for an executable on PATH: one resolves to a path inside it, or a launcher directory could + * not be inspected, so none can be ruled out. Null when every directory was read and none points into it. + */ +export function launcherReason(folder: string, scan: LauncherScan): string | null { + const root = canonicalPath(folder); + const target = scan.targets.find((path) => path.startsWith(`${root}${sep}`)); + if (target) return `an executable on PATH runs from it (${target})`; + if (scan.incomplete.length) + return `an executable on PATH cannot be ruled out: ${scan.incomplete.join(', ')} could not be read`; + return null; +} diff --git a/packages/stim-cli/src/workspace/regenerable-dirs.ts b/packages/stim-cli/src/workspace/regenerable-dirs.ts index b0921cf60a..d7adb29866 100644 --- a/packages/stim-cli/src/workspace/regenerable-dirs.ts +++ b/packages/stim-cli/src/workspace/regenerable-dirs.ts @@ -3,6 +3,7 @@ import { lstat, readdir, readFile } from 'node:fs/promises'; import { basename, dirname, join, relative, sep } from 'node:path'; import { canonicalPath } from '../commands/gc/paths.ts'; import { gitQuietAsync } from './git.ts'; +import { launcherReason, type LauncherScan } from './launchers.ts'; import { othersRunningAsync, readRunningProcessesAsync, runningFromReason } from './running-from.ts'; export type ArtifactKind = 'build' | 'dependencies'; @@ -219,7 +220,7 @@ export async function regenerableDirProblem( root: string, path: string, signal?: AbortSignal, - { processes = true }: { processes?: boolean } = {}, + { processes = true, launchers }: { processes?: boolean; launchers?: () => Promise } = {}, ): Promise { const rel = relative(root, path); try { @@ -240,6 +241,10 @@ export async function regenerableDirProblem( const unignored = await gitQuietAsync(root, ['ls-files', '--others', '--exclude-standard', '--', rel], { signal }); if (unignored === null) return 'git cannot tell whether it ignores every file in it'; if (unignored.trim()) return 'it holds files git does not ignore'; + if (launchers) { + const launcher = launcherReason(root, await launchers()); + if (launcher) return launcher; + } if ( processes && basename(path) === CARGO_TARGET && diff --git a/packages/stim-cli/src/workspace/running-from.ts b/packages/stim-cli/src/workspace/running-from.ts index 2d7e9697e1..7cc9bb76f1 100644 --- a/packages/stim-cli/src/workspace/running-from.ts +++ b/packages/stim-cli/src/workspace/running-from.ts @@ -1,7 +1,5 @@ import { readdirSync, readlinkSync, realpathSync } from 'node:fs'; -import { lstat, readdir, realpath } from 'node:fs/promises'; -import { homedir } from 'node:os'; -import { delimiter, join, sep } from 'node:path'; +import { join, sep } from 'node:path'; import { canonicalPath } from '../commands/gc/paths.ts'; import { getExecutor } from '../exec.ts'; import { serviceRunningFrom } from './service-guard.ts'; @@ -133,46 +131,6 @@ export function processReader(signal?: AbortSignal): () => Promise (table ??= readRunningProcessesAsync(signal).then((running) => othersRunningAsync(running, signal))); } -/** - * Where the executables a person installs by linking them live: the PATH directories of this process, and the usual - * user bin folders a daemon's minimal PATH leaves out. - */ -function launcherDirectories(env: NodeJS.ProcessEnv): string[] { - const home = homedir(); - const extra = [join(home, '.local', 'bin'), join(home, 'bin'), '/usr/local/bin', '/opt/homebrew/bin']; - return [...new Set([...(env.PATH ?? '').split(delimiter).filter(Boolean), ...extra])]; -} - -/** - * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader: where each - * executable reachable through PATH really lives. A wrapper script that merely names a path is not followed. - */ -export function launcherReader(signal?: AbortSignal, env: NodeJS.ProcessEnv = {}): () => Promise { - let targets: Promise | undefined; - const read = async (): Promise => { - const found: string[] = []; - for (const directory of launcherDirectories(env)) { - signal?.throwIfAborted(); - const names = await readdir(directory).catch(() => []); - for (const name of names) { - const entry = join(directory, name); - if (!(await lstat(entry).catch(() => null))?.isSymbolicLink()) continue; - const target = await realpath(entry).catch(() => null); - if (target) found.push(target); - } - } - return found; - }; - return () => (targets ??= read()); -} - -/** Why an executable on PATH, as `targets` lists them, keeps `folder`: one resolves to a path inside it, or null. */ -export function launcherReason(folder: string, targets: readonly string[]): string | null { - const root = canonicalPath(folder); - const target = targets.find((path) => path.startsWith(`${root}${sep}`)); - return target ? `an executable on PATH runs from it (${target})` : null; -} - function stimOnPath(): string | null { const found = getExecutor().findExecutable('stim'); if (!found) return null; diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 4bc5e5a867..417ce6894b 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` links into is kept whole. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. If one of those folders exists but cannot be read, such checkouts are kept for that pass. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From ec0b77a2e10f927b1fa11d72e0f45ebc64be22b3 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 02:15:12 -0500 Subject: [PATCH 09/25] test: a PATH launcher keeps a reduce-only checkout through the artifact and dependency route --- .../__tests__/discovered-reduce-only.test.ts | 89 ++++++++++++++++++- 1 file changed, 88 insertions(+), 1 deletion(-) diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index c3a17b2d0a..0bee626777 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -1,5 +1,6 @@ import { execFileSync } from 'node:child_process'; import { + chmodSync, existsSync, mkdirSync, mkdtempSync, @@ -15,7 +16,13 @@ import { join } from 'node:path'; import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; import { testContext } from './_context.ts'; -import { artifactCheckoutsAsync } from '../stimd/passes/artifacts.ts'; +import * as worktreeModule from '../workspace/worktree.ts'; +import { + artifactCheckoutsAsync, + artifactWorktreesAsync, + clearArtifact, + collectArtifactsAsync, +} from '../stimd/passes/artifacts.ts'; import { stoppedOutputCandidates } from '../daemon/stages.ts'; import { saveConfig } from '../workspace/config.ts'; import { collectWorktreeSweep, removeWorktrees } from '../commands/gc/worktrees.ts'; @@ -207,6 +214,86 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(trim).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); }); + describe('the artifact and dependency route for a scan-found linked worktree', () => { + const operation = () => testContext({ env: {} }); + const scanFoundWorktree = () => { + const found = worktree(join(base, 'home', 'Documents', 'feature-wt'), 'feature'); + mkdirSync(join(found, 'node_modules', '.bin'), { recursive: true }); + writeFileSync(join(found, 'node_modules', '.bin', 'tool'), '#!/bin/sh\n'); + age(found); + writeScan([found]); + return { found, modules: join(found, 'node_modules') }; + }; + const candidates = async () => { + const { signal } = new AbortController(); + return collectArtifactsAsync( + 'dependencies', + { now: Date.now(), windowMs: 0, protectedRoot: '', pinned: () => false }, + signal, + operation(), + ); + }; + const clear = async (modules: string) => { + const { signal } = new AbortController(); + return clearArtifact( + { kind: 'would-delete-dependencies', target: modules, bytes: 0, reason: 'test' }, + 'dependencies', + { + windowMs: 0, + protectedRoot: '', + pinned: () => false, + worktrees: await artifactWorktreesAsync(signal), + }, + ); + }; + + it('collects and clears node_modules without a PATH launcher', async () => { + const { modules } = scanFoundWorktree(); + const found = await candidates(); + expect(found.candidates.map((entry) => entry.path)).toEqual([modules]); + expect(await clear(modules)).toMatchObject({ status: 'done' }); + expect(existsSync(modules)).toBe(false); + }); + + it('keeps it, naming the launcher, when an executable on PATH resolves into the checkout, at the plan and at the check under the lock', async () => { + const { modules } = scanFoundWorktree(); + const bin = join(base, 'home', 'bin'); + mkdirSync(bin); + const link = join(bin, 'tool'); + symlinkSync(join(modules, '.bin', 'tool'), link); + const planned = await candidates(); + expect(planned.candidates).toEqual([]); + expect(planned.skips).toMatchObject([{ target: modules, reason: expect.stringContaining('on PATH') }]); + expect(await clear(modules)).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); + + rmSync(link); + const appears = vi.spyOn(worktreeModule, 'worktreeLockAsync').mockImplementation(async () => { + symlinkSync(join(modules, '.bin', 'tool'), link); + return null; + }); + expect(await clear(modules)).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); + expect(appears).toHaveBeenCalled(); + expect(existsSync(modules)).toBe(true); + }); + + it.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( + 'keeps it when a launcher directory cannot be read, naming the directory', + async () => { + const { modules } = scanFoundWorktree(); + const locked = join(base, 'home', 'locked-bin'); + mkdirSync(locked, { mode: 0o000 }); + launcherDirs.list = [locked]; + try { + const planned = await candidates(); + expect(planned.candidates).toEqual([]); + expect(planned.skips).toMatchObject([{ target: modules, reason: expect.stringContaining(locked) }]); + } finally { + chmodSync(locked, 0o700); + } + }, + ); + }); + it('drops a cached checkout that was replaced by a symlink to a folder the scan never enters, and plans nothing under it', async () => { const moved = join(base, 'elsewhere', 'app'); mkdirSync(moved, { recursive: true }); From b8df58de1d66f18794920def65cf6e0c1ca042cd Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 02:15:40 -0500 Subject: [PATCH 10/25] test: use test.skipIf for the unreadable launcher directory case --- packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 0bee626777..68ffd8fc4e 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -276,7 +276,7 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(modules)).toBe(true); }); - it.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( + test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( 'keeps it when a launcher directory cannot be read, naming the directory', async () => { const { modules } = scanFoundWorktree(); From 60fdb107ff377873253cd80ec9db80999edcf2f5 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 02:38:45 -0500 Subject: [PATCH 11/25] fix: read launchers anew before each deletion, check a Cargo target against its checkout, time out hung launcher directories and read the process PATH without an operation --- .oxlintrc.json | 1 + llp/0004-discovered-checkouts.decision.md | 8 ++ .../__tests__/discovered-reduce-only.test.ts | 96 +++++++++++++------ .../stim-cli/src/__tests__/launchers.test.ts | 21 ++++ packages/stim-cli/src/daemon/stages.ts | 1 + packages/stim-cli/src/guide/cleanup.ts | 7 +- .../stim-cli/src/stimd/passes/artifacts.ts | 13 ++- .../stim-cli/src/stimd/passes/cargo-target.ts | 27 +++--- .../src/workspace/discovered-reads.ts | 7 +- .../src/workspace/launcher-directories.ts | 2 +- packages/stim-cli/src/workspace/launchers.ts | 74 +++++++++++--- .../src/workspace/regenerable-dirs.ts | 8 +- 12 files changed, 199 insertions(+), 66 deletions(-) diff --git a/.oxlintrc.json b/.oxlintrc.json index 69f27b3e6f..aa931bc94c 100644 --- a/.oxlintrc.json +++ b/.oxlintrc.json @@ -522,6 +522,7 @@ "packages/stim-cli/src/web/chrome.ts", "packages/stim-cli/src/workspace/config.ts", "packages/stim-cli/src/workspace/git-read.ts", + "packages/stim-cli/src/workspace/launcher-directories.ts", "packages/stim-cli/src/workspace/settings.ts", "packages/stim-cli/src/workspace/workspace-state.ts", "packages/stim-cli/src/workspace/worktree-refresh.ts" diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index b36472d8bf..304d08f5b8 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -143,6 +143,14 @@ target measurement, the pressure plan and the acting trim those on PATH and the per-user bin folders a daemon started by launchd does not list: `~/.local/bin`, `~/bin`, `~/.cargo/bin`, `~/.bun/bin`, `~/.volta/bin`, `~/Library/pnpm`, the active nvm bin, `/usr/local/bin` and `/opt/homebrew/bin`. +Planning shares one reading per operation. The check that immediately precedes a +deletion (under the lock in `clearArtifact` and `clearSourceTrees`, in +`clearStoppedOutputs`, and before `trimCargoTargetAction` trims) reads the +directories anew, and the next deletion of the same operation reuses that +reading for at most 5 seconds, so a long pass never deletes on an old reading +and a run of deletions does not each pay the read (55 to 72 ms here). A Cargo +target is checked against its owning checkout, not its app folder. Each call to a +directory has 3 seconds to answer; a timeout is an incomplete inspection. A directory or entry that does not exist, and a dangling symlink, is no launcher. Any other failure to inspect a directory (permission, I/O, a timeout) keeps every reduce-only checkout for that operation, with a reason naming the diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 68ffd8fc4e..d08d26f02a 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -16,7 +16,6 @@ import { join } from 'node:path'; import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; import { testContext } from './_context.ts'; -import * as worktreeModule from '../workspace/worktree.ts'; import { artifactCheckoutsAsync, artifactWorktreesAsync, @@ -214,6 +213,47 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(trim).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); }); + it('keeps a nested app Cargo target when a PATH launcher resolves into the checkout outside that app, in the plan and at the trim', async () => { + const mono = join(base, 'home', 'tools', 'mono'); + const app = join(mono, 'apps', 'native'); + mkdirSync(join(app, 'ios', 'App.xcodeproj'), { recursive: true }); + mkdirSync(join(app, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(app, 'target', 'debug', 'incremental'), { recursive: true }); + mkdirSync(join(mono, 'bin'), { recursive: true }); + writeFileSync(join(app, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(app, '.gitignore'), 'target/\n'); + writeFileSync(join(app, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(app, 'target', 'debug', 'incremental', 'data'), 'x'.repeat(10_000)); + writeFileSync(join(mono, 'bin', 'tool'), '#!/bin/sh\n'); + execFileSync('git', ['-C', mono, 'init', '-q']); + execFileSync('git', ['-C', mono, 'add', '-A']); + execFileSync('git', ['-C', mono, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x']); + writeScan([mono]); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + const plan = () => + planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + const planned = await plan(); + expect(planned.actions).toMatchObject([{ kind: 'would-clear-build-dirs', target: join(app, 'target') }]); + + mkdirSync(join(base, 'home', 'bin')); + symlinkSync(join(mono, 'bin', 'tool'), join(base, 'home', 'bin', 'tool')); + const kept = await plan(); + expect(kept.actions).toEqual([]); + expect(kept.skips).toMatchObject([{ target: join(app, 'target'), reason: expect.stringContaining('on PATH') }]); + const [action] = planned.actions; + if (!action) throw new Error('no trim was planned'); + expect(await trimCargoTargetAction(action, testContext({ env: {} }))).toMatchObject({ + status: 'kept', + detail: expect.stringContaining('on PATH'), + }); + }); + describe('the artifact and dependency route for a scan-found linked worktree', () => { const operation = () => testContext({ env: {} }); const scanFoundWorktree = () => { @@ -224,17 +264,10 @@ describe('checkouts found through a tool folder or the full scan', () => { writeScan([found]); return { found, modules: join(found, 'node_modules') }; }; - const candidates = async () => { - const { signal } = new AbortController(); - return collectArtifactsAsync( - 'dependencies', - { now: Date.now(), windowMs: 0, protectedRoot: '', pinned: () => false }, - signal, - operation(), - ); - }; - const clear = async (modules: string) => { - const { signal } = new AbortController(); + const { signal } = new AbortController(); + const collectOptions = { now: Date.now(), windowMs: 0, protectedRoot: '', pinned: () => false }; + const candidates = async () => collectArtifactsAsync('dependencies', collectOptions, signal, operation()); + const clear = async (modules: string, shared?: ReturnType) => { return clearArtifact( { kind: 'would-delete-dependencies', target: modules, bytes: 0, reason: 'test' }, 'dependencies', @@ -242,7 +275,8 @@ describe('checkouts found through a tool folder or the full scan', () => { windowMs: 0, protectedRoot: '', pinned: () => false, - worktrees: await artifactWorktreesAsync(signal), + worktrees: await artifactWorktreesAsync(signal, shared), + ...(shared ? { operation: shared } : {}), }, ); }; @@ -255,38 +289,42 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(modules)).toBe(false); }); - it('keeps it, naming the launcher, when an executable on PATH resolves into the checkout, at the plan and at the check under the lock', async () => { + it('keeps it, naming the launcher, when one appears after the plan, with the same operation at both', async () => { const { modules } = scanFoundWorktree(); const bin = join(base, 'home', 'bin'); mkdirSync(bin); const link = join(bin, 'tool'); symlinkSync(join(modules, '.bin', 'tool'), link); - const planned = await candidates(); - expect(planned.candidates).toEqual([]); - expect(planned.skips).toMatchObject([{ target: modules, reason: expect.stringContaining('on PATH') }]); - expect(await clear(modules)).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); - + const withLauncher = await candidates(); + expect(withLauncher.candidates).toEqual([]); + expect(withLauncher.skips).toMatchObject([{ target: modules, reason: expect.stringContaining('on PATH') }]); rmSync(link); - const appears = vi.spyOn(worktreeModule, 'worktreeLockAsync').mockImplementation(async () => { - symlinkSync(join(modules, '.bin', 'tool'), link); - return null; + + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + expect(planned.candidates.map((entry) => entry.path)).toEqual([modules]); + symlinkSync(join(modules, '.bin', 'tool'), link); + expect(await clear(modules, shared)).toMatchObject({ + status: 'kept', + detail: expect.stringContaining('on PATH'), }); - expect(await clear(modules)).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); - expect(appears).toHaveBeenCalled(); expect(existsSync(modules)).toBe(true); }); test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( - 'keeps it when a launcher directory cannot be read, naming the directory', + 'keeps it when a launcher directory becomes unreadable after the plan, naming the directory', async () => { const { modules } = scanFoundWorktree(); const locked = join(base, 'home', 'locked-bin'); mkdirSync(locked, { mode: 0o000 }); - launcherDirs.list = [locked]; try { - const planned = await candidates(); - expect(planned.candidates).toEqual([]); - expect(planned.skips).toMatchObject([{ target: modules, reason: expect.stringContaining(locked) }]); + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + expect(planned.candidates.map((entry) => entry.path)).toEqual([modules]); + launcherDirs.list = [...launcherDirs.list, locked]; + const outcome = await clear(modules, shared); + expect(outcome).toMatchObject({ status: 'kept', detail: expect.stringContaining(locked) }); + expect(existsSync(modules)).toBe(true); } finally { chmodSync(locked, 0o700); } diff --git a/packages/stim-cli/src/__tests__/launchers.test.ts b/packages/stim-cli/src/__tests__/launchers.test.ts index 4742cd32eb..403d35575d 100644 --- a/packages/stim-cli/src/__tests__/launchers.test.ts +++ b/packages/stim-cli/src/__tests__/launchers.test.ts @@ -1,3 +1,4 @@ +import { launcherDirectories } from '../workspace/launcher-directories.ts'; import { launcherReader, launcherReason } from '../workspace/launchers.ts'; const missing = (code: string) => Object.assign(new Error(code), { code }); @@ -47,4 +48,24 @@ describe('the PATH launcher scan', () => { expect(unresolved.incomplete).toEqual(['/bin1']); expect(launcherReason('/work/app', unresolved)).toContain('cannot be ruled out'); }); + + it('treats a directory that does not answer in time as not inspected, and names the way to restore reduction', async () => { + const hung = { ...io({ '/bin1': ['tool'] }), readdir: () => new Promise(() => {}) }; + const pending = launcherReader(undefined, { directories: ['/hung'], io: hung, timeoutMs: 10 })(); + await vi.advanceTimersByTimeAsync(10); + const result = await pending; + expect(result.incomplete).toEqual(['/hung']); + expect(launcherReason('/work/app', result)).toContain('permissions'); + }); +}); + +describe('the launcher directories', () => { + afterEach(() => vi.unstubAllEnvs()); + + it('lists the PATH of the process when no environment is given, and of the given environment otherwise', () => { + vi.stubEnv('PATH', '/from/process:/also/process'); + expect(launcherDirectories()).toEqual(expect.arrayContaining(['/from/process', '/also/process'])); + expect(launcherDirectories({ PATH: '/given' })).toContain('/given'); + expect(launcherDirectories({ PATH: '/given' })).not.toContain('/from/process'); + }); }); diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index 2c6591ecf6..543f9f6ce3 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -345,6 +345,7 @@ export async function clearStoppedOutputs( (await idleFor(path, thresholdMs, PRESSURE_THRESHOLD, { locked: true, operation, + reads: discoveredReads(operation?.signal, operation, true), ...(candidate.reduceOnly ? { reduceOnly: true } : {}), })) === null ) diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index a51016fb1f..dbbc1904b1 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1245,8 +1245,11 @@ PATH directory, ~/.local/bin, ~/bin, ~/.cargo/bin, ~/.bun/bin, ~/.volta/bin, ~/Library/pnpm, the active nvm bin, /usr/local/bin or /opt/homebrew/bin) resolves into is kept whole, by every pass that deletes its outputs, dependencies or Cargo target. A directory that cannot be read (not one that is -missing) keeps every reduce-only checkout for that pass, and the reason names -it. Reduce-only checkouts have their build outputs and, in +missing, or one that does not answer within 3 seconds) keeps every reduce-only +checkout for that pass, and the reason names it and says fixing its permissions +restores reduction. The check before each deletion reads the directories anew +(reused for at most 5 seconds by the next deletion), not the pass's planning +reading. Reduce-only checkouts have their build outputs and, in a linked worktree, node_modules and Pods reduced by the rules below. Passes evaluate at most 40 of them at a time, in a window that moves every 15 minutes; stim gc --dry-run lists all of them and says which are left for a later pass. A diff --git a/packages/stim-cli/src/stimd/passes/artifacts.ts b/packages/stim-cli/src/stimd/passes/artifacts.ts index ad0ffa99c6..29a6d59595 100644 --- a/packages/stim-cli/src/stimd/passes/artifacts.ts +++ b/packages/stim-cli/src/stimd/passes/artifacts.ts @@ -20,7 +20,7 @@ import { import { reclaimKeys } from '../../commands/worktree.ts'; import { loadConfig } from '../../workspace/config.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; -import { launcherReason, launchersOf } from '../../workspace/launchers.ts'; +import { freshLaunchers, launcherReason, launchersOf } from '../../workspace/launchers.ts'; import { gitQuiet, gitQuietAsync } from '../../workspace/git.ts'; import { childEnvironment } from '../../workspace/git-read.ts'; import { withIdleWorkspace, workspaceInUse, workspaceInUseAsync } from '../../workspace/in-use.ts'; @@ -272,6 +272,7 @@ async function artifactReads( kind: ArtifactKind, signal: AbortSignal, operation?: StimContext, + { fresh = false }: { fresh?: boolean } = {}, ): Promise { const { worktree, path, keys } = candidate; const tracked = await gitQuietAsync(worktree, ['ls-files', '--', relative(worktree, path)], { @@ -284,7 +285,9 @@ async function artifactReads( const windowed = tracked !== null && !tracked.trim() && kind === 'dependencies'; const activity = windowed ? await worktreeActivityOfAsync(worktree, keys, operation) : null; const written = windowed ? await newestWriteAsync(path, SCAN_DEPTH[kind], signal) : null; - const launcher = candidate.reduceOnly ? launcherReason(worktree, await launchersOf(operation, signal)()) : null; + const launcher = candidate.reduceOnly + ? launcherReason(worktree, await (fresh ? freshLaunchers : launchersOf)(operation, signal)()) + : null; return { tracked, activity, written, launcher }; } @@ -632,7 +635,7 @@ export async function clearArtifact( candidate, kind, { ...options, pinned, now: Date.now(), running, inUse: await inUseOf(candidate, kind, signal, operation) }, - await artifactReads(candidate, kind, signal, operation), + await artifactReads(candidate, kind, signal, operation, { fresh: true }), ); if (early) return { status: 'kept', bytes: 0, detail: early }; const unrestorable = @@ -654,7 +657,7 @@ export async function clearArtifact( inUse: await inUseOf(candidate, kind, signal, operation, key), ...(key ? { locked: key } : {}), }, - await artifactReads(candidate, kind, signal, operation), + await artifactReads(candidate, kind, signal, operation, { fresh: true }), ); if (reason) return reason; const late = key && options.recheck ? await options.recheck(candidate.keys, candidate.worktree) : null; @@ -767,7 +770,7 @@ export async function clearSourceTrees( inUse: await inUseOf(candidate, 'dependencies', signal, operation, held), ...(held ? { locked: held } : {}), }, - await artifactReads(candidate, 'dependencies', signal, operation), + await artifactReads(candidate, 'dependencies', signal, operation, { fresh: true }), ) ?? (await regenerableDirProblem(worktree.path, path, signal)) ?? linkedWorktreeUsing(candidate, options.worktrees) diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index fc643554d9..dac4de64ed 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -9,7 +9,7 @@ import { measuredDirectorySizeAsync } from '../../fs-util.ts'; import { canonicalPath } from '../../commands/gc/paths.ts'; import { listedRoots } from '../../commands/gc/worktrees.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; -import { launchersOf } from '../../workspace/launchers.ts'; +import { freshLaunchers, launchersOf } from '../../workspace/launchers.ts'; import { cargoTargetDirsAsync, isBuildDirPath, regenerableDirProblem } from '../../workspace/regenerable-dirs.ts'; import { appRoots, artifactCheckoutsAsync } from './artifacts.ts'; import { readVolumeSpaceAsync } from './budget.ts'; @@ -35,6 +35,8 @@ export interface CargoTargetCandidate { discovered: boolean; /** Found through a tool folder or the full scan: a PATH launcher into the checkout keeps its target. */ reduceOnly?: boolean; + /** The checkout that owns the target, which a PATH launcher can point anywhere into; `root` is the app inside it. */ + checkout?: string; measuredAt: number | null; } @@ -97,7 +99,7 @@ async function cargoTargetCandidates( ): Promise { const signal = operation?.signal ?? new AbortController().signal; const found: CargoTargetCandidate[] = []; - const add = async (base: string, discovered: boolean, reduceOnly = false) => { + const add = async (base: string, discovered: boolean, reduceOnly = false, owner?: string) => { const root = canonicalPath(base); for (const path of await cargoTargetDirsAsync(root)) { try { @@ -109,7 +111,7 @@ async function cargoTargetCandidates( dir: path, identity, discovered, - ...(reduceOnly ? { reduceOnly } : {}), + ...(reduceOnly ? { reduceOnly, checkout: owner ?? root } : {}), measuredAt: lastMeasured.get(path) ?? null, }); } catch {} @@ -123,7 +125,7 @@ async function cargoTargetCandidates( ); for (const checkout of await artifactCheckoutsAsync(signal, operation)) for (const base of appRoots(checkout, 'build')) - await add(base, discovered.has(canonicalPath(checkout.path)), checkout.reduceOnly === true); + await add(base, discovered.has(canonicalPath(checkout.path)), checkout.reduceOnly === true, checkout.path); return found; } @@ -150,11 +152,11 @@ export async function measureCargoTargets( now, ); scan.unlisted = unlisted.map(({ dir }) => dir); - for (const { root, dir, reduceOnly } of measured) { + for (const { root, dir, reduceOnly, checkout } of measured) { lastMeasured.set(dir, now); const problem = await regenerableDirProblem(root, dir, operation?.signal, { processes: false, - ...(reduceOnly ? { launchers: launchersOf(operation) } : {}), + ...(reduceOnly ? { launchers: launchersOf(operation), launcherRoot: checkout ?? root } : {}), }); if (problem) { scan.refused.set(dir, { workspace: root, reason: problem }); @@ -359,13 +361,13 @@ export async function planCargoTargetsAtPressure({ if (settings.cargoTargetMaxGb === 0) return plan; const registered = Object.keys(loadConfig()?.projects ?? {}); const { measured } = selectCargoTargets(await cargoTargetCandidates(registered, operation), Infinity, now); - for (const { root: workspace, dir, reduceOnly } of measured) { + for (const { root: workspace, dir, reduceOnly, checkout } of measured) { signal?.throwIfAborted(); const tightened = level ?? (await diskLevel(dir, signal)); if (tightened === 'normal') continue; const problem = await regenerableDirProblem(workspace, dir, signal, { processes: false, - ...(reduceOnly ? { launchers: launchersOf(operation, signal) } : {}), + ...(reduceOnly ? { launchers: launchersOf(operation, signal), launcherRoot: checkout ?? workspace } : {}), }); if (problem) { plan.skips.push({ target: dir, reason: problem, workspace }); @@ -402,14 +404,17 @@ export async function planCargoTargetsAtPressure({ return plan; } -async function managedAs(root: string, operation?: StimContext): Promise<{ reduceOnly: boolean } | null> { +async function managedAs( + root: string, + operation?: StimContext, +): Promise<{ reduceOnly: boolean; checkout?: string } | null> { const canonical = canonicalPath(root); if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) return { reduceOnly: false }; const signal = operation?.signal ?? new AbortController().signal; const checkout = (await artifactCheckoutsAsync(signal, operation)).find((entry) => appRoots(entry, 'build').some((base) => canonicalPath(base) === canonical), ); - return checkout ? { reduceOnly: checkout.reduceOnly === true } : null; + return checkout ? { reduceOnly: checkout.reduceOnly === true, checkout: checkout.path } : null; } async function targetVolume(target: string, signal?: AbortSignal): Promise<{ freeMb: number; device: number } | null> { @@ -431,7 +436,7 @@ export async function trimCargoTargetAction( if (!managed) return kept('its checkout is no longer in a workspace or discovery scope'); const problem = await regenerableDirProblem(root, action.target, operation?.signal, { processes: false, - ...(managed.reduceOnly ? { launchers: launchersOf(operation) } : {}), + ...(managed.reduceOnly ? { launchers: freshLaunchers(operation), launcherRoot: managed.checkout ?? root } : {}), }); if (problem) return kept(problem); const measure = action.level === undefined; diff --git a/packages/stim-cli/src/workspace/discovered-reads.ts b/packages/stim-cli/src/workspace/discovered-reads.ts index a997666ccb..81dfa97954 100644 --- a/packages/stim-cli/src/workspace/discovered-reads.ts +++ b/packages/stim-cli/src/workspace/discovered-reads.ts @@ -1,7 +1,7 @@ import type { AgentSession } from '@stim-cli/core/state'; import { agentKey, attributeAgentSessions, discoverAgentSessions } from '../agent-sessions.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; -import { launchersOf, type LauncherScan } from './launchers.ts'; +import { freshLaunchers, launchersOf, type LauncherScan } from './launchers.ts'; import { processReader, type RunningProcesses } from './running-from.ts'; import type { StimContext } from '@stim-cli/core'; @@ -14,11 +14,12 @@ export interface DiscoveredReads { sessions: (path: string, now: number) => Promise; } -export function discoveredReads(signal?: AbortSignal, operation?: StimContext): DiscoveredReads { +/** `fresh` reads the launcher directories anew, for the check that immediately precedes a deletion. */ +export function discoveredReads(signal?: AbortSignal, operation?: StimContext, fresh = false): DiscoveredReads { let found: Promise | undefined; return { processes: processReader(signal), - launchers: launchersOf(operation, signal), + launchers: fresh ? freshLaunchers(operation, signal) : launchersOf(operation, signal), sessions: async (path, now) => { found ??= discoverAgentSessions({ now }); const sessions = await found; diff --git a/packages/stim-cli/src/workspace/launcher-directories.ts b/packages/stim-cli/src/workspace/launcher-directories.ts index e758c8a3f4..64a2fae716 100644 --- a/packages/stim-cli/src/workspace/launcher-directories.ts +++ b/packages/stim-cli/src/workspace/launcher-directories.ts @@ -7,7 +7,7 @@ import { delimiter, join } from 'node:path'; * links), `~/.cargo/bin` (cargo install), `~/.bun/bin` (bun), `~/.volta/bin` (Volta), `~/Library/pnpm` (pnpm's global * bin), the active nvm bin (`NVM_BIN`), and `/usr/local/bin` and `/opt/homebrew/bin` (Homebrew). */ -export function launcherDirectories(env: NodeJS.ProcessEnv): string[] { +export function launcherDirectories(env: NodeJS.ProcessEnv = process.env): string[] { const home = homedir(); const extra = [ join(home, '.local', 'bin'), diff --git a/packages/stim-cli/src/workspace/launchers.ts b/packages/stim-cli/src/workspace/launchers.ts index 03165e6522..b06ff9bfee 100644 --- a/packages/stim-cli/src/workspace/launchers.ts +++ b/packages/stim-cli/src/workspace/launchers.ts @@ -29,13 +29,36 @@ const REAL_IO: LauncherIo = { const ABSENT = ['ENOENT', 'ENOTDIR']; const absent = (error: unknown) => error instanceof Error && 'code' in error && ABSENT.includes(String(error.code)); +const READ_TIMEOUT_MS = 3000; +/** How long a reading taken just before a deletion may be reused by the next deletion of the same operation. */ +const FRESH_MS = 5000; + +/** `read`'s result, or an ETIMEDOUT rejection when it has not answered in time, as a hung mount would not. */ +async function answered(read: Promise, timeoutMs: number): Promise { + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + read, + new Promise((_, fail) => { + timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), timeoutMs); + }), + ]); + } finally { + clearTimeout(timer); + } +} + /** * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader. A wrapper - * script that merely names a path is not followed. + * script that merely names a path is not followed. Every call to the directory has 3 seconds to answer. */ export function launcherReader( signal?: AbortSignal, - { directories, io = REAL_IO }: { directories: readonly string[]; io?: LauncherIo } = { directories: [] }, + { + directories, + io = REAL_IO, + timeoutMs = READ_TIMEOUT_MS, + }: { directories: readonly string[]; io?: LauncherIo; timeoutMs?: number } = { directories: [] }, ): () => Promise { let scan: Promise | undefined; const read = async (): Promise => { @@ -43,11 +66,11 @@ export function launcherReader( for (const directory of directories) { signal?.throwIfAborted(); try { - for (const name of await io.readdir(directory)) { + for (const name of await answered(io.readdir(directory), timeoutMs)) { const entry = join(directory, name); try { - if (!(await io.isSymlink(entry))) continue; - found.targets.push(await io.realpath(entry)); + if (!(await answered(io.isSymlink(entry), timeoutMs))) continue; + found.targets.push(await answered(io.realpath(entry), timeoutMs)); } catch (error) { if (!absent(error)) throw error; } @@ -63,23 +86,48 @@ export function launcherReader( } const readers = new WeakMap Promise>(); +const fresh = new WeakMap }>(); + +const newReader = ( + operation: StimContext | undefined, + signal: AbortSignal | undefined, + directories?: readonly string[], +) => launcherReader(signal ?? operation?.signal, { directories: directories ?? launcherDirectories(operation?.env) }); -/** The launcher reader of an operation, so every route of one pass reads the directories once. */ +/** The launcher reader of an operation for planning: every route of one pass shares one reading. */ export function launchersOf( operation: StimContext | undefined, signal?: AbortSignal, directories?: readonly string[], ): () => Promise { - const make = () => - launcherReader(signal ?? operation?.signal, { - directories: directories ?? launcherDirectories(operation?.env ?? {}), - }); - if (!operation) return make(); + if (!operation) return newReader(operation, signal, directories); let reader = readers.get(operation); - if (!reader) readers.set(operation, (reader = make())); + if (!reader) readers.set(operation, (reader = newReader(operation, signal, directories))); return reader; } +/** + * The launcher reader for the check that immediately precedes a deletion. It never uses the operation's planning + * reading, which can be minutes old: a launcher may have appeared or a directory become unreadable since. Deletions of + * one operation within 5 seconds of each other share one fresh reading, so many deletions in a row do not each pay the + * read (55 to 72 ms here). + */ +export function freshLaunchers( + operation: StimContext | undefined, + signal?: AbortSignal, + directories?: readonly string[], +): () => Promise { + if (!operation) return () => newReader(operation, signal, directories)(); + return () => { + const now = Date.now(); + const prior = fresh.get(operation); + if (prior && now >= prior.at && now - prior.at < FRESH_MS) return prior.scan; + const scan = newReader(operation, signal, directories)(); + fresh.set(operation, { at: now, scan }); + return scan; + }; +} + /** * Why `folder` is kept for an executable on PATH: one resolves to a path inside it, or a launcher directory could * not be inspected, so none can be ruled out. Null when every directory was read and none points into it. @@ -89,6 +137,6 @@ export function launcherReason(folder: string, scan: LauncherScan): string | nul const target = scan.targets.find((path) => path.startsWith(`${root}${sep}`)); if (target) return `an executable on PATH runs from it (${target})`; if (scan.incomplete.length) - return `an executable on PATH cannot be ruled out: ${scan.incomplete.join(', ')} could not be read`; + return `an executable on PATH cannot be ruled out: ${scan.incomplete.join(', ')} could not be read; fixing its permissions or access restores reduction`; return null; } diff --git a/packages/stim-cli/src/workspace/regenerable-dirs.ts b/packages/stim-cli/src/workspace/regenerable-dirs.ts index d7adb29866..de0605cc96 100644 --- a/packages/stim-cli/src/workspace/regenerable-dirs.ts +++ b/packages/stim-cli/src/workspace/regenerable-dirs.ts @@ -220,7 +220,11 @@ export async function regenerableDirProblem( root: string, path: string, signal?: AbortSignal, - { processes = true, launchers }: { processes?: boolean; launchers?: () => Promise } = {}, + { + processes = true, + launchers, + launcherRoot = root, + }: { processes?: boolean; launchers?: () => Promise; launcherRoot?: string } = {}, ): Promise { const rel = relative(root, path); try { @@ -242,7 +246,7 @@ export async function regenerableDirProblem( if (unignored === null) return 'git cannot tell whether it ignores every file in it'; if (unignored.trim()) return 'it holds files git does not ignore'; if (launchers) { - const launcher = launcherReason(root, await launchers()); + const launcher = launcherReason(launcherRoot, await launchers()); if (launcher) return launcher; } if ( From ee888f6c2adb0cd2e4228b1e30cae23c0fd7d31c Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 03:06:34 -0500 Subject: [PATCH 12/25] fix: read launchers with the process evidence at every deletion, trim included, and allow one outstanding launcher read --- llp/0004-discovered-checkouts.decision.md | 27 ++++--- .../src/__tests__/cargo-target.test.ts | 31 ++++++++ .../__tests__/discovered-reduce-only.test.ts | 22 ++++++ .../stim-cli/src/__tests__/launchers.test.ts | 30 ++++++-- packages/stim-cli/src/daemon/idle.ts | 18 ++++- packages/stim-cli/src/daemon/stages.ts | 8 ++- packages/stim-cli/src/guide/cleanup.ts | 7 +- .../stim-cli/src/stimd/passes/cargo-target.ts | 1 + .../stim-cli/src/workspace/cargo-target.ts | 42 ++++++++--- packages/stim-cli/src/workspace/launchers.ts | 71 ++++++++++++------- website/docs/worktrees.md | 2 +- 11 files changed, 205 insertions(+), 54 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 304d08f5b8..6ae0e524d8 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -143,14 +143,25 @@ target measurement, the pressure plan and the acting trim those on PATH and the per-user bin folders a daemon started by launchd does not list: `~/.local/bin`, `~/bin`, `~/.cargo/bin`, `~/.bun/bin`, `~/.volta/bin`, `~/Library/pnpm`, the active nvm bin, `/usr/local/bin` and `/opt/homebrew/bin`. -Planning shares one reading per operation. The check that immediately precedes a -deletion (under the lock in `clearArtifact` and `clearSourceTrees`, in -`clearStoppedOutputs`, and before `trimCargoTargetAction` trims) reads the -directories anew, and the next deletion of the same operation reuses that -reading for at most 5 seconds, so a long pass never deletes on an old reading -and a run of deletions does not each pay the read (55 to 72 ms here). A Cargo -target is checked against its owning checkout, not its app folder. Each call to a -directory has 3 seconds to answer; a timeout is an incomplete inspection. +A launcher resolving into a path is the same kind of fact as a process executing +from it, so a deletion route reads it at the same moment as the running-process +evidence, with a reading of its own, never the pass's planning reading (which +can be minutes old and is shared by the operation). The Cargo trim adds the +launcher targets under its target to the paths in use at each of its three +process reads. The artifact and dependency routes read the launchers in +`artifactReads`, after the process read and after their slow checks, in the same +`keptReason` that decides the deletion. `clearStoppedOutputs` reads them with +the idle evidence and again after measuring sizes, immediately before each +removal (`deleteOutputs`). The plan-time whole-checkout keep stays. The window +that remains is the one for a process that starts executing from the path after +the last evidence read: Stim accepts it for processes and gives launchers no +larger one. A Cargo target is checked against its owning checkout, not its app +folder. Each call to a directory has 3 seconds to answer, and a timeout ends the +reading as incomplete at once. At most one launcher read is outstanding per +thread (in the daemon only the passes thread reads launchers): while a read that +timed out is pending, every reading reports incomplete immediately without +starting a read, so a hung mount holds one libuv thread, and reduction of +reduce-only checkouts stays off until that read settles. A directory or entry that does not exist, and a dangling symlink, is no launcher. Any other failure to inspect a directory (permission, I/O, a timeout) keeps every reduce-only checkout for that operation, with a reason naming the diff --git a/packages/stim-cli/src/__tests__/cargo-target.test.ts b/packages/stim-cli/src/__tests__/cargo-target.test.ts index ea8d2b02cf..bd844762d8 100644 --- a/packages/stim-cli/src/__tests__/cargo-target.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-target.test.ts @@ -288,3 +288,34 @@ test('a trim stopped at a deadline keeps every unit it has not moved, finishes t resetExecutor(); } }); + +test('a launcher on PATH that appears after the first reading keeps what it points into at the later reading, and an incomplete reading keeps everything', async () => { + try { + makeProfile('release', 0); + mkdirSync(join(target, 'release', 'incremental', 'x'), { recursive: true }); + const real = realpathSync.native(target); + stubProcesses({}); + let readings = 0; + const launchers = async () => ({ + targets: readings++ === 0 ? [] : [`${real}/release/incremental/x/tool`], + incomplete: [], + }); + + const result = await trimCargoTarget(target, { needBytes: 1, incremental: 'all', launchers }); + + expect(readings).toBe(2); + expect(result.removed).toEqual([]); + expect(result.kept).toEqual(['release/incremental: a process uses it']); + expect(existsSync(join(target, 'release', 'incremental', 'x'))).toBe(true); + + makeProfile('debug', 3 * DAY); + const unreadable = await trimCargoTarget(target, { + needBytes: 0, + launchers: async () => ({ targets: [], incomplete: ['/locked'] }), + }); + expect(unreadable.removed).toEqual([]); + expect(existsSync(join(target, 'debug', 'deps', 'libx.rlib'))).toBe(true); + } finally { + resetExecutor(); + } +}); diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index d08d26f02a..c8977726fc 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -15,6 +15,7 @@ import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; +import * as runningFrom from '../workspace/running-from.ts'; import { testContext } from './_context.ts'; import { artifactCheckoutsAsync, @@ -311,6 +312,27 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(modules)).toBe(true); }); + it('keeps it when a launcher appears after the process read that precedes the deletion', async () => { + const { modules } = scanFoundWorktree(); + const link = join(base, 'home', 'bin', 'tool'); + mkdirSync(join(base, 'home', 'bin')); + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + expect(planned.candidates.map((entry) => entry.path)).toEqual([modules]); + const real = runningFrom.readRunningProcessesAsync; + let reads = 0; + vi.spyOn(runningFrom, 'readRunningProcessesAsync').mockImplementation(async (...args) => { + if (++reads === 2) symlinkSync(join(modules, '.bin', 'tool'), link); + return real(...args); + }); + expect(await clear(modules, shared)).toMatchObject({ + status: 'kept', + detail: expect.stringContaining('on PATH'), + }); + expect(reads).toBeGreaterThanOrEqual(2); + expect(existsSync(modules)).toBe(true); + }); + test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( 'keeps it when a launcher directory becomes unreadable after the plan, naming the directory', async () => { diff --git a/packages/stim-cli/src/__tests__/launchers.test.ts b/packages/stim-cli/src/__tests__/launchers.test.ts index 403d35575d..b915d40e6e 100644 --- a/packages/stim-cli/src/__tests__/launchers.test.ts +++ b/packages/stim-cli/src/__tests__/launchers.test.ts @@ -49,13 +49,35 @@ describe('the PATH launcher scan', () => { expect(launcherReason('/work/app', unresolved)).toContain('cannot be ruled out'); }); - it('treats a directory that does not answer in time as not inspected, and names the way to restore reduction', async () => { - const hung = { ...io({ '/bin1': ['tool'] }), readdir: () => new Promise(() => {}) }; - const pending = launcherReader(undefined, { directories: ['/hung'], io: hung, timeoutMs: 10 })(); + it('ends a reading at once as incomplete when a read does not answer, and starts no read while it is pending', async () => { + let started = 0; + let release: (names: string[]) => void = () => {}; + const stuck = { + ...io({ '/bin1': ['tool'] }), + readdir: (path: string) => { + if (path !== '/hung') return Promise.resolve([]); + started++; + return new Promise((resolve) => { + release = resolve; + }); + }, + }; + const reading = () => launcherReader(undefined, { directories: ['/hung', '/bin1'], io: stuck, timeoutMs: 10 })(); + + const first = reading(); await vi.advanceTimersByTimeAsync(10); - const result = await pending; + const result = await first; expect(result.incomplete).toEqual(['/hung']); expect(launcherReason('/work/app', result)).toContain('permissions'); + + const later = await Promise.all([reading(), reading(), reading()]); + expect(later.map((entry) => entry.incomplete)).toEqual([['/hung'], ['/hung'], ['/hung']]); + expect(started).toBe(1); + + release([]); + await vi.advanceTimersByTimeAsync(0); + const recovered = await launcherReader(undefined, { directories: ['/bin1'], io: stuck })(); + expect(recovered.incomplete).toEqual([]); }); }); diff --git a/packages/stim-cli/src/daemon/idle.ts b/packages/stim-cli/src/daemon/idle.ts index 15807eb65b..f58fee8d48 100644 --- a/packages/stim-cli/src/daemon/idle.ts +++ b/packages/stim-cli/src/daemon/idle.ts @@ -15,6 +15,7 @@ import { type IdleStageRecord, } from '@stim-cli/core/state'; import { formatBytes, measuredDirectorySizeAsync } from '../fs-util.ts'; +import { launcherReason, type LauncherScan } from '../workspace/launchers.ts'; import { collectOwnedDeviceActivityAsync, type OwnedDeviceActivity } from '../commands/gc/idle.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; import { workspaceOutputPaths } from '../commands/gc/workspaces.ts'; @@ -209,10 +210,23 @@ export async function idleOutputs(path: string, config: Config): Promise<{ paths return { paths, kept }; } -export async function deleteOutputs(paths: readonly string[]): Promise { +/** + * Deletes `paths`, measuring each first. For a reduce-only checkout `guard` reads the executables on PATH anew after the + * measurement, immediately before each removal, and keeps a path whose checkout one resolves into, saying so in `kept`. + */ +export async function deleteOutputs( + paths: readonly string[], + guard?: { checkout: string; launchers: () => Promise; kept: string[] }, +): Promise { let freed = 0; for (const path of paths) { - freed += (await measuredDirectorySizeAsync(path, { timeoutMs: SIZE_TIMEOUT_MS })) ?? 0; + const bytes = (await measuredDirectorySizeAsync(path, { timeoutMs: SIZE_TIMEOUT_MS })) ?? 0; + const launcher = guard ? launcherReason(guard.checkout, await guard.launchers()) : null; + if (launcher) { + guard?.kept.push(`kept ${path}: ${launcher}`); + continue; + } + freed += bytes; await rm(path, { recursive: true, force: true }); } return freed; diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index 543f9f6ce3..3bab016215 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -39,6 +39,7 @@ import { getProject, loadConfig, type Config, type ProjectRecord } from '../work import { checkoutRoleOf } from '../workspace/checkout-role.ts'; import { discoverCheckouts } from '../workspace/discovery.ts'; import { discoveredEvidence } from '../workspace/discovered-activity.ts'; +import { freshLaunchers } from '../workspace/launchers.ts'; import { discoveredReads, type DiscoveredReads } from '../workspace/discovered-reads.ts'; import { listedRoots } from '../commands/gc/worktrees.ts'; import { withIdleWorkspace } from '../workspace/in-use.ts'; @@ -352,7 +353,12 @@ export async function clearStoppedOutputs( return { outcome: 'kept', bytes: 0, detail: 'it is no longer idle' }; const outputs = await idleOutputs(path, config); if (!outputs.paths.length) return null; - const bytes = await deleteOutputs(outputs.paths); + const bytes = await deleteOutputs( + outputs.paths, + candidate.reduceOnly + ? { checkout: path, launchers: freshLaunchers(operation), kept: outputs.kept } + : undefined, + ); return { outcome: 'done', bytes, diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index dbbc1904b1..7195004846 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1247,9 +1247,10 @@ resolves into is kept whole, by every pass that deletes its outputs, dependencies or Cargo target. A directory that cannot be read (not one that is missing, or one that does not answer within 3 seconds) keeps every reduce-only checkout for that pass, and the reason names it and says fixing its permissions -restores reduction. The check before each deletion reads the directories anew -(reused for at most 5 seconds by the next deletion), not the pass's planning -reading. Reduce-only checkouts have their build outputs and, in +restores reduction. Each deletion reads the directories anew, at the moment it +reads the processes using the path (not the pass's planning reading); a +directory that hangs ends the reading as incomplete at once, and while that read +is pending no other launcher read starts. Reduce-only checkouts have their build outputs and, in a linked worktree, node_modules and Pods reduced by the rules below. Passes evaluate at most 40 of them at a time, in a window that moves every 15 minutes; stim gc --dry-run lists all of them and says which are left for a later pass. A diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index dac4de64ed..db60874875 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -448,6 +448,7 @@ export async function trimCargoTargetAction( signal: operation?.signal, measure, ...(stop ? { stop } : {}), + ...(managed.reduceOnly ? { launchers: freshLaunchers(operation) } : {}), }); if (!measure) { const after = before ? await targetVolume(action.target, operation?.signal) : null; diff --git a/packages/stim-cli/src/workspace/cargo-target.ts b/packages/stim-cli/src/workspace/cargo-target.ts index fb7046d4d2..58dc2f508c 100644 --- a/packages/stim-cli/src/workspace/cargo-target.ts +++ b/packages/stim-cli/src/workspace/cargo-target.ts @@ -6,6 +6,7 @@ import { releaseClaim, tryAcquireClaim, type ClaimHandle } from '../ownership-cl import { getExecutor } from '../exec.ts'; import { measuredDirectorySizeAsync } from '../fs-util.ts'; import { withCargoLocks } from './cargo-lock.ts'; +import type { LauncherScan } from './launchers.ts'; import { cargoUnitPaths, cargoUnitWasRead, idleCargoUnits, readCargoUnits, removalOrder } from './cargo-units.ts'; const CARGO_PROFILE_IDLE_MS: number = 24 * 3_600_000; @@ -299,10 +300,16 @@ function inUse(used: readonly string[], dir: string): boolean { } /** - * The paths under `target` a running process executes from or holds open, from one `lsof` and one `ps` read; null when - * either cannot be read, which keeps everything. + * The paths under `target` a running process executes from or holds open, from one `lsof` and one `ps` read, and for a + * reduce-only checkout the paths under it that an executable on PATH resolves to, from a launcher reading taken at the + * same moment; null when any of them cannot be read, which keeps everything. */ -async function readTargetUse(target: string, signal?: AbortSignal, lowPriority = false): Promise { +async function readTargetUse( + target: string, + signal?: AbortSignal, + lowPriority = false, + launchers?: () => Promise, +): Promise { const exec = getExecutor(); try { const run = (file: string, args: string[], timeoutMs: number) => @@ -311,7 +318,13 @@ async function readTargetUse(target: string, signal?: AbortSignal, lowPriority = : exec.runFileAsync(file, args, { timeoutMs, signal }); const lsof = await run('lsof', ['-nP', '-w', '-Fn'], 60_000); const ps = await run('ps', ['-axww', '-o', 'args='], 30_000); - return targetPathsInUse(target, lsof, ps); + const used = targetPathsInUse(target, lsof, ps); + if (!launchers) return used; + const scan = await launchers(); + if (scan.incomplete.length) return null; + const prefix = `${folded(target)}/`; + for (const path of scan.targets) if (folded(path).startsWith(prefix)) used.push(folded(path)); + return used; } catch { signal?.throwIfAborted(); return null; @@ -385,6 +398,11 @@ interface CargoTrimOptions { stop?: AbortSignal; /** Whether to walk what was removed to report its bytes; false reports 0 and removes at once. Default true. */ measure?: boolean; + /** + * For a reduce-only checkout: a fresh reading of the executables on PATH, taken wherever the processes using the target + * are read, so a path one of them resolves to is kept like a path a process runs from. + */ + launchers?: () => Promise; } async function trimClaimed( @@ -398,9 +416,13 @@ async function trimClaimed( signal, stop, measure = true, + launchers, }: CargoTrimOptions, ): Promise { const result: CargoTrimResult = { bytes: 0, removed: [], kept: [] }; + const cannotRead = launchers + ? 'the files processes hold open or the executables on PATH cannot be read' + : 'the files processes hold open cannot be read'; const stopped = (): boolean => stop?.aborted === true; const measured = (path: string): Promise => measure ? sizeOf(path, signal, lowPriority) : Promise.resolve(0); @@ -419,8 +441,8 @@ async function trimClaimed( kept: [`a leftover ${aside} cannot be deleted: ${error instanceof Error ? error.message : error}`], }; } - const first = await readTargetUse(target, signal, lowPriority); - if (first === null) return { ...result, kept: ['the files processes hold open cannot be read'] }; + const first = await readTargetUse(target, signal, lowPriority, launchers); + if (first === null) return { ...result, kept: [cannotRead] }; let used: string[] = first; const stale = staleCargoTargetPaths(target, now); if ('refusal' in stale) return { ...result, kept: [stale.refusal] }; @@ -504,8 +526,8 @@ async function trimClaimed( if (!all) return { ...none, kept: 'its units cannot be listed' }; const candidates = [...cargoUnitPaths(profile, idleCargoUnits(all, now, CARGO_PROFILE_IDLE_MS))]; if (!candidates.length) return none; - const fresh = await readTargetUse(target, signal, lowPriority); - if (fresh === null) return { ...none, kept: 'the files processes hold open cannot be read' }; + const fresh = await readTargetUse(target, signal, lowPriority, launchers); + if (fresh === null) return { ...none, kept: cannotRead }; let swept = 0; let bytes = 0; // @ref LLP 0005#never-delete-what-a-build-took-back -- Exact serializes access-time resets and ledger replacement @@ -577,11 +599,11 @@ async function trimClaimed( const refreshed = stopped() ? null : incremental === 'all' || result.bytes < needBytes - ? await readTargetUse(target, signal, lowPriority) + ? await readTargetUse(target, signal, lowPriority, launchers) : used; if (stopped()) { if (!result.kept.includes(DEADLINE_KEPT)) result.kept.push(DEADLINE_KEPT); - } else if (refreshed === null) result.kept.push('the files processes hold open cannot be read'); + } else if (refreshed === null) result.kept.push(cannotRead); else used = refreshed; if (refreshed !== null && (incremental === 'all' || result.bytes < needBytes)) { const candidates: { path: string; bytes: number }[] = []; diff --git a/packages/stim-cli/src/workspace/launchers.ts b/packages/stim-cli/src/workspace/launchers.ts index b06ff9bfee..31be961e76 100644 --- a/packages/stim-cli/src/workspace/launchers.ts +++ b/packages/stim-cli/src/workspace/launchers.ts @@ -30,19 +30,39 @@ const ABSENT = ['ENOENT', 'ENOTDIR']; const absent = (error: unknown) => error instanceof Error && 'code' in error && ABSENT.includes(String(error.code)); const READ_TIMEOUT_MS = 3000; -/** How long a reading taken just before a deletion may be reused by the next deletion of the same operation. */ -const FRESH_MS = 5000; -/** `read`'s result, or an ETIMEDOUT rejection when it has not answered in time, as a hung mount would not. */ -async function answered(read: Promise, timeoutMs: number): Promise { +/** + * The one launcher read that timed out and has not settled. While it is pending, no new read starts: a hung mount holds + * exactly one libuv thread however many readings are asked for, and every reading reports it incomplete at once. + * Module state, so the bound is per thread of the process; in the daemon only the passes thread reads launchers. + */ +let hung: { directory: string } | null = null; +/** Readings run one at a time, so the single read in flight is the only one outstanding. */ +let queue: Promise = Promise.resolve(); + +class Unanswered extends Error {} + +/** `read`'s result, or an `Unanswered` rejection when it has not answered in time; the read itself cannot be cancelled. */ +async function answered(directory: string, start: () => Promise, timeoutMs: number): Promise { + const read = start(); let timer: NodeJS.Timeout | undefined; try { return await Promise.race([ read, new Promise((_, fail) => { - timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), timeoutMs); + timer = setTimeout(() => fail(new Unanswered('timed out')), timeoutMs); }), ]); + } catch (error) { + if (error instanceof Unanswered) { + const mine = { directory }; + hung = mine; + const settled = () => { + if (hung === mine) hung = null; + }; + read.then(settled, settled); + } + throw error; } finally { clearTimeout(timer); } @@ -50,7 +70,8 @@ async function answered(read: Promise, timeoutMs: number): Promise { /** * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader. A wrapper - * script that merely names a path is not followed. Every call to the directory has 3 seconds to answer. + * script that merely names a path is not followed. Every call to the directory has 3 seconds to answer; one that does + * not ends the reading at once as incomplete, and no reading starts a read while an earlier one is still pending. */ export function launcherReader( signal?: AbortSignal, @@ -65,28 +86,37 @@ export function launcherReader( const found: LauncherScan = { targets: [], incomplete: [] }; for (const directory of directories) { signal?.throwIfAborted(); + if (hung) return { targets: found.targets, incomplete: [...found.incomplete, hung.directory] }; try { - for (const name of await answered(io.readdir(directory), timeoutMs)) { + for (const name of await answered(directory, () => io.readdir(directory), timeoutMs)) { const entry = join(directory, name); try { - if (!(await answered(io.isSymlink(entry), timeoutMs))) continue; - found.targets.push(await answered(io.realpath(entry), timeoutMs)); + if (!(await answered(directory, () => io.isSymlink(entry), timeoutMs))) continue; + found.targets.push(await answered(directory, () => io.realpath(entry), timeoutMs)); } catch (error) { if (!absent(error)) throw error; } } } catch (error) { signal?.throwIfAborted(); - if (!absent(error)) found.incomplete.push(directory); + if (absent(error)) continue; + found.incomplete.push(directory); + if (error instanceof Unanswered) return found; } } return found; }; - return () => (scan ??= read()); + return () => { + if (!scan) { + const run = queue.then(read, read); + queue = run; + scan = run; + } + return scan; + }; } const readers = new WeakMap Promise>(); -const fresh = new WeakMap }>(); const newReader = ( operation: StimContext | undefined, @@ -107,25 +137,16 @@ export function launchersOf( } /** - * The launcher reader for the check that immediately precedes a deletion. It never uses the operation's planning - * reading, which can be minutes old: a launcher may have appeared or a directory become unreadable since. Deletions of - * one operation within 5 seconds of each other share one fresh reading, so many deletions in a row do not each pay the - * read (55 to 72 ms here). + * A launcher reader that reads anew on every call, never the operation's planning reading, which can be minutes old. A + * deletion route calls it at the moment it reads the processes that use the path, and no earlier, so a launcher is + * treated like a running process: the window that remains is the one after that read. */ export function freshLaunchers( operation: StimContext | undefined, signal?: AbortSignal, directories?: readonly string[], ): () => Promise { - if (!operation) return () => newReader(operation, signal, directories)(); - return () => { - const now = Date.now(); - const prior = fresh.get(operation); - if (prior && now >= prior.at && now - prior.at < FRESH_MS) return prior.scan; - const scan = newReader(operation, signal, directories)(); - fresh.set(operation, { at: now, scan }); - return scan; - }; + return () => newReader(operation, signal, directories)(); } /** diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 417ce6894b..fa67588ddc 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. If one of those folders exists but cannot be read, such checkouts are kept for that pass. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher read starts; fixing the folder restores reduction. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 3b03132acb5a831559c0d5b0206e371912561322 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 03:49:59 -0500 Subject: [PATCH 13/25] fix: make the reduce-only flag a required argument on every launcher evidence path, bound hung home scan reads with the launcher guard, and test the trim and deleteOutputs guards --- llp/0004-discovered-checkouts.decision.md | 18 ++- .../src/__tests__/cargo-target-cap.test.ts | 2 + .../src/__tests__/cargo-target.test.ts | 27 ++-- .../src/__tests__/cargo-units.test.ts | 31 +++-- .../__tests__/discovered-reduce-only.test.ts | 120 ++++++++++++++++++ .../__tests__/discovered-worktrees.test.ts | 8 +- .../src/__tests__/discovery-scan.test.ts | 49 ++++--- .../stim-cli/src/__tests__/launchers.test.ts | 12 ++ .../stim-cli/src/commands/gc/workspaces.ts | 7 +- .../src/commands/gc/worktree-sizes.ts | 8 +- packages/stim-cli/src/daemon/dependencies.ts | 27 +++- packages/stim-cli/src/daemon/idle.ts | 24 ++-- packages/stim-cli/src/daemon/ladder.ts | 13 +- packages/stim-cli/src/daemon/passes.ts | 11 +- packages/stim-cli/src/daemon/stages.ts | 54 ++++---- packages/stim-cli/src/guide/cleanup.ts | 9 +- .../stim-cli/src/stimd/passes/artifacts.ts | 47 ++++--- .../stim-cli/src/stimd/passes/cargo-target.ts | 31 +++-- .../stim-cli/src/stimd/passes/level-run.ts | 2 +- .../stim-cli/src/workspace/bounded-read.ts | 43 +++++++ .../stim-cli/src/workspace/cargo-target.ts | 11 +- .../src/workspace/discovered-activity.ts | 4 +- .../src/workspace/discovered-removal.ts | 6 +- .../stim-cli/src/workspace/discovery-scan.ts | 63 +++++---- packages/stim-cli/src/workspace/launchers.ts | 72 +++++------ .../src/workspace/regenerable-dirs.ts | 14 +- website/docs/worktrees.md | 2 +- 27 files changed, 500 insertions(+), 215 deletions(-) create mode 100644 packages/stim-cli/src/workspace/bounded-read.ts diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 6ae0e524d8..30d7d4be19 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -157,11 +157,19 @@ that remains is the one for a process that starts executing from the path after the last evidence read: Stim accepts it for processes and gives launchers no larger one. A Cargo target is checked against its owning checkout, not its app folder. Each call to a directory has 3 seconds to answer, and a timeout ends the -reading as incomplete at once. At most one launcher read is outstanding per -thread (in the daemon only the passes thread reads launchers): while a read that -timed out is pending, every reading reports incomplete immediately without -starting a read, so a hung mount holds one libuv thread, and reduction of -reduce-only checkouts stays off until that read settles. +reading as incomplete at once. A native read cannot be cancelled, so at most one +read that timed out is outstanding per JavaScript isolate, shared by the launcher +readings and the home scan (`bounded-read.ts`): while it is pending, no further +launcher or scan read starts, every launcher reading reports incomplete +immediately naming the folder, and a scheduled scan is skipped. A hung mount +therefore holds one libuv thread per isolate, and reduction of reduce-only +checkouts stays off until that read settles. The daemon has two isolates that +read launchers, the main thread (the dormant stage and the pressure ladder, which +reach `clearStoppedOutputs`) and the passes thread (the artifact, dependency and +Cargo routes), so up to two reads can be outstanding in the daemon; the scan runs +only on the passes thread, and CLI commands are separate processes. A scan stops +at the first read that times out: the result is truncated and the previous finds +are kept. A directory or entry that does not exist, and a dangling symlink, is no launcher. Any other failure to inspect a directory (permission, I/O, a timeout) keeps every reduce-only checkout for that operation, with a reason naming the diff --git a/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts b/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts index d26c894783..2036917f91 100644 --- a/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts @@ -31,6 +31,8 @@ const candidate = ( dir, identity, discovered, + reduceOnly: false, + checkout: dir.replace(/\/target$/, ''), measuredAt, }); diff --git a/packages/stim-cli/src/__tests__/cargo-target.test.ts b/packages/stim-cli/src/__tests__/cargo-target.test.ts index bd844762d8..f32e6b8c94 100644 --- a/packages/stim-cli/src/__tests__/cargo-target.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-target.test.ts @@ -105,7 +105,7 @@ test('trimming removes the stale profiles, keeps the rest and leaves nothing asi makeProfile('aarch64-linux-android/host-dev', 3 * DAY); makeProfile('release', 0); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed.toSorted()).toEqual(['aarch64-linux-android', 'debug']); expect(existsSync(join(target, 'debug'))).toBe(false); @@ -139,7 +139,7 @@ test('a profile a process works in or runs a binary from stays, and an unreadabl ps: () => `${real}/release/deps/mytest-0123456789abcdef --nocapture\n/bin/zsh -l\n`, }); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed).toEqual(['host-dev']); expect(result.kept).toEqual(['debug: a process uses it', 'release: a process uses it']); @@ -150,7 +150,7 @@ test('a profile a process works in or runs a binary from stays, and an unreadabl }, }); makeProfile('profile-b', 3 * DAY); - const unreadable = await trimCargoTarget(target, { needBytes: 0 }); + const unreadable = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(unreadable.removed).toEqual([]); expect(existsSync(join(target, 'profile-b', 'deps', 'libx.rlib'))).toBe(true); } finally { @@ -175,7 +175,7 @@ test('a directory that cannot be put back after a build took its lock is never d return '4242\n'; }, }); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed).toEqual([]); expect(result.kept.join(' ')).toContain('could not be put back'); @@ -198,7 +198,7 @@ test('a profile whose Cargo 1.97 build lock is held stays, whichever of the lock lsof: (args) => (args.includes('-t') && args.some((arg) => arg.endsWith('.cargo-build-lock')) ? '4242\n' : ''), }); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed).toEqual([]); expect(result.kept).toEqual(['debug: a build holds its .cargo-build-lock']); @@ -229,13 +229,13 @@ test('a second trim of a target, from another Stim home, keeps what the first mo return file === 'lsof' || file === 'ps' ? '' : real.runFileAsync(file, args, options); }, }); - const first = trimCargoTarget(target, { needBytes: 0 }); + const first = trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); await reachedPause; const staged = join(realpathSync.native(target), '.stim-trimmed', '0', '.fingerprint'); expect(existsSync(staged)).toBe(true); vi.stubEnv('STIM_HOME', otherHome); - const second = await trimCargoTarget(target, { needBytes: 0 }); + const second = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(second.removed).toEqual([]); expect(second.kept.join(' ')).toContain('another trim of it holds'); @@ -268,6 +268,7 @@ test('a trim stopped at a deadline keeps every unit it has not moved, finishes t }); const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, needBytes: 0, incremental: 'all', measure: false, @@ -301,7 +302,11 @@ test('a launcher on PATH that appears after the first reading keeps what it poin incomplete: [], }); - const result = await trimCargoTarget(target, { needBytes: 1, incremental: 'all', launchers }); + const result = await trimCargoTarget(target, { + needBytes: 1, + incremental: 'all', + launching: { reduceOnly: true, launchers, checkout: target }, + }); expect(readings).toBe(2); expect(result.removed).toEqual([]); @@ -311,7 +316,11 @@ test('a launcher on PATH that appears after the first reading keeps what it poin makeProfile('debug', 3 * DAY); const unreadable = await trimCargoTarget(target, { needBytes: 0, - launchers: async () => ({ targets: [], incomplete: ['/locked'] }), + launching: { + reduceOnly: true, + launchers: async () => ({ targets: [], incomplete: ['/locked'] }), + checkout: target, + }, }); expect(unreadable.removed).toEqual([]); expect(existsSync(join(target, 'debug', 'deps', 'libx.rlib'))).toBe(true); diff --git a/packages/stim-cli/src/__tests__/cargo-units.test.ts b/packages/stim-cli/src/__tests__/cargo-units.test.ts index 9871fb7f39..33bc7b5e46 100644 --- a/packages/stim-cli/src/__tests__/cargo-units.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-units.test.ts @@ -117,7 +117,12 @@ describe('a profile on disk with Exact ledger', () => { writeLedger(); stub({}); - const result = await trimCargoTarget(target, { needBytes: 0, incremental: 'all', units: 'all' }); + const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, + needBytes: 0, + incremental: 'all', + units: 'all', + }); expect(result.removed).toContain('debug: 1 idle units'); expect(readdirSync(join(profile(), '.fingerprint')).filter((name) => name.startsWith('foo-'))).toHaveLength(8); @@ -132,7 +137,12 @@ describe('a profile on disk with Exact ledger', () => { stub({}); expect(idleCargoUnitPaths(target, Date.now())).toEqual([]); - const result = await trimCargoTarget(target, { needBytes: 0, incremental: 'all', units: 'all' }); + const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, + needBytes: 0, + incremental: 'all', + units: 'all', + }); expect(result.removed).not.toContain('debug: 1 idle units'); expect(existsSync(join(profile(), 'deps', `libserde-${h(1)}.rlib`))).toBe(true); @@ -146,7 +156,12 @@ describe('a profile on disk with Exact ledger', () => { stub({}); expect(idleCargoUnitPaths(target, Date.now())).toEqual([]); - const result = await trimCargoTarget(target, { needBytes: 0, incremental: 'all', units: 'all' }); + const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, + needBytes: 0, + incremental: 'all', + units: 'all', + }); expect(result.removed).not.toContain('debug: 1 idle units'); expect(existsSync(join(profile(), 'deps', `libserde-${h(1)}.rlib`))).toBe(true); @@ -160,7 +175,7 @@ describe('a profile on disk with Exact ledger', () => { writeLedger(); stub({ ps: () => `${join(profile(), 'deps', `tokio-${h(3)}.d`)} --nocapture\n` }); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual(['debug: 1 idle units']); expect(existsSync(join(profile(), 'deps', `libtokio-${h(3)}.rlib`))).toBe(true); @@ -178,7 +193,7 @@ describe('a profile on disk with Exact ledger', () => { writeLedger(); stub({}); - const sweep = () => trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const sweep = () => trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); const locked = await withCargoLocks([join(target, dir, lock)], sweep); expect(locked).toMatchObject({ value: { removed: [], kept: [expect.stringContaining(`holds ${lock}`)] } }); @@ -193,7 +208,7 @@ describe('a profile on disk with Exact ledger', () => { mkdirSync(join(profile(), '.cargo-build-lock')); stub({}); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual([]); expect(result.kept.join(' ')).toContain('.cargo-build-lock could not be locked'); @@ -213,7 +228,7 @@ describe('a profile on disk with Exact ledger', () => { }, }); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual([]); expect(existsSync(fp)).toBe(true); @@ -234,7 +249,7 @@ describe('a profile on disk with Exact ledger', () => { }, }); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual(['debug: 25 idle units']); expect(result.kept.join(' ')).toContain('Exact updated its ledger'); diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index c8977726fc..6ce055fa17 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -15,8 +15,13 @@ import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; +import * as idleStop from '../stimd/passes/idle-stop.ts'; +import { clearDependency } from '../daemon/dependencies.ts'; import * as runningFrom from '../workspace/running-from.ts'; import { testContext } from './_context.ts'; +import * as fsUtil from '../fs-util.ts'; +import { deleteOutputs } from '../daemon/idle.ts'; +import { getExecutor, resetExecutor, setExecutor } from '../exec.ts'; import { artifactCheckoutsAsync, artifactWorktreesAsync, @@ -214,6 +219,76 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(trim).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); }); + it('keeps what a launcher that appears during the trim own process read points into, and trims it when none appears', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(join(tool, 'ios', 'App.xcodeproj'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', 'incremental', 'x'), { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(tool, '.gitignore'), 'target/\n'); + writeFileSync(join(tool, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'), 'x'.repeat(10_000)); + execFileSync('git', ['-C', tool, 'init', '-q']); + execFileSync('git', ['-C', tool, 'add', '-A']); + execFileSync('git', ['-C', tool, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x']); + writeScan([tool]); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + mkdirSync(join(base, 'home', 'bin')); + const link = join(base, 'home', 'bin', 'tool'); + const real = getExecutor(); + const trimWith = async (appears: boolean) => { + setExecutor({ + ...real, + runFileAsync: async (file, args, options) => { + if (file === 'ps') return ''; + if (file !== 'lsof') return real.runFileAsync(file, args, options); + if (appears && !existsSync(link)) + symlinkSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'), link); + return ''; + }, + }); + const planned = await planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + const [action] = planned.actions; + if (!action) throw new Error('no trim was planned'); + return trimCargoTargetAction(action, testContext({ env: {} })); + }; + try { + expect(await trimWith(true)).toMatchObject({ status: 'kept' }); + expect(existsSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'))).toBe(true); + rmSync(link); + expect(await trimWith(false)).toMatchObject({ status: 'done' }); + expect(existsSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'))).toBe(false); + } finally { + resetExecutor(); + } + }); + + it('keeps an output a launcher resolves into when it appears while deleteOutputs measures, and deletes it otherwise', async () => { + const checkout = join(base, 'home', 'tools', 'out'); + const output = join(checkout, 'build'); + mkdirSync(output, { recursive: true }); + writeFileSync(join(output, 'tool'), 'x'); + let measured = false; + vi.spyOn(fsUtil, 'measuredDirectorySizeAsync').mockImplementation(async () => { + measured = true; + return 1; + }); + const launchers = async () => ({ targets: measured ? [join(output, 'tool')] : [], incomplete: [] }); + const kept: string[] = []; + expect(await deleteOutputs([output], { reduceOnly: true, launchers, checkout }, kept)).toBe(0); + expect(existsSync(output)).toBe(true); + expect(kept).toEqual([expect.stringContaining('on PATH')]); + + expect(await deleteOutputs([output], { reduceOnly: false }, kept)).toBe(1); + expect(existsSync(output)).toBe(false); + }); + it('keeps a nested app Cargo target when a PATH launcher resolves into the checkout outside that app, in the plan and at the trim', async () => { const mono = join(base, 'home', 'tools', 'mono'); const app = join(mono, 'apps', 'native'); @@ -333,6 +408,51 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(modules)).toBe(true); }); + describe('through clearDependency, whose final recheck is the idleness evidence', () => { + const clearThroughRecheck = async (appears: boolean) => { + const { modules } = scanFoundWorktree(); + const link = join(base, 'home', 'bin', 'tool'); + mkdirSync(join(base, 'home', 'bin')); + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + const [candidate] = planned.candidates; + if (!candidate) throw new Error('no dependency candidate'); + const real = idleStop.idleStopEvidence; + vi.spyOn(idleStop, 'idleStopEvidence').mockImplementation(async (...args) => { + if (appears) symlinkSync(join(modules, '.bin', 'tool'), link); + return real(...args); + }); + const outcome = await clearDependency( + { + candidate, + source: false, + workspace: candidate.keys[0] ?? candidate.worktree, + bytes: null, + privateBytes: null, + }, + day, + '', + await artifactWorktreesAsync(signal, shared), + 'test', + shared, + false, + ); + return { outcome, modules }; + }; + + it('clears it when no launcher appears', async () => { + const { outcome, modules } = await clearThroughRecheck(false); + expect(outcome).toMatchObject({ status: 'done' }); + expect(existsSync(modules)).toBe(false); + }); + + it('keeps it when a launcher appears during the recheck', async () => { + const { outcome, modules } = await clearThroughRecheck(true); + expect(outcome).toMatchObject({ status: 'kept', detail: 'it is no longer idle' }); + expect(existsSync(modules)).toBe(true); + }); + }); + test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( 'keeps it when a launcher directory becomes unreadable after the plan, naming the directory', async () => { diff --git a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts index 71b4ed45bf..b70c542eb6 100644 --- a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts @@ -162,10 +162,16 @@ describe('a discovered worktree on the removal path', () => { main, idleMs: day, reads: reads.discoveredReads(), + reduceOnly: false, }); expect(hold).toContain('managed by another tool'); expect( - await discoveredRemovalHold(join(main, '.worktrees', 'x'), { main, idleMs: day, reads: reads.discoveredReads() }), + await discoveredRemovalHold(join(main, '.worktrees', 'x'), { + main, + idleMs: day, + reads: reads.discoveredReads(), + reduceOnly: false, + }), ).not.toContain('another tool'); }); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index 9636764d8e..ba1eeb20ab 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -9,7 +9,6 @@ import { symlinkSync, writeFileSync, } from 'node:fs'; -import { lstat } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { testContext } from './_context.ts'; @@ -97,26 +96,6 @@ describe('scanForCheckouts', () => { } }); - it('stops reading siblings at once when its budget is spent, as repeated timed-out stats spend it', async () => { - for (let i = 0; i < 100; i++) project(`p${String(i).padStart(3, '0')}`); - let stats = 0; - const hung = async (path: string) => { - if (path === home) return lstat(path); - stats++; - return new Promise(() => {}); - }; - const pending = scanForCheckouts(home, marked, { budgetMs: 40, readTimeoutMs: 10, stat: hung }); - const state = { done: false }; - void pending.then(() => (state.done = true)); - while (!state.done) { - if (stats > 0) await vi.advanceTimersByTimeAsync(10); - await new Promise((resolve) => setImmediate(resolve)); - } - const scan = await pending; - expect(scan.truncated).toBe(true); - expect(stats).toBeLessThan(10); - }); - it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); @@ -217,6 +196,34 @@ describe('runDiscoveryScanIfDue', () => { rmSync(base, { recursive: true, force: true }); }); + it('stops at the first read that does not answer, starts no read while it is pending, and scans again once it settles', async () => { + let started = 0; + let release: (names: []) => void = () => {}; + const hungEntries = () => { + started++; + return new Promise<[]>((resolve) => { + release = resolve; + }); + }; + const home = join(base, 'home'); + const first = scanForCheckouts(home, marked, { entries: hungEntries, readTimeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await first).toMatchObject({ truncated: true, unreadable: [home], found: [] }); + + const later = await scanForCheckouts(home, marked, { entries: hungEntries, readTimeoutMs: 10 }); + expect(later.truncated).toBe(true); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toBeNull(); + expect(started).toBe(1); + + release([]); + await vi.advanceTimersByTimeAsync(0); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ found: [join(base, 'home', 'dev', 'app')], truncated: false }); + }); + it('never scans for a command, only for a daemon tick', async () => { await runDiscoveryScanIfDue(undefined, options(t0)); expect(readDiscoveryScan()).toBeNull(); diff --git a/packages/stim-cli/src/__tests__/launchers.test.ts b/packages/stim-cli/src/__tests__/launchers.test.ts index b915d40e6e..358cff856f 100644 --- a/packages/stim-cli/src/__tests__/launchers.test.ts +++ b/packages/stim-cli/src/__tests__/launchers.test.ts @@ -1,4 +1,5 @@ import { launcherDirectories } from '../workspace/launcher-directories.ts'; +import { scanForCheckouts } from '../workspace/discovery-scan.ts'; import { launcherReader, launcherReason } from '../workspace/launchers.ts'; const missing = (code: string) => Object.assign(new Error(code), { code }); @@ -74,6 +75,17 @@ describe('the PATH launcher scan', () => { expect(later.map((entry) => entry.incomplete)).toEqual([['/hung'], ['/hung'], ['/hung']]); expect(started).toBe(1); + let scanReads = 0; + const homeScan = await scanForCheckouts('/', () => false, { + stat: async () => ({ dev: 1 }), + entries: async () => { + scanReads++; + return []; + }, + }); + expect(homeScan.truncated).toBe(true); + expect(scanReads).toBe(0); + release([]); await vi.advanceTimersByTimeAsync(0); const recovered = await launcherReader(undefined, { directories: ['/bin1'], io: stuck })(); diff --git a/packages/stim-cli/src/commands/gc/workspaces.ts b/packages/stim-cli/src/commands/gc/workspaces.ts index 0c29fe0549..5db730c079 100644 --- a/packages/stim-cli/src/commands/gc/workspaces.ts +++ b/packages/stim-cli/src/commands/gc/workspaces.ts @@ -366,7 +366,7 @@ export async function workspaceBuildReasonsAsync( async function inTreeBuildDirs(root: string, signal?: AbortSignal): Promise { const dirs: string[] = []; for (const path of await existingBuildDirsAsync(root)) - if (!(await regenerableDirProblem(root, path, signal))) dirs.push(path); + if (!(await regenerableDirProblem(root, path, signal, { launching: { reduceOnly: false } }))) dirs.push(path); return dirs; } @@ -661,7 +661,10 @@ export async function clearWorkspaceOutputs( if (again?.keptReason) return again.keptReason; const inTree: { path: string; problem: string | null }[] = []; for (const path of lighten ? await existingBuildDirsAsync(root) : []) - inTree.push({ path, problem: await regenerableDirProblem(root, path, signal) }); + inTree.push({ + path, + problem: await regenerableDirProblem(root, path, signal, { launching: { reduceOnly: false } }), + }); for (const path of [ ...workspaceOutputPaths(entry.dir), ...inTree.filter((dir) => !dir.problem).map((dir) => dir.path), diff --git a/packages/stim-cli/src/commands/gc/worktree-sizes.ts b/packages/stim-cli/src/commands/gc/worktree-sizes.ts index 03e043b411..053fede546 100644 --- a/packages/stim-cli/src/commands/gc/worktree-sizes.ts +++ b/packages/stim-cli/src/commands/gc/worktree-sizes.ts @@ -53,7 +53,13 @@ async function buildStateDirs(path: string, keys: readonly string[], operation?: }); const own = keys.filter((key) => matchWorktreeEntry(entries, key)?.path === path); const root = canonicalPath(path); - const worktree = { path, linked: true, locked: Boolean(matchWorktreeEntry(entries, path)?.locked), keys: own }; + const worktree = { + path, + linked: true, + locked: Boolean(matchWorktreeEntry(entries, path)?.locked), + keys: own, + reduceOnly: false, + }; const dirs = new Map(); const privateSized = new Set(); const reportOnly = new Set(); diff --git a/packages/stim-cli/src/daemon/dependencies.ts b/packages/stim-cli/src/daemon/dependencies.ts index 7ec60b8fdc..956e775b1a 100644 --- a/packages/stim-cli/src/daemon/dependencies.ts +++ b/packages/stim-cli/src/daemon/dependencies.ts @@ -194,7 +194,10 @@ export async function rankSourceDependencies( }); continue; } - const verdict = await sourceCheckoutIdle(candidate.worktree, rule.thresholdMs, rule.setting, operation); + const verdict = await sourceCheckoutIdle(candidate.worktree, rule.thresholdMs, rule.setting, { + operation, + reduceOnly: candidate.reduceOnly, + }); if ('kept' in verdict) { kept.push({ target: candidate.path, workspace, reason: verdict.kept }); continue; @@ -242,15 +245,19 @@ async function stillUnused( keys: readonly string[], worktree: string, idleMs: number, + reduceOnly: boolean, operation?: StimContext, setting?: string, ): Promise { const [first] = keys; if (!source) - return first && !(await stillIdle(first, idleMs, operation, false, true, setting)) ? 'it is no longer idle' : null; + return first && !(await stillIdle(first, idleMs, { operation, setting, reduceOnly })) + ? 'it is no longer idle' + : null; const held = new Set(workspaceKeys(keys)); for (const key of new Set([...held, worktree])) - if (!(await stillIdle(key, idleMs, operation, true, held.has(key), setting))) return `${key} is no longer idle`; + if (!(await stillIdle(key, idleMs, { operation, full: true, locked: held.has(key), setting, reduceOnly }))) + return `${key} is no longer idle`; return Date.now() - warmedFromAt(keys) < idleMs ? 'a worktree warm copied from it since' : null; } @@ -284,7 +291,8 @@ export async function clearDependency( pinned: maintenancePinnedReader(operation), worktrees, requireRestorable, - recheck: (keys, worktree) => stillUnused(false, keys, worktree, idleMs, operation), + recheck: (keys, worktree) => + stillUnused(false, keys, worktree, idleMs, dependency.candidate.reduceOnly, operation), level, operation, }, @@ -341,7 +349,16 @@ export async function clearSourceDependencies( protectedRoot, pinned: maintenancePinnedReader(operation), worktrees, - recheck: (keys, worktree) => stillUnused(true, keys, worktree, idleMs, operation, setting), + recheck: (keys, worktree) => + stillUnused( + true, + keys, + worktree, + idleMs, + dependencies.some(({ candidate }) => candidate.reduceOnly), + operation, + setting, + ), level, operation, }, diff --git a/packages/stim-cli/src/daemon/idle.ts b/packages/stim-cli/src/daemon/idle.ts index f58fee8d48..8e767e72e7 100644 --- a/packages/stim-cli/src/daemon/idle.ts +++ b/packages/stim-cli/src/daemon/idle.ts @@ -15,7 +15,7 @@ import { type IdleStageRecord, } from '@stim-cli/core/state'; import { formatBytes, measuredDirectorySizeAsync } from '../fs-util.ts'; -import { launcherReason, type LauncherScan } from '../workspace/launchers.ts'; +import { launcherReason, type Launching } from '../workspace/launchers.ts'; import { collectOwnedDeviceActivityAsync, type OwnedDeviceActivity } from '../commands/gc/idle.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; import { workspaceOutputPaths } from '../commands/gc/workspaces.ts'; @@ -187,11 +187,15 @@ async function decide( } /** The regenerable outputs the idle stage deletes, and why each other one stays. */ -export async function idleOutputs(path: string, config: Config): Promise<{ paths: string[]; kept: string[] }> { +export async function idleOutputs( + path: string, + config: Config, + launching: Launching, +): Promise<{ paths: string[]; kept: string[] }> { const paths = workspaceOutputPaths(workspaceDir(path)); const kept: string[] = []; for (const dir of await existingBuildDirsAsync(path)) { - const problem = await regenerableDirProblem(path, dir); + const problem = await regenerableDirProblem(path, dir, undefined, { launching }); if (problem) kept.push(`kept ${dir}: ${problem}`); else paths.push(dir); } @@ -214,16 +218,13 @@ export async function idleOutputs(path: string, config: Config): Promise<{ paths * Deletes `paths`, measuring each first. For a reduce-only checkout `guard` reads the executables on PATH anew after the * measurement, immediately before each removal, and keeps a path whose checkout one resolves into, saying so in `kept`. */ -export async function deleteOutputs( - paths: readonly string[], - guard?: { checkout: string; launchers: () => Promise; kept: string[] }, -): Promise { +export async function deleteOutputs(paths: readonly string[], launching: Launching, kept: string[]): Promise { let freed = 0; for (const path of paths) { const bytes = (await measuredDirectorySizeAsync(path, { timeoutMs: SIZE_TIMEOUT_MS })) ?? 0; - const launcher = guard ? launcherReason(guard.checkout, await guard.launchers()) : null; + const launcher = launching.reduceOnly ? launcherReason(launching.checkout, await launching.launchers()) : null; if (launcher) { - guard?.kept.push(`kept ${path}: ${launcher}`); + kept.push(`kept ${path}: ${launcher}`); continue; } freed += bytes; @@ -320,8 +321,9 @@ async function enterIdleStage( const [kind, slot = 'default'] = key.split(':'); return outcome?.status === 'shut-down' && booted.has(`${kind}:${slot}`) ? [key] : []; }); - const outputs = await idleOutputs(path, config); - const freed = await deleteOutputs(outputs.paths); + const launching: Launching = { reduceOnly: false }; + const outputs = await idleOutputs(path, config, launching); + const freed = await deleteOutputs(outputs.paths, launching, outputs.kept); await recordStage(path, minutes, shutDown, freed); const idle = Math.floor(again.decision.idleMs / MINUTE_MS); const restore = revivable(path) ? 'stim revive' : 'stim ios or stim android'; diff --git a/packages/stim-cli/src/daemon/ladder.ts b/packages/stim-cli/src/daemon/ladder.ts index 5f66b32471..96c6f11ecd 100644 --- a/packages/stim-cli/src/daemon/ladder.ts +++ b/packages/stim-cli/src/daemon/ladder.ts @@ -21,6 +21,7 @@ import { projectDeviceSlots } from '../devices/device-slots.ts'; import type { IosSimRecord } from '../devices/ios.ts'; import { readParked } from '../devices/sim-pool.ts'; import { restoreProblem } from '../workspace/lightened.ts'; +import { launchingOf, type Launching } from '../workspace/launchers.ts'; import { formatBytes, measuredDirectorySizeAsync } from '../fs-util.ts'; import { executeAction, type ActContext, type ActionOutcome } from '../stimd/passes/act.ts'; import { artifactWorktreesFor } from '../stimd/passes/artifacts.ts'; @@ -151,8 +152,8 @@ function recordAction( } /** The bytes of the regenerable build outputs the idle stage would delete in a workspace. */ -export async function idleOutputBytes(path: string, config: Config): Promise { - return pathsBytes((await idleOutputs(path, config)).paths); +export async function idleOutputBytes(path: string, config: Config, launching: Launching): Promise { + return pathsBytes((await idleOutputs(path, config, launching)).paths); } async function pathsBytes(paths: readonly string[]): Promise { @@ -274,7 +275,13 @@ async function idleOutputsRung(context: RungContext): Promise { ? await clearStoppedOutputs(candidate, context.config, PRESSURE_IDLE_MS, context.operation) : { outcome: 'would' as const, - bytes: context.measure ? await idleOutputBytes(candidate.path, context.config) : 0, + bytes: context.measure + ? await idleOutputBytes( + candidate.path, + context.config, + launchingOf(candidate.reduceOnly, candidate.path, context.operation), + ) + : 0, detail: wouldDetail(context), }; if (!result) continue; diff --git a/packages/stim-cli/src/daemon/passes.ts b/packages/stim-cli/src/daemon/passes.ts index 8382ff2e64..36a1477f81 100644 --- a/packages/stim-cli/src/daemon/passes.ts +++ b/packages/stim-cli/src/daemon/passes.ts @@ -12,6 +12,7 @@ import { import { processContext } from '../cli-operation.ts'; import { runMaintenance } from '../stimd/passes/run.ts'; import { createNdjsonWriter } from '../ndjson.ts'; +import { launchingOf } from '../workspace/launchers.ts'; import { loadConfig, type Config } from '../workspace/config.ts'; import { requestDaemonTick, stimdRefused } from './client.ts'; import { runBootedParkedPass } from './booted-parked.ts'; @@ -457,7 +458,9 @@ export async function runForcedLadder({ target: entry.path, workspace: entry.path, outcome: entry.outcome === 'idle' ? 'done' : entry.outcome === 'would-idle' ? 'would' : 'failed', - bytes: entry.freedBytes ?? (entry.outcome === 'would-idle' ? await idleOutputBytes(entry.path, config) : 0), + bytes: + entry.freedBytes ?? + (entry.outcome === 'would-idle' ? await idleOutputBytes(entry.path, config, { reduceOnly: false }) : 0), detail: entry.outcome === 'would-idle' ? 'stim gc would do this' : entry.detail, ...(entry.outcome === 'idle' ? { restore: 'stim revive' } : {}), }); @@ -527,7 +530,11 @@ export async function runDormantReduction({ ? await clearStoppedOutputs(candidate, config, afterMs, operation) : { outcome: 'would' as const, - bytes: await idleOutputBytes(candidate.path, config), + bytes: await idleOutputBytes( + candidate.path, + config, + launchingOf(candidate.reduceOnly, candidate.path, operation), + ), detail: 'stim gc would do this', }; if (result) diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index 3bab016215..398a67257b 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -39,7 +39,7 @@ import { getProject, loadConfig, type Config, type ProjectRecord } from '../work import { checkoutRoleOf } from '../workspace/checkout-role.ts'; import { discoverCheckouts } from '../workspace/discovery.ts'; import { discoveredEvidence } from '../workspace/discovered-activity.ts'; -import { freshLaunchers } from '../workspace/launchers.ts'; +import { launchingOf } from '../workspace/launchers.ts'; import { discoveredReads, type DiscoveredReads } from '../workspace/discovered-reads.ts'; import { listedRoots } from '../commands/gc/worktrees.ts'; import { withIdleWorkspace } from '../workspace/in-use.ts'; @@ -66,7 +66,7 @@ export interface StageCandidate { /** A source checkout is taken after every linked worktree. */ role: CheckoutRole; /** Found through a tool folder or the full scan; a PATH launcher into it keeps its outputs. */ - reduceOnly?: boolean; + reduceOnly: boolean; } /** `idle.dormantDays` in milliseconds; 0 when the dormant stage is off. */ @@ -150,14 +150,15 @@ async function idleVerdict( operation, reads = discoveredReads(operation?.signal, operation), full = false, - reduceOnly = false, + reduceOnly, }: { locked?: boolean; operation?: StimContext; reads?: DiscoveredReads; full?: boolean; - reduceOnly?: boolean; - } = {}, + /** Found through a tool folder or the full scan: a PATH launcher into the checkout keeps it. Every caller states it. */ + reduceOnly: boolean; + }, ): Promise<{ idleMs: number } | { kept: string }> { const now = Date.now(); const pressured = setting === PRESSURE_THRESHOLD; @@ -184,7 +185,7 @@ async function idleFor( path: string, thresholdMs: number, setting: string, - options: Parameters[3] = {}, + options: Parameters[3], ): Promise { const verdict = await idleVerdict(path, thresholdMs, setting, options); return 'idleMs' in verdict ? verdict.idleMs : null; @@ -199,19 +200,18 @@ export function sourceCheckoutIdle( path: string, thresholdMs: number, setting: string, - operation?: StimContext, - locked = false, + { operation, locked = false, reduceOnly }: { operation?: StimContext; locked?: boolean; reduceOnly: boolean }, ): Promise<{ idleMs: number } | { kept: string }> { - return idleVerdict(path, thresholdMs, setting, { operation, full: true, locked }); + return idleVerdict(path, thresholdMs, setting, { operation, full: true, locked, reduceOnly }); } /** Why a discovered checkout is not idle `thresholdMs` by the activity definition, or null when it is. */ export async function discoveredKeepReason( path: string, thresholdMs: number, + reduceOnly: boolean, operation?: StimContext, reads?: DiscoveredReads, - reduceOnly = false, ): Promise { const verdict = await idleVerdict(path, thresholdMs, 'idle.dormantDays', { operation, @@ -268,7 +268,7 @@ async function idleWorkspaces( if ('idleMs' in verdict) idleMs = verdict.idleMs; else onKept?.(path, verdict.kept); } - readings.push({ path, project, idleMs, ...(reduceOnly.has(path) ? { reduceOnly: true } : {}) }); + readings.push({ path, project, idleMs, reduceOnly: reduceOnly.has(path) }); } catch (error) { if (isClaimUnavailable(error) && 'reportCleanup' in error) throw error; reading.operation?.signal.throwIfAborted(); @@ -293,12 +293,16 @@ async function hasOutputs(path: string): Promise { export async function stillIdle( path: string, thresholdMs: number, - operation?: StimContext, - full = false, - locked = true, - setting: string = PRESSURE_THRESHOLD, + { + operation, + full = false, + locked = true, + setting = PRESSURE_THRESHOLD, + reduceOnly, + }: { operation?: StimContext; full?: boolean; locked?: boolean; setting?: string; reduceOnly: boolean }, ): Promise { - return (await idleFor(path, thresholdMs, setting, { locked, operation, full })) !== null; + const reads = discoveredReads(operation?.signal, operation, true); + return (await idleFor(path, thresholdMs, setting, { locked, operation, full, reads, reduceOnly })) !== null; } /** Workspaces with nothing running, idle `thresholdMs` by the activity definition: linked worktrees, then source checkouts, longest-idle first in each. */ @@ -347,18 +351,14 @@ export async function clearStoppedOutputs( locked: true, operation, reads: discoveredReads(operation?.signal, operation, true), - ...(candidate.reduceOnly ? { reduceOnly: true } : {}), + reduceOnly: candidate.reduceOnly, })) === null ) return { outcome: 'kept', bytes: 0, detail: 'it is no longer idle' }; - const outputs = await idleOutputs(path, config); + const launching = launchingOf(candidate.reduceOnly, path, operation); + const outputs = await idleOutputs(path, config, launching); if (!outputs.paths.length) return null; - const bytes = await deleteOutputs( - outputs.paths, - candidate.reduceOnly - ? { checkout: path, launchers: freshLaunchers(operation), kept: outputs.kept } - : undefined, - ); + const bytes = await deleteOutputs(outputs.paths, launching, outputs.kept); return { outcome: 'done', bytes, @@ -507,7 +507,11 @@ export async function enterDormantStage(action: MaintenanceAction, operation?: S const run = await withIdleWorkspace( path, async (): Promise => { - const idleMs = await idleFor(path, thresholdMs, 'idle.dormantDays', { locked: true, operation }); + const idleMs = await idleFor(path, thresholdMs, 'idle.dormantDays', { + locked: true, + operation, + reduceOnly: config.projects[path] === undefined, + }); if (idleMs === null) return { status: 'kept', bytes: 0, detail: 'it is no longer idle' }; const owns = (kind: 'ios' | 'android') => projectDeviceSlots(project).some(({ platforms }) => platforms[kind]?.owned === true); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 7195004846..0eefc17180 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1224,8 +1224,8 @@ cleanup, never by a command): weekly, and also while the disk level is above normal when no scan has run for an hour; a failed scan is retried after an hour. The schedule is kept with the result, so a restarted daemon forgets nothing. It reads ~/Documents (macOS may ask once; a folder that does not answer within 3 -seconds, such as one waiting for that answer, is recorded as unreadable and is -read on the next scan), skips ~/Desktop, ~/Downloads, ~/Library, the media +seconds, such as one waiting for that answer, is recorded as unreadable, ends +that scan early with the previous finds kept, and is read on the next scan), skips ~/Desktop, ~/Downloads, ~/Library, the media folders, other volumes, symlinks, dependency or build folders, deeper dot-folders, the Stim home and every dot-folder starting with .stim, the build worker root, and the tool homes and caches among the home folder's other @@ -1250,7 +1250,10 @@ checkout for that pass, and the reason names it and says fixing its permissions restores reduction. Each deletion reads the directories anew, at the moment it reads the processes using the path (not the pass's planning reading); a directory that hangs ends the reading as incomplete at once, and while that read -is pending no other launcher read starts. Reduce-only checkouts have their build outputs and, in +is pending (at most one per thread: the daemon's main thread and passes thread +each have one; the scan runs on the passes thread) no other launcher or scan +read starts and a scheduled scan is skipped. A scan stops at the first read that +times out, keeping the previous finds. Reduce-only checkouts have their build outputs and, in a linked worktree, node_modules and Pods reduced by the rules below. Passes evaluate at most 40 of them at a time, in a window that moves every 15 minutes; stim gc --dry-run lists all of them and says which are left for a later pass. A diff --git a/packages/stim-cli/src/stimd/passes/artifacts.ts b/packages/stim-cli/src/stimd/passes/artifacts.ts index 29a6d59595..46b0b362df 100644 --- a/packages/stim-cli/src/stimd/passes/artifacts.ts +++ b/packages/stim-cli/src/stimd/passes/artifacts.ts @@ -20,7 +20,7 @@ import { import { reclaimKeys } from '../../commands/worktree.ts'; import { loadConfig } from '../../workspace/config.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; -import { freshLaunchers, launcherReason, launchersOf } from '../../workspace/launchers.ts'; +import { freshLaunchers, launcherReason, launchersOf, launchingOf } from '../../workspace/launchers.ts'; import { gitQuiet, gitQuietAsync } from '../../workspace/git.ts'; import { childEnvironment } from '../../workspace/git-read.ts'; import { withIdleWorkspace, workspaceInUse, workspaceInUseAsync } from '../../workspace/in-use.ts'; @@ -69,7 +69,7 @@ export interface ArtifactWorktree { /** Why a running agent session keeps the worktree, read when the worktrees were listed. */ agents?: string[]; /** Found through a tool folder or the full scan, where an installed tool cannot be told from a checkout. */ - reduceOnly?: boolean; + reduceOnly: boolean; } export interface ArtifactCandidate { @@ -80,7 +80,7 @@ export interface ArtifactCandidate { agents?: string[]; /** In a source checkout, whose dependencies no worktree warm restores. */ source?: boolean; - reduceOnly?: boolean; + reduceOnly: boolean; } function isDirectory(path: string): boolean { @@ -108,7 +108,7 @@ function registeredRoots(roots: string[]): string[] { export function artifactWorktrees(): ArtifactWorktree[] { const found = new Map(); for (const root of registeredRoots(candidateRoots())) { - const entry = artifactWorktree(root, listWorktrees(root)); + const entry = artifactWorktree(root, listWorktrees(root), false); if (entry) found.set(entry.path, entry); } return [...found.values()]; @@ -117,6 +117,7 @@ export function artifactWorktrees(): ArtifactWorktree[] { function artifactWorktree( root: string, entries: WorktreeEntry[], + reduceOnly: boolean, source: SourceCheckout = sourceCheckoutOf(entries), ): ArtifactWorktree | null { const entry = matchWorktreeEntry(entries, root); @@ -128,6 +129,7 @@ function artifactWorktree( linked: entry.path !== source.path, locked: Boolean(entry.locked), keys, + reduceOnly, }; } @@ -155,12 +157,8 @@ export async function artifactCheckoutsAsync( signal, declareFor: operation, }); - const entry = artifactWorktree(root, entries, source); - if (entry) - found.set( - entry.path, - reduceOnly.has(root) || found.get(entry.path)?.reduceOnly ? { ...entry, reduceOnly: true } : entry, - ); + const entry = artifactWorktree(root, entries, reduceOnly.has(root), source); + if (entry) found.set(entry.path, found.get(entry.path)?.reduceOnly ? { ...entry, reduceOnly: true } : entry); } return [...found.values()]; } @@ -271,8 +269,8 @@ async function artifactReads( candidate: ArtifactCandidate, kind: ArtifactKind, signal: AbortSignal, - operation?: StimContext, - { fresh = false }: { fresh?: boolean } = {}, + operation: StimContext | undefined, + { fresh }: { fresh: boolean }, ): Promise { const { worktree, path, keys } = candidate; const tracked = await gitQuietAsync(worktree, ['ls-files', '--', relative(worktree, path)], { @@ -466,7 +464,7 @@ export async function collectArtifactsAsync( keys: worktree.keys, locked: worktree.locked, agents: worktree.agents, - ...(worktree.reduceOnly ? { reduceOnly: true } : {}), + reduceOnly: worktree.reduceOnly, ...(options.source ? { source: true } : {}), }; const inUse = protectedHere ? null : await inUseOf(candidate, kind, signal, operation); @@ -479,10 +477,13 @@ export async function collectArtifactsAsync( candidate, kind, { ...options, pinned, running, ...(inUse ? { inUse } : {}) }, - await artifactReads(candidate, kind, signal, operation), + await artifactReads(candidate, kind, signal, operation, { fresh: false }), ); if (!reason && candidate.source) - reason = (await regenerableDirProblem(worktree.path, path, signal)) ?? linkedWorktreeUsing(candidate, all); + reason = + (await regenerableDirProblem(worktree.path, path, signal, { + launching: launchingOf(worktree.reduceOnly, worktree.path, operation, signal), + })) ?? linkedWorktreeUsing(candidate, all); if (reason) skips.push({ target: path, reason }); else candidates.push(candidate); } @@ -581,7 +582,13 @@ export function collectArtifacts( } catch { continue; } - const candidate = { worktree: worktree.path, path, keys: worktree.keys, locked: worktree.locked }; + const candidate = { + worktree: worktree.path, + path, + keys: worktree.keys, + locked: worktree.locked, + reduceOnly: worktree.reduceOnly, + }; const reason = protectedHere ? 'the command that started this pass runs inside its worktree' : keptReason(candidate, kind, { ...options, running }); @@ -627,7 +634,7 @@ export async function clearArtifact( keys: worktree.keys, locked: worktree.locked, agents, - ...(worktree.reduceOnly ? { reduceOnly: true } : {}), + reduceOnly: worktree.reduceOnly, }; const running = kind === 'dependencies' ? await readRunningProcessesAsync() : null; const pinned = await pinsOf(candidate.keys, options.pinned); @@ -753,7 +760,7 @@ export async function clearSourceTrees( locked: worktree.locked, agents: (await agentSessionUse([worktree.path], Date.now(), signal))[0] ?? [], source: true, - ...(worktree.reduceOnly ? { reduceOnly: true } : {}), + reduceOnly: worktree.reduceOnly, }; const checkTree = async (path: string, held?: readonly string[]): Promise => { const candidate = { ...base, path, ...(held ? { source: false } : {}) }; @@ -772,7 +779,9 @@ export async function clearSourceTrees( }, await artifactReads(candidate, 'dependencies', signal, operation, { fresh: true }), ) ?? - (await regenerableDirProblem(worktree.path, path, signal)) ?? + (await regenerableDirProblem(worktree.path, path, signal, { + launching: launchingOf(worktree.reduceOnly, worktree.path, operation, signal), + })) ?? linkedWorktreeUsing(candidate, options.worktrees) ); }; diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index db60874875..3dd7158fc8 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -9,7 +9,7 @@ import { measuredDirectorySizeAsync } from '../../fs-util.ts'; import { canonicalPath } from '../../commands/gc/paths.ts'; import { listedRoots } from '../../commands/gc/worktrees.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; -import { freshLaunchers, launchersOf } from '../../workspace/launchers.ts'; +import { launchingOf, planningLaunching } from '../../workspace/launchers.ts'; import { cargoTargetDirsAsync, isBuildDirPath, regenerableDirProblem } from '../../workspace/regenerable-dirs.ts'; import { appRoots, artifactCheckoutsAsync } from './artifacts.ts'; import { readVolumeSpaceAsync } from './budget.ts'; @@ -34,9 +34,9 @@ export interface CargoTargetCandidate { identity: string; discovered: boolean; /** Found through a tool folder or the full scan: a PATH launcher into the checkout keeps its target. */ - reduceOnly?: boolean; + reduceOnly: boolean; /** The checkout that owns the target, which a PATH launcher can point anywhere into; `root` is the app inside it. */ - checkout?: string; + checkout: string; measuredAt: number | null; } @@ -99,7 +99,7 @@ async function cargoTargetCandidates( ): Promise { const signal = operation?.signal ?? new AbortController().signal; const found: CargoTargetCandidate[] = []; - const add = async (base: string, discovered: boolean, reduceOnly = false, owner?: string) => { + const add = async (base: string, discovered: boolean, reduceOnly: boolean, checkout: string) => { const root = canonicalPath(base); for (const path of await cargoTargetDirsAsync(root)) { try { @@ -111,13 +111,14 @@ async function cargoTargetCandidates( dir: path, identity, discovered, - ...(reduceOnly ? { reduceOnly, checkout: owner ?? root } : {}), + reduceOnly, + checkout, measuredAt: lastMeasured.get(path) ?? null, }); } catch {} } }; - for (const root of registered) await add(root, false); + for (const root of registered) await add(root, false, false, root); const discovered = new Set( (await discoverCheckouts(loadConfig(), listedRoots(), operation)) .filter(({ deferred }) => !deferred) @@ -125,7 +126,7 @@ async function cargoTargetCandidates( ); for (const checkout of await artifactCheckoutsAsync(signal, operation)) for (const base of appRoots(checkout, 'build')) - await add(base, discovered.has(canonicalPath(checkout.path)), checkout.reduceOnly === true, checkout.path); + await add(base, discovered.has(canonicalPath(checkout.path)), checkout.reduceOnly, checkout.path); return found; } @@ -156,7 +157,7 @@ export async function measureCargoTargets( lastMeasured.set(dir, now); const problem = await regenerableDirProblem(root, dir, operation?.signal, { processes: false, - ...(reduceOnly ? { launchers: launchersOf(operation), launcherRoot: checkout ?? root } : {}), + launching: planningLaunching(reduceOnly, checkout, operation), }); if (problem) { scan.refused.set(dir, { workspace: root, reason: problem }); @@ -367,7 +368,7 @@ export async function planCargoTargetsAtPressure({ if (tightened === 'normal') continue; const problem = await regenerableDirProblem(workspace, dir, signal, { processes: false, - ...(reduceOnly ? { launchers: launchersOf(operation, signal), launcherRoot: checkout ?? workspace } : {}), + launching: planningLaunching(reduceOnly, checkout, operation, signal), }); if (problem) { plan.skips.push({ target: dir, reason: problem, workspace }); @@ -407,14 +408,15 @@ export async function planCargoTargetsAtPressure({ async function managedAs( root: string, operation?: StimContext, -): Promise<{ reduceOnly: boolean; checkout?: string } | null> { +): Promise<{ reduceOnly: boolean; checkout: string } | null> { const canonical = canonicalPath(root); - if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) return { reduceOnly: false }; + if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) + return { reduceOnly: false, checkout: canonical }; const signal = operation?.signal ?? new AbortController().signal; const checkout = (await artifactCheckoutsAsync(signal, operation)).find((entry) => appRoots(entry, 'build').some((base) => canonicalPath(base) === canonical), ); - return checkout ? { reduceOnly: checkout.reduceOnly === true, checkout: checkout.path } : null; + return checkout ? { reduceOnly: checkout.reduceOnly, checkout: checkout.path } : null; } async function targetVolume(target: string, signal?: AbortSignal): Promise<{ freeMb: number; device: number } | null> { @@ -434,9 +436,10 @@ export async function trimCargoTargetAction( if (!root || !isBuildDirPath(action.target)) return kept('it is no longer a Cargo target/ Stim recognises'); const managed = await managedAs(root, operation); if (!managed) return kept('its checkout is no longer in a workspace or discovery scope'); + const launching = launchingOf(managed.reduceOnly, managed.checkout, operation); const problem = await regenerableDirProblem(root, action.target, operation?.signal, { processes: false, - ...(managed.reduceOnly ? { launchers: freshLaunchers(operation), launcherRoot: managed.checkout ?? root } : {}), + launching, }); if (problem) return kept(problem); const measure = action.level === undefined; @@ -448,7 +451,7 @@ export async function trimCargoTargetAction( signal: operation?.signal, measure, ...(stop ? { stop } : {}), - ...(managed.reduceOnly ? { launchers: freshLaunchers(operation) } : {}), + launching, }); if (!measure) { const after = before ? await targetVolume(action.target, operation?.signal) : null; diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index 17ba0137d7..a7c45f4ab8 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -253,7 +253,7 @@ export async function runMaintenanceAtLevel({ ? 'left for a later pass: at most 40 checkouts found through a tool folder or the full scan are handled at a time' : taken || idleMs <= 0 || !existsSync(checkout.path) ? null - : await discoveredKeepReason(checkout.path, idleMs, operation, reads, checkout.reduceOnly === true); + : await discoveredKeepReason(checkout.path, idleMs, checkout.reduceOnly === true, operation, reads); if (kept && /source entries to check|cannot be read/.test(kept)) blocked.push(`Discovered checkout ${checkout.path}: ${kept}, so it is never reduced`); discovered.push({ ...checkout, actions: mine, ...(kept ? { kept } : {}) }); diff --git a/packages/stim-cli/src/workspace/bounded-read.ts b/packages/stim-cli/src/workspace/bounded-read.ts new file mode 100644 index 0000000000..e1dd658630 --- /dev/null +++ b/packages/stim-cli/src/workspace/bounded-read.ts @@ -0,0 +1,43 @@ +/** + * The one filesystem read, of the launcher directories or the home scan, that timed out and has not settled. A timed-out + * native read cannot be cancelled, so while one is pending no further read of either kind starts: a hung mount holds + * exactly one libuv thread however many readings and scans are asked for. Module state, so the bound is one outstanding + * read per JavaScript isolate: the daemon's main thread and its passes thread each have their own. + */ +let hung: { label: string } | null = null; + +export class Unanswered extends Error {} + +/** The folder whose read timed out and is still pending, or null when none is. */ +export function hungRead(): string | null { + return hung?.label ?? null; +} + +/** + * `start()`'s result, or an `Unanswered` rejection when it has not answered in `timeoutMs`; the read keeps running, and + * is the pending one until it settles. + */ +export async function answered(label: string, start: () => Promise, timeoutMs: number): Promise { + const read = start(); + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + read, + new Promise((_, fail) => { + timer = setTimeout(() => fail(new Unanswered('timed out')), timeoutMs); + }), + ]); + } catch (error) { + if (error instanceof Unanswered) { + const mine = { label }; + hung = mine; + const settled = () => { + if (hung === mine) hung = null; + }; + read.then(settled, settled); + } + throw error; + } finally { + clearTimeout(timer); + } +} diff --git a/packages/stim-cli/src/workspace/cargo-target.ts b/packages/stim-cli/src/workspace/cargo-target.ts index 58dc2f508c..4b1f9e9a23 100644 --- a/packages/stim-cli/src/workspace/cargo-target.ts +++ b/packages/stim-cli/src/workspace/cargo-target.ts @@ -6,7 +6,7 @@ import { releaseClaim, tryAcquireClaim, type ClaimHandle } from '../ownership-cl import { getExecutor } from '../exec.ts'; import { measuredDirectorySizeAsync } from '../fs-util.ts'; import { withCargoLocks } from './cargo-lock.ts'; -import type { LauncherScan } from './launchers.ts'; +import type { LauncherScan, Launching } from './launchers.ts'; import { cargoUnitPaths, cargoUnitWasRead, idleCargoUnits, readCargoUnits, removalOrder } from './cargo-units.ts'; const CARGO_PROFILE_IDLE_MS: number = 24 * 3_600_000; @@ -399,10 +399,10 @@ interface CargoTrimOptions { /** Whether to walk what was removed to report its bytes; false reports 0 and removes at once. Default true. */ measure?: boolean; /** - * For a reduce-only checkout: a fresh reading of the executables on PATH, taken wherever the processes using the target - * are read, so a path one of them resolves to is kept like a path a process runs from. + * For a reduce-only checkout, a fresh reading of the executables on PATH, taken wherever the processes using the target + * are read, so a path one of them resolves to is kept like a path a process runs from. Every caller states it. */ - launchers?: () => Promise; + launching: Launching; } async function trimClaimed( @@ -416,10 +416,11 @@ async function trimClaimed( signal, stop, measure = true, - launchers, + launching, }: CargoTrimOptions, ): Promise { const result: CargoTrimResult = { bytes: 0, removed: [], kept: [] }; + const launchers = launching.reduceOnly ? launching.launchers : undefined; const cannotRead = launchers ? 'the files processes hold open or the executables on PATH cannot be read' : 'the files processes hold open cannot be read'; diff --git a/packages/stim-cli/src/workspace/discovered-activity.ts b/packages/stim-cli/src/workspace/discovered-activity.ts index deea1f6615..c1fc94c3c5 100644 --- a/packages/stim-cli/src/workspace/discovered-activity.ts +++ b/packages/stim-cli/src/workspace/discovered-activity.ts @@ -111,14 +111,14 @@ export async function discoveredEvidence( idleMs, reads, operation, - reduceOnly = false, + reduceOnly, }: { now: number; idleMs: number; reads: DiscoveredReads; operation?: StimContext; /** Found through a tool folder or the full scan, where an installed tool cannot be told from a checkout. */ - reduceOnly?: boolean; + reduceOnly: boolean; }, ): Promise { const signal = operation?.signal; diff --git a/packages/stim-cli/src/workspace/discovered-removal.ts b/packages/stim-cli/src/workspace/discovered-removal.ts index 1a0eca555c..1db3632b0c 100644 --- a/packages/stim-cli/src/workspace/discovered-removal.ts +++ b/packages/stim-cli/src/workspace/discovered-removal.ts @@ -111,9 +111,9 @@ export async function discoveredRemovalHold( main, idleMs, reads, - reduceOnly = false, + reduceOnly, operation, - }: { main: string | null; idleMs: number; reads: DiscoveredReads; reduceOnly?: boolean; operation?: StimContext }, + }: { main: string | null; idleMs: number; reads: DiscoveredReads; reduceOnly: boolean; operation?: StimContext }, ): Promise { if (reduceOnly) return 'it was found in a tool folder or by the full scan, so Stim reduces its outputs and never removes it'; @@ -122,7 +122,7 @@ export async function discoveredRemovalHold( return `it is managed by another tool (inside ${folder} of its main checkout)`; if (!(idleMs > 0)) return 'idle.dormantDays is 0, so its idleness cannot be established'; const now = Date.now(); - const decision = decideIdleStop(await discoveredEvidence(path, { now, idleMs, reads, operation }), { + const decision = decideIdleStop(await discoveredEvidence(path, { now, idleMs, reads, operation, reduceOnly }), { idleMs, now, setting: 'idle.dormantDays', diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 7b56dd4777..bd8796a8b7 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -12,6 +12,7 @@ import { } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; import { diskPressure, diskThresholds } from '../daemon/pressure.ts'; +import { answered, hungRead, Unanswered } from './bounded-read.ts'; import { loadConfig } from './config.ts'; import { recognises } from './discovery.ts'; import { MAX_DEPTH, scanDescends, scanSkipper } from './scan-rules.ts'; @@ -31,28 +32,15 @@ interface ScanOutcome { const errorCode = (error: unknown) => (error instanceof Error && 'code' in error ? String(error.code) : ''); -/** `read`'s result, or an ETIMEDOUT rejection when it has not answered in time, as a hung mount or a permission prompt would not. */ -async function answered(read: Promise, timeoutMs: number): Promise { - let timer: NodeJS.Timeout | undefined; - try { - return await Promise.race([ - read, - new Promise((_, fail) => { - timer = setTimeout(() => fail(Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })), timeoutMs); - }), - ]); - } finally { - clearTimeout(timer); - } -} - -const unanswered = (error: unknown) => ['EACCES', 'EPERM', 'ETIMEDOUT'].includes(errorCode(error)); +const refused = (error: unknown) => ['EACCES', 'EPERM'].includes(errorCode(error)); /** * A pruned walk below `home` (`scanDescends`, `skip`): folders for which `isCheckout` holds are results and are not - * entered; symlinks and other volumes are skipped; a folder the system refuses or that does not answer within 3 seconds - * is recorded as unreadable, never an error. It stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or `budgetMs`, and - * says so in `truncated`. An abort rejects. + * entered; symlinks and other volumes are skipped; a folder the system refuses is recorded as unreadable, never an error. + * A read that does not answer within 3 seconds stops the walk at once: the folder is recorded unreadable, the result is + * truncated, and the read stays the pending one (`bounded-read.ts`), so no further read starts, here or in a later scan, + * until it settles. It also stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or `budgetMs`, and says so in + * `truncated`. An abort rejects. */ export async function scanForCheckouts( home: string, @@ -63,31 +51,50 @@ export async function scanForCheckouts( skip = () => false, readTimeoutMs = READ_TIMEOUT_MS, stat: statOf = (path) => lstat(path), + entries: entriesOf = (path) => readdir(path, { withFileTypes: true }), }: { budgetMs?: number; signal?: AbortSignal; skip?: (path: string) => boolean; readTimeoutMs?: number; stat?: (path: string) => Promise<{ dev: number }>; + entries?: (path: string) => Promise<{ name: string; isDirectory(): boolean; isSymbolicLink(): boolean }[]>; } = {}, ): Promise { const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false }; const deadline = Date.now() + budgetMs; - const volume = (await statOf(home)).dev; const exhausted = (): boolean => { signal?.throwIfAborted(); - if (!outcome.truncated && (Date.now() > deadline || outcome.visited >= MAX_FOLDERS)) outcome.truncated = true; + if (!outcome.truncated && (Date.now() > deadline || outcome.visited >= MAX_FOLDERS || hungRead() !== null)) + outcome.truncated = true; return outcome.truncated; }; + const stopAt = (folder: string, error: unknown): boolean => { + signal?.throwIfAborted(); + if (error instanceof Unanswered) { + outcome.unreadable.push(folder); + outcome.truncated = true; + return true; + } + if (refused(error)) outcome.unreadable.push(folder); + return false; + }; + if (hungRead() !== null) return { ...outcome, truncated: true }; + let volume: number; + try { + volume = (await answered(home, () => statOf(home), readTimeoutMs)).dev; + } catch (error) { + if (!stopAt(home, error)) throw error; + return outcome; + } const walk = async (folder: string, depth: number): Promise => { if (exhausted()) return; outcome.visited++; let entries; try { - entries = await answered(readdir(folder, { withFileTypes: true }), readTimeoutMs); + entries = await answered(folder, () => entriesOf(folder), readTimeoutMs); } catch (error) { - signal?.throwIfAborted(); - if (unanswered(error)) outcome.unreadable.push(folder); + stopAt(folder, error); return; } if (depth > 0 && isCheckout(folder)) { @@ -102,10 +109,9 @@ export async function scanForCheckouts( if (skip(path)) continue; let info; try { - info = await answered(statOf(path), readTimeoutMs); + info = await answered(path, () => statOf(path), readTimeoutMs); } catch (error) { - signal?.throwIfAborted(); - if (unanswered(error)) outcome.unreadable.push(path); + if (stopAt(path, error)) return; continue; } if (info.dev === volume) await walk(path, depth + 1); @@ -185,7 +191,8 @@ export async function runDiscoveryScanIfDue( } catch { return; } - if (!discoveryScanDue(last, now, low) || (now >= attempted.at && now - attempted.at < HOUR_MS)) return; + if (!discoveryScanDue(last, now, low) || (now >= attempted.at && now - attempted.at < HOUR_MS) || hungRead() !== null) + return; attempted.at = now; const attempt = { version: 2, diff --git a/packages/stim-cli/src/workspace/launchers.ts b/packages/stim-cli/src/workspace/launchers.ts index 31be961e76..184695d5e1 100644 --- a/packages/stim-cli/src/workspace/launchers.ts +++ b/packages/stim-cli/src/workspace/launchers.ts @@ -2,6 +2,7 @@ import { lstat, readdir, realpath } from 'node:fs/promises'; import { join, sep } from 'node:path'; import type { StimContext } from '@stim-cli/core'; import { canonicalPath } from '../commands/gc/paths.ts'; +import { answered, hungRead, Unanswered } from './bounded-read.ts'; import { launcherDirectories } from './launcher-directories.ts'; /** @@ -31,47 +32,14 @@ const absent = (error: unknown) => error instanceof Error && 'code' in error && const READ_TIMEOUT_MS = 3000; -/** - * The one launcher read that timed out and has not settled. While it is pending, no new read starts: a hung mount holds - * exactly one libuv thread however many readings are asked for, and every reading reports it incomplete at once. - * Module state, so the bound is per thread of the process; in the daemon only the passes thread reads launchers. - */ -let hung: { directory: string } | null = null; -/** Readings run one at a time, so the single read in flight is the only one outstanding. */ +/** Readings run one at a time, so a reading never adds a second read in flight. */ let queue: Promise = Promise.resolve(); -class Unanswered extends Error {} - -/** `read`'s result, or an `Unanswered` rejection when it has not answered in time; the read itself cannot be cancelled. */ -async function answered(directory: string, start: () => Promise, timeoutMs: number): Promise { - const read = start(); - let timer: NodeJS.Timeout | undefined; - try { - return await Promise.race([ - read, - new Promise((_, fail) => { - timer = setTimeout(() => fail(new Unanswered('timed out')), timeoutMs); - }), - ]); - } catch (error) { - if (error instanceof Unanswered) { - const mine = { directory }; - hung = mine; - const settled = () => { - if (hung === mine) hung = null; - }; - read.then(settled, settled); - } - throw error; - } finally { - clearTimeout(timer); - } -} - /** * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader. A wrapper * script that merely names a path is not followed. Every call to the directory has 3 seconds to answer; one that does - * not ends the reading at once as incomplete, and no reading starts a read while an earlier one is still pending. + * not ends the reading at once as incomplete, and no read of any kind starts while an earlier one is still pending + * (`bounded-read.ts`). */ export function launcherReader( signal?: AbortSignal, @@ -86,7 +54,8 @@ export function launcherReader( const found: LauncherScan = { targets: [], incomplete: [] }; for (const directory of directories) { signal?.throwIfAborted(); - if (hung) return { targets: found.targets, incomplete: [...found.incomplete, hung.directory] }; + const stuck = hungRead(); + if (stuck) return { targets: found.targets, incomplete: [...found.incomplete, stuck] }; try { for (const name of await answered(directory, () => io.readdir(directory), timeoutMs)) { const entry = join(directory, name); @@ -149,6 +118,35 @@ export function freshLaunchers( return () => newReader(operation, signal, directories)(); } +/** + * Whether a PATH launcher can keep a checkout's outputs, which every caller on a deletion path states: `reduceOnly: false` + * for a registered workspace, which has no launcher keep, or a fresh launcher reading and the checkout it is checked + * against. + */ +export type Launching = + | { reduceOnly: false } + | { reduceOnly: true; launchers: () => Promise; checkout: string }; + +/** The `Launching` of a checkout for planning: the operation's shared reading for a reduce-only one, none otherwise. */ +export function planningLaunching( + reduceOnly: boolean, + checkout: string, + operation: StimContext | undefined, + signal?: AbortSignal, +): Launching { + return reduceOnly ? { reduceOnly, launchers: launchersOf(operation, signal), checkout } : { reduceOnly }; +} + +/** The `Launching` of a checkout for a deletion: a fresh launcher reading for a reduce-only one, none otherwise. */ +export function launchingOf( + reduceOnly: boolean, + checkout: string, + operation: StimContext | undefined, + signal?: AbortSignal, +): Launching { + return reduceOnly ? { reduceOnly, launchers: freshLaunchers(operation, signal), checkout } : { reduceOnly }; +} + /** * Why `folder` is kept for an executable on PATH: one resolves to a path inside it, or a launcher directory could * not be inspected, so none can be ruled out. Null when every directory was read and none points into it. diff --git a/packages/stim-cli/src/workspace/regenerable-dirs.ts b/packages/stim-cli/src/workspace/regenerable-dirs.ts index de0605cc96..a3669cdf82 100644 --- a/packages/stim-cli/src/workspace/regenerable-dirs.ts +++ b/packages/stim-cli/src/workspace/regenerable-dirs.ts @@ -3,7 +3,7 @@ import { lstat, readdir, readFile } from 'node:fs/promises'; import { basename, dirname, join, relative, sep } from 'node:path'; import { canonicalPath } from '../commands/gc/paths.ts'; import { gitQuietAsync } from './git.ts'; -import { launcherReason, type LauncherScan } from './launchers.ts'; +import { launcherReason, type Launching } from './launchers.ts'; import { othersRunningAsync, readRunningProcessesAsync, runningFromReason } from './running-from.ts'; export type ArtifactKind = 'build' | 'dependencies'; @@ -219,12 +219,8 @@ export function isBuildDirPath(path: string): boolean { export async function regenerableDirProblem( root: string, path: string, - signal?: AbortSignal, - { - processes = true, - launchers, - launcherRoot = root, - }: { processes?: boolean; launchers?: () => Promise; launcherRoot?: string } = {}, + signal: AbortSignal | undefined, + { processes = true, launching }: { processes?: boolean; launching: Launching }, ): Promise { const rel = relative(root, path); try { @@ -245,8 +241,8 @@ export async function regenerableDirProblem( const unignored = await gitQuietAsync(root, ['ls-files', '--others', '--exclude-standard', '--', rel], { signal }); if (unignored === null) return 'git cannot tell whether it ignores every file in it'; if (unignored.trim()) return 'it holds files git does not ignore'; - if (launchers) { - const launcher = launcherReason(launcherRoot, await launchers()); + if (launching.reduceOnly) { + const launcher = launcherReason(launching.checkout, await launching.launchers()); if (launcher) return launcher; } if ( diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index fa67588ddc..a8c6341f44 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher read starts; fixing the folder restores reduction. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From a64b27731fde09b20cb255f4c09949a0523b8787 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 04:17:38 -0500 Subject: [PATCH 14/25] fix: prove project detection cannot block before the scan or revalidation runs it, retry a truncated scan after an hour and never overwrite the pending read --- llp/0004-discovered-checkouts.decision.md | 37 +++- packages/core/state/discovery-scan.ts | 2 +- .../__tests__/discovered-reduce-only.test.ts | 29 +++ .../src/__tests__/discovery-scan.test.ts | 72 ++++++++ .../stim-cli/src/__tests__/scan-rules.test.ts | 77 +++++++- packages/stim-cli/src/daemon/stages.ts | 2 +- packages/stim-cli/src/guide/cleanup.ts | 8 +- .../stim-cli/src/workspace/bounded-read.ts | 3 +- .../stim-cli/src/workspace/discovery-scan.ts | 44 +++-- packages/stim-cli/src/workspace/discovery.ts | 13 +- packages/stim-cli/src/workspace/scan-rules.ts | 166 ++++++++++++++++-- website/docs/worktrees.md | 2 +- 12 files changed, 405 insertions(+), 50 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 30d7d4be19..e3084a9c75 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -131,6 +131,28 @@ installed command. The record lets doctor report what the folder holds and the command that adds it to `maintenance.discoverRoots`; that listing is not part of this change. +Project detection (`recognises`) on a scan or revalidation path is bounded by +construction, because it reads files synchronously and a stalled mount, an +iCloud file waiting for a download or a FIFO would block the thread. Per folder it +reads `package.json` (existence and content), checks the existence of +`settings.gradle(.kts)`, `gradlew(.bat)`, `Package.swift`, `exact.mjs` and +`app.json`, lists the folder and stats its `*.xcodeproj` and `*.xcworkspace` +entries, lists `apps`, `examples` and `packages` and repeats this for up to 50 +children of each, and for a native Android or Xcode layout without an app +dependency walks every ancestor up to `/` doing the same. Before `recognises` +runs, the folder's own listing is checked: a detection file that is not a regular +file makes the folder not a checkout, a regular `package.json` is read first +through the bounded reader (a read that does not answer ends the scan truncated, +or leaves a cached find unselected and in the file), the children of `apps`, +`examples` and `packages` get the same treatment, and a native Android layout +below a folder with such a non-regular file is not recognised. An Xcode +container is not treated as a scan checkout, because its project files name +other files that cannot be listed ahead of time; sibling and root discovery still +recognise it. What stays unbounded: the ancestor reads of a native Android layout +for `/Users` and `/` (local system volumes), and the synchronous read of a file +that became unreadable between the bounded read and detection. Registered +projects and sibling discovery recognise as before. + A scan find outside a home dot-folder can also be an installed tool. Every route that deletes outputs, dependencies or Cargo data of a reduce-only checkout keeps the checkout while an executable reachable through a PATH directory is a symlink @@ -163,13 +185,12 @@ readings and the home scan (`bounded-read.ts`): while it is pending, no further launcher or scan read starts, every launcher reading reports incomplete immediately naming the folder, and a scheduled scan is skipped. A hung mount therefore holds one libuv thread per isolate, and reduction of reduce-only -checkouts stays off until that read settles. The daemon has two isolates that -read launchers, the main thread (the dormant stage and the pressure ladder, which -reach `clearStoppedOutputs`) and the passes thread (the artifact, dependency and -Cargo routes), so up to two reads can be outstanding in the daemon; the scan runs -only on the passes thread, and CLI commands are separate processes. A scan stops -at the first read that times out: the result is truncated and the previous finds -are kept. +checkouts stays off until that read settles. Two isolates in the daemon can +read launchers, the main thread and the passes thread, so up to two reads can be +outstanding there; the scan runs only on the passes thread, and CLI commands are +separate processes. A scan stops at the first read that times out: the result is +truncated and the previous finds are kept. A truncated scan, like a failed one, is +retried after an hour; the weekly schedule counts only from a scan that finished. A directory or entry that does not exist, and a dangling symlink, is no launcher. Any other failure to inspect a directory (permission, I/O, a timeout) keeps every reduce-only checkout for that operation, with a reason naming the @@ -220,7 +241,7 @@ minutes (not "least recently handled first"), the depth of seven, the folders pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and `~/Applications` left out, the low-disk scan running hourly while the daemon's disk level is above normal (high or critical, `disk.minFreeGb`) and no scan ran for an hour, the -retry of a failed scan after an hour, the scan running only in the daemon +retry of a failed or truncated scan after an hour (its result is partial), the scan running only in the daemon after the pass, and the PATH launcher keep. The scan is not deferred under host load; it takes about a second and a half on one MacBook. diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts index 4220039d79..461c12a538 100644 --- a/packages/core/state/discovery-scan.ts +++ b/packages/core/state/discovery-scan.ts @@ -9,7 +9,7 @@ export interface DiscoveryScan { version: 2; /** When the last attempt started, epoch ms. A failed or truncated attempt is stamped too. */ ranAt: number; - /** When the last attempt that did not fail started; 0 when none did. The weekly schedule counts from it. */ + /** When the last attempt that finished, neither failed nor truncated, started; 0 when none did. The weekly schedule counts from it. */ completedAt: number; /** When the last attempt that ran while a volume's level was above normal started; 0 when none did. */ lowScanAt: number; diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 6ce055fa17..30e5df09bc 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -2,6 +2,7 @@ import { execFileSync } from 'node:child_process'; import { chmodSync, existsSync, + readFileSync, mkdirSync, mkdtempSync, readdirSync, @@ -18,6 +19,7 @@ import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; import * as idleStop from '../stimd/passes/idle-stop.ts'; import { clearDependency } from '../daemon/dependencies.ts'; import * as runningFrom from '../workspace/running-from.ts'; +import { answered } from '../workspace/bounded-read.ts'; import { testContext } from './_context.ts'; import * as fsUtil from '../fs-util.ts'; import { deleteOutputs } from '../daemon/idle.ts'; @@ -474,6 +476,33 @@ describe('checkouts found through a tool folder or the full scan', () => { ); }); + it('leaves a cached scan find unselected while a read is pending, keeps it in the scan file, and selects it once the read settles', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(tool, { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeScan([tool]); + const file = join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'); + const before = readFileSync(file, 'utf8'); + let release: (value: string) => void = () => {}; + const hung = answered( + 'a stalled mount', + () => + new Promise((resolve) => { + release = resolve; + }), + 10, + ).catch(() => null); + await vi.advanceTimersByTimeAsync(10); + await hung; + const config = { projects: { [main]: {} }, repos: {} }; + expect(await discoverCheckouts(config, [main])).toEqual([]); + expect(readFileSync(file, 'utf8')).toBe(before); + + release('done'); + await vi.advanceTimersByTimeAsync(0); + expect((await discoverCheckouts(config, [main])).map((entry) => entry.path)).toEqual([tool]); + }); + it('drops a cached checkout that was replaced by a symlink to a folder the scan never enters, and plans nothing under it', async () => { const moved = join(base, 'elsewhere', 'app'); mkdirSync(moved, { recursive: true }); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index ba1eeb20ab..dc645e6974 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -9,6 +9,7 @@ import { symlinkSync, writeFileSync, } from 'node:fs'; +import { readdir } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { testContext } from './_context.ts'; @@ -96,6 +97,68 @@ describe('scanForCheckouts', () => { } }); + it('does not call detection for a folder whose manifest read does not answer, and ends the scan truncated with the folder unreadable', async () => { + project('dev', 'app'); + const detect = vi.fn<(path: string) => boolean>((path) => marked(path)); + let started = 0; + let release: (text: string) => void = () => {}; + const stuck = { + readEntries: (path: string) => readdir(path, { withFileTypes: true }), + readText: () => { + started++; + return new Promise((resolve) => { + release = resolve; + }); + }, + }; + const pending = scanForCheckouts(home, detect, { recognition: stuck, readTimeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await pending).toMatchObject({ truncated: true, unreadable: [join(home, 'dev', 'app')], found: [] }); + expect(detect).not.toHaveBeenCalledWith(join(home, 'dev', 'app')); + release('{}'); + await vi.advanceTimersByTimeAsync(0); + }); + + it('treats a folder whose manifest is not a regular file as not a checkout, and goes on', async () => { + mkdirSync(join(home, 'dev', 'fifo'), { recursive: true }); + project('dev', 'ok'); + const detect = vi.fn<(path: string) => boolean>((path) => marked(path)); + const entries = async (path: string) => { + const listing = await readdir(path, { withFileTypes: true }); + return path.endsWith('fifo') + ? [{ name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }] + : listing; + }; + const scan = await scanForCheckouts(home, detect, { entries }); + expect(scan.found).toEqual([join(home, 'dev', 'ok')]); + expect(detect).not.toHaveBeenCalledWith(join(home, 'dev', 'fifo')); + }); + + it('does not recognise a native Android layout below a folder whose manifest is not a regular file, since detection reads every ancestor', async () => { + mkdirSync(join(home, 'dev', 'app'), { recursive: true }); + const regular = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const fifo = { name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }; + const directory = { name: 'app', isFile: () => false, isDirectory: () => true, isSymbolicLink: () => false }; + const listing = (blocked: boolean) => async (path: string) => { + if (path === home) return [{ ...directory, name: 'dev' }]; + if (path === join(home, 'dev')) return blocked ? [fifo, directory] : [directory]; + return [regular('settings.gradle'), regular('gradlew')]; + }; + const detect = vi.fn<(path: string) => boolean>((path) => path.endsWith('app')); + expect((await scanForCheckouts(home, detect, { entries: listing(true) })).found).toEqual([]); + expect(detect).not.toHaveBeenCalledWith(join(home, 'dev', 'app')); + expect((await scanForCheckouts(home, detect, { entries: listing(false) })).found).toEqual([ + join(home, 'dev', 'app'), + ]); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); @@ -275,6 +338,15 @@ describe('runDiscoveryScanIfDue', () => { }); }); + it('retries a truncated scan after an hour, like a failed one, instead of waiting a week', async () => { + await runDiscoveryScanIfDue(tick(), { ...options(t0), budgetMs: -1 }); + expect(readDiscoveryScan()).toMatchObject({ truncated: true, completedAt: 0, ranAt: t0 }); + await runDiscoveryScanIfDue(tick(), options(t0 + hour / 2)); + expect(readDiscoveryScan()?.ranAt).toBe(t0); + await runDiscoveryScanIfDue(tick(), options(t0 + 2 * hour)); + expect(readDiscoveryScan()).toMatchObject({ truncated: false, completedAt: t0 + 2 * hour }); + }); + it('keeps the previous finds when a scan fails and records the attempt', async () => { await runDiscoveryScanIfDue(tick(), options(t0)); rmSync(join(base, 'home'), { recursive: true }); diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts index 44264a51ed..411a7cb03b 100644 --- a/packages/stim-cli/src/__tests__/scan-rules.test.ts +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -19,11 +19,11 @@ describe('scanEntryValid with a hand-edited scan result', () => { afterEach(() => rmSync(home, { recursive: true, force: true })); - it('accepts a real checkout the scan could have found', () => { - expect(scanEntryValid(project('dev', 'app'), rules())).toBe(true); + it('accepts a real checkout the scan could have found', async () => { + expect(await scanEntryValid(project('dev', 'app'), rules())).toBe('valid'); }); - it('refuses entries the scan never finds: the filesystem root, another home, the Stim home, pruned and skipped folders', () => { + it('refuses entries the scan never finds: the filesystem root, another home, the Stim home, pruned and skipped folders', async () => { const other = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-other-'))); try { writeFileSync(join(other, 'package.json'), '{}'); @@ -36,22 +36,81 @@ describe('scanEntryValid with a hand-edited scan result', () => { project('dev', 'node_modules', 'x'), project('Desktop', 'x'), ]) - expect(scanEntryValid(path, rules())).toBe(false); + expect(await scanEntryValid(path, rules())).toBe('invalid'); } finally { rmSync(other, { recursive: true, force: true }); } }); - it('refuses a path through a symlinked ancestor, a symlinked checkout and a different-case spelling of a pruned folder', () => { + it('refuses a path through a symlinked ancestor, a symlinked checkout and a different-case spelling of a pruned folder', async () => { const real = project('elsewhere', 'app'); mkdirSync(join(home, 'dev')); symlinkSync(join(home, 'elsewhere'), join(home, 'dev', 'link')); symlinkSync(real, join(home, 'dev', 'app-link')); - expect(scanEntryValid(join(home, 'dev', 'link', 'app'), rules())).toBe(false); - expect(scanEntryValid(join(home, 'dev', 'app-link'), rules())).toBe(false); + expect(await scanEntryValid(join(home, 'dev', 'link', 'app'), rules())).toBe('invalid'); + expect(await scanEntryValid(join(home, 'dev', 'app-link'), rules())).toBe('invalid'); const cached = project('.claude', 'x'); - expect(scanEntryValid(cached, rules())).toBe(false); + expect(await scanEntryValid(cached, rules())).toBe('invalid'); const upper = join(home, '.CLAUDE', 'x'); - expect(scanEntryValid(upper, rules())).toBe(false); + expect(await scanEntryValid(upper, rules())).toBe('invalid'); + }); + + describe('before project detection runs', () => { + const file = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const fifo = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const folder = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + + it('does not call detection for a detection file that is not regular, or for an Xcode container', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + for (const entries of [ + [fifo('package.json')], + [file('package.json'), fifo('gradlew')], + [folder('App.xcodeproj')], + ]) { + const io = { readEntries: async () => entries, readText: async () => '{"dependencies":{"expo":"*"}}' }; + const path = project('dev', 'app'); + const result = await scanEntryValid(path, { ...rules(), isCheckout: detect, io }); + expect(result).toBe('invalid'); + } + expect(detect).not.toHaveBeenCalled(); + }); + + it('does not call detection while a manifest read is pending, leaves the entry pending, and recovers once it settles', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + let release: (text: string) => void = () => {}; + const stuck = { + readEntries: async () => [file('package.json')], + readText: () => + new Promise((resolve) => { + release = resolve; + }), + }; + const path = project('dev', 'app'); + const first = scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 }); + await vi.advanceTimersByTimeAsync(10); + expect(await first).toBe('pending'); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 })).toBe('pending'); + expect(detect).not.toHaveBeenCalled(); + + release('{}'); + await vi.advanceTimersByTimeAsync(0); + const ok = { readEntries: async () => [file('package.json')], readText: async () => '{}' }; + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: ok })).toBe('valid'); + }); }); }); diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index 398a67257b..e06dbc9f3c 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -510,7 +510,7 @@ export async function enterDormantStage(action: MaintenanceAction, operation?: S const idleMs = await idleFor(path, thresholdMs, 'idle.dormantDays', { locked: true, operation, - reduceOnly: config.projects[path] === undefined, + reduceOnly: false, }); if (idleMs === null) return { status: 'kept', bytes: 0, detail: 'it is no longer idle' }; const owns = (kind: 'ios' | 'android') => diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 0eefc17180..2c20007de3 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1221,7 +1221,7 @@ Two more sources find checkouts that Stim only reduces and never removes and the checkouts of the full scan. The scan is a pruned walk of the home folder, run by the daemon after a maintenance pass has acted (never before its cleanup, never by a command): weekly, and also while the disk level is above -normal when no scan has run for an hour; a failed scan is retried after an hour. +normal when no scan has run for an hour; a failed or truncated scan is retried after an hour. The schedule is kept with the result, so a restarted daemon forgets nothing. It reads ~/Documents (macOS may ask once; a folder that does not answer within 3 seconds, such as one waiting for that answer, is recorded as unreadable, ends @@ -1230,7 +1230,11 @@ folders, other volumes, symlinks, dependency or build folders, deeper dot-folders, the Stim home and every dot-folder starting with .stim, the build worker root, and the tool homes and caches among the home folder's other dot-folders (.npm, .gradle, .cache and the like; .exact is entered), stops at a -recognised project and goes seven levels deep within 45 seconds. The result is +recognised project and goes seven levels deep within 45 seconds. Before project +detection runs on a folder, its detection files are proved safe to read: a +manifest that is not a regular file, or an Xcode container, makes the folder not a +scan checkout, and a manifest that does not answer in 3 seconds ends the scan +(cached finds stay unselected, not dropped). The result is kept in maintenance/discovery-scan.json under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a pass or gc never walks the disk. A scan that fails or stops early keeps the previous finds, diff --git a/packages/stim-cli/src/workspace/bounded-read.ts b/packages/stim-cli/src/workspace/bounded-read.ts index e1dd658630..abcdfc4d3f 100644 --- a/packages/stim-cli/src/workspace/bounded-read.ts +++ b/packages/stim-cli/src/workspace/bounded-read.ts @@ -15,9 +15,10 @@ export function hungRead(): string | null { /** * `start()`'s result, or an `Unanswered` rejection when it has not answered in `timeoutMs`; the read keeps running, and - * is the pending one until it settles. + * is the pending one until it settles. It starts no read at all while one is pending, so the slot is never overwritten. */ export async function answered(label: string, start: () => Promise, timeoutMs: number): Promise { + if (hung) throw new Unanswered(`a read of ${hung.label} is still pending`); const read = start(); let timer: NodeJS.Timeout | undefined; try { diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index bd8796a8b7..51b5f0d8f0 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -15,7 +15,15 @@ import { diskPressure, diskThresholds } from '../daemon/pressure.ts'; import { answered, hungRead, Unanswered } from './bounded-read.ts'; import { loadConfig } from './config.ts'; import { recognises } from './discovery.ts'; -import { MAX_DEPTH, scanDescends, scanSkipper } from './scan-rules.ts'; +import { + MAX_DEPTH, + REAL_RECOGNITION_IO, + needsAncestors, + recognitionReads, + scanDescends, + scanSkipper, + type RecognitionIo, +} from './scan-rules.ts'; const HOUR_MS = 60 * 60_000; const WEEK_MS = 7 * 24 * HOUR_MS; @@ -40,7 +48,9 @@ const refused = (error: unknown) => ['EACCES', 'EPERM'].includes(errorCode(error * A read that does not answer within 3 seconds stops the walk at once: the folder is recorded unreadable, the result is * truncated, and the read stays the pending one (`bounded-read.ts`), so no further read starts, here or in a later scan, * until it settles. It also stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or `budgetMs`, and says so in - * `truncated`. An abort rejects. + * `truncated`. Before `isCheckout` runs on a folder, `recognitionReads` proves that project detection cannot block + * there: a detection file that is not regular, or an Xcode container, makes the folder not a checkout, and a manifest + * read that does not answer ends the walk like any other timed-out read. An abort rejects. */ export async function scanForCheckouts( home: string, @@ -52,13 +62,15 @@ export async function scanForCheckouts( readTimeoutMs = READ_TIMEOUT_MS, stat: statOf = (path) => lstat(path), entries: entriesOf = (path) => readdir(path, { withFileTypes: true }), + recognition = REAL_RECOGNITION_IO, }: { budgetMs?: number; signal?: AbortSignal; skip?: (path: string) => boolean; readTimeoutMs?: number; stat?: (path: string) => Promise<{ dev: number }>; - entries?: (path: string) => Promise<{ name: string; isDirectory(): boolean; isSymbolicLink(): boolean }[]>; + entries?: RecognitionIo['readEntries']; + recognition?: RecognitionIo; } = {}, ): Promise { const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false }; @@ -87,7 +99,7 @@ export async function scanForCheckouts( if (!stopAt(home, error)) throw error; return outcome; } - const walk = async (folder: string, depth: number): Promise => { + const walk = async (folder: string, depth: number, unsafeAbove: boolean): Promise => { if (exhausted()) return; outcome.visited++; let entries; @@ -97,7 +109,14 @@ export async function scanForCheckouts( stopAt(folder, error); return; } - if (depth > 0 && isCheckout(folder)) { + const reads = await recognitionReads(folder, entries, readTimeoutMs, recognition, depth > 0); + if (reads === 'unreadable') { + outcome.unreadable.push(folder); + outcome.truncated = true; + return; + } + const above = unsafeAbove || reads === 'unsafe'; + if (depth > 0 && reads === 'safe' && !(above && needsAncestors(entries)) && isCheckout(folder)) { outcome.found.push(folder); return; } @@ -114,10 +133,10 @@ export async function scanForCheckouts( if (stopAt(path, error)) return; continue; } - if (info.dev === volume) await walk(path, depth + 1); + if (info.dev === volume) await walk(path, depth + 1, above); } }; - await walk(home, 0); + await walk(home, 0, false); return outcome; } @@ -134,8 +153,9 @@ export function mergeFound(previous: readonly string[], outcome: ScanOutcome): s /** * Whether a scan is due. Never twice within an hour, so a failing or truncated scan is retried hourly and a rise and - * fall of the disk level cannot start a run of scans; then weekly from the last attempt that did not fail, or hourly - * while the level is above normal and no scan has run during it for an hour. + * fall of the disk level cannot start a run of scans; then weekly from the last scan that finished (a failed or + * truncated one is partial and does not count), or hourly while the level is above normal and no scan has run during + * it for an hour. */ export function discoveryScanDue( last: Pick | null, @@ -178,7 +198,8 @@ export async function runDiscoveryScanIfDue( home = homedir(), now = Date.now(), attempted = attempts, - }: { home?: string; now?: number; attempted?: { at: number } } = {}, + budgetMs, + }: { home?: string; now?: number; attempted?: { at: number }; budgetMs?: number } = {}, ): Promise { if (!tick || isScopedHome(tick.operation.env)) return; const { operation } = tick; @@ -207,10 +228,11 @@ export async function runDiscoveryScanIfDue( const outcome = await scanForCheckouts(realHome, recognises, { signal: operation.signal, skip: scanSkipper(realHome), + ...(budgetMs === undefined ? {} : { budgetMs }), }); writeScan({ ...attempt, - completedAt: now, + completedAt: outcome.truncated ? (last?.completedAt ?? 0) : now, durationMs: Date.now() - now, ...outcome, found: mergeFound(last?.found ?? [], { ...outcome, found: outcome.found.filter((p) => !dotFolder(p)) }), diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index 9e79d2be1e..2b17887322 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -261,12 +261,15 @@ function discovery( } /** The checkouts of the last full scan that are still what the scan found, see `scanEntryValid`. */ -function revalidatedScan(): DiscoveryChild[] { +async function revalidatedScan(): Promise { const home = canonicalPath(homedir()); const rules = { home, isCheckout: recognises, skip: scanSkipper(home) }; - return (readDiscoveryScan()?.found ?? []) - .filter((path) => isAbsolute(path) && scanEntryValid(path, rules)) - .map((path) => ({ path, origin: 'scan', from: 'full scan', recognised: true })); + const valid: DiscoveryChild[] = []; + const ancestors = new Map(); + for (const path of readDiscoveryScan()?.found ?? []) + if (isAbsolute(path) && (await scanEntryValid(path, { ...rules, ancestors })) === 'valid') + valid.push({ path, origin: 'scan', from: 'full scan', recognised: true }); + return valid; } async function computeDiscovered( @@ -337,7 +340,7 @@ async function computeDiscovered( else outside.push({ path, from: main }); } } - const scanned: DiscoveryChild[] = scoped ? [] : revalidatedScan(); + const scanned: DiscoveryChild[] = scoped ? [] : await revalidatedScan(); const allowed = [...children, ...scanned].filter( (child) => canonicalRoots.some((root) => inside(child.path, root)) || !isProtectedLocation(child.path), ); diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index e5d905fcc1..c4f59f4381 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -1,9 +1,12 @@ import { lstatSync, realpathSync, statSync } from 'node:fs'; +import { readdir, readFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; -import { relative, sep } from 'node:path'; +import { dirname, join, relative, sep } from 'node:path'; import { configDir } from '@stim-cli/core'; import { buildWorkerRoot } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; +import { answered, hungRead, Unanswered } from './bounded-read.ts'; +import { declaresAppDependency } from './project-files.ts'; import { loadConfig } from './config.ts'; export const MAX_DEPTH = 7; @@ -123,24 +126,165 @@ export function scanSkipper(home: string): (path: string) => boolean { stim.some((folder) => within(path, folder)) || (!homeIsTemporary && temps.some((temp) => within(path, temp))); } +/** The files of a folder that project detection reads or stats: the manifests every integration's `inspect` looks at. */ +const DETECTION_FILES = new Set([ + 'package.json', + 'settings.gradle', + 'settings.gradle.kts', + 'gradlew', + 'gradlew.bat', + 'Package.swift', + 'exact.mjs', + 'app.json', +]); +const PROJECT_FOLDERS = ['apps', 'examples', 'packages']; +const PROJECT_FOLDER_ENTRIES = 50; +const XCODE_CONTAINER = /\.(xcodeproj|xcworkspace)$/; + +interface Entry { + name: string; + isFile(): boolean; + isDirectory(): boolean; + isSymbolicLink(): boolean; +} + +/** How the recognition gate reads the disk; injected by tests. */ +export interface RecognitionIo { + readEntries: (path: string) => Promise; + readText: (path: string) => Promise; +} + +export const REAL_RECOGNITION_IO: RecognitionIo = { + readEntries: (path) => readdir(path, { withFileTypes: true }), + readText: (path) => readFile(path, 'utf8'), +}; + +/** + * Whether `recognises` can be called on `folder` without a read that could block: `safe`; `unsafe` when it could (a + * detection file that is not a regular file, such as a FIFO or a symlink into a stalled mount, or an Xcode container, + * whose project files `containers()` follows by the paths they name), so the folder is not treated as a checkout; + * `unreadable` when a read did not answer in time, or another read is still pending (`bounded-read.ts`). `entries` is + * the folder's listing. A regular `package.json`, the one detection file project detection reads rather than stats, is + * read here first through the bounded reader, so an evicted or stalled file is found out before the synchronous read. + * `apps`, `examples` and `packages` children are checked the same way, because `recognises` looks inside them. + */ +export async function recognitionReads( + folder: string, + entries: readonly Entry[], + timeoutMs: number, + io: RecognitionIo = REAL_RECOGNITION_IO, + nested = true, +): Promise<'safe' | 'unsafe' | 'unreadable'> { + if (hungRead() !== null) return 'unreadable'; + const detection = entries.filter((entry) => DETECTION_FILES.has(entry.name)); + if (detection.some((entry) => !entry.isFile())) return 'unsafe'; + const hasContainer = entries.some((entry) => XCODE_CONTAINER.test(entry.name)); + try { + if (detection.some((entry) => entry.name === 'package.json')) { + const text = await answered(folder, () => io.readText(join(folder, 'package.json')), timeoutMs); + let declares = false; + try { + declares = declaresAppDependency(JSON.parse(text)); + } catch {} + if (hasContainer && !declares) return 'unsafe'; + } else if (hasContainer) return 'unsafe'; + if (!nested) return 'safe'; + for (const name of PROJECT_FOLDERS) { + const entry = entries.find((candidate) => candidate.name === name); + if (!entry) continue; + if (entry.isSymbolicLink()) return 'unsafe'; + if (!entry.isDirectory()) continue; + const children = (await answered(folder, () => io.readEntries(join(folder, name)), timeoutMs)) + .filter((child) => child.isDirectory() && !child.isSymbolicLink()) + .slice(0, PROJECT_FOLDER_ENTRIES); + for (const child of children) { + const path = join(folder, name, child.name); + const inner = await answered(folder, () => io.readEntries(path), timeoutMs).catch((error: unknown) => { + if (error instanceof Unanswered) throw error; + return null; + }); + if (inner === null) continue; + const result = await recognitionReads(path, inner, timeoutMs, io, false); + if (result !== 'safe') return result; + } + } + return 'safe'; + } catch (error) { + if (error instanceof Unanswered) return 'unreadable'; + return 'unsafe'; + } +} + +/** Whether detection in `entries`' folder walks up the ancestors: a native Android layout is a candidate root. */ +export const needsAncestors = (entries: readonly Entry[]): boolean => + entries.some((entry) => entry.name === 'settings.gradle' || entry.name === 'settings.gradle.kts'); + /** * Whether a checkout a scan recorded may still be used: it is a real directory (not a symlink), its canonical path is * the recorded one, it lies below the home folder on the home volume, no folder between the two is one the scan skips - * (`scanDescends`, `skip`), and `isCheckout` still recognises it. Anything else, such as a checkout moved away with a - * symlink left behind, is dropped. + * (`scanDescends`, `skip`), its detection reads are bounded (`recognitionReads`) and `isCheckout` still recognises it. + * `invalid` drops it, such as a checkout moved away with a symlink left behind; `pending` means a read did not answer, + * so it is not selected this time and stays in the scan file. */ -export function scanEntryValid( +export async function scanEntryValid( path: string, - { home, isCheckout, skip }: { home: string; isCheckout: (path: string) => boolean; skip: (path: string) => boolean }, -): boolean { + { + home, + isCheckout, + skip, + timeoutMs = 3000, + io = REAL_RECOGNITION_IO, + ancestors = new Map(), + }: { + home: string; + isCheckout: (path: string) => boolean; + skip: (path: string) => boolean; + timeoutMs?: number; + io?: RecognitionIo; + /** The gate results of folders above checkouts, shared by the entries of one revalidation. */ + ancestors?: Map>; + }, +): Promise<'valid' | 'invalid' | 'pending'> { try { if (!within(path, home) || path === home || lstatSync(path).isSymbolicLink() || !statSync(path).isDirectory()) - return false; - if (realpathSync.native(path) !== path || statSync(path).dev !== statSync(home).dev || skip(path)) return false; + return 'invalid'; + if (realpathSync.native(path) !== path || statSync(path).dev !== statSync(home).dev || skip(path)) return 'invalid'; const names = relative(home, path).split(sep); - if (names.length > MAX_DEPTH) return false; - return names.every((name, i) => scanDescends(name, i + 1)) && isCheckout(path); + if (names.length > MAX_DEPTH || !names.every((name, i) => scanDescends(name, i + 1))) return 'invalid'; } catch { - return false; + return 'invalid'; + } + if (hungRead() !== null) return 'pending'; + let entries: Entry[]; + try { + entries = await answered(path, () => io.readEntries(path), timeoutMs); + } catch (error) { + return error instanceof Unanswered ? 'pending' : 'invalid'; + } + const reads = await recognitionReads(path, entries, timeoutMs, io); + if (reads === 'unreadable') return 'pending'; + if (reads !== 'safe') return 'invalid'; + if (needsAncestors(entries)) { + const above = [ + home, + ...relative(home, dirname(path)) + .split(sep) + .filter(Boolean) + .map((_, i, names) => join(home, ...names.slice(0, i + 1))), + ]; + for (const folder of above) { + let result = ancestors.get(folder); + if (!result) { + result = answered(folder, () => io.readEntries(folder), timeoutMs).then( + (listing) => recognitionReads(folder, listing, timeoutMs, io, false), + (error: unknown) => (error instanceof Unanswered ? ('unreadable' as const) : ('unsafe' as const)), + ); + ancestors.set(folder, result); + } + const gate = await result; + if (gate === 'unreadable') return 'pending'; + if (gate === 'unsafe') return 'invalid'; + } } + return isCheckout(path) ? 'valid' : 'invalid'; } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index a8c6341f44..db8700b652 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed or truncated scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests there can be read without hanging: a folder whose manifest is not a regular file, or that holds an Xcode project, is not treated as a checkout by the scan, and a manifest that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 9b4a97f9e40ba3c60ceb26107b65bab60a0a47fb Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 04:42:41 -0500 Subject: [PATCH 15/25] fix: check every folder on the way to a checkout structurally, read cached entry metadata through the bounded reader and schedule retries by how a scan ended --- llp/0004-discovered-checkouts.decision.md | 57 +++--- packages/core/state/discovery-scan.ts | 14 +- .../__tests__/discovered-reduce-only.test.ts | 1 + .../src/__tests__/discovery-scan.test.ts | 86 +++++++- .../stim-cli/src/__tests__/discovery.test.ts | 1 + .../stim-cli/src/__tests__/scan-rules.test.ts | 89 +++++++-- packages/stim-cli/src/guide/cleanup.ts | 15 +- .../stim-cli/src/workspace/discovery-scan.ts | 98 +++++++--- packages/stim-cli/src/workspace/scan-rules.ts | 183 ++++++++++-------- website/docs/worktrees.md | 2 +- 10 files changed, 388 insertions(+), 158 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index e3084a9c75..715db3765d 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -131,27 +131,34 @@ installed command. The record lets doctor report what the folder holds and the command that adds it to `maintenance.discoverRoots`; that listing is not part of this change. -Project detection (`recognises`) on a scan or revalidation path is bounded by -construction, because it reads files synchronously and a stalled mount, an -iCloud file waiting for a download or a FIFO would block the thread. Per folder it -reads `package.json` (existence and content), checks the existence of -`settings.gradle(.kts)`, `gradlew(.bat)`, `Package.swift`, `exact.mjs` and -`app.json`, lists the folder and stats its `*.xcodeproj` and `*.xcworkspace` -entries, lists `apps`, `examples` and `packages` and repeats this for up to 50 -children of each, and for a native Android or Xcode layout without an app -dependency walks every ancestor up to `/` doing the same. Before `recognises` -runs, the folder's own listing is checked: a detection file that is not a regular -file makes the folder not a checkout, a regular `package.json` is read first -through the bounded reader (a read that does not answer ends the scan truncated, -or leaves a cached find unselected and in the file), the children of `apps`, -`examples` and `packages` get the same treatment, and a native Android layout -below a folder with such a non-regular file is not recognised. An Xcode -container is not treated as a scan checkout, because its project files name -other files that cannot be listed ahead of time; sibling and root discovery still -recognise it. What stays unbounded: the ancestor reads of a native Android layout -for `/Users` and `/` (local system volumes), and the synchronous read of a file -that became unreadable between the bounded read and detection. Registered -projects and sibling discovery recognise as before. +Project detection (`recognises`) reads files synchronously, and a stalled mount, +an iCloud file waiting for a download or a FIFO would block the thread. When the +scan or the revalidation decides whether a folder is a checkout, every read +detection makes is bounded by construction. Detection inspects the folder, up to +50 children of each of its `apps`, `examples` and `packages` folders, those +container folders themselves and every ancestor up to `/`; each inspection reads +`package.json`, checks the existence of `settings.gradle(.kts)`, `gradlew(.bat)`, +`Package.swift`, `exact.mjs` and `app.json`, and lists the folder, and an Xcode +container without an app dependency makes it parse project files that name other +files. So one check, `folderSafety`, applies to every folder the walk passes from +the home folder down and to each folder detection inspects below the one being +decided (`nestedSafety`), with no condition: a detection file that is not a +regular file, or an Xcode container with no app dependency in `package.json`, makes +the folder unsafe; a regular `package.json` is read once through the bounded +reader first. An unsafe folder makes its whole subtree unrecognisable and the walk +does not descend into it. A manifest read that does not answer ends the scan +truncated with the folder unreadable (`truncation: read`), or leaves a cached find +unselected and in the file. Revalidation checks the chain from the home folder to +the entry with the same function, memoised per discovery call, and reads the +entry's own metadata (`lstat`, `stat`, `realpath`) through the bounded reader too. +Not bounded, and stated as such: the ancestor reads above the home folder (`/Users` +and `/`); a file that becomes unreadable between the bounded read and detection; +and everything a pass later reads synchronously inside a checkout once it is +discovered (artifact collection, app roots, state), which is the same for +registered and sibling-discovered checkouts today. Registered projects and +sibling discovery recognise as before. An Xcode container without an app +dependency is not a scan checkout, though sibling and root discovery still +recognise it. A scan find outside a home dot-folder can also be an installed tool. Every route that deletes outputs, dependencies or Cargo data of a reduce-only checkout keeps @@ -189,8 +196,10 @@ checkouts stays off until that read settles. Two isolates in the daemon can read launchers, the main thread and the passes thread, so up to two reads can be outstanding there; the scan runs only on the passes thread, and CLI commands are separate processes. A scan stops at the first read that times out: the result is -truncated and the previous finds are kept. A truncated scan, like a failed one, is -retried after an hour; the weekly schedule counts only from a scan that finished. +truncated and the previous finds are kept. A scan that failed or ended on a timed-out +read is retried after an hour whatever the disk level; one that stopped at the folder +cap or the time budget follows the weekly schedule (hourly while the disk level is above +normal), since a retry would hit the same limit. A directory or entry that does not exist, and a dangling symlink, is no launcher. Any other failure to inspect a directory (permission, I/O, a timeout) keeps every reduce-only checkout for that operation, with a reason naming the @@ -241,7 +250,7 @@ minutes (not "least recently handled first"), the depth of seven, the folders pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and `~/Applications` left out, the low-disk scan running hourly while the daemon's disk level is above normal (high or critical, `disk.minFreeGb`) and no scan ran for an hour, the -retry of a failed or truncated scan after an hour (its result is partial), the scan running only in the daemon +retry after an hour of a scan that failed or ended on a read that timed out, while one that stopped at the folder cap or the time budget counts as finished (`truncation` records which), the scan running only in the daemon after the pass, and the PATH launcher keep. The scan is not deferred under host load; it takes about a second and a half on one MacBook. diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts index 461c12a538..bdc19f061d 100644 --- a/packages/core/state/discovery-scan.ts +++ b/packages/core/state/discovery-scan.ts @@ -1,6 +1,12 @@ import { isJsonObject, readJsonObject } from './json-file.ts'; import { discoveryScanFile } from './paths.ts'; +/** How an attempt ended: finished, cut short by the folder cap or the time budget, or by a read that timed out or a failure. */ +export type ScanTruncation = 'none' | 'budget' | 'folders' | 'read' | 'failed'; + +const isTruncation = (value: unknown): value is ScanTruncation => + value === 'none' || value === 'budget' || value === 'folders' || value === 'read' || value === 'failed'; + /** * What the last full scan of the home folder found: the checkouts Stim's project detection recognises that * `maintenance.discoverRoots`, registered projects and the common tool folders do not reach, and what it could not read. @@ -29,8 +35,10 @@ export interface DiscoveryScan { dotFolderFound: string[]; /** Folders the system refused or that did not answer, skipped without error. */ unreadable: string[]; - /** The scan stopped at its time or folder limit before it finished. */ + /** The scan stopped at its time or folder limit, or on a read or failure, before it finished. */ truncated: boolean; + /** How the attempt ended. `read` and `failed` are retried after an hour; the others follow the weekly schedule. */ + truncation: ScanTruncation; } const strings = (value: unknown): value is string[] => @@ -51,7 +59,8 @@ export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { !strings(value.found) || !strings(value.dotFolderFound) || !strings(value.unreadable) || - typeof value.truncated !== 'boolean' + typeof value.truncated !== 'boolean' || + !isTruncation(value.truncation) ) return null; return { @@ -66,6 +75,7 @@ export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { dotFolderFound: value.dotFolderFound, unreadable: value.unreadable, truncated: value.truncated, + truncation: value.truncation, }; } diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 30e5df09bc..d0e93cecf6 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -133,6 +133,7 @@ describe('checkouts found through a tool folder or the full scan', () => { dotFolderFound, unreadable: [], truncated: false, + truncation: 'none', }; writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); }; diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index dc645e6974..585d3ec111 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -15,7 +15,8 @@ import { join } from 'node:path'; import { testContext } from './_context.ts'; import { discoveryScanFile, parseDiscoveryScan, readDiscoveryScan } from '@stim-cli/core/state'; import { discoveryScanDue, mergeFound, runDiscoveryScanIfDue, scanForCheckouts } from '../workspace/discovery-scan.ts'; -import { scanDescends } from '../workspace/scan-rules.ts'; +import { REAL_RECOGNITION_IO, scanDescends } from '../workspace/scan-rules.ts'; +import type { ScanTruncation } from '@stim-cli/core/state'; const marked = (path: string) => existsSync(join(path, 'package.json')); @@ -103,6 +104,7 @@ describe('scanForCheckouts', () => { let started = 0; let release: (text: string) => void = () => {}; const stuck = { + ...REAL_RECOGNITION_IO, readEntries: (path: string) => readdir(path, { withFileTypes: true }), readText: () => { started++; @@ -159,6 +161,37 @@ describe('scanForCheckouts', () => { ]); }); + it('calls no detection on a folder or below it when its apps folder holds a manifest that is not a regular file', async () => { + mkdirSync(join(home, 'mono', 'apps', 'x'), { recursive: true }); + const regular = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const fifo = { name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }; + const listing = (blocked: boolean) => async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return blocked ? [fifo, directory('x')] : [directory('x')]; + return [regular('settings.gradle'), regular('gradlew')]; + }; + const recognition = { ...REAL_RECOGNITION_IO, readEntries: async (path: string) => listing(true)(path) }; + const detect = vi.fn<(path: string) => boolean>(() => true); + expect((await scanForCheckouts(home, detect, { entries: listing(true), recognition })).found).toEqual([]); + expect(detect).not.toHaveBeenCalled(); + const clean = { ...REAL_RECOGNITION_IO, readEntries: async (path: string) => listing(false)(path) }; + expect((await scanForCheckouts(home, detect, { entries: listing(false), recognition: clean })).found).toEqual([ + join(home, 'mono'), + ]); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); @@ -168,10 +201,11 @@ describe('scanForCheckouts', () => { describe('discoveryScanDue', () => { const now = Date.UTC(2026, 10, 14); - const at = (ranAgo: number, completedAgo = ranAgo, lowAgo = 1e12) => ({ + const at = (ranAgo: number, completedAgo = ranAgo, lowAgo = 1e12, truncation: ScanTruncation = 'none') => ({ ranAt: now - ranAgo, completedAt: now - completedAgo, lowScanAt: now - lowAgo, + truncation, }); const hour = 3_600_000; const day = 24 * hour; @@ -183,6 +217,17 @@ describe('discoveryScanDue', () => { expect(discoveryScanDue(at(7 * day), now, false)).toBe(true); }); + it('retries an attempt that ended on a read or a failure after an hour at any disk level, even right after a success', () => { + expect(discoveryScanDue(at(2 * hour, 24 * hour, 2 * hour, 'read'), now, false)).toBe(true); + expect(discoveryScanDue(at(2 * hour, 24 * hour, 2 * hour, 'failed'), now, false)).toBe(true); + expect(discoveryScanDue(at(30 * 60_000, 24 * hour, 2 * hour, 'read'), now, false)).toBe(false); + }); + + it('treats an attempt that stopped at the folder cap or the time budget as finished, so it is not retried hourly at normal disk', () => { + expect(discoveryScanDue(at(2 * hour, 2 * hour, 1e12, 'budget'), now, false)).toBe(false); + expect(discoveryScanDue(at(2 * hour, 2 * hour, 1e12, 'folders'), now, true)).toBe(true); + }); + it('retries a failed scan after an hour, not after a week, and never twice within an hour', () => { expect(discoveryScanDue(at(30 * 60_000, 8 * day), now, false)).toBe(false); expect(discoveryScanDue(at(2 * hour, 8 * day), now, false)).toBe(true); @@ -201,6 +246,7 @@ describe('mergeFound', () => { found, unreadable, truncated, + truncation: truncated ? ('read' as const) : ('none' as const), visited: 1, }); it('keeps the previous finds under an unreadable folder, drops the others, and keeps all of them when truncated', () => { @@ -338,13 +384,37 @@ describe('runDiscoveryScanIfDue', () => { }); }); - it('retries a truncated scan after an hour, like a failed one, instead of waiting a week', async () => { - await runDiscoveryScanIfDue(tick(), { ...options(t0), budgetMs: -1 }); - expect(readDiscoveryScan()).toMatchObject({ truncated: true, completedAt: 0, ranAt: t0 }); - await runDiscoveryScanIfDue(tick(), options(t0 + hour / 2)); - expect(readDiscoveryScan()?.ranAt).toBe(t0); + it('counts a scan that stopped at its time budget as finished, so it is not retried hourly at normal disk', async () => { + await runDiscoveryScanIfDue(tick(), { ...options(t0), scanOptions: { budgetMs: -1 } }); + expect(readDiscoveryScan()).toMatchObject({ truncated: true, truncation: 'budget', completedAt: t0, ranAt: t0 }); await runDiscoveryScanIfDue(tick(), options(t0 + 2 * hour)); - expect(readDiscoveryScan()).toMatchObject({ truncated: false, completedAt: t0 + 2 * hour }); + expect(readDiscoveryScan()?.ranAt).toBe(t0); + }); + + it('retries a scan that ended on a read that timed out after an hour at any disk level, though a scan finished recently', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ truncation: 'none', completedAt: t0 }); + let started = 0; + let release: (names: []) => void = () => {}; + const entries = () => { + started++; + return new Promise<[]>((resolve) => { + release = resolve; + }); + }; + const second = runDiscoveryScanIfDue(tick(low), { + ...options(t0 + 25 * hour), + scanOptions: { entries, readTimeoutMs: 10 }, + }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + await second; + expect(readDiscoveryScan()).toMatchObject({ truncation: 'read', completedAt: t0, ranAt: t0 + 25 * hour }); + release([]); + await vi.advanceTimersByTimeAsync(0); + await runDiscoveryScanIfDue(tick(), options(t0 + 27 * hour)); + expect(readDiscoveryScan()).toMatchObject({ truncation: 'none', ranAt: t0 + 27 * hour }); }); it('keeps the previous finds when a scan fails and records the attempt', async () => { diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index 4aac6eeae0..272c80ce6f 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -172,6 +172,7 @@ describe('discoverCheckouts', () => { dotFolderFound: [], unreadable: [], truncated: false, + truncation: 'none', }; writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); expect( diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts index 411a7cb03b..d662caa257 100644 --- a/packages/stim-cli/src/__tests__/scan-rules.test.ts +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -1,7 +1,7 @@ import { existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { scanEntryValid } from '../workspace/scan-rules.ts'; +import { REAL_RECOGNITION_IO, scanEntryValid } from '../workspace/scan-rules.ts'; describe('scanEntryValid with a hand-edited scan result', () => { let home: string; @@ -75,33 +75,96 @@ describe('scanEntryValid with a hand-edited scan result', () => { isSymbolicLink: () => false, }); - it('does not call detection for a detection file that is not regular, or for an Xcode container', async () => { + const listing = (listings: Record[]>) => ({ + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => listings[path] ?? [], + readText: async () => '{}', + }); + + it('does not call detection for a detection file that is not regular, or for an Xcode container without an app dependency', async () => { const detect = vi.fn<(path: string) => boolean>(() => true); + const path = project('dev', 'app'); for (const entries of [ [fifo('package.json')], [file('package.json'), fifo('gradlew')], [folder('App.xcodeproj')], - ]) { - const io = { readEntries: async () => entries, readText: async () => '{"dependencies":{"expo":"*"}}' }; - const path = project('dev', 'app'); - const result = await scanEntryValid(path, { ...rules(), isCheckout: detect, io }); - expect(result).toBe('invalid'); + [file('package.json'), folder('App.xcodeproj')], + ]) + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: listing({ [path]: entries }) })).toBe( + 'invalid', + ); + expect(detect).not.toHaveBeenCalled(); + const app = { + ...listing({ [path]: [file('package.json'), folder('App.xcodeproj')] }), + readText: async () => '{"dependencies":{"expo":"*"}}', + }; + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: app })).toBe('valid'); + }); + + it('does not call detection when an ancestor, the home folder or an apps folder holds a manifest that is not a regular file', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + const path = project('mono', 'apps', 'x'); + const android = [file('settings.gradle'), file('gradlew')]; + const control = listing({ + [join(home, 'mono')]: [folder('apps')], + [join(home, 'mono', 'apps')]: [folder('x')], + [path]: android, + }); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: control })).toBe('valid'); + detect.mockClear(); + for (const unsafe of [home, join(home, 'mono'), join(home, 'mono', 'apps')]) { + const io = { + ...control, + readEntries: async (at: string) => (at === unsafe ? [fifo('package.json')] : await control.readEntries(at)), + }; + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io })).toBe('invalid'); } expect(detect).not.toHaveBeenCalled(); }); + it('treats a metadata read that does not answer as pending, before any detection', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + let release: (value: { isSymbolicLink(): boolean }) => void = () => {}; + let started = 0; + const stuck = { + ...listing({}), + lstat: () => { + started++; + return new Promise<{ isSymbolicLink(): boolean }>((resolve) => { + release = resolve; + }); + }, + }; + const path = project('dev', 'app'); + const first = scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await first).toBe('pending'); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck })).toBe('pending'); + expect(started).toBe(1); + expect(detect).not.toHaveBeenCalled(); + release({ isSymbolicLink: () => false }); + await vi.advanceTimersByTimeAsync(0); + }); + it('does not call detection while a manifest read is pending, leaves the entry pending, and recovers once it settles', async () => { const detect = vi.fn<(path: string) => boolean>(() => true); let release: (text: string) => void = () => {}; + let started = 0; + const path = project('dev', 'app'); const stuck = { - readEntries: async () => [file('package.json')], - readText: () => - new Promise((resolve) => { + ...listing({ [path]: [file('package.json')] }), + readText: () => { + started++; + return new Promise((resolve) => { release = resolve; - }), + }); + }, }; - const path = project('dev', 'app'); const first = scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); await vi.advanceTimersByTimeAsync(10); expect(await first).toBe('pending'); expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 })).toBe('pending'); @@ -109,7 +172,7 @@ describe('scanEntryValid with a hand-edited scan result', () => { release('{}'); await vi.advanceTimersByTimeAsync(0); - const ok = { readEntries: async () => [file('package.json')], readText: async () => '{}' }; + const ok = listing({ [path]: [file('package.json')] }); expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: ok })).toBe('valid'); }); }); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 2c20007de3..2342b37665 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1221,7 +1221,8 @@ Two more sources find checkouts that Stim only reduces and never removes and the checkouts of the full scan. The scan is a pruned walk of the home folder, run by the daemon after a maintenance pass has acted (never before its cleanup, never by a command): weekly, and also while the disk level is above -normal when no scan has run for an hour; a failed or truncated scan is retried after an hour. +normal when no scan has run for an hour; a scan that failed or ended on a read that timed out is retried after an hour, +one that stopped at its folder cap or time budget is not. The schedule is kept with the result, so a restarted daemon forgets nothing. It reads ~/Documents (macOS may ask once; a folder that does not answer within 3 seconds, such as one waiting for that answer, is recorded as unreadable, ends @@ -1231,10 +1232,14 @@ dot-folders, the Stim home and every dot-folder starting with .stim, the build worker root, and the tool homes and caches among the home folder's other dot-folders (.npm, .gradle, .cache and the like; .exact is entered), stops at a recognised project and goes seven levels deep within 45 seconds. Before project -detection runs on a folder, its detection files are proved safe to read: a -manifest that is not a regular file, or an Xcode container, makes the folder not a -scan checkout, and a manifest that does not answer in 3 seconds ends the scan -(cached finds stay unselected, not dropped). The result is +detection runs on a folder, the manifests detection reads are proved safe, for +the folder, each ancestor down from the home folder, its apps, examples and +packages folders and their children: a manifest that is not a regular file, or an +Xcode container with no app dependency in package.json, makes that subtree not a +scan checkout, and a manifest or file metadata read that does not answer in 3 +seconds ends the scan (cached finds stay unselected, not dropped). Not bounded: +the ancestors above the home folder, a file that becomes unreadable between that +check and detection, and what a pass reads inside a checkout once discovered. The result is kept in maintenance/discovery-scan.json under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a pass or gc never walks the disk. A scan that fails or stops early keeps the previous finds, diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 51b5f0d8f0..584be43eed 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -4,6 +4,7 @@ import { homedir } from 'node:os'; import { join, relative, sep } from 'node:path'; import { isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; import { + type ScanTruncation, discoveryScanFile, maintenanceDir, readDiscoveryScan, @@ -18,8 +19,8 @@ import { recognises } from './discovery.ts'; import { MAX_DEPTH, REAL_RECOGNITION_IO, - needsAncestors, - recognitionReads, + folderSafety, + nestedSafety, scanDescends, scanSkipper, type RecognitionIo, @@ -36,6 +37,8 @@ interface ScanOutcome { unreadable: string[]; visited: number; truncated: boolean; + /** Why it stopped early, for the schedule and the doctor listing. */ + truncation: Exclude; } const errorCode = (error: unknown) => (error instanceof Error && 'code' in error ? String(error.code) : ''); @@ -48,9 +51,11 @@ const refused = (error: unknown) => ['EACCES', 'EPERM'].includes(errorCode(error * A read that does not answer within 3 seconds stops the walk at once: the folder is recorded unreadable, the result is * truncated, and the read stays the pending one (`bounded-read.ts`), so no further read starts, here or in a later scan, * until it settles. It also stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or `budgetMs`, and says so in - * `truncated`. Before `isCheckout` runs on a folder, `recognitionReads` proves that project detection cannot block - * there: a detection file that is not regular, or an Xcode container, makes the folder not a checkout, and a manifest - * read that does not answer ends the walk like any other timed-out read. An abort rejects. + * `truncated`, with the reason in `truncation`. Before `isCheckout` runs on a folder, `folderSafety` has proved, for it and + * for every folder above it down from the home folder, and `nestedSafety` for what detection inspects below it, that + * project detection cannot block: an unsafe folder (a non-regular detection file, or an Xcode container without an app + * dependency) makes its whole subtree unrecognisable, and a manifest read that does not answer ends the walk like any + * other timed-out read. An abort rejects. */ export async function scanForCheckouts( home: string, @@ -73,25 +78,34 @@ export async function scanForCheckouts( recognition?: RecognitionIo; } = {}, ): Promise { - const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false }; + const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false, truncation: 'none' }; const deadline = Date.now() + budgetMs; + const stop = (truncation: Exclude): void => { + if (outcome.truncated) return; + outcome.truncated = true; + outcome.truncation = truncation; + }; const exhausted = (): boolean => { signal?.throwIfAborted(); - if (!outcome.truncated && (Date.now() > deadline || outcome.visited >= MAX_FOLDERS || hungRead() !== null)) - outcome.truncated = true; + if (hungRead() !== null) stop('read'); + else if (Date.now() > deadline) stop('budget'); + else if (outcome.visited >= MAX_FOLDERS) stop('folders'); return outcome.truncated; }; const stopAt = (folder: string, error: unknown): boolean => { signal?.throwIfAborted(); if (error instanceof Unanswered) { outcome.unreadable.push(folder); - outcome.truncated = true; + stop('read'); return true; } if (refused(error)) outcome.unreadable.push(folder); return false; }; - if (hungRead() !== null) return { ...outcome, truncated: true }; + if (hungRead() !== null) { + stop('read'); + return outcome; + } let volume: number; try { volume = (await answered(home, () => statOf(home), readTimeoutMs)).dev; @@ -109,16 +123,25 @@ export async function scanForCheckouts( stopAt(folder, error); return; } - const reads = await recognitionReads(folder, entries, readTimeoutMs, recognition, depth > 0); - if (reads === 'unreadable') { + const safety = await folderSafety(folder, entries, readTimeoutMs, recognition); + if (safety === 'unreadable') { outcome.unreadable.push(folder); - outcome.truncated = true; + stop('read'); return; } - const above = unsafeAbove || reads === 'unsafe'; - if (depth > 0 && reads === 'safe' && !(above && needsAncestors(entries)) && isCheckout(folder)) { - outcome.found.push(folder); - return; + const above = unsafeAbove || safety === 'unsafe'; + if (above) return; + if (depth > 0) { + const nested = await nestedSafety(folder, entries, readTimeoutMs, recognition); + if (nested === 'unreadable') { + outcome.unreadable.push(folder); + stop('read'); + return; + } + if (nested === 'safe' && isCheckout(folder)) { + outcome.found.push(folder); + return; + } } if (depth >= MAX_DEPTH) return; for (const entry of entries) { @@ -133,7 +156,7 @@ export async function scanForCheckouts( if (stopAt(path, error)) return; continue; } - if (info.dev === volume) await walk(path, depth + 1, above); + if (info.dev === volume) await walk(path, depth + 1, false); } }; await walk(home, 0, false); @@ -152,18 +175,20 @@ export function mergeFound(previous: readonly string[], outcome: ScanOutcome): s } /** - * Whether a scan is due. Never twice within an hour, so a failing or truncated scan is retried hourly and a rise and - * fall of the disk level cannot start a run of scans; then weekly from the last scan that finished (a failed or - * truncated one is partial and does not count), or hourly while the level is above normal and no scan has run during - * it for an hour. + * Whether a scan is due. Never twice within an hour, so a rise and fall of the disk level cannot start a run of scans. + * After that hour an attempt that ended on a failure or a read that timed out is retried whatever the disk level; one + * that finished, or stopped at the folder cap or the time budget (the same limit would stop a retry), follows the weekly + * schedule from the last scan that did not end on a read or failure, and runs hourly while the level is above normal + * and no scan has run during it for an hour. */ export function discoveryScanDue( - last: Pick | null, + last: Pick | null, now: number, low: boolean, ): boolean { if (!last || last.ranAt > now) return true; if (now - last.ranAt < HOUR_MS) return false; + if (last.truncation === 'read' || last.truncation === 'failed') return true; return now - last.completedAt >= WEEK_MS || (low && now - last.lowScanAt >= HOUR_MS); } @@ -189,7 +214,8 @@ function writeScan(scan: DiscoveryScan): void { /** * Scans the home folder when the schedule says so (`discoveryScanDue`) and stores the result under the Stim home, with * the schedule's own times, so a retired worker or a restarted daemon forgets nothing. A failed scan stores nothing but - * its attempt time, keeps the previous finds and is retried after an hour. Only a daemon tick scans, never a command, + * its attempt time, keeps the previous finds and is retried after an hour, as is a scan that ended on a read that timed + * out; one that stopped at the folder cap or the time budget counts as finished, since a retry would hit the same limit. Only a daemon tick scans, never a command, * and a scoped STIM_HOME never does. Only an abort escapes, so a scan can never end the maintenance pass it follows. */ export async function runDiscoveryScanIfDue( @@ -198,8 +224,14 @@ export async function runDiscoveryScanIfDue( home = homedir(), now = Date.now(), attempted = attempts, - budgetMs, - }: { home?: string; now?: number; attempted?: { at: number }; budgetMs?: number } = {}, + scanOptions = {}, + }: { + home?: string; + now?: number; + attempted?: { at: number }; + /** Limits and readers of the walk, for tests. */ + scanOptions?: Parameters[2]; + } = {}, ): Promise { if (!tick || isScopedHome(tick.operation.env)) return; const { operation } = tick; @@ -221,18 +253,26 @@ export async function runDiscoveryScanIfDue( lowScanAt: low ? now : (last?.lowScanAt ?? 0), trigger: low ? 'low-disk' : 'weekly', } as const; - const failed = { ...attempt, completedAt: last?.completedAt ?? 0, durationMs: 0, visited: 0, truncated: true }; + const failedTruncation: ScanTruncation = 'failed'; + const failed = { + ...attempt, + completedAt: last?.completedAt ?? 0, + durationMs: 0, + visited: 0, + truncated: true, + truncation: failedTruncation, + }; const realHome = canonicalPath(home); const dotFolder = (path: string) => relative(realHome, path).startsWith('.'); try { const outcome = await scanForCheckouts(realHome, recognises, { signal: operation.signal, skip: scanSkipper(realHome), - ...(budgetMs === undefined ? {} : { budgetMs }), + ...scanOptions, }); writeScan({ ...attempt, - completedAt: outcome.truncated ? (last?.completedAt ?? 0) : now, + completedAt: outcome.truncation === 'read' ? (last?.completedAt ?? 0) : now, durationMs: Date.now() - now, ...outcome, found: mergeFound(last?.found ?? [], { ...outcome, found: outcome.found.filter((p) => !dotFolder(p)) }), diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index c4f59f4381..fa21fd9067 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -1,7 +1,6 @@ -import { lstatSync, realpathSync, statSync } from 'node:fs'; -import { readdir, readFile } from 'node:fs/promises'; +import { lstat, readdir, readFile, realpath, stat } from 'node:fs/promises'; import { tmpdir } from 'node:os'; -import { dirname, join, relative, sep } from 'node:path'; +import { join, relative, sep } from 'node:path'; import { configDir } from '@stim-cli/core'; import { buildWorkerRoot } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; @@ -152,77 +151,117 @@ interface Entry { export interface RecognitionIo { readEntries: (path: string) => Promise; readText: (path: string) => Promise; + /** Metadata reads of a cached entry; `realpath` is the native one, which folds case on macOS. */ + lstat: (path: string) => Promise<{ isSymbolicLink(): boolean }>; + stat: (path: string) => Promise<{ isDirectory(): boolean; dev: number }>; + realpath: (path: string) => Promise; } export const REAL_RECOGNITION_IO: RecognitionIo = { readEntries: (path) => readdir(path, { withFileTypes: true }), readText: (path) => readFile(path, 'utf8'), + lstat: (path) => lstat(path), + stat: (path) => stat(path), + realpath: (path) => realpath(path), }; +type Safety = 'safe' | 'unsafe' | 'unreadable'; + /** - * Whether `recognises` can be called on `folder` without a read that could block: `safe`; `unsafe` when it could (a - * detection file that is not a regular file, such as a FIFO or a symlink into a stalled mount, or an Xcode container, - * whose project files `containers()` follows by the paths they name), so the folder is not treated as a checkout; - * `unreadable` when a read did not answer in time, or another read is still pending (`bounded-read.ts`). `entries` is - * the folder's listing. A regular `package.json`, the one detection file project detection reads rather than stats, is - * read here first through the bounded reader, so an evicted or stalled file is found out before the synchronous read. - * `apps`, `examples` and `packages` children are checked the same way, because `recognises` looks inside them. + * Whether project detection can read in `folder` without blocking, from the folder's own listing: `unsafe` when a + * detection file is not a regular file (a FIFO, or a symlink into a stalled mount) or the folder holds an Xcode + * container with no app dependency, whose project files `containers()` follows by the paths they name; `unreadable` + * when a read did not answer in time or another read is still pending (`bounded-read.ts`). A regular `package.json`, + * the one detection file detection reads rather than stats, is read here first through the bounded reader, so an + * evicted or stalled file is found out before the synchronous read. Detection also inspects every ancestor of a folder, + * so a folder is recognisable only when this holds for it and for each ancestor from the home folder down, with no + * condition: the walk and the revalidation both apply it to every folder on the way. */ -export async function recognitionReads( +export async function folderSafety( folder: string, entries: readonly Entry[], timeoutMs: number, io: RecognitionIo = REAL_RECOGNITION_IO, - nested = true, -): Promise<'safe' | 'unsafe' | 'unreadable'> { +): Promise { if (hungRead() !== null) return 'unreadable'; const detection = entries.filter((entry) => DETECTION_FILES.has(entry.name)); if (detection.some((entry) => !entry.isFile())) return 'unsafe'; const hasContainer = entries.some((entry) => XCODE_CONTAINER.test(entry.name)); + if (!detection.some((entry) => entry.name === 'package.json')) return hasContainer ? 'unsafe' : 'safe'; + try { + const text = await answered(folder, () => io.readText(join(folder, 'package.json')), timeoutMs); + let declares = false; + try { + declares = declaresAppDependency(JSON.parse(text)); + } catch {} + return hasContainer && !declares ? 'unsafe' : 'safe'; + } catch (error) { + return error instanceof Unanswered ? 'unreadable' : 'unsafe'; + } +} + +/** + * `folderSafety` for what detection inspects below `folder` itself: each of `apps`, `examples` and `packages` (the + * container folder, with its own `package.json`) and up to 50 children of each. A symlinked container, or any unsafe or + * unreadable member, makes the result that. The ancestors of `folder` are the caller's: they have passed `folderSafety`. + */ +export async function nestedSafety( + folder: string, + entries: readonly Entry[], + timeoutMs: number, + io: RecognitionIo = REAL_RECOGNITION_IO, +): Promise { + const read = (path: string) => answered(path, () => io.readEntries(path), timeoutMs); try { - if (detection.some((entry) => entry.name === 'package.json')) { - const text = await answered(folder, () => io.readText(join(folder, 'package.json')), timeoutMs); - let declares = false; - try { - declares = declaresAppDependency(JSON.parse(text)); - } catch {} - if (hasContainer && !declares) return 'unsafe'; - } else if (hasContainer) return 'unsafe'; - if (!nested) return 'safe'; for (const name of PROJECT_FOLDERS) { const entry = entries.find((candidate) => candidate.name === name); if (!entry) continue; if (entry.isSymbolicLink()) return 'unsafe'; if (!entry.isDirectory()) continue; - const children = (await answered(folder, () => io.readEntries(join(folder, name)), timeoutMs)) - .filter((child) => child.isDirectory() && !child.isSymbolicLink()) - .slice(0, PROJECT_FOLDER_ENTRIES); - for (const child of children) { - const path = join(folder, name, child.name); - const inner = await answered(folder, () => io.readEntries(path), timeoutMs).catch((error: unknown) => { - if (error instanceof Unanswered) throw error; - return null; - }); - if (inner === null) continue; - const result = await recognitionReads(path, inner, timeoutMs, io, false); + const container = join(folder, name); + const members = await read(container); + const own = await folderSafety(container, members, timeoutMs, io); + if (own !== 'safe') return own; + for (const child of members + .filter((member) => member.isDirectory() && !member.isSymbolicLink()) + .slice(0, PROJECT_FOLDER_ENTRIES)) { + const path = join(container, child.name); + let inner: Entry[]; + try { + inner = await read(path); + } catch (error) { + if (error instanceof Unanswered) return 'unreadable'; + continue; + } + const result = await folderSafety(path, inner, timeoutMs, io); if (result !== 'safe') return result; } } return 'safe'; } catch (error) { - if (error instanceof Unanswered) return 'unreadable'; - return 'unsafe'; + return error instanceof Unanswered ? 'unreadable' : 'unsafe'; } } -/** Whether detection in `entries`' folder walks up the ancestors: a native Android layout is a candidate root. */ -export const needsAncestors = (entries: readonly Entry[]): boolean => - entries.some((entry) => entry.name === 'settings.gradle' || entry.name === 'settings.gradle.kts'); +/** A folder's listing and `folderSafety`, for the chain of folders above a cached entry. */ +async function gateOf( + folder: string, + timeoutMs: number, + io: RecognitionIo, +): Promise<{ listing: Entry[]; safety: Safety }> { + try { + const listing = await answered(folder, () => io.readEntries(folder), timeoutMs); + return { listing, safety: await folderSafety(folder, listing, timeoutMs, io) }; + } catch (error) { + return { listing: [], safety: error instanceof Unanswered ? 'unreadable' : 'unsafe' }; + } +} /** * Whether a checkout a scan recorded may still be used: it is a real directory (not a symlink), its canonical path is * the recorded one, it lies below the home folder on the home volume, no folder between the two is one the scan skips - * (`scanDescends`, `skip`), its detection reads are bounded (`recognitionReads`) and `isCheckout` still recognises it. + * (`scanDescends`, `skip`), every folder from the home folder down to it passes `folderSafety` (memoised in `ancestors`), + * what it inspects below itself passes `nestedSafety`, and `isCheckout` still recognises it. * `invalid` drops it, such as a checkout moved away with a symlink left behind; `pending` means a read did not answer, * so it is not selected this time and stays in the scan file. */ @@ -241,50 +280,42 @@ export async function scanEntryValid( skip: (path: string) => boolean; timeoutMs?: number; io?: RecognitionIo; - /** The gate results of folders above checkouts, shared by the entries of one revalidation. */ - ancestors?: Map>; + /** The listing and safety of folders on the way to a checkout, shared by the entries of one revalidation. */ + ancestors?: Map>; }, ): Promise<'valid' | 'invalid' | 'pending'> { - try { - if (!within(path, home) || path === home || lstatSync(path).isSymbolicLink() || !statSync(path).isDirectory()) - return 'invalid'; - if (realpathSync.native(path) !== path || statSync(path).dev !== statSync(home).dev || skip(path)) return 'invalid'; - const names = relative(home, path).split(sep); - if (names.length > MAX_DEPTH || !names.every((name, i) => scanDescends(name, i + 1))) return 'invalid'; - } catch { - return 'invalid'; - } if (hungRead() !== null) return 'pending'; - let entries: Entry[]; + if (!within(path, home) || path === home || skip(path)) return 'invalid'; + const names = relative(home, path).split(sep); + if (names.length > MAX_DEPTH || !names.every((name, i) => scanDescends(name, i + 1))) return 'invalid'; try { - entries = await answered(path, () => io.readEntries(path), timeoutMs); + if ((await answered(path, () => io.lstat(path), timeoutMs)).isSymbolicLink()) return 'invalid'; + const info = await answered(path, () => io.stat(path), timeoutMs); + if (!info.isDirectory()) return 'invalid'; + if ((await answered(path, () => io.realpath(path), timeoutMs)) !== path) return 'invalid'; + if ((await answered(home, () => io.stat(home), timeoutMs)).dev !== info.dev) return 'invalid'; } catch (error) { return error instanceof Unanswered ? 'pending' : 'invalid'; } - const reads = await recognitionReads(path, entries, timeoutMs, io); - if (reads === 'unreadable') return 'pending'; - if (reads !== 'safe') return 'invalid'; - if (needsAncestors(entries)) { - const above = [ - home, - ...relative(home, dirname(path)) - .split(sep) - .filter(Boolean) - .map((_, i, names) => join(home, ...names.slice(0, i + 1))), - ]; - for (const folder of above) { - let result = ancestors.get(folder); - if (!result) { - result = answered(folder, () => io.readEntries(folder), timeoutMs).then( - (listing) => recognitionReads(folder, listing, timeoutMs, io, false), - (error: unknown) => (error instanceof Unanswered ? ('unreadable' as const) : ('unsafe' as const)), - ); - ancestors.set(folder, result); - } - const gate = await result; - if (gate === 'unreadable') return 'pending'; - if (gate === 'unsafe') return 'invalid'; + const chain = [ + home, + ...relative(home, path) + .split(sep) + .map((_, i, parts) => join(home, ...parts.slice(0, i + 1))), + ]; + let own: Entry[] = []; + for (const folder of chain) { + let gate = ancestors.get(folder); + if (!gate) { + gate = gateOf(folder, timeoutMs, io); + ancestors.set(folder, gate); } + const { listing, safety } = await gate; + if (safety === 'unreadable') return 'pending'; + if (safety === 'unsafe') return 'invalid'; + if (folder === path) own = listing; } - return isCheckout(path) ? 'valid' : 'invalid'; + const nested = await nestedSafety(path, own, timeoutMs, io); + if (nested === 'unreadable') return 'pending'; + return nested === 'safe' && isCheckout(path) ? 'valid' : 'invalid'; } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index db8700b652..8841a73938 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A failed or truncated scan is retried after an hour. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests there can be read without hanging: a folder whose manifest is not a regular file, or that holds an Xcode project, is not treated as a checkout by the scan, and a manifest that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest is not a regular file, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 6e870625d35cb4449d88cc4c74735ad7684c7f4d Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 05:14:43 -0500 Subject: [PATCH 16/25] fix: compare detection names case-insensitively, gate ios and android roots and make an unsafe nested result cover the subtree --- llp/0004-discovered-checkouts.decision.md | 48 +++++++++---------- .../src/__tests__/discovery-scan.test.ts | 28 +++++++++++ .../stim-cli/src/__tests__/scan-rules.test.ts | 36 +++++++++++++- packages/stim-cli/src/guide/cleanup.ts | 7 +-- .../stim-cli/src/workspace/discovery-scan.ts | 3 +- packages/stim-cli/src/workspace/scan-rules.ts | 40 ++++++++++------ website/docs/worktrees.md | 2 +- 7 files changed, 119 insertions(+), 45 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 715db3765d..de0bd1bc57 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -132,30 +132,30 @@ command that adds it to `maintenance.discoverRoots`; that listing is not part of this change. Project detection (`recognises`) reads files synchronously, and a stalled mount, -an iCloud file waiting for a download or a FIFO would block the thread. When the -scan or the revalidation decides whether a folder is a checkout, every read -detection makes is bounded by construction. Detection inspects the folder, up to -50 children of each of its `apps`, `examples` and `packages` folders, those -container folders themselves and every ancestor up to `/`; each inspection reads -`package.json`, checks the existence of `settings.gradle(.kts)`, `gradlew(.bat)`, -`Package.swift`, `exact.mjs` and `app.json`, and lists the folder, and an Xcode -container without an app dependency makes it parse project files that name other -files. So one check, `folderSafety`, applies to every folder the walk passes from -the home folder down and to each folder detection inspects below the one being -decided (`nestedSafety`), with no condition: a detection file that is not a -regular file, or an Xcode container with no app dependency in `package.json`, makes -the folder unsafe; a regular `package.json` is read once through the bounded -reader first. An unsafe folder makes its whole subtree unrecognisable and the walk -does not descend into it. A manifest read that does not answer ends the scan -truncated with the folder unreadable (`truncation: read`), or leaves a cached find -unselected and in the file. Revalidation checks the chain from the home folder to -the entry with the same function, memoised per discovery call, and reads the -entry's own metadata (`lstat`, `stat`, `realpath`) through the bounded reader too. -Not bounded, and stated as such: the ancestor reads above the home folder (`/Users` -and `/`); a file that becomes unreadable between the bounded read and detection; -and everything a pass later reads synchronously inside a checkout once it is -discovered (artifact collection, app roots, state), which is the same for -registered and sibling-discovered checkouts today. Registered projects and +an iCloud file waiting for a download or a FIFO would block the thread. Before +detection runs on a folder, the scan or the revalidation proves its reads cannot +block, as far as this list goes. For the folder, each ancestor from the home +folder down (`folderSafety`), and each folder detection inspects below it +(`nestedSafety`: each `apps`, `examples` and `packages` container and up to 50 +children of each), with every name compared case-insensitively on every volume: +a detection file (`package.json`, `settings.gradle(.kts)`, `gradlew(.bat)`, +`Package.swift`, `exact.mjs`, `app.json`) that is not a regular file makes the +folder unsafe; so does an `ios` or `android` entry that is not a real directory +(detection resolves them with `realpath` as React Native's owned roots), an Xcode +container with no app dependency in `package.json` (detection parses the project +files it names), and a symlinked container; every regular `package.json` is read +once through the bounded reader first. An unsafe folder, or an unsafe result for +anything detection inspects below it, makes its whole subtree unrecognisable: no +detection runs on it or below it, and the walk does not descend. A read that does +not answer ends the scan truncated with the folder unreadable (`truncation: read`), +or leaves a cached find unselected and in the file. Revalidation applies the same +functions to the chain from the home folder to the entry, memoised per discovery +call, and reads the entry's own metadata (`lstat`, `stat`, `realpath`) through the +bounded reader. Not bounded, and stated as such: the ancestor reads above the home +folder (`/Users` and `/`); a file that becomes unreadable between the bounded read +and detection; and everything a pass later reads synchronously inside a checkout +once it is discovered (artifact collection, app roots, state), which is the same +for registered and sibling-discovered checkouts today. Registered projects and sibling discovery recognise as before. An Xcode container without an app dependency is not a scan checkout, though sibling and root discovery still recognise it. diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index 585d3ec111..d8331582c3 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -192,6 +192,34 @@ describe('scanForCheckouts', () => { ]); }); + it('makes the whole subtree unrecognisable when a child that detection inspects is unsafe, so a sibling is not a find either', async () => { + mkdirSync(join(home, 'mono', 'apps', 'x'), { recursive: true }); + const regular = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const fifo = { name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }; + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('x'), directory('y')]; + if (path === join(home, 'mono', 'apps', 'y')) return [fifo]; + return [regular('settings.gradle'), regular('gradlew')]; + }; + const detect = vi.fn<(path: string) => boolean>((path) => path.endsWith('x')); + const recognition = { ...REAL_RECOGNITION_IO, readEntries: listing }; + expect((await scanForCheckouts(home, detect, { entries: listing, recognition })).found).toEqual([]); + expect(detect).not.toHaveBeenCalled(); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts index d662caa257..8fedf38f10 100644 --- a/packages/stim-cli/src/__tests__/scan-rules.test.ts +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -1,7 +1,7 @@ import { existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { REAL_RECOGNITION_IO, scanEntryValid } from '../workspace/scan-rules.ts'; +import { REAL_RECOGNITION_IO, folderSafety, nestedSafety, scanEntryValid } from '../workspace/scan-rules.ts'; describe('scanEntryValid with a hand-edited scan result', () => { let home: string; @@ -122,6 +122,40 @@ describe('scanEntryValid with a hand-edited scan result', () => { expect(detect).not.toHaveBeenCalled(); }); + describe('names compare case-insensitively, as a case-insensitive volume reads them', () => { + const symlink = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => false, + isSymbolicLink: () => true, + }); + const where = '/work/x'; + + it('finds a FIFO manifest, a symlinked container and a symlinked or non-directory ios or android root whatever the case', async () => { + expect(await folderSafety(where, [fifo('PACKAGE.JSON')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [fifo('App.JSON')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [symlink('iOS')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [file('Android')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [folder('IOS'), folder('android')], 10)).toBe('safe'); + expect(await nestedSafety(where, [symlink('APPS')], 10)).toBe('unsafe'); + expect(await nestedSafety(where, [symlink('Packages')], 10)).toBe('unsafe'); + }); + + it('reads a mixed-case regular manifest through the bounded reader instead of skipping it', async () => { + const reads: string[] = []; + const io = { + ...REAL_RECOGNITION_IO, + readText: async (path: string) => { + reads.push(path); + return '{}'; + }, + }; + expect(await folderSafety(where, [file('Package.JSON')], 10, io)).toBe('safe'); + expect(reads).toEqual([join(where, 'Package.JSON')]); + expect(await folderSafety(where, [file('PACKAGE.JSON'), folder('App.XCODEPROJ')], 10, io)).toBe('unsafe'); + }); + }); + it('treats a metadata read that does not answer as pending, before any detection', async () => { const detect = vi.fn<(path: string) => boolean>(() => true); let release: (value: { isSymbolicLink(): boolean }) => void = () => {}; diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 2342b37665..3f07e1d2b9 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1234,9 +1234,10 @@ dot-folders (.npm, .gradle, .cache and the like; .exact is entered), stops at a recognised project and goes seven levels deep within 45 seconds. Before project detection runs on a folder, the manifests detection reads are proved safe, for the folder, each ancestor down from the home folder, its apps, examples and -packages folders and their children: a manifest that is not a regular file, or an -Xcode container with no app dependency in package.json, makes that subtree not a -scan checkout, and a manifest or file metadata read that does not answer in 3 +packages folders and their children: a manifest (any case) that is not a regular file, an +ios or android entry that is not a real directory, a symlinked apps, examples or +packages folder, or an Xcode container with no app dependency in package.json, +makes that subtree not a scan checkout, and a manifest or file metadata read that does not answer in 3 seconds ends the scan (cached finds stay unselected, not dropped). Not bounded: the ancestors above the home folder, a file that becomes unreadable between that check and detection, and what a pass reads inside a checkout once discovered. The result is diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 584be43eed..1bf1a63cae 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -138,7 +138,8 @@ export async function scanForCheckouts( stop('read'); return; } - if (nested === 'safe' && isCheckout(folder)) { + if (nested === 'unsafe') return; + if (isCheckout(folder)) { outcome.found.push(folder); return; } diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index fa21fd9067..b9c02a7c4c 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -126,19 +126,26 @@ export function scanSkipper(home: string): (path: string) => boolean { } /** The files of a folder that project detection reads or stats: the manifests every integration's `inspect` looks at. */ +/** + * Compared case-insensitively on every volume: a case-insensitive one reads `Package.JSON` as `package.json`, and a case- + * sensitive one only loses a folder it could have accepted, the safe direction. + */ const DETECTION_FILES = new Set([ 'package.json', 'settings.gradle', 'settings.gradle.kts', 'gradlew', 'gradlew.bat', - 'Package.swift', + 'package.swift', 'exact.mjs', 'app.json', ]); const PROJECT_FOLDERS = ['apps', 'examples', 'packages']; const PROJECT_FOLDER_ENTRIES = 50; -const XCODE_CONTAINER = /\.(xcodeproj|xcworkspace)$/; +const XCODE_CONTAINER = /\.(xcodeproj|xcworkspace)$/i; +/** React Native's owned roots, which detection resolves with `realpath` when it walks up from a native layout. */ +const OWNED_ROOTS = new Set(['ios', 'android']); +const folded = (name: string): string => name.toLowerCase(); interface Entry { name: string; @@ -184,20 +191,25 @@ export async function folderSafety( io: RecognitionIo = REAL_RECOGNITION_IO, ): Promise { if (hungRead() !== null) return 'unreadable'; - const detection = entries.filter((entry) => DETECTION_FILES.has(entry.name)); + const detection = entries.filter((entry) => DETECTION_FILES.has(folded(entry.name))); if (detection.some((entry) => !entry.isFile())) return 'unsafe'; + if (entries.some((entry) => OWNED_ROOTS.has(folded(entry.name)) && !entry.isDirectory())) return 'unsafe'; const hasContainer = entries.some((entry) => XCODE_CONTAINER.test(entry.name)); - if (!detection.some((entry) => entry.name === 'package.json')) return hasContainer ? 'unsafe' : 'safe'; - try { - const text = await answered(folder, () => io.readText(join(folder, 'package.json')), timeoutMs); - let declares = false; + const manifests = detection.filter((entry) => folded(entry.name) === 'package.json'); + if (manifests.length === 0) return hasContainer ? 'unsafe' : 'safe'; + let declares = false; + for (const manifest of manifests) { + let text: string; try { - declares = declaresAppDependency(JSON.parse(text)); + text = await answered(folder, () => io.readText(join(folder, manifest.name)), timeoutMs); + } catch (error) { + return error instanceof Unanswered ? 'unreadable' : 'unsafe'; + } + try { + declares ||= declaresAppDependency(JSON.parse(text)); } catch {} - return hasContainer && !declares ? 'unsafe' : 'safe'; - } catch (error) { - return error instanceof Unanswered ? 'unreadable' : 'unsafe'; } + return hasContainer && !declares ? 'unsafe' : 'safe'; } /** @@ -213,12 +225,10 @@ export async function nestedSafety( ): Promise { const read = (path: string) => answered(path, () => io.readEntries(path), timeoutMs); try { - for (const name of PROJECT_FOLDERS) { - const entry = entries.find((candidate) => candidate.name === name); - if (!entry) continue; + for (const entry of entries.filter((candidate) => PROJECT_FOLDERS.includes(folded(candidate.name)))) { if (entry.isSymbolicLink()) return 'unsafe'; if (!entry.isDirectory()) continue; - const container = join(folder, name); + const container = join(folder, entry.name); const members = await read(container); const own = await folderSafety(container, members, timeoutMs, io); if (own !== 'safe') return own; diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 8841a73938..1ccaac2b37 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest is not a regular file, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From b8c9206d343b1edee34f31907c0de8db9831ed7c Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 05:33:38 -0500 Subject: [PATCH 17/25] fix: make the recognition gate fail closed on every listing error and require every folded manifest to declare an app dependency for the Xcode exemption --- llp/0004-discovered-checkouts.decision.md | 7 ++++- .../src/__tests__/discovery-scan.test.ts | 21 +++++++++++++++ .../stim-cli/src/__tests__/scan-rules.test.ts | 27 +++++++++++++++++++ packages/stim-cli/src/guide/cleanup.ts | 4 ++- packages/stim-cli/src/workspace/scan-rules.ts | 20 +++++++++----- website/docs/worktrees.md | 2 +- 6 files changed, 72 insertions(+), 9 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index de0bd1bc57..9a78676cfa 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -153,7 +153,12 @@ functions to the chain from the home folder to the entry, memoised per discovery call, and reads the entry's own metadata (`lstat`, `stat`, `realpath`) through the bounded reader. Not bounded, and stated as such: the ancestor reads above the home folder (`/Users` and `/`); a file that becomes unreadable between the bounded read -and detection; and everything a pass later reads synchronously inside a checkout +and detection; the `ios` and `android` directories of the folder and of each +ancestor, which detection resolves with `realpath` without the walk entering +them (the gate checks that each is a real directory, not that it answers, so a +stalled mount point there is not detected; the walk's own bounded `readdir` covers +only the folders it enters, and the `apps`, `examples` and `packages` containers +and their children are listed through the bounded reader by the gate); and everything a pass later reads synchronously inside a checkout once it is discovered (artifact collection, app roots, state), which is the same for registered and sibling-discovered checkouts today. Registered projects and sibling discovery recognise as before. An Xcode container without an app diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index d8331582c3..a46d8b080e 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -220,6 +220,27 @@ describe('scanForCheckouts', () => { expect(detect).not.toHaveBeenCalled(); }); + it('calls no detection on a folder or its child when that child cannot be listed, however readable its manifest is', async () => { + mkdirSync(join(home, 'mono', 'apps', 'app'), { recursive: true }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('app')]; + throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + }; + const detect = vi.fn<(path: string) => boolean>(() => true); + const recognition = { ...REAL_RECOGNITION_IO, readEntries: listing }; + const scan = await scanForCheckouts(home, detect, { entries: listing, recognition }); + expect(scan.found).toEqual([]); + expect(detect).not.toHaveBeenCalled(); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts index 8fedf38f10..ffc836cbd9 100644 --- a/packages/stim-cli/src/__tests__/scan-rules.test.ts +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -141,6 +141,33 @@ describe('scanEntryValid with a hand-edited scan result', () => { expect(await nestedSafety(where, [symlink('Packages')], 10)).toBe('unsafe'); }); + it('keeps the Xcode exemption only when every manifest that folds to package.json declares an app dependency', async () => { + const declaring = JSON.stringify({ dependencies: { expo: '*' } }); + const texts: Record = {}; + const io = { ...REAL_RECOGNITION_IO, readText: async (path: string) => texts[path] ?? '{}' }; + const entries = [file('package.json'), file('Package.JSON'), folder('X.xcodeproj')]; + texts['/work/x/Package.JSON'] = declaring; + expect(await folderSafety(where, entries, 10, io)).toBe('unsafe'); + texts['/work/x/package.json'] = declaring; + expect(await folderSafety(where, entries, 10, io)).toBe('safe'); + }); + + it('is unsafe, never safe, when a child that detection enumerates cannot be listed', async () => { + const denied = Object.assign(new Error('EACCES'), { code: 'EACCES' }); + const gone = Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); + const io = (failure: Error) => ({ + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === '/work/x/apps') return [folder('app')]; + if (path === '/work/x/apps/app') throw failure; + return []; + }, + }); + const entries = [folder('apps')]; + expect(await nestedSafety(where, entries, 10, io(denied))).toBe('unsafe'); + expect(await nestedSafety(where, entries, 10, io(gone))).toBe('safe'); + }); + it('reads a mixed-case regular manifest through the bounded reader instead of skipping it', async () => { const reads: string[] = []; const io = { diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 3f07e1d2b9..678434d3ea 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1240,7 +1240,9 @@ packages folder, or an Xcode container with no app dependency in package.json, makes that subtree not a scan checkout, and a manifest or file metadata read that does not answer in 3 seconds ends the scan (cached finds stay unselected, not dropped). Not bounded: the ancestors above the home folder, a file that becomes unreadable between that -check and detection, and what a pass reads inside a checkout once discovered. The result is +check and detection, the ios and android directories detection resolves without +the walk entering them (checked to be real directories, not that they answer, so +a stalled mount point there is not detected), and what a pass reads inside a checkout once discovered. The result is kept in maintenance/discovery-scan.json under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a pass or gc never walks the disk. A scan that fails or stops early keeps the previous finds, diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index b9c02a7c4c..8ba2cd7fdf 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -127,8 +127,10 @@ export function scanSkipper(home: string): (path: string) => boolean { /** The files of a folder that project detection reads or stats: the manifests every integration's `inspect` looks at. */ /** - * Compared case-insensitively on every volume: a case-insensitive one reads `Package.JSON` as `package.json`, and a case- - * sensitive one only loses a folder it could have accepted, the safe direction. + * Compared case-insensitively on every volume: a case-insensitive one reads `Package.JSON` as `package.json`. On a case- + * sensitive one detection reads only the exact `package.json`, so this can only reject a folder it could have accepted, + * except that the Xcode exemption below must hold for every manifest that folds to `package.json`, since which one + * detection reads depends on the volume. */ const DETECTION_FILES = new Set([ 'package.json', @@ -146,6 +148,9 @@ const XCODE_CONTAINER = /\.(xcodeproj|xcworkspace)$/i; /** React Native's owned roots, which detection resolves with `realpath` when it walks up from a native layout. */ const OWNED_ROOTS = new Set(['ios', 'android']); const folded = (name: string): string => name.toLowerCase(); +/** The entry is not there: the only read failure that is not unsafe, and only for an entry a successful listing showed. */ +const absent = (error: unknown): boolean => + error instanceof Error && 'code' in error && (error.code === 'ENOENT' || error.code === 'ENOTDIR'); interface Entry { name: string; @@ -197,7 +202,7 @@ export async function folderSafety( const hasContainer = entries.some((entry) => XCODE_CONTAINER.test(entry.name)); const manifests = detection.filter((entry) => folded(entry.name) === 'package.json'); if (manifests.length === 0) return hasContainer ? 'unsafe' : 'safe'; - let declares = false; + let declares = true; for (const manifest of manifests) { let text: string; try { @@ -206,8 +211,10 @@ export async function folderSafety( return error instanceof Unanswered ? 'unreadable' : 'unsafe'; } try { - declares ||= declaresAppDependency(JSON.parse(text)); - } catch {} + declares &&= declaresAppDependency(JSON.parse(text)); + } catch { + declares = false; + } } return hasContainer && !declares ? 'unsafe' : 'safe'; } @@ -241,7 +248,8 @@ export async function nestedSafety( inner = await read(path); } catch (error) { if (error instanceof Unanswered) return 'unreadable'; - continue; + if (absent(error)) continue; + return 'unsafe'; } const result = await folderSafety(path, inner, timeoutMs, io); if (result !== 'safe') return result; diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 1ccaac2b37..05d2aacc3d 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 0032b44abbaa4c2b6d1faec9d3f03021be49e438 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 05:56:24 -0500 Subject: [PATCH 18/25] fix: check the scan budget before every read of the recognition gate --- llp/0004-discovered-checkouts.decision.md | 9 ++++++ .../src/__tests__/discovery-scan.test.ts | 32 +++++++++++++++++++ packages/stim-cli/src/guide/cleanup.ts | 10 +++++- .../stim-cli/src/workspace/discovery-scan.ts | 6 ++-- packages/stim-cli/src/workspace/scan-rules.ts | 23 +++++++++---- website/docs/worktrees.md | 2 +- 6 files changed, 71 insertions(+), 11 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 9a78676cfa..94ec13ede6 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -131,6 +131,15 @@ installed command. The record lets doctor report what the folder holds and the command that adds it to `maintenance.discoverRoots`; that listing is not part of this change. +The scan has a 45 second budget (and a 200 thousand folder cap, and an abort), +checked before every read it makes, including every read of the safety checks +below: when it runs out the folder is not recognised and the scan ends truncated +as `budget`, keeping the previous finds. It can overrun the budget by at most one +read's 3 second timeout plus the time detection itself takes on the folder it is +recognising, which is synchronous and not interruptible. Revalidation has no total +budget; each read has the 3 second timeout, and the number of reads is bounded by +detection's own caps (50 children per container, seven folders in the chain). + Project detection (`recognises`) reads files synchronously, and a stalled mount, an iCloud file waiting for a download or a FIFO would block the thread. Before detection runs on a folder, the scan or the revalidation proves its reads cannot diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index a46d8b080e..ec0e72a577 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -241,6 +241,38 @@ describe('scanForCheckouts', () => { expect(detect).not.toHaveBeenCalled(); }); + it("ends the scan truncated as budget, without calling detection, when a folder's nested checks outlast the remaining budget", async () => { + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + mkdirSync(join(home, 'mono', 'apps'), { recursive: true }); + const children = Array.from({ length: 50 }, (_, i) => directory(`app${i}`)); + let reads = 0; + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return children; + return []; + }; + const slow = async (path: string) => { + reads++; + vi.setSystemTime(Date.now() + 1000); + return listing(path); + }; + const detect = vi.fn<(path: string) => boolean>(() => true); + const scan = await scanForCheckouts(home, detect, { + entries: listing, + recognition: { ...REAL_RECOGNITION_IO, readEntries: slow }, + budgetMs: 5000, + }); + expect(scan).toMatchObject({ truncated: true, truncation: 'budget', found: [] }); + expect(detect).not.toHaveBeenCalled(); + expect(reads).toBeLessThan(10); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 678434d3ea..866ea47794 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1231,7 +1231,15 @@ folders, other volumes, symlinks, dependency or build folders, deeper dot-folders, the Stim home and every dot-folder starting with .stim, the build worker root, and the tool homes and caches among the home folder's other dot-folders (.npm, .gradle, .cache and the like; .exact is entered), stops at a -recognised project and goes seven levels deep within 45 seconds. Before project +recognised project and goes seven levels deep. The scan has a 45 second budget, +checked before every read it makes (including every read of the project checks +below); when it runs out the scan ends truncated as a budget stop and keeps the +previous finds. It can overrun the budget by at most one read's 3 second timeout, +plus the time project detection itself takes on the folder it is recognising, +which is synchronous and cannot be interrupted. Revalidation of cached finds has +no total budget: each read is bounded by the 3 second timeout, and the number of +reads by detection's own caps (50 children per container, seven folders in the +chain). Before project detection runs on a folder, the manifests detection reads are proved safe, for the folder, each ancestor down from the home folder, its apps, examples and packages folders and their children: a manifest (any case) that is not a regular file, an diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 1bf1a63cae..cd1cc29f08 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -123,7 +123,8 @@ export async function scanForCheckouts( stopAt(folder, error); return; } - const safety = await folderSafety(folder, entries, readTimeoutMs, recognition); + const safety = await folderSafety(folder, entries, readTimeoutMs, recognition, exhausted); + if (safety === 'stopped') return; if (safety === 'unreadable') { outcome.unreadable.push(folder); stop('read'); @@ -132,7 +133,8 @@ export async function scanForCheckouts( const above = unsafeAbove || safety === 'unsafe'; if (above) return; if (depth > 0) { - const nested = await nestedSafety(folder, entries, readTimeoutMs, recognition); + const nested = await nestedSafety(folder, entries, readTimeoutMs, recognition, exhausted); + if (nested === 'stopped') return; if (nested === 'unreadable') { outcome.unreadable.push(folder); stop('read'); diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index 8ba2cd7fdf..4825d0b9e9 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -177,7 +177,10 @@ export const REAL_RECOGNITION_IO: RecognitionIo = { realpath: (path) => realpath(path), }; -type Safety = 'safe' | 'unsafe' | 'unreadable'; +type Safety = 'safe' | 'unsafe' | 'unreadable' | 'stopped'; + +/** Asked before every read the gate makes: true when the scan's own limit (abort, time budget, folder cap) has run out. */ +type Stop = () => boolean; /** * Whether project detection can read in `folder` without blocking, from the folder's own listing: `unsafe` when a @@ -194,6 +197,7 @@ export async function folderSafety( entries: readonly Entry[], timeoutMs: number, io: RecognitionIo = REAL_RECOGNITION_IO, + stop?: Stop, ): Promise { if (hungRead() !== null) return 'unreadable'; const detection = entries.filter((entry) => DETECTION_FILES.has(folded(entry.name))); @@ -204,6 +208,7 @@ export async function folderSafety( if (manifests.length === 0) return hasContainer ? 'unsafe' : 'safe'; let declares = true; for (const manifest of manifests) { + if (stop?.()) return 'stopped'; let text: string; try { text = await answered(folder, () => io.readText(join(folder, manifest.name)), timeoutMs); @@ -223,12 +228,15 @@ export async function folderSafety( * `folderSafety` for what detection inspects below `folder` itself: each of `apps`, `examples` and `packages` (the * container folder, with its own `package.json`) and up to 50 children of each. A symlinked container, or any unsafe or * unreadable member, makes the result that. The ancestors of `folder` are the caller's: they have passed `folderSafety`. + * `stop` is asked before every read, so a scan's budget is honoured between them (`stopped`); revalidation passes none, + * and is bounded per read and by detection's own caps (50 children per container, at most seven folders in the chain). */ export async function nestedSafety( folder: string, entries: readonly Entry[], timeoutMs: number, io: RecognitionIo = REAL_RECOGNITION_IO, + stop?: Stop, ): Promise { const read = (path: string) => answered(path, () => io.readEntries(path), timeoutMs); try { @@ -236,13 +244,15 @@ export async function nestedSafety( if (entry.isSymbolicLink()) return 'unsafe'; if (!entry.isDirectory()) continue; const container = join(folder, entry.name); + if (stop?.()) return 'stopped'; const members = await read(container); - const own = await folderSafety(container, members, timeoutMs, io); + const own = await folderSafety(container, members, timeoutMs, io, stop); if (own !== 'safe') return own; for (const child of members .filter((member) => member.isDirectory() && !member.isSymbolicLink()) .slice(0, PROJECT_FOLDER_ENTRIES)) { const path = join(container, child.name); + if (stop?.()) return 'stopped'; let inner: Entry[]; try { inner = await read(path); @@ -251,7 +261,7 @@ export async function nestedSafety( if (absent(error)) continue; return 'unsafe'; } - const result = await folderSafety(path, inner, timeoutMs, io); + const result = await folderSafety(path, inner, timeoutMs, io, stop); if (result !== 'safe') return result; } } @@ -329,11 +339,10 @@ export async function scanEntryValid( ancestors.set(folder, gate); } const { listing, safety } = await gate; - if (safety === 'unreadable') return 'pending'; - if (safety === 'unsafe') return 'invalid'; + if (safety !== 'safe') return safety === 'unsafe' ? 'invalid' : 'pending'; if (folder === path) own = listing; } const nested = await nestedSafety(path, own, timeoutMs, io); - if (nested === 'unreadable') return 'pending'; - return nested === 'safe' && isCheckout(path) ? 'valid' : 'invalid'; + if (nested !== 'safe') return nested === 'unsafe' ? 'invalid' : 'pending'; + return isCheckout(path) ? 'valid' : 'invalid'; } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 05d2aacc3d..0dc81b9abc 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 3d6edcf06bdecc656ed9e3703c2f6a4d2f0a637c Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 06:33:35 -0500 Subject: [PATCH 19/25] fix: let an abort escape the recognition gate, correct the chain length and state what dormant dependency deletion does for each checkout kind --- llp/0004-discovered-checkouts.decision.md | 12 ++-- .../src/__tests__/discovery-scan.test.ts | 64 +++++++++++++++++++ packages/stim-cli/src/guide/cleanup.ts | 18 ++++-- .../stim-cli/src/workspace/discovery-scan.ts | 2 +- packages/stim-cli/src/workspace/scan-rules.ts | 60 +++++++++-------- website/docs/worktrees.md | 4 +- 6 files changed, 119 insertions(+), 41 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 94ec13ede6..c00fe5945d 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -138,7 +138,7 @@ as `budget`, keeping the previous finds. It can overrun the budget by at most on read's 3 second timeout plus the time detection itself takes on the folder it is recognising, which is synchronous and not interruptible. Revalidation has no total budget; each read has the 3 second timeout, and the number of reads is bounded by -detection's own caps (50 children per container, seven folders in the chain). +detection's own caps (50 children per container, the home folder and up to seven below it). Project detection (`recognises`) reads files synchronously, and a stalled mount, an iCloud file waiting for a download or a FIFO would block the thread. Before @@ -219,9 +219,13 @@ launcher. Any other failure to inspect a directory (permission, I/O, a timeout) keeps every reduce-only checkout for that operation, with a reason naming the directory. A wrapper script that only names a path is not followed. -A scan-found source checkout follows the existing discovered rule and loses -`node_modules` and Pods after `idle.sourceDormantDays` (7); the maintainer has -not confirmed that consequence. +Every source (main) checkout, registered or discovered by any route (siblings, +`maintenance.discoverRoots` children, tool folders, the scan), follows the +existing dormant rule and loses `node_modules` and Pods after +`idle.sourceDormantDays` (7), a linked worktree's after `idle.dormantDays` (1) +(`daemon/dependencies.ts` `takeIdleDependencies`, over `idleStageCandidates`' +discovered list); the maintainer has not confirmed that consequence for the scan +finds, and the guide wrongly said a main checkout keeps them before this change. Reason: on one MacBook `~/.codex/worktrees` held 53 worktrees and 12 GB that Stim listed once as outside the folders it manages, with no size and no way diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index ec0e72a577..1f3f92c66f 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -273,6 +273,35 @@ describe('scanForCheckouts', () => { expect(reads).toBeLessThan(10); }); + it('rejects the scan as an abort when the abort arrives during the nested checks of the last folder', async () => { + mkdirSync(join(home, 'mono', 'apps', 'app'), { recursive: true }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const controller = new AbortController(); + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('app')]; + return []; + }; + const recognition = { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === join(home, 'mono', 'apps', 'app')) controller.abort(); + return listing(path); + }, + }; + const detect = vi.fn<(path: string) => boolean>(() => true); + await expect( + scanForCheckouts(home, detect, { entries: listing, recognition, signal: controller.signal }), + ).rejects.toThrow(/abort/i); + expect(detect).not.toHaveBeenCalled(); + }); + it('stops at its time budget and says so', async () => { project('dev', 'one'); const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); @@ -510,6 +539,41 @@ describe('runDiscoveryScanIfDue', () => { }); }); + it('leaves the stored result and its finds unchanged when an abort arrives during the nested checks of the last folder', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const before = readFileSync(discoveryScanFile(), 'utf8'); + mkdirSync(join(base, 'home', 'mono', 'apps', 'app'), { recursive: true }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const controller = new AbortController(); + const listing = async (path: string) => { + const home = join(base, 'home'); + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('app')]; + return []; + }; + const recognition = { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path.endsWith('app')) controller.abort(); + return listing(path); + }, + }; + await expect( + runDiscoveryScanIfDue(tick(normal, {}, testContext({ signal: controller.signal })), { + ...options(t0 + 8 * 24 * hour), + scanOptions: { entries: listing, recognition }, + }), + ).rejects.toThrow(/abort/i); + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + expect(readDiscoveryScan()?.found).toEqual([join(base, 'home', 'dev', 'app')]); + }); + it('lets an abort escape instead of returning normally, and stores nothing', async () => { const controller = new AbortController(); controller.abort(); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 866ea47794..98a3c7ce95 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1238,8 +1238,8 @@ previous finds. It can overrun the budget by at most one read's 3 second timeout plus the time project detection itself takes on the folder it is recognising, which is synchronous and cannot be interrupted. Revalidation of cached finds has no total budget: each read is bounded by the 3 second timeout, and the number of -reads by detection's own caps (50 children per container, seven folders in the -chain). Before project +reads by detection's own caps (50 children per container, the home folder and up to +seven below it). Before project detection runs on a folder, the manifests detection reads are proved safe, for the folder, each ancestor down from the home folder, its apps, examples and packages folders and their children: a manifest (any case) that is not a regular file, an @@ -1273,8 +1273,9 @@ directory that hangs ends the reading as incomplete at once, and while that read is pending (at most one per thread: the daemon's main thread and passes thread each have one; the scan runs on the passes thread) no other launcher or scan read starts and a scheduled scan is skipped. A scan stops at the first read that -times out, keeping the previous finds. Reduce-only checkouts have their build outputs and, in -a linked worktree, node_modules and Pods reduced by the rules below. Passes +times out, keeping the previous finds. Reduce-only checkouts have their build outputs, and +their node_modules and Pods (a linked worktree's after idle.dormantDays, a source +checkout's after idle.sourceDormantDays), reduced by the rules below. Passes evaluate at most 40 of them at a time, in a window that moves every 15 minutes; stim gc --dry-run lists all of them and says which are left for a later pass. A folder named in maintenance.discoverRoots @@ -1286,8 +1287,13 @@ and build, dist, target, .cache and similar folders skipped only where git ignores them), its last commit and its git writes, for idle.dormantDays. A process running from or working in it, an agent session in it, a pin by maintenance.keep, more source files than the scan reads, or evidence that -cannot be read keeps it. Its regenerable build outputs go when it is idle; a -linked one also loses node_modules and Pods, a main checkout keeps them. A +cannot be read keeps it. Its regenerable build outputs go when it is idle, and its +node_modules and Pods go too: a linked worktree's after idle.dormantDays (default +1 day), a main (source) checkout's after idle.sourceDormantDays (default 7 days; 0 +never), by the same evidence. That holds for every kind of discovered checkout +(children of the folders holding registered checkouts, children of +maintenance.discoverRoots, reduce-only finds) even where no stim command ever ran, +and a PATH launcher keeps a reduce-only one. A target directory at a checkout root whose CACHEDIR.TAG starts with the standard CACHEDIR.TAG signature line and whose files git ignores counts as build output for every checkout. diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index cd1cc29f08..c74c5da472 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -140,7 +140,7 @@ export async function scanForCheckouts( stop('read'); return; } - if (nested === 'unsafe') return; + if (nested === 'unsafe' || exhausted()) return; if (isCheckout(folder)) { outcome.found.push(folder); return; diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index 4825d0b9e9..1d81313fea 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -229,7 +229,7 @@ export async function folderSafety( * container folder, with its own `package.json`) and up to 50 children of each. A symlinked container, or any unsafe or * unreadable member, makes the result that. The ancestors of `folder` are the caller's: they have passed `folderSafety`. * `stop` is asked before every read, so a scan's budget is honoured between them (`stopped`); revalidation passes none, - * and is bounded per read and by detection's own caps (50 children per container, at most seven folders in the chain). + * and is bounded per read and by detection's own caps (50 children per container; the chain is the home folder and up to seven below it). */ export async function nestedSafety( folder: string, @@ -238,37 +238,41 @@ export async function nestedSafety( io: RecognitionIo = REAL_RECOGNITION_IO, stop?: Stop, ): Promise { - const read = (path: string) => answered(path, () => io.readEntries(path), timeoutMs); - try { - for (const entry of entries.filter((candidate) => PROJECT_FOLDERS.includes(folded(candidate.name)))) { - if (entry.isSymbolicLink()) return 'unsafe'; - if (!entry.isDirectory()) continue; - const container = join(folder, entry.name); + /** The listing, or why it cannot be trusted; `stop` is never asked inside it, so an abort is never caught as a read error. */ + const list = async (path: string): Promise => { + try { + return await answered(path, () => io.readEntries(path), timeoutMs); + } catch (error) { + return error instanceof Unanswered ? 'unreadable' : 'unsafe'; + } + }; + for (const entry of entries.filter((candidate) => PROJECT_FOLDERS.includes(folded(candidate.name)))) { + if (entry.isSymbolicLink()) return 'unsafe'; + if (!entry.isDirectory()) continue; + const container = join(folder, entry.name); + if (stop?.()) return 'stopped'; + const members = await list(container); + if (typeof members === 'string') return members; + const own = await folderSafety(container, members, timeoutMs, io, stop); + if (own !== 'safe') return own; + for (const child of members + .filter((member) => member.isDirectory() && !member.isSymbolicLink()) + .slice(0, PROJECT_FOLDER_ENTRIES)) { + const path = join(container, child.name); if (stop?.()) return 'stopped'; - const members = await read(container); - const own = await folderSafety(container, members, timeoutMs, io, stop); - if (own !== 'safe') return own; - for (const child of members - .filter((member) => member.isDirectory() && !member.isSymbolicLink()) - .slice(0, PROJECT_FOLDER_ENTRIES)) { - const path = join(container, child.name); - if (stop?.()) return 'stopped'; - let inner: Entry[]; - try { - inner = await read(path); - } catch (error) { - if (error instanceof Unanswered) return 'unreadable'; - if (absent(error)) continue; - return 'unsafe'; - } - const result = await folderSafety(path, inner, timeoutMs, io, stop); - if (result !== 'safe') return result; + let inner: Entry[]; + try { + inner = await answered(path, () => io.readEntries(path), timeoutMs); + } catch (error) { + if (error instanceof Unanswered) return 'unreadable'; + if (absent(error)) continue; + return 'unsafe'; } + const result = await folderSafety(path, inner, timeoutMs, io, stop); + if (result !== 'safe') return result; } - return 'safe'; - } catch (error) { - return error instanceof Unanswered ? 'unreadable' : 'unsafe'; } + return 'safe'; } /** A folder's listing and `folderSafety`, for the chain of folders above a cached entry. */ diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 0dc81b9abc..cc1e4f0dca 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,9 +86,9 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs and, in a linked worktree, `node_modules` and Pods by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. -With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. +With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. A clean linked worktree whose branch is merged or whose pull request ended is removed only when it is idle by that same evidence at every pressure level, with no grace shortcut, and every git-ignored file also exists, byte for byte, in the main checkout. The exception is regenerable folders: directories git reports as ignored that are named `node_modules` or like build output (`build`, `dist`, `target`, `.next`, `.turbo`, `.expo` and similar), whose files are deleted with the worktree. An ignored file elsewhere that the main checkout lacks, even inside a tracked `dist/` folder, keeps the worktree. Any change to a tracked file, even `Podfile.lock` or `project.pbxproj`, is dirty work and keeps it, and so does a tracked file marked assume-unchanged or skip-worktree, which hides edits from `git status`. A worktree inside a dot-folder of its main checkout other than `.worktrees` (for example `.claude/worktrees`) belongs to another tool and is never removed, although its build output and dependencies follow the normal idle rules. The same holds for a checkout found through `~/.codex/worktrees` or the full scan. Dirty work is never archived, where a registered finished worktree's dirty work is archived first. From 9ad3222e27019f798a90cda699469247dd23bcd6 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 06:54:37 -0500 Subject: [PATCH 20/25] fix: record a denied project check as unreadable and keep its finds, check the abort after every gate result and correct the reduce-only sentence --- llp/0004-discovered-checkouts.decision.md | 12 +++- .../src/__tests__/discovery-scan.test.ts | 71 +++++++++++++++++++ .../stim-cli/src/__tests__/scan-rules.test.ts | 4 +- packages/stim-cli/src/guide/cleanup.ts | 4 +- .../stim-cli/src/workspace/discovery-scan.ts | 14 +++- packages/stim-cli/src/workspace/scan-rules.ts | 16 +++-- website/docs/worktrees.md | 2 +- 7 files changed, 111 insertions(+), 12 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index c00fe5945d..fbd9b399fb 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -65,7 +65,8 @@ Everything else in this section was chosen by the coordinator and the review of ## Tool folders and the full scan Discovery also reads two more sources, and what they find is reduce-only: Stim -reduces its build outputs and, in a linked worktree, `node_modules` and Pods by +reduces its build outputs and its `node_modules` and Pods (a linked worktree's +after `idle.dormantDays`, a source checkout's after `idle.sourceDormantDays`) by the discovered-checkout rules, and never removes the checkout or worktree. The hold that keeps a worktree inside a dot-folder of its main checkout (`.claude/worktrees`) applies to it. @@ -89,7 +90,11 @@ hold that keeps a worktree inside a dot-folder of its main checkout - The stored result keeps the schedule (last attempt, last attempt that did not fail, last attempt at low disk) and the finds. A failed or truncated scan keeps the previous finds, and a finished one keeps those under a folder it - could not read, so a worse result never replaces a better one. Discovery + could not read, so a worse result never replaces a better one. A read the system + denies during the project checks (a manifest, a nested listing) is not a + structural fact: it lists the folder unreadable, runs no detection on it or + below it, leaves the scan untruncated and keeps its finds; in revalidation it is + `pending`. Discovery revalidates each find before using it: a real folder, not a symlink, on the home volume, below no skipped folder, still recognised. - The scan runs after a maintenance pass has acted, so it never delays cleanup. @@ -251,6 +256,9 @@ decision leaves open: in a dot-folder of its main checkout (`.claude/worktrees`). Removal is what doctor suggests to the person. A folder named in `maintenance.discoverRoots` keeps the full discovered rules, removal of finished worktrees included. + (Added by the implementing lane, not part of the recorded reading: the rule + in force also removes a source checkout's `node_modules` and Pods after + `idle.sourceDormantDays`, as the section that states it describes.) 2. Common paths start with `~/.codex/worktrees`. Other tools' folders are added only with evidence that the tool uses that path. 3. "Once in a while" is weekly. On low disk the scan runs when a volume's level diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index 1f3f92c66f..d618de3d26 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -574,6 +574,77 @@ describe('runDiscoveryScanIfDue', () => { expect(readDiscoveryScan()?.found).toEqual([join(base, 'home', 'dev', 'app')]); }); + describe('a read that fails for a reason other than a timeout or absence', () => { + const denied = () => Object.assign(new Error('EACCES'), { code: 'EACCES' }); + const later = () => options(t0 + 8 * 24 * hour); + + it('keeps the previous find and lists its folder unreadable when its manifest read is denied', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const app = join(base, 'home', 'dev', 'app'); + await runDiscoveryScanIfDue(tick(), { + ...later(), + scanOptions: { + recognition: { + ...REAL_RECOGNITION_IO, + readText: async () => { + throw denied(); + }, + }, + }, + }); + expect(readDiscoveryScan()).toMatchObject({ found: [app], unreadable: [app], truncation: 'none' }); + }); + + it('keeps the previous find and lists its folder unreadable when a nested listing is denied', async () => { + const mono = join(base, 'home', 'mono'); + mkdirSync(join(mono, 'apps', 'app'), { recursive: true }); + writeFileSync(join(mono, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()?.found).toContain(mono); + await runDiscoveryScanIfDue(tick(), { + ...later(), + scanOptions: { + recognition: { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === join(mono, 'apps', 'app')) throw denied(); + return REAL_RECOGNITION_IO.readEntries(path); + }, + }, + }, + }); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [mono], truncation: 'none' }); + expect(readDiscoveryScan()?.found).toContain(mono); + }); + + it('rejects as an abort and leaves the file unchanged when the abort arrives during a final nested listing that also fails', async () => { + const mono = join(base, 'home', 'mono'); + mkdirSync(join(mono, 'apps', 'app'), { recursive: true }); + writeFileSync(join(mono, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + await runDiscoveryScanIfDue(tick(), options(t0)); + const before = readFileSync(discoveryScanFile(), 'utf8'); + const controller = new AbortController(); + await expect( + runDiscoveryScanIfDue(tick(normal, {}, testContext({ signal: controller.signal })), { + ...later(), + scanOptions: { + recognition: { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === join(mono, 'apps', 'app')) { + controller.abort(); + throw denied(); + } + return REAL_RECOGNITION_IO.readEntries(path); + }, + }, + }, + }), + ).rejects.toThrow(/abort/i); + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + }); + }); + it('lets an abort escape instead of returning normally, and stores nothing', async () => { const controller = new AbortController(); controller.abort(); diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts index ffc836cbd9..cd9e29ecef 100644 --- a/packages/stim-cli/src/__tests__/scan-rules.test.ts +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -152,7 +152,7 @@ describe('scanEntryValid with a hand-edited scan result', () => { expect(await folderSafety(where, entries, 10, io)).toBe('safe'); }); - it('is unsafe, never safe, when a child that detection enumerates cannot be listed', async () => { + it('is denied, never safe, when a child that detection enumerates cannot be listed', async () => { const denied = Object.assign(new Error('EACCES'), { code: 'EACCES' }); const gone = Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); const io = (failure: Error) => ({ @@ -164,7 +164,7 @@ describe('scanEntryValid with a hand-edited scan result', () => { }, }); const entries = [folder('apps')]; - expect(await nestedSafety(where, entries, 10, io(denied))).toBe('unsafe'); + expect(await nestedSafety(where, entries, 10, io(denied))).toBe('denied'); expect(await nestedSafety(where, entries, 10, io(gone))).toBe('safe'); }); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 98a3c7ce95..a25ff24415 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1254,7 +1254,9 @@ a stalled mount point there is not detected), and what a pass reads inside a che kept in maintenance/discovery-scan.json under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a pass or gc never walks the disk. A scan that fails or stops early keeps the previous finds, -and a finished one keeps those under a folder it could not read. Discovery checks +and a finished one keeps those under a folder it could not read, which includes a +folder whose project check was denied by the system (it is listed unreadable, not +rejected). Discovery checks each recorded checkout again before using it: still a real folder (not a symlink), on the home volume, below no skipped folder and still a project; anything else is dropped. Checkouts the scan finds under a dot-folder of the diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index c74c5da472..6bf6ccca2c 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -124,7 +124,12 @@ export async function scanForCheckouts( return; } const safety = await folderSafety(folder, entries, readTimeoutMs, recognition, exhausted); + signal?.throwIfAborted(); if (safety === 'stopped') return; + if (safety === 'denied') { + outcome.unreadable.push(folder); + return; + } if (safety === 'unreadable') { outcome.unreadable.push(folder); stop('read'); @@ -134,7 +139,12 @@ export async function scanForCheckouts( if (above) return; if (depth > 0) { const nested = await nestedSafety(folder, entries, readTimeoutMs, recognition, exhausted); + signal?.throwIfAborted(); if (nested === 'stopped') return; + if (nested === 'denied') { + outcome.unreadable.push(folder); + return; + } if (nested === 'unreadable') { outcome.unreadable.push(folder); stop('read'); @@ -172,7 +182,8 @@ export async function scanForCheckouts( * `scanEntryValid`, not here. */ export function mergeFound(previous: readonly string[], outcome: ScanOutcome): string[] { - const under = (path: string) => outcome.unreadable.some((folder) => path.startsWith(`${folder}${sep}`)); + const under = (path: string) => + outcome.unreadable.some((folder) => path === folder || path.startsWith(`${folder}${sep}`)); const kept = outcome.truncated ? previous : previous.filter(under); return [...new Set([...outcome.found, ...kept])].toSorted(); } @@ -273,6 +284,7 @@ export async function runDiscoveryScanIfDue( skip: scanSkipper(realHome), ...scanOptions, }); + operation.signal.throwIfAborted(); writeScan({ ...attempt, completedAt: outcome.truncation === 'read' ? (last?.completedAt ?? 0) : now, diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index 1d81313fea..27ad3d7eed 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -177,7 +177,13 @@ export const REAL_RECOGNITION_IO: RecognitionIo = { realpath: (path) => realpath(path), }; -type Safety = 'safe' | 'unsafe' | 'unreadable' | 'stopped'; +/** + * `unsafe`: a structural fact, the folder is not a checkout and there is nothing to record. `denied`: a read failed for a + * reason that is neither a timeout nor "absent" (EACCES, EPERM, EIO), so the scan could not look, as when a folder's own + * listing fails: it is recorded unreadable and its finds are kept. `unreadable`: a read timed out or another is pending. + * `stopped`: the scan's limit ran out. + */ +type Safety = 'safe' | 'unsafe' | 'denied' | 'unreadable' | 'stopped'; /** Asked before every read the gate makes: true when the scan's own limit (abort, time budget, folder cap) has run out. */ type Stop = () => boolean; @@ -213,7 +219,7 @@ export async function folderSafety( try { text = await answered(folder, () => io.readText(join(folder, manifest.name)), timeoutMs); } catch (error) { - return error instanceof Unanswered ? 'unreadable' : 'unsafe'; + return error instanceof Unanswered ? 'unreadable' : 'denied'; } try { declares &&= declaresAppDependency(JSON.parse(text)); @@ -243,7 +249,7 @@ export async function nestedSafety( try { return await answered(path, () => io.readEntries(path), timeoutMs); } catch (error) { - return error instanceof Unanswered ? 'unreadable' : 'unsafe'; + return error instanceof Unanswered ? 'unreadable' : 'denied'; } }; for (const entry of entries.filter((candidate) => PROJECT_FOLDERS.includes(folded(candidate.name)))) { @@ -266,7 +272,7 @@ export async function nestedSafety( } catch (error) { if (error instanceof Unanswered) return 'unreadable'; if (absent(error)) continue; - return 'unsafe'; + return 'denied'; } const result = await folderSafety(path, inner, timeoutMs, io, stop); if (result !== 'safe') return result; @@ -285,7 +291,7 @@ async function gateOf( const listing = await answered(folder, () => io.readEntries(folder), timeoutMs); return { listing, safety: await folderSafety(folder, listing, timeoutMs, io) }; } catch (error) { - return { listing: [], safety: error instanceof Unanswered ? 'unreadable' : 'unsafe' }; + return { listing: [], safety: error instanceof Unanswered ? 'unreadable' : 'denied' }; } } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index cc1e4f0dca..ebd8106e03 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 1c22881c803f215dc60399a0ecaf7c8a420975c6 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 07:15:22 -0500 Subject: [PATCH 21/25] fix: record any failed scan read that is not absent or a timeout as unreadable --- .../src/__tests__/discovery-scan.test.ts | 39 ++++++++++++++++++- .../stim-cli/src/workspace/discovery-scan.ts | 13 ++++--- packages/stim-cli/src/workspace/scan-rules.ts | 2 +- 3 files changed, 46 insertions(+), 8 deletions(-) diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index d618de3d26..c4c657158f 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -9,7 +9,7 @@ import { symlinkSync, writeFileSync, } from 'node:fs'; -import { readdir } from 'node:fs/promises'; +import { lstat, readdir } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { testContext } from './_context.ts'; @@ -595,6 +595,43 @@ describe('runDiscoveryScanIfDue', () => { expect(readDiscoveryScan()).toMatchObject({ found: [app], unreadable: [app], truncation: 'none' }); }); + const failure = (code: string) => Object.assign(new Error(code), { code }); + + it('lists a folder unreadable and keeps the finds beneath it when its listing fails with EIO, without truncating', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const dev = join(base, 'home', 'dev'); + await runDiscoveryScanIfDue(tick(), { + ...later(), + scanOptions: { + entries: async (path: string) => { + if (path === dev) throw failure('EIO'); + return readdir(path, { withFileTypes: true }); + }, + }, + }); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [dev], truncated: false, truncation: 'none' }); + expect(readDiscoveryScan()?.found).toEqual([join(dev, 'app')]); + }); + + it('does the same when the lstat of a child fails with ESTALE, and records nothing when the child is absent', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const dev = join(base, 'home', 'dev'); + const failing = (code: string) => ({ + ...later(), + scanOptions: { + stat: async (path: string) => { + if (path === dev) throw failure(code); + return lstat(path); + }, + }, + }); + await runDiscoveryScanIfDue(tick(), failing('ESTALE')); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [dev], truncation: 'none' }); + expect(readDiscoveryScan()?.found).toEqual([join(dev, 'app')]); + await runDiscoveryScanIfDue(tick(), { ...failing('ENOENT'), now: t0 + 16 * 24 * hour }); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [], truncation: 'none', found: [] }); + }); + it('keeps the previous find and lists its folder unreadable when a nested listing is denied', async () => { const mono = join(base, 'home', 'mono'); mkdirSync(join(mono, 'apps', 'app'), { recursive: true }); diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 6bf6ccca2c..2e4e8da531 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -23,6 +23,7 @@ import { nestedSafety, scanDescends, scanSkipper, + absent, type RecognitionIo, } from './scan-rules.ts'; @@ -41,10 +42,6 @@ interface ScanOutcome { truncation: Exclude; } -const errorCode = (error: unknown) => (error instanceof Error && 'code' in error ? String(error.code) : ''); - -const refused = (error: unknown) => ['EACCES', 'EPERM'].includes(errorCode(error)); - /** * A pruned walk below `home` (`scanDescends`, `skip`): folders for which `isCheckout` holds are results and are not * entered; symlinks and other volumes are skipped; a folder the system refuses is recorded as unreadable, never an error. @@ -92,6 +89,10 @@ export async function scanForCheckouts( else if (outcome.visited >= MAX_FOLDERS) stop('folders'); return outcome.truncated; }; + /** + * A read of `folder` failed. A timeout ends the scan (true). Anything but "it does not exist" means the scan could not + * look, so the folder is recorded unreadable and its earlier finds are kept; the walk carries on (false). + */ const stopAt = (folder: string, error: unknown): boolean => { signal?.throwIfAborted(); if (error instanceof Unanswered) { @@ -99,7 +100,7 @@ export async function scanForCheckouts( stop('read'); return true; } - if (refused(error)) outcome.unreadable.push(folder); + if (!absent(error)) outcome.unreadable.push(folder); return false; }; if (hungRead() !== null) { @@ -110,7 +111,7 @@ export async function scanForCheckouts( try { volume = (await answered(home, () => statOf(home), readTimeoutMs)).dev; } catch (error) { - if (!stopAt(home, error)) throw error; + if (!stopAt(home, error) && absent(error)) throw error; return outcome; } const walk = async (folder: string, depth: number, unsafeAbove: boolean): Promise => { diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts index 27ad3d7eed..f0564096d5 100644 --- a/packages/stim-cli/src/workspace/scan-rules.ts +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -149,7 +149,7 @@ const XCODE_CONTAINER = /\.(xcodeproj|xcworkspace)$/i; const OWNED_ROOTS = new Set(['ios', 'android']); const folded = (name: string): string => name.toLowerCase(); /** The entry is not there: the only read failure that is not unsafe, and only for an entry a successful listing showed. */ -const absent = (error: unknown): boolean => +export const absent = (error: unknown): boolean => error instanceof Error && 'code' in error && (error.code === 'ENOENT' || error.code === 'ENOTDIR'); interface Entry { From f3e3d277de1b27551c613febd67157a978624fb0 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 07:39:27 -0500 Subject: [PATCH 22/25] fix: a background worktree sweep inspects no reduce-only checkout left for a later pass --- llp/0004-discovered-checkouts.decision.md | 11 ++++-- .../__tests__/discovered-reduce-only.test.ts | 38 +++++++++++++++++++ .../stim-cli/src/commands/gc/worktrees.ts | 14 ++++++- packages/stim-cli/src/guide/cleanup.ts | 7 ++-- packages/stim-cli/src/stimd/passes/preview.ts | 1 + packages/stim-cli/src/stimd/passes/sweep.ts | 4 ++ website/docs/worktrees.md | 2 +- 7 files changed, 69 insertions(+), 8 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index fbd9b399fb..bb04f3eb21 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -84,9 +84,14 @@ hold that keeps a worktree inside a dot-folder of its main checkout an error. The result is stored in `maintenance/discovery-scan.json` under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a maintenance pass or `stim gc` never walks the disk. -- Passes evaluate at most 40 reduce-only checkouts at a time, in a window over - them that moves every 15 minutes. Discovery still lists every one, and the - report says which are left for a later pass. +- A background maintenance pass evaluates and inspects (git status, activity, + HEAD, unpushed, pins; the artifact, dependency, stage and Cargo routes) at most + 40 reduce-only checkouts at a time, in a window over them that moves every 15 + minutes; the worktree sweep leaves the others out entirely (`skipDeferred`), + which cannot change what is removed because a reduce-only checkout is held + from removal before any evidence is read. Discovery still lists every one, and + `stim gc` and its dry run inspect and list all of them, saying which are left + for a later pass. The ladder's finished-worktree rung does not skip them. - The stored result keeps the schedule (last attempt, last attempt that did not fail, last attempt at low disk) and the finds. A failed or truncated scan keeps the previous finds, and a finished one keeps those under a folder it diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index d0e93cecf6..31eed920a9 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -16,6 +16,7 @@ import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; +import * as discovery from '../workspace/discovery.ts'; import * as idleStop from '../stimd/passes/idle-stop.ts'; import { clearDependency } from '../daemon/dependencies.ts'; import * as runningFrom from '../workspace/running-from.ts'; @@ -539,6 +540,43 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(codex)).toBe(true); }); + it('inspects no deferred reduce-only worktree in a background sweep, and still lists every one in a report', async () => { + const first = worktree(join(base, 'home', '.codex', 'worktrees', 'aa11', 'main'), 'first'); + const second = worktree(join(base, 'home', '.codex', 'worktrees', 'bb22', 'main'), 'second'); + const real = discovery.discoverCheckouts; + vi.spyOn(discovery, 'discoverCheckouts').mockImplementation(async (...args) => { + const found = await real(...args); + return found.map((entry) => { + const deferred = entry.path === second; + return deferred ? Object.assign({}, entry, { deferred }) : entry; + }); + }); + const inspected: string[] = []; + const executor = getExecutor(); + setExecutor({ + ...executor, + runFileAsync: async (file, args, options) => { + if (file === 'git' && args.includes('status')) + inspected.push(String(options?.cwd ?? args[args.indexOf('-C') + 1])); + return executor.runFileAsync(file, args, options); + }, + }); + const run = (skipDeferred: boolean) => + collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false, skipDeferred }); + + const background = await run(true); + expect(background.worktrees.map((entry) => entry.path)).toContain(first); + expect(background.worktrees.map((entry) => entry.path)).not.toContain(second); + expect(inspected).toContain(first); + expect(inspected).not.toContain(second); + + inspected.length = 0; + const report = await run(false); + expect(report.worktrees.map((entry) => entry.path)).toEqual(expect.arrayContaining([first, second])); + expect(inspected).toContain(second); + resetExecutor(); + }); + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); diff --git a/packages/stim-cli/src/commands/gc/worktrees.ts b/packages/stim-cli/src/commands/gc/worktrees.ts index 1c13b488b5..bb84f8c46d 100644 --- a/packages/stim-cli/src/commands/gc/worktrees.ts +++ b/packages/stim-cli/src/commands/gc/worktrees.ts @@ -814,6 +814,7 @@ export async function collectWorktreeSweep({ now, graceMs, refresh = true, + skipDeferred = false, operation, }: { idle: boolean; @@ -821,6 +822,12 @@ export async function collectWorktreeSweep({ now: number; graceMs?: number; refresh?: boolean; + /** + * Leaves out the reduce-only checkouts discovery defers to a later pass, so a background pass inspects at most its + * window of them. A reduce-only checkout is never removed (`discoveredRemovalHold` decides before any evidence is read), + * so leaving one out cannot change what is removed. Reports keep every one. + */ + skipDeferred?: boolean; operation?: StimContext; }): Promise { const days = idle ? (olderThan ?? DEFAULT_WORKTREE_IDLE_DAYS) : null; @@ -835,7 +842,12 @@ export async function collectWorktreeSweep({ const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); const discoveredPaths = new Set(discovered.map(({ path }) => path)); const reduceOnlyPaths = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); - const candidates = await candidateRootsAsync(operation, discovered); + const deferredPaths = new Set( + skipDeferred ? discovered.filter((entry) => entry.deferred).map(({ path }) => path) : [], + ); + const candidates = (await candidateRootsAsync(operation, discovered)).filter( + (root) => !deferredPaths.has(canonicalPath(root)), + ); if (candidates.length) phase('worktrees', `checking ${plural(candidates.length, 'project')} for merged or idle worktrees`, operation); const matched: ReturnType[] = []; diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index a25ff24415..8fe9e63019 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1277,9 +1277,10 @@ each have one; the scan runs on the passes thread) no other launcher or scan read starts and a scheduled scan is skipped. A scan stops at the first read that times out, keeping the previous finds. Reduce-only checkouts have their build outputs, and their node_modules and Pods (a linked worktree's after idle.dormantDays, a source -checkout's after idle.sourceDormantDays), reduced by the rules below. Passes -evaluate at most 40 of them at a time, in a window that moves every 15 minutes; -stim gc --dry-run lists all of them and says which are left for a later pass. A +checkout's after idle.sourceDormantDays), reduced by the rules below. A background +maintenance pass evaluates, and inspects with git, at most 40 of them at a time, +in a window that moves every 15 minutes; stim gc and its dry run inspect and list +all of them and say which are left for a later pass. A folder named in maintenance.discoverRoots keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine diff --git a/packages/stim-cli/src/stimd/passes/preview.ts b/packages/stim-cli/src/stimd/passes/preview.ts index cf29de7d40..89671fa0e4 100644 --- a/packages/stim-cli/src/stimd/passes/preview.ts +++ b/packages/stim-cli/src/stimd/passes/preview.ts @@ -251,6 +251,7 @@ export async function plannedMaintenance( idleDays: sweep && settings.removeIdleWorktrees && settings.olderThanDays > 0 ? settings.olderThanDays : null, operation, deviceActivity, + skipDeferred: background, }); merge('worktree', planned); result.worktreeSweep = planned.sweep; diff --git a/packages/stim-cli/src/stimd/passes/sweep.ts b/packages/stim-cli/src/stimd/passes/sweep.ts index 6973a8fd97..598dd54933 100644 --- a/packages/stim-cli/src/stimd/passes/sweep.ts +++ b/packages/stim-cli/src/stimd/passes/sweep.ts @@ -347,6 +347,7 @@ export async function planWorktrees({ protectRecentMs = 0, refresh, idleDays = null, + skipDeferred = false, operation, deviceActivity, }: { @@ -358,6 +359,8 @@ export async function planWorktrees({ sizes?: readonly MaintenanceSize[]; protectRecentMs?: number; refresh?: boolean; + /** A background pass does not inspect the reduce-only checkouts left for a later pass. */ + skipDeferred?: boolean; operation?: StimContext; /** Reads owned device activity for an operation; a new listing when absent. */ deviceActivity?: (withSims: boolean) => Promise; @@ -367,6 +370,7 @@ export async function planWorktrees({ olderThan: idleDays, now, operation, + skipDeferred, ...(graceMs === undefined ? {} : { graceMs }), ...(refresh === undefined ? {} : { refresh }), }); diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index ebd8106e03..c0bd2fa793 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. A background maintenance pass evaluates, and checks with git, at most 40 of them at a time; `stim gc` and `stim gc --dry-run` check and list all of them and say which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 4b5265a84183be51e2451b4ab2c690964ec8320a Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 08:13:04 -0500 Subject: [PATCH 23/25] Revert "fix: a background worktree sweep inspects no reduce-only checkout left for a later pass" This reverts commit f3e3d277de1b27551c613febd67157a978624fb0. --- llp/0004-discovered-checkouts.decision.md | 11 ++---- .../__tests__/discovered-reduce-only.test.ts | 38 ------------------- .../stim-cli/src/commands/gc/worktrees.ts | 14 +------ packages/stim-cli/src/guide/cleanup.ts | 7 ++-- packages/stim-cli/src/stimd/passes/preview.ts | 1 - packages/stim-cli/src/stimd/passes/sweep.ts | 4 -- website/docs/worktrees.md | 2 +- 7 files changed, 8 insertions(+), 69 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index bb04f3eb21..fbd9b399fb 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -84,14 +84,9 @@ hold that keeps a worktree inside a dot-folder of its main checkout an error. The result is stored in `maintenance/discovery-scan.json` under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a maintenance pass or `stim gc` never walks the disk. -- A background maintenance pass evaluates and inspects (git status, activity, - HEAD, unpushed, pins; the artifact, dependency, stage and Cargo routes) at most - 40 reduce-only checkouts at a time, in a window over them that moves every 15 - minutes; the worktree sweep leaves the others out entirely (`skipDeferred`), - which cannot change what is removed because a reduce-only checkout is held - from removal before any evidence is read. Discovery still lists every one, and - `stim gc` and its dry run inspect and list all of them, saying which are left - for a later pass. The ladder's finished-worktree rung does not skip them. +- Passes evaluate at most 40 reduce-only checkouts at a time, in a window over + them that moves every 15 minutes. Discovery still lists every one, and the + report says which are left for a later pass. - The stored result keeps the schedule (last attempt, last attempt that did not fail, last attempt at low disk) and the finds. A failed or truncated scan keeps the previous finds, and a finished one keeps those under a folder it diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 31eed920a9..d0e93cecf6 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -16,7 +16,6 @@ import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; -import * as discovery from '../workspace/discovery.ts'; import * as idleStop from '../stimd/passes/idle-stop.ts'; import { clearDependency } from '../daemon/dependencies.ts'; import * as runningFrom from '../workspace/running-from.ts'; @@ -540,43 +539,6 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(codex)).toBe(true); }); - it('inspects no deferred reduce-only worktree in a background sweep, and still lists every one in a report', async () => { - const first = worktree(join(base, 'home', '.codex', 'worktrees', 'aa11', 'main'), 'first'); - const second = worktree(join(base, 'home', '.codex', 'worktrees', 'bb22', 'main'), 'second'); - const real = discovery.discoverCheckouts; - vi.spyOn(discovery, 'discoverCheckouts').mockImplementation(async (...args) => { - const found = await real(...args); - return found.map((entry) => { - const deferred = entry.path === second; - return deferred ? Object.assign({}, entry, { deferred }) : entry; - }); - }); - const inspected: string[] = []; - const executor = getExecutor(); - setExecutor({ - ...executor, - runFileAsync: async (file, args, options) => { - if (file === 'git' && args.includes('status')) - inspected.push(String(options?.cwd ?? args[args.indexOf('-C') + 1])); - return executor.runFileAsync(file, args, options); - }, - }); - const run = (skipDeferred: boolean) => - collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false, skipDeferred }); - - const background = await run(true); - expect(background.worktrees.map((entry) => entry.path)).toContain(first); - expect(background.worktrees.map((entry) => entry.path)).not.toContain(second); - expect(inspected).toContain(first); - expect(inspected).not.toContain(second); - - inspected.length = 0; - const report = await run(false); - expect(report.worktrees.map((entry) => entry.path)).toEqual(expect.arrayContaining([first, second])); - expect(inspected).toContain(second); - resetExecutor(); - }); - it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); diff --git a/packages/stim-cli/src/commands/gc/worktrees.ts b/packages/stim-cli/src/commands/gc/worktrees.ts index bb84f8c46d..1c13b488b5 100644 --- a/packages/stim-cli/src/commands/gc/worktrees.ts +++ b/packages/stim-cli/src/commands/gc/worktrees.ts @@ -814,7 +814,6 @@ export async function collectWorktreeSweep({ now, graceMs, refresh = true, - skipDeferred = false, operation, }: { idle: boolean; @@ -822,12 +821,6 @@ export async function collectWorktreeSweep({ now: number; graceMs?: number; refresh?: boolean; - /** - * Leaves out the reduce-only checkouts discovery defers to a later pass, so a background pass inspects at most its - * window of them. A reduce-only checkout is never removed (`discoveredRemovalHold` decides before any evidence is read), - * so leaving one out cannot change what is removed. Reports keep every one. - */ - skipDeferred?: boolean; operation?: StimContext; }): Promise { const days = idle ? (olderThan ?? DEFAULT_WORKTREE_IDLE_DAYS) : null; @@ -842,12 +835,7 @@ export async function collectWorktreeSweep({ const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); const discoveredPaths = new Set(discovered.map(({ path }) => path)); const reduceOnlyPaths = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); - const deferredPaths = new Set( - skipDeferred ? discovered.filter((entry) => entry.deferred).map(({ path }) => path) : [], - ); - const candidates = (await candidateRootsAsync(operation, discovered)).filter( - (root) => !deferredPaths.has(canonicalPath(root)), - ); + const candidates = await candidateRootsAsync(operation, discovered); if (candidates.length) phase('worktrees', `checking ${plural(candidates.length, 'project')} for merged or idle worktrees`, operation); const matched: ReturnType[] = []; diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 8fe9e63019..a25ff24415 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1277,10 +1277,9 @@ each have one; the scan runs on the passes thread) no other launcher or scan read starts and a scheduled scan is skipped. A scan stops at the first read that times out, keeping the previous finds. Reduce-only checkouts have their build outputs, and their node_modules and Pods (a linked worktree's after idle.dormantDays, a source -checkout's after idle.sourceDormantDays), reduced by the rules below. A background -maintenance pass evaluates, and inspects with git, at most 40 of them at a time, -in a window that moves every 15 minutes; stim gc and its dry run inspect and list -all of them and say which are left for a later pass. A +checkout's after idle.sourceDormantDays), reduced by the rules below. Passes +evaluate at most 40 of them at a time, in a window that moves every 15 minutes; +stim gc --dry-run lists all of them and says which are left for a later pass. A folder named in maintenance.discoverRoots keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine diff --git a/packages/stim-cli/src/stimd/passes/preview.ts b/packages/stim-cli/src/stimd/passes/preview.ts index 89671fa0e4..cf29de7d40 100644 --- a/packages/stim-cli/src/stimd/passes/preview.ts +++ b/packages/stim-cli/src/stimd/passes/preview.ts @@ -251,7 +251,6 @@ export async function plannedMaintenance( idleDays: sweep && settings.removeIdleWorktrees && settings.olderThanDays > 0 ? settings.olderThanDays : null, operation, deviceActivity, - skipDeferred: background, }); merge('worktree', planned); result.worktreeSweep = planned.sweep; diff --git a/packages/stim-cli/src/stimd/passes/sweep.ts b/packages/stim-cli/src/stimd/passes/sweep.ts index 598dd54933..6973a8fd97 100644 --- a/packages/stim-cli/src/stimd/passes/sweep.ts +++ b/packages/stim-cli/src/stimd/passes/sweep.ts @@ -347,7 +347,6 @@ export async function planWorktrees({ protectRecentMs = 0, refresh, idleDays = null, - skipDeferred = false, operation, deviceActivity, }: { @@ -359,8 +358,6 @@ export async function planWorktrees({ sizes?: readonly MaintenanceSize[]; protectRecentMs?: number; refresh?: boolean; - /** A background pass does not inspect the reduce-only checkouts left for a later pass. */ - skipDeferred?: boolean; operation?: StimContext; /** Reads owned device activity for an operation; a new listing when absent. */ deviceActivity?: (withSims: boolean) => Promise; @@ -370,7 +367,6 @@ export async function planWorktrees({ olderThan: idleDays, now, operation, - skipDeferred, ...(graceMs === undefined ? {} : { graceMs }), ...(refresh === undefined ? {} : { refresh }), }); diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index c0bd2fa793..ebd8106e03 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. A background maintenance pass evaluates, and checks with git, at most 40 of them at a time; `stim gc` and `stim gc --dry-run` check and list all of them and say which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From cc647e11daec854d5d78e0b38c52600c19aad386 Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 08:14:37 -0500 Subject: [PATCH 24/25] docs: state what the reduce-only window bounds and add a regression that a nested discover root cannot release a held tool-folder worktree --- llp/0004-discovered-checkouts.decision.md | 16 +++++++++-- .../__tests__/discovered-reduce-only.test.ts | 28 +++++++++++++++++++ packages/stim-cli/src/guide/cleanup.ts | 10 +++++-- website/docs/worktrees.md | 2 +- 4 files changed, 49 insertions(+), 7 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index fbd9b399fb..72fa033671 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -84,9 +84,19 @@ hold that keeps a worktree inside a dot-folder of its main checkout an error. The result is stored in `maintenance/discovery-scan.json` under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a maintenance pass or `stim gc` never walks the disk. -- Passes evaluate at most 40 reduce-only checkouts at a time, in a window over - them that moves every 15 minutes. Discovery still lists every one, and the - report says which are left for a later pass. +- The reduction passes (the artifact and dependency collection in + `artifactCheckoutsAsync`, the idle and dormant stages in `idleWorkspaces`, and + the Cargo target candidates in `cargoTargetCandidates`) skip a reduce-only + checkout left for a later pass, so each evaluates at most 40 at a time, in a + window over them that moves every 15 minutes. Discovery still lists every one, + and `stim gc --dry-run` marks those left for a later pass. The window does not + bound the finished-worktree check (`collectWorktreeSweep`), in a background + pass or in the pressure ladder: it classifies and inspects every discovered + worktree with git (status, activity, HEAD, unpushed), measured at about 0.14 + seconds each on one MacBook, whether or not it is left for a later pass. Skipping + the deferred ones there was tried and reverted, because worktrees are grouped by + path prefix and a group is held when any of its roots is reduce-only, so + dropping a root could release a held worktree. - The stored result keeps the schedule (last attempt, last attempt that did not fail, last attempt at low disk) and the finds. A failed or truncated scan keeps the previous finds, and a finished one keeps those under a folder it diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index d0e93cecf6..11da17e69c 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -539,6 +539,34 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(codex)).toBe(true); }); + it('holds a tool-folder worktree from removal even when a discover root inside it adds a non-reduce-only child root', async () => { + mkdirSync(join(main, 'packages', 'app'), { recursive: true }); + writeFileSync(join(main, 'packages', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + git(main, 'add', '-A'); + git(main, 'commit', '-q', '-m', 'app'); + git(main, 'push', '-q', 'origin', 'main'); + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'aa11', 'main'), 'codex'); + const config = { + version: 2 as const, + projects: { [main]: {} }, + repos: {}, + maintenance: { discoverRoots: [join(codex, 'packages')] }, + }; + saveConfig(config); + const found = await discoverCheckouts(config, [main]); + expect(found.map(({ path, reduceOnly }) => [path, reduceOnly])).toEqual( + expect.arrayContaining([ + [codex, true], + [join(codex, 'packages', 'app'), undefined], + ]), + ); + const planned = await collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false }); + expect(planned.worktrees.find((entry) => entry.path === codex)).toMatchObject({ + reduceOnly: true, + skipped: expect.stringContaining('never removes'), + }); + }); + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index a25ff24415..56edd1c3c7 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1277,9 +1277,13 @@ each have one; the scan runs on the passes thread) no other launcher or scan read starts and a scheduled scan is skipped. A scan stops at the first read that times out, keeping the previous finds. Reduce-only checkouts have their build outputs, and their node_modules and Pods (a linked worktree's after idle.dormantDays, a source -checkout's after idle.sourceDormantDays), reduced by the rules below. Passes -evaluate at most 40 of them at a time, in a window that moves every 15 minutes; -stim gc --dry-run lists all of them and says which are left for a later pass. A +checkout's after idle.sourceDormantDays), reduced by the rules below. The +reduction passes (artifact and dependency collection, the idle and dormant +stages, and the Cargo target measurement) evaluate at most 40 of them at a time, +in a window that moves every 15 minutes; stim gc --dry-run lists all of them and +says which are left for a later pass. The finished-worktree check is not bounded +by that window: in a background pass and in the pressure ladder it still inspects +every discovered worktree with git (about 0.14 seconds each where measured). A folder named in maintenance.discoverRoots keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index ebd8106e03..049f38c56b 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Passes evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. The cleanup passes that reduce build outputs, dependencies and Cargo targets evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. The check for finished worktrees is not limited that way: it still inspects every discovered worktree with git, which took about 0.14 seconds each where it was measured. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. From 938fee291b810731cd08795a10cb54e9a53ee70d Mon Sep 17 00:00:00 2001 From: Janic Duplessis Date: Sun, 15 Nov 2026 08:41:08 -0500 Subject: [PATCH 25/25] fix: remove the reduce-only window and skip the scan when its file cannot be read --- llp/0004-discovered-checkouts.decision.md | 28 +++--- packages/core/state/discovery-scan.ts | 31 +++++- packages/stim-cli/api.api.txt | 1 - .../__tests__/discovered-reduce-only.test.ts | 97 ++++++++++++------- .../src/__tests__/discovery-scan.test.ts | 25 +++++ packages/stim-cli/src/daemon/stages.ts | 4 +- packages/stim-cli/src/guide/cleanup.ts | 11 +-- .../stim-cli/src/stimd/passes/artifacts.ts | 4 +- .../stim-cli/src/stimd/passes/cargo-target.ts | 4 +- .../stim-cli/src/stimd/passes/level-run.ts | 5 +- .../stim-cli/src/workspace/discovery-scan.ts | 6 +- packages/stim-cli/src/workspace/discovery.ts | 44 +-------- website/docs/worktrees.md | 2 +- 13 files changed, 149 insertions(+), 113 deletions(-) diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 72fa033671..77dee7a409 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -84,19 +84,18 @@ hold that keeps a worktree inside a dot-folder of its main checkout an error. The result is stored in `maintenance/discovery-scan.json` under the Stim home (when it ran, what it found, what it could not read) and discovery reads that file, so a maintenance pass or `stim gc` never walks the disk. -- The reduction passes (the artifact and dependency collection in - `artifactCheckoutsAsync`, the idle and dormant stages in `idleWorkspaces`, and - the Cargo target candidates in `cargoTargetCandidates`) skip a reduce-only - checkout left for a later pass, so each evaluates at most 40 at a time, in a - window over them that moves every 15 minutes. Discovery still lists every one, - and `stim gc --dry-run` marks those left for a later pass. The window does not - bound the finished-worktree check (`collectWorktreeSweep`), in a background - pass or in the pressure ladder: it classifies and inspects every discovered - worktree with git (status, activity, HEAD, unpushed), measured at about 0.14 - seconds each on one MacBook, whether or not it is left for a later pass. Skipping - the deferred ones there was tried and reverted, because worktrees are grouped by - path prefix and a group is held when any of its roots is reduce-only, so - dropping a root could release a held worktree. +- Every reduce-only checkout is evaluated on every pass, like a sibling-discovered + checkout, and classified from the complete discovery list. A window of 40 per + pass was built and removed: filtering by it before classification changed + what was protected, twice (a worktree grouped with a nested discover root lost + its reduce-only status). Bounding per-pass cost for many checkouts belongs to + #5531; the finished-worktree check (`collectWorktreeSweep`) inspects every + discovered worktree with git (status, activity, HEAD, unpushed), measured at about + 0.14 seconds each on one MacBook. +- An existing scan file that cannot be read (any error other than it not existing) + stops the scan for that tick and writes nothing, as an unreadable config does; + one that reads but does not parse, or is another version, is treated as no + result, since its contents are unusable either way. - The stored result keeps the schedule (last attempt, last attempt that did not fail, last attempt at low disk) and the finds. A failed or truncated scan keeps the previous finds, and a finished one keeps those under a folder it @@ -281,8 +280,7 @@ decision leaves open: 7. "Idle for a long time" has no number yet. Chosen by the implementing lane, not by the maintainer or the coordinator: the -bound of 40 reduce-only checkouts per discovery in a window that moves every 15 -minutes (not "least recently handled first"), the depth of seven, the folders +depth of seven, the folders pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and `~/Applications` left out, the low-disk scan running hourly while the daemon's disk level is above normal (high or critical, `disk.minFreeGb`) and no scan ran for an hour, the diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts index bdc19f061d..b251234286 100644 --- a/packages/core/state/discovery-scan.ts +++ b/packages/core/state/discovery-scan.ts @@ -1,4 +1,4 @@ -import { isJsonObject, readJsonObject } from './json-file.ts'; +import { isJsonObject, readStateFile } from './json-file.ts'; import { discoveryScanFile } from './paths.ts'; /** How an attempt ended: finished, cut short by the folder cap or the time budget, or by a read that timed out or a failure. */ @@ -79,6 +79,33 @@ export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { }; } +/** + * The scan file as the scheduler needs it: `absent` (no previous result), `unreadable` (it exists and could not be + * read, so nothing is known and no scan may run on top of it), or `read` with the result, which is null when the + * contents do not parse or are another version, since they are unusable either way. + */ +export type DiscoveryScanFile = + | { status: 'absent' } + | { status: 'unreadable' } + | { status: 'read'; scan: DiscoveryScan | null }; + +export function readDiscoveryScanFile(): DiscoveryScanFile { + let text: string; + try { + text = readStateFile(discoveryScanFile()); + } catch (error) { + return error instanceof Error && 'code' in error && error.code === 'ENOENT' + ? { status: 'absent' } + : { status: 'unreadable' }; + } + try { + return { status: 'read', scan: parseDiscoveryScan(JSON.parse(text)) }; + } catch { + return { status: 'read', scan: null }; + } +} + export function readDiscoveryScan(): DiscoveryScan | null { - return parseDiscoveryScan(readJsonObject(discoveryScanFile())); + const file = readDiscoveryScanFile(); + return file.status === 'read' ? file.scan : null; } diff --git a/packages/stim-cli/api.api.txt b/packages/stim-cli/api.api.txt index 03f5d1fd7d..10df1fb611 100644 --- a/packages/stim-cli/api.api.txt +++ b/packages/stim-cli/api.api.txt @@ -357,7 +357,6 @@ [*] interface DevicePlacement { @@1 machine?: string; [*] interface DevicePlacement { @@1 reason: string; [*] interface DiscoveredCheckout { -[*] interface DiscoveredCheckout { @@1 deferred?: boolean; [*] interface DiscoveredCheckout { @@1 from: string; [*] interface DiscoveredCheckout { @@1 origin: DiscoveryOrigin; [*] interface DiscoveredCheckout { @@1 path: string; diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts index 11da17e69c..804f11423f 100644 --- a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -33,12 +33,7 @@ import { import { stoppedOutputCandidates } from '../daemon/stages.ts'; import { saveConfig } from '../workspace/config.ts'; import { collectWorktreeSweep, removeWorktrees } from '../commands/gc/worktrees.ts'; -import { - windowReduceOnly, - discoverCheckouts, - discoveredOutside, - type DiscoveredCheckout, -} from '../workspace/discovery.ts'; +import { discoverCheckouts, discoveredOutside } from '../workspace/discovery.ts'; vi.mock('../workspace/protected-locations.ts', async (original) => { const real = await original(); @@ -567,6 +562,68 @@ describe('checkouts found through a tool folder or the full scan', () => { }); }); + it('treats a tool-folder worktree as reduce-only for the artifact and Cargo passes too when a discover root inside it adds a child root', async () => { + mkdirSync(join(main, 'packages', 'app'), { recursive: true }); + mkdirSync(join(main, 'ios', 'App.xcodeproj'), { recursive: true }); + writeFileSync(join(main, 'packages', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(main, 'ios', 'App.xcodeproj', 'project.pbxproj'), 'x'); + writeFileSync(join(main, '.gitignore'), 'target/\nnode_modules/\n'); + git(main, 'add', '-A'); + git(main, 'commit', '-q', '-m', 'app'); + git(main, 'push', '-q', 'origin', 'main'); + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'aa11', 'main'), 'codex'); + mkdirSync(join(codex, 'node_modules', '.bin'), { recursive: true }); + writeFileSync(join(codex, 'node_modules', '.bin', 'tool'), '#!/bin/sh\n'); + mkdirSync(join(codex, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(codex, 'target', 'debug', 'incremental'), { recursive: true }); + mkdirSync(join(codex, 'target', 'release'), { recursive: true }); + writeFileSync(join(codex, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(codex, 'target', 'debug', 'incremental', 'data'), 'x'.repeat(10_000)); + writeFileSync(join(codex, 'target', 'release', 'tool'), '#!/bin/sh\n'); + age(codex); + saveConfig({ + version: 2, + projects: { [main]: {} }, + repos: {}, + maintenance: { discoverRoots: [join(codex, 'packages')] }, + }); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + const { signal } = new AbortController(); + const checkouts = await artifactCheckoutsAsync(signal, testContext({ env: {} })); + expect(checkouts.find((entry) => entry.path === codex)?.reduceOnly).toBe(true); + + const options = { now: Date.now(), windowMs: 0, protectedRoot: '', pinned: () => false }; + const dependencies = (await collectArtifactsAsync('dependencies', options, signal, testContext({ env: {} }))) + .candidates; + expect(dependencies.map((entry) => entry.path)).toContain(join(codex, 'node_modules')); + const plan = () => + planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + expect((await plan()).actions.map((action) => action.target)).toContain(join(codex, 'target')); + + mkdirSync(join(base, 'home', 'bin')); + symlinkSync(join(codex, 'target', 'release', 'tool'), join(base, 'home', 'bin', 'tool')); + const kept = await collectArtifactsAsync('dependencies', options, signal, testContext({ env: {} })); + expect(kept.candidates.map((entry) => entry.path)).not.toContain(join(codex, 'node_modules')); + expect(kept.skips).toEqual( + expect.arrayContaining([ + expect.objectContaining({ target: join(codex, 'node_modules'), reason: expect.stringContaining('on PATH') }), + ]), + ); + const after = await plan(); + expect(after.actions.map((action) => action.target)).not.toContain(join(codex, 'target')); + expect(after.skips).toEqual( + expect.arrayContaining([ + expect.objectContaining({ target: join(codex, 'target'), reason: expect.stringContaining('on PATH') }), + ]), + ); + }); + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); @@ -586,31 +643,3 @@ describe('checkouts found through a tool folder or the full scan', () => { expect(existsSync(codex)).toBe(true); }); }); - -describe('windowReduceOnly', () => { - const entry = (n: number, reduceOnly = true): DiscoveredCheckout => ({ - path: `/p/${String(n).padStart(3, '0')}`, - origin: 'scan', - from: 'full scan', - ...(reduceOnly ? { reduceOnly: true } : {}), - }); - const window = 15 * 60_000; - const handled = (found: DiscoveredCheckout[]) => found.filter((e) => !e.deferred).map((e) => e.path); - - it('lists every checkout, defers all but a moving window of the reduce-only ones, and takes each in turn', () => { - const all = [entry(1, false), ...Array.from({ length: 10 }, (_, i) => entry(i + 10))]; - const now = 100 * window; - const first = windowReduceOnly(all, now, 4); - expect(first.map((e) => e.path)).toEqual(all.map((e) => e.path)); - expect(first.filter((e) => e.reduceOnly && !e.deferred)).toHaveLength(4); - expect(first[0]?.deferred).toBeUndefined(); - const seen = new Set(); - for (let i = 0; i < 10; i++) for (const path of handled(windowReduceOnly(all, now + i * window, 4))) seen.add(path); - expect(seen.size).toBe(all.length); - }); - - it('defers nothing when the reduce-only ones fit the limit', () => { - const all = [entry(1), entry(2)]; - expect(windowReduceOnly(all, 0, 4)).toEqual(all); - }); -}); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts index c4c657158f..700a154e0c 100644 --- a/packages/stim-cli/src/__tests__/discovery-scan.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -574,6 +574,31 @@ describe('runDiscoveryScanIfDue', () => { expect(readDiscoveryScan()?.found).toEqual([join(base, 'home', 'dev', 'app')]); }); + test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( + 'starts no scan and writes nothing when an existing scan file cannot be read', + async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const before = readFileSync(discoveryScanFile(), 'utf8'); + chmodSync(discoveryScanFile(), 0o000); + try { + let started = 0; + await runDiscoveryScanIfDue(tick(), { + ...options(t0 + 8 * 24 * hour), + scanOptions: { + entries: async () => { + started++; + return []; + }, + }, + }); + expect(started).toBe(0); + } finally { + chmodSync(discoveryScanFile(), 0o600); + } + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + }, + ); + describe('a read that fails for a reason other than a timeout or absence', () => { const denied = () => Object.assign(new Error('EACCES'), { code: 'EACCES' }); const later = () => options(t0 + 8 * 24 * hour); diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index e06dbc9f3c..ee90658c7f 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -243,9 +243,7 @@ async function idleWorkspaces( ): Promise { const readings: (Omit & { idleMs: number | null })[] = []; const reads = discoveredReads(reading.operation?.signal, reading.operation); - const found = discover - ? (await discoverCheckouts(config, listedRoots(), reading.operation)).filter(({ deferred }) => !deferred) - : []; + const found = discover ? await discoverCheckouts(config, listedRoots(), reading.operation) : []; const discovered = found.map(({ path }): [string, ProjectRecord] => [path, {}]); const reduceOnly = new Set(found.filter((entry) => entry.reduceOnly).map(({ path }) => path)); for (const [path, project] of [...Object.entries(config.projects), ...discovered]) { diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 56edd1c3c7..595a65299f 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -1277,13 +1277,10 @@ each have one; the scan runs on the passes thread) no other launcher or scan read starts and a scheduled scan is skipped. A scan stops at the first read that times out, keeping the previous finds. Reduce-only checkouts have their build outputs, and their node_modules and Pods (a linked worktree's after idle.dormantDays, a source -checkout's after idle.sourceDormantDays), reduced by the rules below. The -reduction passes (artifact and dependency collection, the idle and dormant -stages, and the Cargo target measurement) evaluate at most 40 of them at a time, -in a window that moves every 15 minutes; stim gc --dry-run lists all of them and -says which are left for a later pass. The finished-worktree check is not bounded -by that window: in a background pass and in the pressure ladder it still inspects -every discovered worktree with git (about 0.14 seconds each where measured). A +checkout's after idle.sourceDormantDays), reduced by the rules below. Every one is +evaluated on every pass, like a sibling-discovered checkout; the +finished-worktree check inspects every discovered worktree with git (about 0.14 +seconds each where measured). A folder named in maintenance.discoverRoots keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine diff --git a/packages/stim-cli/src/stimd/passes/artifacts.ts b/packages/stim-cli/src/stimd/passes/artifacts.ts index 46b0b362df..4c63ccdaf1 100644 --- a/packages/stim-cli/src/stimd/passes/artifacts.ts +++ b/packages/stim-cli/src/stimd/passes/artifacts.ts @@ -145,9 +145,7 @@ export async function artifactCheckoutsAsync( operation?: StimContext, ): Promise { const found = new Map(); - const discovered = (await discoverCheckouts(loadConfig(), listedRoots(), operation)).filter( - (entry) => !entry.deferred, - ); + const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); const known = new Set([...listedRoots(), ...discovered.map(({ path }) => path)]); const reduceOnly = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); for (const root of (await candidateRootsAsync(operation, discovered)).filter((entry) => known.has(entry))) { diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index 3dd7158fc8..f118bda1ac 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -120,9 +120,7 @@ async function cargoTargetCandidates( }; for (const root of registered) await add(root, false, false, root); const discovered = new Set( - (await discoverCheckouts(loadConfig(), listedRoots(), operation)) - .filter(({ deferred }) => !deferred) - .map(({ path }) => canonicalPath(path)), + (await discoverCheckouts(loadConfig(), listedRoots(), operation)).map(({ path }) => canonicalPath(path)), ); for (const checkout of await artifactCheckoutsAsync(signal, operation)) for (const base of appRoots(checkout, 'build')) diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index a7c45f4ab8..7dad378d38 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -249,9 +249,8 @@ export async function runMaintenanceAtLevel({ for (const checkout of found) { const mine = actions.filter((action) => ownerOf(action.target) === checkout.path); const taken = mine.some((action) => action.outcome === 'would' || action.outcome === 'done'); - const kept = checkout.deferred - ? 'left for a later pass: at most 40 checkouts found through a tool folder or the full scan are handled at a time' - : taken || idleMs <= 0 || !existsSync(checkout.path) + const kept = + taken || idleMs <= 0 || !existsSync(checkout.path) ? null : await discoveredKeepReason(checkout.path, idleMs, checkout.reduceOnly === true, operation, reads); if (kept && /source entries to check|cannot be read/.test(kept)) diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts index 2e4e8da531..9e51acd326 100644 --- a/packages/stim-cli/src/workspace/discovery-scan.ts +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -7,7 +7,7 @@ import { type ScanTruncation, discoveryScanFile, maintenanceDir, - readDiscoveryScan, + readDiscoveryScanFile, type DaemonMeasurement, type DiscoveryScan, } from '@stim-cli/core/state'; @@ -255,7 +255,9 @@ export async function runDiscoveryScanIfDue( try { const thresholds = diskThresholds(loadConfig(), operation.env).thresholds; low = diskPressure(tick.measured.measurement.disk, thresholds).level !== 'normal'; - last = readDiscoveryScan(); + const file = readDiscoveryScanFile(); + if (file.status === 'unreadable') return; + last = file.status === 'read' ? file.scan : null; } catch { return; } diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index 2b17887322..82cee508d0 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -21,10 +21,6 @@ type DiscoveryOrigin = 'worktree' | 'sibling' | 'root' | 'scan'; */ const TOOL_FOLDERS = ['.codex/worktrees']; -/** At most this many checkouts found only through a tool folder or the full scan are handled per discovery. */ -const MAX_REDUCE_ONLY = 40; -const REDUCE_ONLY_WINDOW_MS = 15 * 60_000; - export interface DiscoveredCheckout { /** The canonical path of the checkout. */ path: string; @@ -33,11 +29,6 @@ export interface DiscoveredCheckout { from: string; /** Found through a tool folder or the full scan: Stim reduces its outputs and never removes it. */ reduceOnly?: boolean; - /** - * A reduce-only checkout outside the window of at most `MAX_REDUCE_ONLY` that this discovery hands to the passes that - * evaluate each checkout; a later window takes it. Reports still list it. - */ - deferred?: boolean; } export interface DiscoveryChild { @@ -180,23 +171,6 @@ export function inToolFolder(path: string, home: string): boolean { return TOOL_FOLDERS.some((folder) => inside(path, join(home, folder))); } -/** - * Marks all but `limit` of the checkouts found only through a tool folder or the full scan as `deferred`: a window over - * them that moves with `now`, so every one is handled within a few windows and one pass evaluates a bounded number. - * Every checkout stays in the list, so no report or classification loses one. - */ -export function windowReduceOnly( - found: readonly DiscoveredCheckout[], - now: number, - limit: number = MAX_REDUCE_ONLY, -): DiscoveredCheckout[] { - const only = found.filter((entry) => entry.reduceOnly); - if (only.length <= limit) return [...found]; - const start = (Math.floor(now / REDUCE_ONLY_WINDOW_MS) * limit) % only.length; - const kept = new Set(Array.from({ length: limit }, (_, i) => only[(start + i) % only.length])); - return found.map((entry) => (entry.reduceOnly && !kept.has(entry) ? { ...entry, deferred: true } : entry)); -} - /** * Whether Stim acts on a discovered linked worktree: it lies in its repository's own main checkout folder, in a folder * that holds a known project, or in a `maintenance.discoverRoots` folder, and not in a protected location. A path inside @@ -232,9 +206,8 @@ export async function discoverCheckouts( config: Config | null, known: readonly string[], operation?: StimContext, - now: number = Date.now(), ): Promise { - return (await discovery(config, known, operation, now)).found; + return (await discovery(config, known, operation)).found; } /** What discovery reached outside the folders Stim manages. */ @@ -248,15 +221,10 @@ export async function discoveredOutside( const memo = new WeakMap>(); -function discovery( - config: Config | null, - known: readonly string[], - operation?: StimContext, - now: number = Date.now(), -): Promise { - if (!operation) return computeDiscovered(config, known, operation, now); +function discovery(config: Config | null, known: readonly string[], operation?: StimContext): Promise { + if (!operation) return computeDiscovered(config, known, operation); let found = memo.get(operation); - if (!found) memo.set(operation, (found = computeDiscovered(config, known, operation, now))); + if (!found) memo.set(operation, (found = computeDiscovered(config, known, operation))); return found; } @@ -276,7 +244,6 @@ async function computeDiscovered( config: Config | null, known: readonly string[], operation?: StimContext, - now: number = Date.now(), ): Promise { const env = operation?.env ?? {}; const scoped = isScopedHome(operation?.env); @@ -346,11 +313,10 @@ async function computeDiscovered( ); const knownRoots = known.map(canonicalPath); const everything = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); - const selected = windowReduceOnly(everything, now); const listed = new Set(everything.map(({ path }) => path)); const coveredByKnown = (path: string) => knownRoots.some((root) => inside(path, root) || inside(root, path)); return { - found: selected, + found: everything, outside: outside.filter(({ path }) => !listed.has(path) && !coveredByKnown(path)), }; } diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 049f38c56b..06e47ce4e4 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -86,7 +86,7 @@ Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. D Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. The cleanup passes that reduce build outputs, dependencies and Cargo targets evaluate at most 40 of them at a time; `stim gc --dry-run` lists all of them and says which are left for a later pass. The check for finished worktrees is not limited that way: it still inspects every discovered worktree with git, which took about 0.14 seconds each where it was measured. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Every one is evaluated on every cleanup pass, like a checkout found next to your registered ones; the check for finished worktrees inspects every discovered worktree with git, which took about 0.14 seconds each where it was measured. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output.