diff --git a/.oxlintrc.json b/.oxlintrc.json index 69f27b3e6..aa931bc94 100644 --- a/.oxlintrc.json +++ b/.oxlintrc.json @@ -522,6 +522,7 @@ "packages/stim-cli/src/web/chrome.ts", "packages/stim-cli/src/workspace/config.ts", "packages/stim-cli/src/workspace/git-read.ts", + "packages/stim-cli/src/workspace/launcher-directories.ts", "packages/stim-cli/src/workspace/settings.ts", "packages/stim-cli/src/workspace/workspace-state.ts", "packages/stim-cli/src/workspace/worktree-refresh.ts" diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index a6f3541f8..77dee7a40 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -16,7 +16,9 @@ registered checkouts, and direct children of the folders in `maintenance.discoverRoots`. A child counts only when Stim's project detection recognises it, itself or through a direct child of its `apps`, `examples` or `packages` folder. Discovery registers nothing, writes nothing, recomputes each -run and never runs `git fetch`. +run and never runs `git fetch`. Two more sources, the common tool folders and the +full scan (see "Tool folders and the full scan"), add checkouts that are only +reduced. Reason: checkouts no `stim` command ran in can hold most of a disk; an audit of one MacBook on 2026-11-11 found 119 of 151 checkouts, 248 GB, unknown to Stim, @@ -39,14 +41,17 @@ checkout folder, a folder that holds a known project, or a `maintenance.discoverRoots` folder. Nothing under the system temporary directory or a dot-folder directly inside the home directory is ever touched, unless it lies inside a `maintenance.discoverRoots` folder: a path inside such a folder -is never a protected location, matched after symlinks resolve. The home +is never a protected location, matched after symlinks resolve. The one other +exception is `~/.codex/worktrees`, whose worktrees are reduce-only (see "Tool +folders and the full scan"). The home directory and `/` are never a folder that holds known projects. A scoped `STIM_HOME` discovers only from its own `maintenance.discoverRoots`. Reason: `git worktree list` also returns worktrees other tools own, such as `~/.codex/worktrees`, pin checkouts under a home dot-folder and agent scratch folders under the temporary directory. Stim cannot tell what those tools still -need. A scratch home must not reach the repositories of the real one. +need, so it removes none of them; it reduces only the outputs of the +`~/.codex/worktrees` ones. A scratch home must not reach the repositories of the real one. Maintainer decision: "Def need to fix it if its the case, paths added to discover root need to always be considered", 2026-11-14 (#5757), about a folder @@ -57,6 +62,235 @@ the maintainer's words. Everything else in this section was chosen by the coordinator and the review of #5519, not by the maintainer. +## Tool folders and the full scan + +Discovery also reads two more sources, and what they find is reduce-only: Stim +reduces its build outputs and its `node_modules` and Pods (a linked worktree's +after `idle.dormantDays`, a source checkout's after `idle.sourceDormantDays`) by +the discovered-checkout rules, and never removes the checkout or worktree. The +hold that keeps a worktree inside a dot-folder of its main checkout +(`.claude/worktrees`) applies to it. + +- Common tool folders: linked worktrees of known repositories in + `~/.codex/worktrees`. The list is one constant with that one path. +- The full scan: a pruned walk of the home folder for checkouts that Stim's + project detection recognises and discovery does not reach. The daemon runs it + weekly and, while the disk level is above normal, when no scan has run for an + hour; a command never runs it. It reads `~/Documents`, skips `~/Desktop`, `~/Downloads`, + `~/Library`, other media folders, symlinks, other volumes, dependency, build + and cache folders, dot-folders below the first level and the named tool homes + among the home folder's dot-folders (see below), and the Stim home, stops at a recognised project and goes + seven levels deep. A folder the system refuses is skipped and recorded, never + an error. The result is stored in `maintenance/discovery-scan.json` under the + Stim home (when it ran, what it found, what it could not read) and discovery + reads that file, so a maintenance pass or `stim gc` never walks the disk. +- Every reduce-only checkout is evaluated on every pass, like a sibling-discovered + checkout, and classified from the complete discovery list. A window of 40 per + pass was built and removed: filtering by it before classification changed + what was protected, twice (a worktree grouped with a nested discover root lost + its reduce-only status). Bounding per-pass cost for many checkouts belongs to + #5531; the finished-worktree check (`collectWorktreeSweep`) inspects every + discovered worktree with git (status, activity, HEAD, unpushed), measured at about + 0.14 seconds each on one MacBook. +- An existing scan file that cannot be read (any error other than it not existing) + stops the scan for that tick and writes nothing, as an unreadable config does; + one that reads but does not parse, or is another version, is treated as no + result, since its contents are unusable either way. +- The stored result keeps the schedule (last attempt, last attempt that did not + fail, last attempt at low disk) and the finds. A failed or truncated scan + keeps the previous finds, and a finished one keeps those under a folder it + could not read, so a worse result never replaces a better one. A read the system + denies during the project checks (a manifest, a nested listing) is not a + structural fact: it lists the folder unreadable, runs no detection on it or + below it, leaves the scan untruncated and keeps its finds; in revalidation it is + `pending`. Discovery + revalidates each find before using it: a real folder, not a symlink, on the + home volume, below no skipped folder, still recognised. +- The scan runs after a maintenance pass has acted, so it never delays cleanup. + +A folder named in `maintenance.discoverRoots` keeps the full rules. A scoped +`STIM_HOME` discovers nothing from either source. + +Reason for entering home dot-folders: checkouts live there too, for example +about 18 checkouts and 80 GB of Cargo `target/` under `~/.exact/exact2` (#5757), +so a blanket skip of dot-folders would never even report the layout that started +this work. The scan records them; it does not act on them (see below). +The scan enters a dot-folder directly in the home folder unless it is in a +named prune list of tool homes, package and build caches and runtimes +(`.git`, `.npm`, `.yarn`, `.bun`, `.gradle`, `.cargo`, `.rustup`, `.nvm`, +`.rvm`, `.cache`, `.android`, `.vscode`, `.cursor`, `.agent-device`, `.claude`, +`.codex` and others in `HOME_DOT_PRUNED`, and by prefix every `.stim*` folder: the Stim +home and other Stim state, where the build worker's client checkouts must never become +reduce-only) or named like a cache (`*-cache`). +The list comes from entry counts to depth four on one machine, led by `.gradle` +(67 thousand), `.agent-device` (45 thousand), the two Metro cache folders +(25 and 17 thousand), `.npm`, `.bun`, `.rvm` and `.yarn` (15 thousand each). +`.codex` is reached through its worktrees folder, and the Stim home is never +scanned. The version managers and package caches `.volta`, `.asdf`, `.mise`, +`.fnm`, `.pyenv`, `.rbenv`, `.sdkman`, `.deno`, `.pub-cache` and `.fvm` are +pruned too: their installs hold thousands of files and no checkouts, and one +large one could truncate the walk on every run. Dot-folders deeper down +(`.worktrees`, `.claude/worktrees`) are skipped because git reaches their +worktrees. + +The coordinator's decision following the review of #5764, not the maintainer's +words: what the scan finds under a home dot-folder is recorded in the result +(`dotFolderFound`) and never selected by discovery, so no pass acts on it. The +protection rule for home dot-folders applies to scan finds in full; the only +ways into a home dot-folder stay `maintenance.discoverRoots` and the tool folder +list. Reason: Stim cannot tell an installed tool from a checkout there, for +example a Swift CLI cloned under `~/.mytool` whose launcher on PATH points into +its ignored `.build/release`; clearing that output once idle would break the +installed command. The record lets doctor report what the folder holds and the +command that adds it to `maintenance.discoverRoots`; that listing is not part of +this change. + +The scan has a 45 second budget (and a 200 thousand folder cap, and an abort), +checked before every read it makes, including every read of the safety checks +below: when it runs out the folder is not recognised and the scan ends truncated +as `budget`, keeping the previous finds. It can overrun the budget by at most one +read's 3 second timeout plus the time detection itself takes on the folder it is +recognising, which is synchronous and not interruptible. Revalidation has no total +budget; each read has the 3 second timeout, and the number of reads is bounded by +detection's own caps (50 children per container, the home folder and up to seven below it). + +Project detection (`recognises`) reads files synchronously, and a stalled mount, +an iCloud file waiting for a download or a FIFO would block the thread. Before +detection runs on a folder, the scan or the revalidation proves its reads cannot +block, as far as this list goes. For the folder, each ancestor from the home +folder down (`folderSafety`), and each folder detection inspects below it +(`nestedSafety`: each `apps`, `examples` and `packages` container and up to 50 +children of each), with every name compared case-insensitively on every volume: +a detection file (`package.json`, `settings.gradle(.kts)`, `gradlew(.bat)`, +`Package.swift`, `exact.mjs`, `app.json`) that is not a regular file makes the +folder unsafe; so does an `ios` or `android` entry that is not a real directory +(detection resolves them with `realpath` as React Native's owned roots), an Xcode +container with no app dependency in `package.json` (detection parses the project +files it names), and a symlinked container; every regular `package.json` is read +once through the bounded reader first. An unsafe folder, or an unsafe result for +anything detection inspects below it, makes its whole subtree unrecognisable: no +detection runs on it or below it, and the walk does not descend. A read that does +not answer ends the scan truncated with the folder unreadable (`truncation: read`), +or leaves a cached find unselected and in the file. Revalidation applies the same +functions to the chain from the home folder to the entry, memoised per discovery +call, and reads the entry's own metadata (`lstat`, `stat`, `realpath`) through the +bounded reader. Not bounded, and stated as such: the ancestor reads above the home +folder (`/Users` and `/`); a file that becomes unreadable between the bounded read +and detection; the `ios` and `android` directories of the folder and of each +ancestor, which detection resolves with `realpath` without the walk entering +them (the gate checks that each is a real directory, not that it answers, so a +stalled mount point there is not detected; the walk's own bounded `readdir` covers +only the folders it enters, and the `apps`, `examples` and `packages` containers +and their children are listed through the bounded reader by the gate); and everything a pass later reads synchronously inside a checkout +once it is discovered (artifact collection, app roots, state), which is the same +for registered and sibling-discovered checkouts today. Registered projects and +sibling discovery recognise as before. An Xcode container without an app +dependency is not a scan checkout, though sibling and root discovery still +recognise it. + +A scan find outside a home dot-folder can also be an installed tool. Every route +that deletes outputs, dependencies or Cargo data of a reduce-only checkout keeps +the checkout while an executable reachable through a PATH directory is a symlink +that resolves into it: the idle stage (`discoveredEvidence`, and again under the +locks in `clearStoppedOutputs`), the artifact and dependency passes +(`artifactReads` into `keptReason`, which `collectArtifactsAsync`, +`clearArtifact` and the source dependency trees all pass through), and the Cargo +target measurement, the pressure plan and the acting trim +(`regenerableDirProblem` with the launcher reader). The directories read are +those on PATH and the per-user bin folders a daemon started by launchd does not +list: `~/.local/bin`, `~/bin`, `~/.cargo/bin`, `~/.bun/bin`, `~/.volta/bin`, +`~/Library/pnpm`, the active nvm bin, `/usr/local/bin` and `/opt/homebrew/bin`. +A launcher resolving into a path is the same kind of fact as a process executing +from it, so a deletion route reads it at the same moment as the running-process +evidence, with a reading of its own, never the pass's planning reading (which +can be minutes old and is shared by the operation). The Cargo trim adds the +launcher targets under its target to the paths in use at each of its three +process reads. The artifact and dependency routes read the launchers in +`artifactReads`, after the process read and after their slow checks, in the same +`keptReason` that decides the deletion. `clearStoppedOutputs` reads them with +the idle evidence and again after measuring sizes, immediately before each +removal (`deleteOutputs`). The plan-time whole-checkout keep stays. The window +that remains is the one for a process that starts executing from the path after +the last evidence read: Stim accepts it for processes and gives launchers no +larger one. A Cargo target is checked against its owning checkout, not its app +folder. Each call to a directory has 3 seconds to answer, and a timeout ends the +reading as incomplete at once. A native read cannot be cancelled, so at most one +read that timed out is outstanding per JavaScript isolate, shared by the launcher +readings and the home scan (`bounded-read.ts`): while it is pending, no further +launcher or scan read starts, every launcher reading reports incomplete +immediately naming the folder, and a scheduled scan is skipped. A hung mount +therefore holds one libuv thread per isolate, and reduction of reduce-only +checkouts stays off until that read settles. Two isolates in the daemon can +read launchers, the main thread and the passes thread, so up to two reads can be +outstanding there; the scan runs only on the passes thread, and CLI commands are +separate processes. A scan stops at the first read that times out: the result is +truncated and the previous finds are kept. A scan that failed or ended on a timed-out +read is retried after an hour whatever the disk level; one that stopped at the folder +cap or the time budget follows the weekly schedule (hourly while the disk level is above +normal), since a retry would hit the same limit. +A directory or entry that does not exist, and a dangling symlink, is no +launcher. Any other failure to inspect a directory (permission, I/O, a timeout) +keeps every reduce-only checkout for that operation, with a reason naming the +directory. A wrapper script that only names a path is not followed. + +Every source (main) checkout, registered or discovered by any route (siblings, +`maintenance.discoverRoots` children, tool folders, the scan), follows the +existing dormant rule and loses `node_modules` and Pods after +`idle.sourceDormantDays` (7), a linked worktree's after `idle.dormantDays` (1) +(`daemon/dependencies.ts` `takeIdleDependencies`, over `idleStageCandidates`' +discovered list); the maintainer has not confirmed that consequence for the scan +finds, and the guide wrongly said a main checkout keeps them before this change. + +Reason: on one MacBook `~/.codex/worktrees` held 53 worktrees and 12 GB that +Stim listed once as outside the folders it manages, with no size and no way +forward. + +Maintainer decision, 2026-11-14 (#5762), verbatim: + +> ok so lets do: +> discovery - current logic + common paths + full scan once in a while (also triggered more often on low disk), full scan can try to be fast, ie like not scan places we know theres gonna be no projects and lots of files so can keep it fast ish, prob will trigger permission promts for Documents folder prob the only one it makes sense to scan but I guess its fine. +> +> discovered workspaces - cleaned using our safer rules we have already, doctor lists them and suggest adding as managed path or delete if idle for long time + +Earlier the same evening, on including common tool folders by default: "Could +these common paths just be included by default in Discovered checkout". + +The coordinator's reading, not the maintainer's words, for the points the +decision leaves open: + +1. Checkouts found only through a common path or the full scan are reduced and + never removed by Stim: build outputs, tagged caches, and a linked worktree's + `node_modules` and Pods go by the discovered-checkout rules, and the checkout + or worktree itself stays. This is the rule that already applies to a worktree + in a dot-folder of its main checkout (`.claude/worktrees`). Removal is what + doctor suggests to the person. A folder named in `maintenance.discoverRoots` + keeps the full discovered rules, removal of finished worktrees included. + (Added by the implementing lane, not part of the recorded reading: the rule + in force also removes a source checkout's `node_modules` and Pods after + `idle.sourceDormantDays`, as the section that states it describes.) +2. Common paths start with `~/.codex/worktrees`. Other tools' folders are added + only with evidence that the tool uses that path. +3. "Once in a while" is weekly. On low disk the scan runs when a volume's level + rises above normal, at most once an hour. +4. Doctor reads the last scan's result and does not run the scan, so it stays + fast. +5. The line Stim commands print on low disk also names what the scan found, its + size and the command that adds it. +6. `~/Desktop` and `~/Downloads` are not scanned. Other volumes are not scanned. +7. "Idle for a long time" has no number yet. + +Chosen by the implementing lane, not by the maintainer or the coordinator: the +depth of seven, the folders +pruned, `~/Library`, `~/Movies`, `~/Music`, `~/Pictures`, `~/Public` and +`~/Applications` left out, the low-disk scan running hourly while the daemon's disk level is above +normal (high or critical, `disk.minFreeGb`) and no scan ran for an hour, the +retry after an hour of a scan that failed or ended on a read that timed out, while one that stopped at the folder cap or the time budget counts as finished (`truncation` records which), the scan running only in the daemon +after the pass, and the PATH launcher keep. The scan is not deferred under host +load; it takes about a second and a half on one MacBook. + +Not part of this change: points 4, 5 and 7, which belong to the doctor listing +and the low-disk line. + ## Removing a discovered worktree needs more evidence A discovered linked worktree follows the registered-workspace rules for build @@ -91,8 +325,9 @@ Chosen by the coordinator and the reviews of #5519, not by the maintainer. ## Links -- Issues: #5512, #5515, #5757. Pull request: #5519. +- Issues: #5512, #5515, #5757, #5762. Pull request: #5519. - Code: `packages/stim-cli/src/workspace/discovery.ts`, + `packages/stim-cli/src/workspace/discovery-scan.ts`, `packages/stim-cli/src/workspace/discovered-removal.ts`, `packages/stim-cli/src/workspace/protected-locations.ts`, `packages/stim-cli/src/workspace/hidden-edits.ts`, diff --git a/packages/core/state/discovery-scan.ts b/packages/core/state/discovery-scan.ts new file mode 100644 index 000000000..b25123428 --- /dev/null +++ b/packages/core/state/discovery-scan.ts @@ -0,0 +1,111 @@ +import { isJsonObject, readStateFile } from './json-file.ts'; +import { discoveryScanFile } from './paths.ts'; + +/** How an attempt ended: finished, cut short by the folder cap or the time budget, or by a read that timed out or a failure. */ +export type ScanTruncation = 'none' | 'budget' | 'folders' | 'read' | 'failed'; + +const isTruncation = (value: unknown): value is ScanTruncation => + value === 'none' || value === 'budget' || value === 'folders' || value === 'read' || value === 'failed'; + +/** + * What the last full scan of the home folder found: the checkouts Stim's project detection recognises that + * `maintenance.discoverRoots`, registered projects and the common tool folders do not reach, and what it could not read. + */ +export interface DiscoveryScan { + version: 2; + /** When the last attempt started, epoch ms. A failed or truncated attempt is stamped too. */ + ranAt: number; + /** When the last attempt that finished, neither failed nor truncated, started; 0 when none did. The weekly schedule counts from it. */ + completedAt: number; + /** When the last attempt that ran while a volume's level was above normal started; 0 when none did. */ + lowScanAt: number; + durationMs: number; + trigger: 'weekly' | 'low-disk'; + /** Folders the scan read. */ + visited: number; + /** + * Canonical paths of the recognised checkouts. A failed or truncated scan keeps the previous scan's finds that it did + * not reach, and a scan keeps those under a folder it could not read; discovery revalidates each before using it. + */ + found: string[]; + /** + * Canonical paths of recognised checkouts under a dot-folder of the home folder, recorded for the doctor listing and + * never used by discovery: Stim cannot tell an installed tool from a checkout there. + */ + dotFolderFound: string[]; + /** Folders the system refused or that did not answer, skipped without error. */ + unreadable: string[]; + /** The scan stopped at its time or folder limit, or on a read or failure, before it finished. */ + truncated: boolean; + /** How the attempt ended. `read` and `failed` are retried after an hour; the others follow the weekly schedule. */ + truncation: ScanTruncation; +} + +const strings = (value: unknown): value is string[] => + Array.isArray(value) && value.every((entry) => typeof entry === 'string'); +const finite = (value: unknown): value is number => typeof value === 'number' && Number.isFinite(value); + +/** Rejects missing, corrupt or unsupported payloads without throwing. */ +export function parseDiscoveryScan(value: unknown): DiscoveryScan | null { + if ( + !isJsonObject(value) || + value.version !== 2 || + !finite(value.ranAt) || + !finite(value.completedAt) || + !finite(value.lowScanAt) || + !finite(value.durationMs) || + (value.trigger !== 'weekly' && value.trigger !== 'low-disk') || + !finite(value.visited) || + !strings(value.found) || + !strings(value.dotFolderFound) || + !strings(value.unreadable) || + typeof value.truncated !== 'boolean' || + !isTruncation(value.truncation) + ) + return null; + return { + version: 2, + ranAt: value.ranAt, + completedAt: value.completedAt, + lowScanAt: value.lowScanAt, + durationMs: value.durationMs, + trigger: value.trigger, + visited: value.visited, + found: value.found, + dotFolderFound: value.dotFolderFound, + unreadable: value.unreadable, + truncated: value.truncated, + truncation: value.truncation, + }; +} + +/** + * The scan file as the scheduler needs it: `absent` (no previous result), `unreadable` (it exists and could not be + * read, so nothing is known and no scan may run on top of it), or `read` with the result, which is null when the + * contents do not parse or are another version, since they are unusable either way. + */ +export type DiscoveryScanFile = + | { status: 'absent' } + | { status: 'unreadable' } + | { status: 'read'; scan: DiscoveryScan | null }; + +export function readDiscoveryScanFile(): DiscoveryScanFile { + let text: string; + try { + text = readStateFile(discoveryScanFile()); + } catch (error) { + return error instanceof Error && 'code' in error && error.code === 'ENOENT' + ? { status: 'absent' } + : { status: 'unreadable' }; + } + try { + return { status: 'read', scan: parseDiscoveryScan(JSON.parse(text)) }; + } catch { + return { status: 'read', scan: null }; + } +} + +export function readDiscoveryScan(): DiscoveryScan | null { + const file = readDiscoveryScanFile(); + return file.status === 'read' ? file.scan : null; +} diff --git a/packages/core/state/index.ts b/packages/core/state/index.ts index 3a26000e9..a710b757f 100644 --- a/packages/core/state/index.ts +++ b/packages/core/state/index.ts @@ -52,6 +52,7 @@ export * from './tailnet.ts'; export * from './server-service.ts'; export * from './hosted-logs.ts'; export * from './maintenance.ts'; +export * from './discovery-scan.ts'; export * from './gc-log.ts'; export * from './build-ledger.ts'; export * from './agent-device.ts'; diff --git a/packages/core/state/paths.ts b/packages/core/state/paths.ts index 11e3460c9..1e2a4996a 100644 --- a/packages/core/state/paths.ts +++ b/packages/core/state/paths.ts @@ -246,6 +246,10 @@ export function maintenanceStateFile(): string { return join(maintenanceDir(), 'state.json'); } +export function discoveryScanFile(): string { + return join(maintenanceDir(), 'discovery-scan.json'); +} + export function gcLogDir(): string { return join(configDir(), 'gc'); } diff --git a/packages/stim-cli/api.api.txt b/packages/stim-cli/api.api.txt index 7f66107a7..10df1fb61 100644 --- a/packages/stim-cli/api.api.txt +++ b/packages/stim-cli/api.api.txt @@ -360,6 +360,7 @@ [*] interface DiscoveredCheckout { @@1 from: string; [*] interface DiscoveredCheckout { @@1 origin: DiscoveryOrigin; [*] interface DiscoveredCheckout { @@1 path: string; +[*] interface DiscoveredCheckout { @@1 reduceOnly?: boolean; [*] interface DoctorReport { [*] interface DoctorReport { @@1 @internal daemon: DaemonStatus & { [*] interface DoctorReport { @@1 @internal maintenance: ReturnType & { @@ -1499,7 +1500,7 @@ [*] type DeviceToolRequest = { @branch4 @@1 platform: "android"; [*] type DeviceToolRequest = { @branch4 @@1 state: "on" | "off"; [*] type DeviceToolRequest = { @branch4 @@1 tool: "network"; -[*] type DiscoveryOrigin = "worktree" | "sibling" | "root"; +[*] type DiscoveryOrigin = "worktree" | "sibling" | "root" | "scan"; [*] type DiskPressureLevel = 0 | 1 | 2 | 3; [*] type DoctorPlatform = "ios" | "android"; [*] type EngineOptions = Omit & { diff --git a/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts b/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts index d26c89478..2036917f9 100644 --- a/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-target-cap.test.ts @@ -31,6 +31,8 @@ const candidate = ( dir, identity, discovered, + reduceOnly: false, + checkout: dir.replace(/\/target$/, ''), measuredAt, }); diff --git a/packages/stim-cli/src/__tests__/cargo-target.test.ts b/packages/stim-cli/src/__tests__/cargo-target.test.ts index ea8d2b02c..f32e6b8c9 100644 --- a/packages/stim-cli/src/__tests__/cargo-target.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-target.test.ts @@ -105,7 +105,7 @@ test('trimming removes the stale profiles, keeps the rest and leaves nothing asi makeProfile('aarch64-linux-android/host-dev', 3 * DAY); makeProfile('release', 0); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed.toSorted()).toEqual(['aarch64-linux-android', 'debug']); expect(existsSync(join(target, 'debug'))).toBe(false); @@ -139,7 +139,7 @@ test('a profile a process works in or runs a binary from stays, and an unreadabl ps: () => `${real}/release/deps/mytest-0123456789abcdef --nocapture\n/bin/zsh -l\n`, }); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed).toEqual(['host-dev']); expect(result.kept).toEqual(['debug: a process uses it', 'release: a process uses it']); @@ -150,7 +150,7 @@ test('a profile a process works in or runs a binary from stays, and an unreadabl }, }); makeProfile('profile-b', 3 * DAY); - const unreadable = await trimCargoTarget(target, { needBytes: 0 }); + const unreadable = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(unreadable.removed).toEqual([]); expect(existsSync(join(target, 'profile-b', 'deps', 'libx.rlib'))).toBe(true); } finally { @@ -175,7 +175,7 @@ test('a directory that cannot be put back after a build took its lock is never d return '4242\n'; }, }); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed).toEqual([]); expect(result.kept.join(' ')).toContain('could not be put back'); @@ -198,7 +198,7 @@ test('a profile whose Cargo 1.97 build lock is held stays, whichever of the lock lsof: (args) => (args.includes('-t') && args.some((arg) => arg.endsWith('.cargo-build-lock')) ? '4242\n' : ''), }); - const result = await trimCargoTarget(target, { needBytes: 0 }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(result.removed).toEqual([]); expect(result.kept).toEqual(['debug: a build holds its .cargo-build-lock']); @@ -229,13 +229,13 @@ test('a second trim of a target, from another Stim home, keeps what the first mo return file === 'lsof' || file === 'ps' ? '' : real.runFileAsync(file, args, options); }, }); - const first = trimCargoTarget(target, { needBytes: 0 }); + const first = trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); await reachedPause; const staged = join(realpathSync.native(target), '.stim-trimmed', '0', '.fingerprint'); expect(existsSync(staged)).toBe(true); vi.stubEnv('STIM_HOME', otherHome); - const second = await trimCargoTarget(target, { needBytes: 0 }); + const second = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0 }); expect(second.removed).toEqual([]); expect(second.kept.join(' ')).toContain('another trim of it holds'); @@ -268,6 +268,7 @@ test('a trim stopped at a deadline keeps every unit it has not moved, finishes t }); const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, needBytes: 0, incremental: 'all', measure: false, @@ -288,3 +289,42 @@ test('a trim stopped at a deadline keeps every unit it has not moved, finishes t resetExecutor(); } }); + +test('a launcher on PATH that appears after the first reading keeps what it points into at the later reading, and an incomplete reading keeps everything', async () => { + try { + makeProfile('release', 0); + mkdirSync(join(target, 'release', 'incremental', 'x'), { recursive: true }); + const real = realpathSync.native(target); + stubProcesses({}); + let readings = 0; + const launchers = async () => ({ + targets: readings++ === 0 ? [] : [`${real}/release/incremental/x/tool`], + incomplete: [], + }); + + const result = await trimCargoTarget(target, { + needBytes: 1, + incremental: 'all', + launching: { reduceOnly: true, launchers, checkout: target }, + }); + + expect(readings).toBe(2); + expect(result.removed).toEqual([]); + expect(result.kept).toEqual(['release/incremental: a process uses it']); + expect(existsSync(join(target, 'release', 'incremental', 'x'))).toBe(true); + + makeProfile('debug', 3 * DAY); + const unreadable = await trimCargoTarget(target, { + needBytes: 0, + launching: { + reduceOnly: true, + launchers: async () => ({ targets: [], incomplete: ['/locked'] }), + checkout: target, + }, + }); + expect(unreadable.removed).toEqual([]); + expect(existsSync(join(target, 'debug', 'deps', 'libx.rlib'))).toBe(true); + } finally { + resetExecutor(); + } +}); diff --git a/packages/stim-cli/src/__tests__/cargo-units.test.ts b/packages/stim-cli/src/__tests__/cargo-units.test.ts index 9871fb7f3..33bc7b5e4 100644 --- a/packages/stim-cli/src/__tests__/cargo-units.test.ts +++ b/packages/stim-cli/src/__tests__/cargo-units.test.ts @@ -117,7 +117,12 @@ describe('a profile on disk with Exact ledger', () => { writeLedger(); stub({}); - const result = await trimCargoTarget(target, { needBytes: 0, incremental: 'all', units: 'all' }); + const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, + needBytes: 0, + incremental: 'all', + units: 'all', + }); expect(result.removed).toContain('debug: 1 idle units'); expect(readdirSync(join(profile(), '.fingerprint')).filter((name) => name.startsWith('foo-'))).toHaveLength(8); @@ -132,7 +137,12 @@ describe('a profile on disk with Exact ledger', () => { stub({}); expect(idleCargoUnitPaths(target, Date.now())).toEqual([]); - const result = await trimCargoTarget(target, { needBytes: 0, incremental: 'all', units: 'all' }); + const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, + needBytes: 0, + incremental: 'all', + units: 'all', + }); expect(result.removed).not.toContain('debug: 1 idle units'); expect(existsSync(join(profile(), 'deps', `libserde-${h(1)}.rlib`))).toBe(true); @@ -146,7 +156,12 @@ describe('a profile on disk with Exact ledger', () => { stub({}); expect(idleCargoUnitPaths(target, Date.now())).toEqual([]); - const result = await trimCargoTarget(target, { needBytes: 0, incremental: 'all', units: 'all' }); + const result = await trimCargoTarget(target, { + launching: { reduceOnly: false }, + needBytes: 0, + incremental: 'all', + units: 'all', + }); expect(result.removed).not.toContain('debug: 1 idle units'); expect(existsSync(join(profile(), 'deps', `libserde-${h(1)}.rlib`))).toBe(true); @@ -160,7 +175,7 @@ describe('a profile on disk with Exact ledger', () => { writeLedger(); stub({ ps: () => `${join(profile(), 'deps', `tokio-${h(3)}.d`)} --nocapture\n` }); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual(['debug: 1 idle units']); expect(existsSync(join(profile(), 'deps', `libtokio-${h(3)}.rlib`))).toBe(true); @@ -178,7 +193,7 @@ describe('a profile on disk with Exact ledger', () => { writeLedger(); stub({}); - const sweep = () => trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const sweep = () => trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); const locked = await withCargoLocks([join(target, dir, lock)], sweep); expect(locked).toMatchObject({ value: { removed: [], kept: [expect.stringContaining(`holds ${lock}`)] } }); @@ -193,7 +208,7 @@ describe('a profile on disk with Exact ledger', () => { mkdirSync(join(profile(), '.cargo-build-lock')); stub({}); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual([]); expect(result.kept.join(' ')).toContain('.cargo-build-lock could not be locked'); @@ -213,7 +228,7 @@ describe('a profile on disk with Exact ledger', () => { }, }); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual([]); expect(existsSync(fp)).toBe(true); @@ -234,7 +249,7 @@ describe('a profile on disk with Exact ledger', () => { }, }); - const result = await trimCargoTarget(target, { needBytes: 0, units: 'all' }); + const result = await trimCargoTarget(target, { launching: { reduceOnly: false }, needBytes: 0, units: 'all' }); expect(result.removed).toEqual(['debug: 25 idle units']); expect(result.kept.join(' ')).toContain('Exact updated its ledger'); diff --git a/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts new file mode 100644 index 000000000..804f11423 --- /dev/null +++ b/packages/stim-cli/src/__tests__/discovered-reduce-only.test.ts @@ -0,0 +1,645 @@ +import { execFileSync } from 'node:child_process'; +import { + chmodSync, + existsSync, + readFileSync, + mkdirSync, + mkdtempSync, + readdirSync, + realpathSync, + rmSync, + symlinkSync, + utimesSync, + writeFileSync, +} from 'node:fs'; +import { homedir, tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { planCargoTargetsAtPressure, trimCargoTargetAction } from '../stimd/passes/cargo-target.ts'; +import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; +import * as idleStop from '../stimd/passes/idle-stop.ts'; +import { clearDependency } from '../daemon/dependencies.ts'; +import * as runningFrom from '../workspace/running-from.ts'; +import { answered } from '../workspace/bounded-read.ts'; +import { testContext } from './_context.ts'; +import * as fsUtil from '../fs-util.ts'; +import { deleteOutputs } from '../daemon/idle.ts'; +import { getExecutor, resetExecutor, setExecutor } from '../exec.ts'; +import { + artifactCheckoutsAsync, + artifactWorktreesAsync, + clearArtifact, + collectArtifactsAsync, +} from '../stimd/passes/artifacts.ts'; +import { stoppedOutputCandidates } from '../daemon/stages.ts'; +import { saveConfig } from '../workspace/config.ts'; +import { collectWorktreeSweep, removeWorktrees } from '../commands/gc/worktrees.ts'; +import { discoverCheckouts, discoveredOutside } from '../workspace/discovery.ts'; + +vi.mock('../workspace/protected-locations.ts', async (original) => { + const real = await original(); + return { + ...real, + isProtectedLocation: (path: string) => real.isProtectedLocation(path, { home: homedir(), temps: [] }), + }; +}); + +const launcherDirs = vi.hoisted(() => ({ list: new Array() })); +vi.mock('../workspace/launcher-directories.ts', () => ({ launcherDirectories: () => launcherDirs.list })); + +const day = 86_400_000; + +describe('checkouts found through a tool folder or the full scan', () => { + let base: string; + let main: string; + const old = new Date(Date.now() - 3 * day); + const git = (cwd: string, ...args: string[]) => + execFileSync('git', ['-C', cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', ...args], { + encoding: 'utf8', + env: { ...process.env, GIT_AUTHOR_DATE: old.toISOString(), GIT_COMMITTER_DATE: old.toISOString() }, + }); + const age = (dir: string) => { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + const path = join(dir, entry.name); + if (entry.isDirectory() && entry.name !== '.git') age(path); + utimesSync(path, old, old); + } + utimesSync(dir, old, old); + }; + const worktree = (path: string, name: string) => { + mkdirSync(join(path, '..'), { recursive: true }); + git(main, 'worktree', 'add', '-q', '-b', name, path); + const gitdir = git(path, 'rev-parse', '--absolute-git-dir').trim(); + age(path); + for (const file of ['index', 'HEAD', join('logs', 'HEAD')]) utimesSync(join(gitdir, file), old, old); + return realpathSync(path); + }; + + beforeEach(() => { + base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-reduce-only-'))); + launcherDirs.list = [join(base, 'home', 'bin')]; + vi.stubEnv('HOME', join(base, 'home')); + vi.stubEnv('USERPROFILE', join(base, 'home')); + vi.stubEnv('STIM_HOME', ''); + main = join(base, 'dev', 'main'); + mkdirSync(main, { recursive: true }); + mkdirSync(join(base, 'home')); + writeFileSync(join(main, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + git(main, 'init', '-q', '-b', 'main'); + git(base, 'init', '-q', '--bare', '-b', 'main', join(base, 'remote.git')); + git(main, 'remote', 'add', 'origin', join(base, 'remote.git')); + git(main, 'add', '-A'); + git(main, 'commit', '-q', '-m', 'init'); + git(main, 'push', '-q', '-u', 'origin', 'main'); + git(main, 'remote', 'set-head', 'origin', 'main'); + saveConfig({ version: 2, projects: { [main]: {} }, repos: {} }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + rmSync(base, { recursive: true, force: true }); + }); + + it('discovers a ~/.codex/worktrees worktree of a known repository as reduce-only, and no other home dot-folder worktree', async () => { + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); + const other = worktree(join(base, 'home', '.other', 'wt'), 'other'); + const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); + const found = await discoverCheckouts({ projects: { [main]: {} }, repos: {} }, [main]); + expect(found.map(({ path, reduceOnly }) => [path, reduceOnly])).toEqual([ + [sibling, undefined], + [codex, true], + ]); + expect((await discoveredOutside({ projects: { [main]: {} }, repos: {} }, [main])).map(({ path }) => path)).toEqual([ + other, + ]); + }); + + const writeScan = (found: string[], dotFolderFound: string[] = []) => { + mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); + const scan = { + version: 2, + ranAt: 1, + completedAt: 1, + lowScanAt: 0, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found, + dotFolderFound, + unreadable: [], + truncated: false, + truncation: 'none', + }; + writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); + }; + + it('never selects a checkout under a home dot-folder from the scan result, only through maintenance.discoverRoots', async () => { + const exact = join(base, 'home', '.exact', 'exact2'); + mkdirSync(exact, { recursive: true }); + writeFileSync(join(exact, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + const config = { projects: { [main]: {} }, repos: {} }; + writeScan([], [exact]); + expect(await discoverCheckouts(config, [main])).toEqual([]); + writeScan([exact], [exact]); + expect(await discoverCheckouts(config, [main])).toEqual([]); + const rooted = { ...config, maintenance: { discoverRoots: [join(base, 'home', '.exact')] } }; + expect( + (await discoverCheckouts(rooted, [main])).map(({ path, origin, reduceOnly }) => [path, origin, reduceOnly]), + ).toEqual([[exact, 'root', undefined]]); + }); + + it('keeps the output of a scan-found checkout that an executable on PATH runs from', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(join(tool, 'android', 'app', 'build', 'bin'), { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(tool, 'android', 'app', 'build', 'bin', 'mytool'), '#!/bin/sh\n'); + execFileSync('git', ['-C', tool, 'init', '-q']); + writeFileSync(join(tool, '.gitignore'), 'build/\n'); + execFileSync('git', ['-C', tool, 'add', '-A']); + execFileSync('git', ['-C', tool, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x'], { + env: { ...process.env, GIT_AUTHOR_DATE: old.toISOString(), GIT_COMMITTER_DATE: old.toISOString() }, + }); + age(tool); + const gitdir = join(tool, '.git'); + for (const file of ['index', 'HEAD', join('logs', 'HEAD')]) utimesSync(join(gitdir, file), old, old); + writeScan([tool]); + const config = { projects: { [main]: {} }, repos: {} }; + const reading = { devices: [], simsFailed: false }; + expect((await stoppedOutputCandidates(config, reading, day)).map((entry) => entry.path)).toEqual([tool]); + const bin = join(base, 'home', 'bin'); + mkdirSync(bin); + symlinkSync(join(tool, 'android', 'app', 'build', 'bin', 'mytool'), join(bin, 'mytool')); + expect(await stoppedOutputCandidates(config, reading, day)).toEqual([]); + }); + + it('keeps the Cargo target of a scan-found checkout at high disk, in the plan and at the trim, while a PATH executable links into it', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(join(tool, 'ios', 'App.xcodeproj'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', 'incremental'), { recursive: true }); + mkdirSync(join(tool, 'target', 'release'), { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(tool, '.gitignore'), 'target/\n'); + writeFileSync(join(tool, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(tool, 'target', 'debug', 'incremental', 'data'), 'x'.repeat(10_000)); + writeFileSync(join(tool, 'target', 'release', 'mytool'), '#!/bin/sh\n'); + execFileSync('git', ['-C', tool, 'init', '-q']); + execFileSync('git', ['-C', tool, 'add', '-A']); + execFileSync('git', ['-C', tool, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x']); + writeScan([tool]); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + const operation = testContext({ env: {} }); + const plan = () => + planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation, + }); + const planned = await plan(); + expect(planned.actions).toMatchObject([{ kind: 'would-clear-build-dirs', target: join(tool, 'target') }]); + + mkdirSync(join(base, 'home', 'bin')); + symlinkSync(join(tool, 'target', 'release', 'mytool'), join(base, 'home', 'bin', 'mytool')); + const kept = await planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + expect(kept.actions).toEqual([]); + expect(kept.skips).toMatchObject([{ target: join(tool, 'target'), reason: expect.stringContaining('on PATH') }]); + const [action] = planned.actions; + if (!action) throw new Error('no trim was planned'); + const trim = await trimCargoTargetAction(action, testContext({ env: {} })); + expect(trim).toMatchObject({ status: 'kept', detail: expect.stringContaining('on PATH') }); + }); + + it('keeps what a launcher that appears during the trim own process read points into, and trims it when none appears', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(join(tool, 'ios', 'App.xcodeproj'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(tool, 'target', 'debug', 'incremental', 'x'), { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(tool, '.gitignore'), 'target/\n'); + writeFileSync(join(tool, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'), 'x'.repeat(10_000)); + execFileSync('git', ['-C', tool, 'init', '-q']); + execFileSync('git', ['-C', tool, 'add', '-A']); + execFileSync('git', ['-C', tool, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x']); + writeScan([tool]); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + mkdirSync(join(base, 'home', 'bin')); + const link = join(base, 'home', 'bin', 'tool'); + const real = getExecutor(); + const trimWith = async (appears: boolean) => { + setExecutor({ + ...real, + runFileAsync: async (file, args, options) => { + if (file === 'ps') return ''; + if (file !== 'lsof') return real.runFileAsync(file, args, options); + if (appears && !existsSync(link)) + symlinkSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'), link); + return ''; + }, + }); + const planned = await planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + const [action] = planned.actions; + if (!action) throw new Error('no trim was planned'); + return trimCargoTargetAction(action, testContext({ env: {} })); + }; + try { + expect(await trimWith(true)).toMatchObject({ status: 'kept' }); + expect(existsSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'))).toBe(true); + rmSync(link); + expect(await trimWith(false)).toMatchObject({ status: 'done' }); + expect(existsSync(join(tool, 'target', 'debug', 'incremental', 'x', 'tool'))).toBe(false); + } finally { + resetExecutor(); + } + }); + + it('keeps an output a launcher resolves into when it appears while deleteOutputs measures, and deletes it otherwise', async () => { + const checkout = join(base, 'home', 'tools', 'out'); + const output = join(checkout, 'build'); + mkdirSync(output, { recursive: true }); + writeFileSync(join(output, 'tool'), 'x'); + let measured = false; + vi.spyOn(fsUtil, 'measuredDirectorySizeAsync').mockImplementation(async () => { + measured = true; + return 1; + }); + const launchers = async () => ({ targets: measured ? [join(output, 'tool')] : [], incomplete: [] }); + const kept: string[] = []; + expect(await deleteOutputs([output], { reduceOnly: true, launchers, checkout }, kept)).toBe(0); + expect(existsSync(output)).toBe(true); + expect(kept).toEqual([expect.stringContaining('on PATH')]); + + expect(await deleteOutputs([output], { reduceOnly: false }, kept)).toBe(1); + expect(existsSync(output)).toBe(false); + }); + + it('keeps a nested app Cargo target when a PATH launcher resolves into the checkout outside that app, in the plan and at the trim', async () => { + const mono = join(base, 'home', 'tools', 'mono'); + const app = join(mono, 'apps', 'native'); + mkdirSync(join(app, 'ios', 'App.xcodeproj'), { recursive: true }); + mkdirSync(join(app, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(app, 'target', 'debug', 'incremental'), { recursive: true }); + mkdirSync(join(mono, 'bin'), { recursive: true }); + writeFileSync(join(app, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(app, '.gitignore'), 'target/\n'); + writeFileSync(join(app, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(app, 'target', 'debug', 'incremental', 'data'), 'x'.repeat(10_000)); + writeFileSync(join(mono, 'bin', 'tool'), '#!/bin/sh\n'); + execFileSync('git', ['-C', mono, 'init', '-q']); + execFileSync('git', ['-C', mono, 'add', '-A']); + execFileSync('git', ['-C', mono, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'x']); + writeScan([mono]); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + const plan = () => + planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + const planned = await plan(); + expect(planned.actions).toMatchObject([{ kind: 'would-clear-build-dirs', target: join(app, 'target') }]); + + mkdirSync(join(base, 'home', 'bin')); + symlinkSync(join(mono, 'bin', 'tool'), join(base, 'home', 'bin', 'tool')); + const kept = await plan(); + expect(kept.actions).toEqual([]); + expect(kept.skips).toMatchObject([{ target: join(app, 'target'), reason: expect.stringContaining('on PATH') }]); + const [action] = planned.actions; + if (!action) throw new Error('no trim was planned'); + expect(await trimCargoTargetAction(action, testContext({ env: {} }))).toMatchObject({ + status: 'kept', + detail: expect.stringContaining('on PATH'), + }); + }); + + describe('the artifact and dependency route for a scan-found linked worktree', () => { + const operation = () => testContext({ env: {} }); + const scanFoundWorktree = () => { + const found = worktree(join(base, 'home', 'Documents', 'feature-wt'), 'feature'); + mkdirSync(join(found, 'node_modules', '.bin'), { recursive: true }); + writeFileSync(join(found, 'node_modules', '.bin', 'tool'), '#!/bin/sh\n'); + age(found); + writeScan([found]); + return { found, modules: join(found, 'node_modules') }; + }; + const { signal } = new AbortController(); + const collectOptions = { now: Date.now(), windowMs: 0, protectedRoot: '', pinned: () => false }; + const candidates = async () => collectArtifactsAsync('dependencies', collectOptions, signal, operation()); + const clear = async (modules: string, shared?: ReturnType) => { + return clearArtifact( + { kind: 'would-delete-dependencies', target: modules, bytes: 0, reason: 'test' }, + 'dependencies', + { + windowMs: 0, + protectedRoot: '', + pinned: () => false, + worktrees: await artifactWorktreesAsync(signal, shared), + ...(shared ? { operation: shared } : {}), + }, + ); + }; + + it('collects and clears node_modules without a PATH launcher', async () => { + const { modules } = scanFoundWorktree(); + const found = await candidates(); + expect(found.candidates.map((entry) => entry.path)).toEqual([modules]); + expect(await clear(modules)).toMatchObject({ status: 'done' }); + expect(existsSync(modules)).toBe(false); + }); + + it('keeps it, naming the launcher, when one appears after the plan, with the same operation at both', async () => { + const { modules } = scanFoundWorktree(); + const bin = join(base, 'home', 'bin'); + mkdirSync(bin); + const link = join(bin, 'tool'); + symlinkSync(join(modules, '.bin', 'tool'), link); + const withLauncher = await candidates(); + expect(withLauncher.candidates).toEqual([]); + expect(withLauncher.skips).toMatchObject([{ target: modules, reason: expect.stringContaining('on PATH') }]); + rmSync(link); + + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + expect(planned.candidates.map((entry) => entry.path)).toEqual([modules]); + symlinkSync(join(modules, '.bin', 'tool'), link); + expect(await clear(modules, shared)).toMatchObject({ + status: 'kept', + detail: expect.stringContaining('on PATH'), + }); + expect(existsSync(modules)).toBe(true); + }); + + it('keeps it when a launcher appears after the process read that precedes the deletion', async () => { + const { modules } = scanFoundWorktree(); + const link = join(base, 'home', 'bin', 'tool'); + mkdirSync(join(base, 'home', 'bin')); + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + expect(planned.candidates.map((entry) => entry.path)).toEqual([modules]); + const real = runningFrom.readRunningProcessesAsync; + let reads = 0; + vi.spyOn(runningFrom, 'readRunningProcessesAsync').mockImplementation(async (...args) => { + if (++reads === 2) symlinkSync(join(modules, '.bin', 'tool'), link); + return real(...args); + }); + expect(await clear(modules, shared)).toMatchObject({ + status: 'kept', + detail: expect.stringContaining('on PATH'), + }); + expect(reads).toBeGreaterThanOrEqual(2); + expect(existsSync(modules)).toBe(true); + }); + + describe('through clearDependency, whose final recheck is the idleness evidence', () => { + const clearThroughRecheck = async (appears: boolean) => { + const { modules } = scanFoundWorktree(); + const link = join(base, 'home', 'bin', 'tool'); + mkdirSync(join(base, 'home', 'bin')); + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + const [candidate] = planned.candidates; + if (!candidate) throw new Error('no dependency candidate'); + const real = idleStop.idleStopEvidence; + vi.spyOn(idleStop, 'idleStopEvidence').mockImplementation(async (...args) => { + if (appears) symlinkSync(join(modules, '.bin', 'tool'), link); + return real(...args); + }); + const outcome = await clearDependency( + { + candidate, + source: false, + workspace: candidate.keys[0] ?? candidate.worktree, + bytes: null, + privateBytes: null, + }, + day, + '', + await artifactWorktreesAsync(signal, shared), + 'test', + shared, + false, + ); + return { outcome, modules }; + }; + + it('clears it when no launcher appears', async () => { + const { outcome, modules } = await clearThroughRecheck(false); + expect(outcome).toMatchObject({ status: 'done' }); + expect(existsSync(modules)).toBe(false); + }); + + it('keeps it when a launcher appears during the recheck', async () => { + const { outcome, modules } = await clearThroughRecheck(true); + expect(outcome).toMatchObject({ status: 'kept', detail: 'it is no longer idle' }); + expect(existsSync(modules)).toBe(true); + }); + }); + + test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( + 'keeps it when a launcher directory becomes unreadable after the plan, naming the directory', + async () => { + const { modules } = scanFoundWorktree(); + const locked = join(base, 'home', 'locked-bin'); + mkdirSync(locked, { mode: 0o000 }); + try { + const shared = operation(); + const planned = await collectArtifactsAsync('dependencies', collectOptions, signal, shared); + expect(planned.candidates.map((entry) => entry.path)).toEqual([modules]); + launcherDirs.list = [...launcherDirs.list, locked]; + const outcome = await clear(modules, shared); + expect(outcome).toMatchObject({ status: 'kept', detail: expect.stringContaining(locked) }); + expect(existsSync(modules)).toBe(true); + } finally { + chmodSync(locked, 0o700); + } + }, + ); + }); + + it('leaves a cached scan find unselected while a read is pending, keeps it in the scan file, and selects it once the read settles', async () => { + const tool = join(base, 'home', 'tools', 'mytool'); + mkdirSync(tool, { recursive: true }); + writeFileSync(join(tool, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeScan([tool]); + const file = join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'); + const before = readFileSync(file, 'utf8'); + let release: (value: string) => void = () => {}; + const hung = answered( + 'a stalled mount', + () => + new Promise((resolve) => { + release = resolve; + }), + 10, + ).catch(() => null); + await vi.advanceTimersByTimeAsync(10); + await hung; + const config = { projects: { [main]: {} }, repos: {} }; + expect(await discoverCheckouts(config, [main])).toEqual([]); + expect(readFileSync(file, 'utf8')).toBe(before); + + release('done'); + await vi.advanceTimersByTimeAsync(0); + expect((await discoverCheckouts(config, [main])).map((entry) => entry.path)).toEqual([tool]); + }); + + it('drops a cached checkout that was replaced by a symlink to a folder the scan never enters, and plans nothing under it', async () => { + const moved = join(base, 'elsewhere', 'app'); + mkdirSync(moved, { recursive: true }); + writeFileSync(join(moved, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + mkdirSync(join(base, 'home', 'Documents'), { recursive: true }); + const stale = join(base, 'home', 'Documents', 'app'); + symlinkSync(moved, stale); + writeScan([stale, moved]); + const config = { projects: { [main]: {} }, repos: {} }; + expect(await discoverCheckouts(config, [main])).toEqual([]); + const planned = await artifactCheckoutsAsync(new AbortController().signal); + expect(planned.filter((entry) => entry.path.includes('elsewhere') || entry.path.includes('Documents'))).toEqual([]); + }); + + it('holds a scan-found linked worktree at the removal plan', async () => { + const found = worktree(join(base, 'home', 'Documents', 'feature-wt'), 'feature'); + writeScan([found]); + const planned = await collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false }); + expect(planned.worktrees.find((entry) => entry.path === found)).toMatchObject({ + discovered: true, + reduceOnly: true, + skipped: expect.stringContaining('never removes'), + }); + }); + + it('clears the build output of an idle reduce-only worktree in the dormant stage and leaves the worktree', async () => { + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); + mkdirSync(join(codex, 'android', 'app', 'build'), { recursive: true }); + age(codex); + const config = { projects: { [main]: {} }, repos: {} }; + const stage = await stoppedOutputCandidates(config, { devices: [], simsFailed: false }, day); + expect(stage.map((entry) => entry.path)).toEqual([codex]); + expect(existsSync(codex)).toBe(true); + }); + + it('holds a tool-folder worktree from removal even when a discover root inside it adds a non-reduce-only child root', async () => { + mkdirSync(join(main, 'packages', 'app'), { recursive: true }); + writeFileSync(join(main, 'packages', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + git(main, 'add', '-A'); + git(main, 'commit', '-q', '-m', 'app'); + git(main, 'push', '-q', 'origin', 'main'); + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'aa11', 'main'), 'codex'); + const config = { + version: 2 as const, + projects: { [main]: {} }, + repos: {}, + maintenance: { discoverRoots: [join(codex, 'packages')] }, + }; + saveConfig(config); + const found = await discoverCheckouts(config, [main]); + expect(found.map(({ path, reduceOnly }) => [path, reduceOnly])).toEqual( + expect.arrayContaining([ + [codex, true], + [join(codex, 'packages', 'app'), undefined], + ]), + ); + const planned = await collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false }); + expect(planned.worktrees.find((entry) => entry.path === codex)).toMatchObject({ + reduceOnly: true, + skipped: expect.stringContaining('never removes'), + }); + }); + + it('treats a tool-folder worktree as reduce-only for the artifact and Cargo passes too when a discover root inside it adds a child root', async () => { + mkdirSync(join(main, 'packages', 'app'), { recursive: true }); + mkdirSync(join(main, 'ios', 'App.xcodeproj'), { recursive: true }); + writeFileSync(join(main, 'packages', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(main, 'ios', 'App.xcodeproj', 'project.pbxproj'), 'x'); + writeFileSync(join(main, '.gitignore'), 'target/\nnode_modules/\n'); + git(main, 'add', '-A'); + git(main, 'commit', '-q', '-m', 'app'); + git(main, 'push', '-q', 'origin', 'main'); + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'aa11', 'main'), 'codex'); + mkdirSync(join(codex, 'node_modules', '.bin'), { recursive: true }); + writeFileSync(join(codex, 'node_modules', '.bin', 'tool'), '#!/bin/sh\n'); + mkdirSync(join(codex, 'target', 'debug', '.fingerprint'), { recursive: true }); + mkdirSync(join(codex, 'target', 'debug', 'incremental'), { recursive: true }); + mkdirSync(join(codex, 'target', 'release'), { recursive: true }); + writeFileSync(join(codex, 'target', 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55\n'); + writeFileSync(join(codex, 'target', 'debug', 'incremental', 'data'), 'x'.repeat(10_000)); + writeFileSync(join(codex, 'target', 'release', 'tool'), '#!/bin/sh\n'); + age(codex); + saveConfig({ + version: 2, + projects: { [main]: {} }, + repos: {}, + maintenance: { discoverRoots: [join(codex, 'packages')] }, + }); + vi.stubEnv('STIM_MAINTENANCE_CARGO_TARGET_MAX_GB', '0.000001'); + const { signal } = new AbortController(); + const checkouts = await artifactCheckoutsAsync(signal, testContext({ env: {} })); + expect(checkouts.find((entry) => entry.path === codex)?.reduceOnly).toBe(true); + + const options = { now: Date.now(), windowMs: 0, protectedRoot: '', pinned: () => false }; + const dependencies = (await collectArtifactsAsync('dependencies', options, signal, testContext({ env: {} }))) + .candidates; + expect(dependencies.map((entry) => entry.path)).toContain(join(codex, 'node_modules')); + const plan = () => + planCargoTargetsAtPressure({ + settings: resolveMaintenanceSettings(), + pinned: () => false, + now: Date.now(), + level: 'high', + operation: testContext({ env: {} }), + }); + expect((await plan()).actions.map((action) => action.target)).toContain(join(codex, 'target')); + + mkdirSync(join(base, 'home', 'bin')); + symlinkSync(join(codex, 'target', 'release', 'tool'), join(base, 'home', 'bin', 'tool')); + const kept = await collectArtifactsAsync('dependencies', options, signal, testContext({ env: {} })); + expect(kept.candidates.map((entry) => entry.path)).not.toContain(join(codex, 'node_modules')); + expect(kept.skips).toEqual( + expect.arrayContaining([ + expect.objectContaining({ target: join(codex, 'node_modules'), reason: expect.stringContaining('on PATH') }), + ]), + ); + const after = await plan(); + expect(after.actions.map((action) => action.target)).not.toContain(join(codex, 'target')); + expect(after.skips).toEqual( + expect.arrayContaining([ + expect.objectContaining({ target: join(codex, 'target'), reason: expect.stringContaining('on PATH') }), + ]), + ); + }); + + it('plans a reduce-only worktree as kept, and keeps it at removal even when the plan said it was removable', async () => { + const codex = worktree(join(base, 'home', '.codex', 'worktrees', 'ab12', 'main'), 'codex'); + const sibling = worktree(join(base, 'dev', 'sibling-wt'), 'sibling'); + const planned = await collectWorktreeSweep({ idle: false, olderThan: null, now: Date.now(), refresh: false }); + const byPath = Object.fromEntries(planned.worktrees.map((entry) => [entry.path, entry])); + expect(byPath[sibling]?.skipped).toBeNull(); + expect(byPath[codex]).toMatchObject({ + discovered: true, + reduceOnly: true, + skipped: expect.stringContaining('never removes'), + }); + const worktrees = planned.worktrees.filter((entry) => entry.path === codex); + for (const entry of worktrees) Object.assign(entry, { skipped: null, skipCode: null }); + vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + await removeWorktrees({ ...planned, worktrees }, { now: Date.now() }); + expect(existsSync(codex)).toBe(true); + }); +}); diff --git a/packages/stim-cli/src/__tests__/discovered-stages.test.ts b/packages/stim-cli/src/__tests__/discovered-stages.test.ts index 30885f8a5..45b75ba93 100644 --- a/packages/stim-cli/src/__tests__/discovered-stages.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-stages.test.ts @@ -21,7 +21,9 @@ const fail = (): never => { throw new Error('no candidate'); }; -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, +})); describe('a discovered checkout in the idle stages', () => { let base: string; diff --git a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts index f8b5df4da..b70c542eb 100644 --- a/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts +++ b/packages/stim-cli/src/__tests__/discovered-worktrees.test.ts @@ -17,7 +17,9 @@ import * as reads from '../workspace/discovered-reads.ts'; import { discoveredRemovalHold } from '../workspace/discovered-removal.ts'; import { worktreeInScope } from '../workspace/discovery.ts'; -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, +})); const day = 86_400_000; @@ -160,10 +162,16 @@ describe('a discovered worktree on the removal path', () => { main, idleMs: day, reads: reads.discoveredReads(), + reduceOnly: false, }); expect(hold).toContain('managed by another tool'); expect( - await discoveredRemovalHold(join(main, '.worktrees', 'x'), { main, idleMs: day, reads: reads.discoveredReads() }), + await discoveredRemovalHold(join(main, '.worktrees', 'x'), { + main, + idleMs: day, + reads: reads.discoveredReads(), + reduceOnly: false, + }), ).not.toContain('another tool'); }); @@ -182,6 +190,7 @@ describe('a discovered worktree on the removal path', () => { it('keeps a worktree a process runs from or an agent session works in', async () => { worktree('held'); vi.spyOn(reads, 'discoveredReads').mockImplementation(() => ({ + launchers: async () => ({ targets: [], incomplete: [] }), processes: async () => ({ commands: [{ pid: 1, line: `node ${join(base, 'roots', 'held')}/server.js` }], cwds: [], diff --git a/packages/stim-cli/src/__tests__/discovery-protected.test.ts b/packages/stim-cli/src/__tests__/discovery-protected.test.ts index e60f2d2ae..ec81f4456 100644 --- a/packages/stim-cli/src/__tests__/discovery-protected.test.ts +++ b/packages/stim-cli/src/__tests__/discovery-protected.test.ts @@ -1,7 +1,8 @@ import { mkdirSync, mkdtempSync, realpathSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { worktreeInScope } from '../workspace/discovery.ts'; +import { inToolFolder, worktreeInScope } from '../workspace/discovery.ts'; +import { isProtectedLocation } from '../workspace/protected-locations.ts'; describe('a protected location and maintenance.discoverRoots, with the real protection rule', () => { let base: string; @@ -21,4 +22,17 @@ describe('a protected location and maintenance.discoverRoots, with the real prot expect(worktreeInScope(join(roots, 'wt'), { ...where, roots: [] })).toBe(false); expect(worktreeInScope(join(base, 'other', 'wt'), { ...where, roots: [roots] })).toBe(false); }); + + it('protects every home dot-folder, yet names only ~/.codex/worktrees as a tool folder', () => { + const home = join(base, 'home'); + for (const dir of ['.codex/worktrees/a/app', '.codex/sessions/a', '.claude/worktrees/a', 'dev/a']) + mkdirSync(join(home, dir), { recursive: true }); + const roots = { home, temps: [] }; + expect(isProtectedLocation(join(home, '.codex', 'worktrees', 'a', 'app'), roots)).toBe(true); + expect(isProtectedLocation(join(home, 'dev', 'a'), roots)).toBe(false); + expect(inToolFolder(join(home, '.codex', 'worktrees', 'a', 'app'), home)).toBe(true); + expect(inToolFolder(join(home, '.codex', 'sessions', 'a'), home)).toBe(false); + expect(inToolFolder(join(home, '.claude', 'worktrees', 'a'), home)).toBe(false); + expect(inToolFolder(join(home, 'dev', 'a'), home)).toBe(false); + }); }); diff --git a/packages/stim-cli/src/__tests__/discovery-scan.test.ts b/packages/stim-cli/src/__tests__/discovery-scan.test.ts new file mode 100644 index 000000000..700a154e0 --- /dev/null +++ b/packages/stim-cli/src/__tests__/discovery-scan.test.ts @@ -0,0 +1,718 @@ +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, + symlinkSync, + writeFileSync, +} from 'node:fs'; +import { lstat, readdir } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { testContext } from './_context.ts'; +import { discoveryScanFile, parseDiscoveryScan, readDiscoveryScan } from '@stim-cli/core/state'; +import { discoveryScanDue, mergeFound, runDiscoveryScanIfDue, scanForCheckouts } from '../workspace/discovery-scan.ts'; +import { REAL_RECOGNITION_IO, scanDescends } from '../workspace/scan-rules.ts'; +import type { ScanTruncation } from '@stim-cli/core/state'; + +const marked = (path: string) => existsSync(join(path, 'package.json')); + +describe('scanDescends', () => { + it('skips dependency and build folders everywhere, and the excluded folders of the home folder only', () => { + for (const name of ['node_modules', 'Pods', 'build', 'target', 'Library']) + expect(scanDescends(name, 3)).toBe(false); + for (const name of ['Desktop', 'Downloads', 'Movies']) { + expect(scanDescends(name, 1)).toBe(false); + expect(scanDescends(name, 2)).toBe(true); + } + expect(scanDescends('Documents', 1)).toBe(true); + expect(scanDescends('dev', 1)).toBe(true); + }); + + it('enters a dot-folder directly in the home folder, but not a Stim one, a cache, or a deeper one', () => { + expect(scanDescends('.exact', 1)).toBe(true); + expect(scanDescends('.stim-agent', 1)).toBe(false); + expect(scanDescends('.npm', 1)).toBe(false); + expect(scanDescends('.some-tool-cache', 1)).toBe(false); + expect(scanDescends('.exact', 2)).toBe(false); + }); +}); + +describe('scanForCheckouts', () => { + let home: string; + const project = (...parts: string[]) => { + mkdirSync(join(home, ...parts), { recursive: true }); + writeFileSync(join(home, ...parts, 'package.json'), '{}'); + }; + + beforeEach(() => { + home = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-'))); + }); + + afterEach(() => rmSync(home, { recursive: true, force: true })); + + it('finds projects at depth, stops at one, and never reads pruned, dot, excluded or linked folders', async () => { + project('Documents', 'a'); + project('dev', 'deep', 'er', 'b'); + project('dev', 'c'); + project('.exact', 'exact2'); + mkdirSync(join(home, 'dev', 'c', 'inner')); + project('dev', 'c', 'inner'); + for (const skipped of [ + ['Desktop', 'x'], + ['Downloads', 'x'], + ['dev', 'node_modules', 'x'], + ['dev', '.hidden', 'x'], + ['.npm', 'x'], + ['dev', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'too-deep'], + ]) + project(...skipped); + symlinkSync(join(home, 'Documents', 'a'), join(home, 'dev', 'linked')); + const scan = await scanForCheckouts(home, marked); + expect(scan.found.toSorted()).toEqual( + [ + join(home, 'Documents', 'a'), + join(home, 'dev', 'c'), + join(home, 'dev', 'deep', 'er', 'b'), + join(home, '.exact', 'exact2'), + ].toSorted(), + ); + expect(scan.truncated).toBe(false); + expect(scan.unreadable).toEqual([]); + }); + + it('records a folder the system refuses as unreadable and goes on', async () => { + project('dev', 'ok'); + mkdirSync(join(home, 'Documents')); + mkdirSync(join(home, 'Documents', 'locked'), { mode: 0o000 }); + try { + const scan = await scanForCheckouts(home, marked); + expect(scan.found).toEqual([join(home, 'dev', 'ok')]); + const refused = process.platform !== 'win32' && process.getuid?.() !== 0; + expect(scan.unreadable).toEqual(refused ? [join(home, 'Documents', 'locked')] : []); + } finally { + chmodSync(join(home, 'Documents', 'locked'), 0o700); + } + }); + + it('does not call detection for a folder whose manifest read does not answer, and ends the scan truncated with the folder unreadable', async () => { + project('dev', 'app'); + const detect = vi.fn<(path: string) => boolean>((path) => marked(path)); + let started = 0; + let release: (text: string) => void = () => {}; + const stuck = { + ...REAL_RECOGNITION_IO, + readEntries: (path: string) => readdir(path, { withFileTypes: true }), + readText: () => { + started++; + return new Promise((resolve) => { + release = resolve; + }); + }, + }; + const pending = scanForCheckouts(home, detect, { recognition: stuck, readTimeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await pending).toMatchObject({ truncated: true, unreadable: [join(home, 'dev', 'app')], found: [] }); + expect(detect).not.toHaveBeenCalledWith(join(home, 'dev', 'app')); + release('{}'); + await vi.advanceTimersByTimeAsync(0); + }); + + it('treats a folder whose manifest is not a regular file as not a checkout, and goes on', async () => { + mkdirSync(join(home, 'dev', 'fifo'), { recursive: true }); + project('dev', 'ok'); + const detect = vi.fn<(path: string) => boolean>((path) => marked(path)); + const entries = async (path: string) => { + const listing = await readdir(path, { withFileTypes: true }); + return path.endsWith('fifo') + ? [{ name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }] + : listing; + }; + const scan = await scanForCheckouts(home, detect, { entries }); + expect(scan.found).toEqual([join(home, 'dev', 'ok')]); + expect(detect).not.toHaveBeenCalledWith(join(home, 'dev', 'fifo')); + }); + + it('does not recognise a native Android layout below a folder whose manifest is not a regular file, since detection reads every ancestor', async () => { + mkdirSync(join(home, 'dev', 'app'), { recursive: true }); + const regular = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const fifo = { name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }; + const directory = { name: 'app', isFile: () => false, isDirectory: () => true, isSymbolicLink: () => false }; + const listing = (blocked: boolean) => async (path: string) => { + if (path === home) return [{ ...directory, name: 'dev' }]; + if (path === join(home, 'dev')) return blocked ? [fifo, directory] : [directory]; + return [regular('settings.gradle'), regular('gradlew')]; + }; + const detect = vi.fn<(path: string) => boolean>((path) => path.endsWith('app')); + expect((await scanForCheckouts(home, detect, { entries: listing(true) })).found).toEqual([]); + expect(detect).not.toHaveBeenCalledWith(join(home, 'dev', 'app')); + expect((await scanForCheckouts(home, detect, { entries: listing(false) })).found).toEqual([ + join(home, 'dev', 'app'), + ]); + }); + + it('calls no detection on a folder or below it when its apps folder holds a manifest that is not a regular file', async () => { + mkdirSync(join(home, 'mono', 'apps', 'x'), { recursive: true }); + const regular = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const fifo = { name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }; + const listing = (blocked: boolean) => async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return blocked ? [fifo, directory('x')] : [directory('x')]; + return [regular('settings.gradle'), regular('gradlew')]; + }; + const recognition = { ...REAL_RECOGNITION_IO, readEntries: async (path: string) => listing(true)(path) }; + const detect = vi.fn<(path: string) => boolean>(() => true); + expect((await scanForCheckouts(home, detect, { entries: listing(true), recognition })).found).toEqual([]); + expect(detect).not.toHaveBeenCalled(); + const clean = { ...REAL_RECOGNITION_IO, readEntries: async (path: string) => listing(false)(path) }; + expect((await scanForCheckouts(home, detect, { entries: listing(false), recognition: clean })).found).toEqual([ + join(home, 'mono'), + ]); + }); + + it('makes the whole subtree unrecognisable when a child that detection inspects is unsafe, so a sibling is not a find either', async () => { + mkdirSync(join(home, 'mono', 'apps', 'x'), { recursive: true }); + const regular = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const fifo = { name: 'package.json', isFile: () => false, isDirectory: () => false, isSymbolicLink: () => false }; + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('x'), directory('y')]; + if (path === join(home, 'mono', 'apps', 'y')) return [fifo]; + return [regular('settings.gradle'), regular('gradlew')]; + }; + const detect = vi.fn<(path: string) => boolean>((path) => path.endsWith('x')); + const recognition = { ...REAL_RECOGNITION_IO, readEntries: listing }; + expect((await scanForCheckouts(home, detect, { entries: listing, recognition })).found).toEqual([]); + expect(detect).not.toHaveBeenCalled(); + }); + + it('calls no detection on a folder or its child when that child cannot be listed, however readable its manifest is', async () => { + mkdirSync(join(home, 'mono', 'apps', 'app'), { recursive: true }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('app')]; + throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + }; + const detect = vi.fn<(path: string) => boolean>(() => true); + const recognition = { ...REAL_RECOGNITION_IO, readEntries: listing }; + const scan = await scanForCheckouts(home, detect, { entries: listing, recognition }); + expect(scan.found).toEqual([]); + expect(detect).not.toHaveBeenCalled(); + }); + + it("ends the scan truncated as budget, without calling detection, when a folder's nested checks outlast the remaining budget", async () => { + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + mkdirSync(join(home, 'mono', 'apps'), { recursive: true }); + const children = Array.from({ length: 50 }, (_, i) => directory(`app${i}`)); + let reads = 0; + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return children; + return []; + }; + const slow = async (path: string) => { + reads++; + vi.setSystemTime(Date.now() + 1000); + return listing(path); + }; + const detect = vi.fn<(path: string) => boolean>(() => true); + const scan = await scanForCheckouts(home, detect, { + entries: listing, + recognition: { ...REAL_RECOGNITION_IO, readEntries: slow }, + budgetMs: 5000, + }); + expect(scan).toMatchObject({ truncated: true, truncation: 'budget', found: [] }); + expect(detect).not.toHaveBeenCalled(); + expect(reads).toBeLessThan(10); + }); + + it('rejects the scan as an abort when the abort arrives during the nested checks of the last folder', async () => { + mkdirSync(join(home, 'mono', 'apps', 'app'), { recursive: true }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const controller = new AbortController(); + const listing = async (path: string) => { + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('app')]; + return []; + }; + const recognition = { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === join(home, 'mono', 'apps', 'app')) controller.abort(); + return listing(path); + }, + }; + const detect = vi.fn<(path: string) => boolean>(() => true); + await expect( + scanForCheckouts(home, detect, { entries: listing, recognition, signal: controller.signal }), + ).rejects.toThrow(/abort/i); + expect(detect).not.toHaveBeenCalled(); + }); + + it('stops at its time budget and says so', async () => { + project('dev', 'one'); + const scan = await scanForCheckouts(home, marked, { budgetMs: -1 }); + expect(scan).toMatchObject({ found: [], truncated: true }); + }); +}); + +describe('discoveryScanDue', () => { + const now = Date.UTC(2026, 10, 14); + const at = (ranAgo: number, completedAgo = ranAgo, lowAgo = 1e12, truncation: ScanTruncation = 'none') => ({ + ranAt: now - ranAgo, + completedAt: now - completedAgo, + lowScanAt: now - lowAgo, + truncation, + }); + const hour = 3_600_000; + const day = 24 * hour; + + it('runs weekly from the last scan that did not fail, and when none or a future one is recorded', () => { + expect(discoveryScanDue(null, now, false)).toBe(true); + expect(discoveryScanDue(at(-1000), now, false)).toBe(true); + expect(discoveryScanDue(at(6 * day), now, false)).toBe(false); + expect(discoveryScanDue(at(7 * day), now, false)).toBe(true); + }); + + it('retries an attempt that ended on a read or a failure after an hour at any disk level, even right after a success', () => { + expect(discoveryScanDue(at(2 * hour, 24 * hour, 2 * hour, 'read'), now, false)).toBe(true); + expect(discoveryScanDue(at(2 * hour, 24 * hour, 2 * hour, 'failed'), now, false)).toBe(true); + expect(discoveryScanDue(at(30 * 60_000, 24 * hour, 2 * hour, 'read'), now, false)).toBe(false); + }); + + it('treats an attempt that stopped at the folder cap or the time budget as finished, so it is not retried hourly at normal disk', () => { + expect(discoveryScanDue(at(2 * hour, 2 * hour, 1e12, 'budget'), now, false)).toBe(false); + expect(discoveryScanDue(at(2 * hour, 2 * hour, 1e12, 'folders'), now, true)).toBe(true); + }); + + it('retries a failed scan after an hour, not after a week, and never twice within an hour', () => { + expect(discoveryScanDue(at(30 * 60_000, 8 * day), now, false)).toBe(false); + expect(discoveryScanDue(at(2 * hour, 8 * day), now, false)).toBe(true); + }); + + it('scans while the disk level is above normal when no scan ran during it for an hour, even right after the weekly one', () => { + expect(discoveryScanDue(at(2 * hour), now, true)).toBe(true); + expect(discoveryScanDue(at(2 * hour, 2 * hour, 3 * hour), now, true)).toBe(true); + expect(discoveryScanDue(at(2 * hour, 2 * hour, 90 * 60_000), now, true)).toBe(true); + expect(discoveryScanDue(at(30 * 60_000, 30 * 60_000, 30 * 60_000), now, true)).toBe(false); + }); +}); + +describe('mergeFound', () => { + const outcome = (found: string[], unreadable: string[], truncated: boolean) => ({ + found, + unreadable, + truncated, + truncation: truncated ? ('read' as const) : ('none' as const), + visited: 1, + }); + it('keeps the previous finds under an unreadable folder, drops the others, and keeps all of them when truncated', () => { + const previous = ['/h/Documents/a', '/h/dev/b']; + expect(mergeFound(previous, outcome(['/h/dev/c'], ['/h/Documents'], false))).toEqual([ + '/h/Documents/a', + '/h/dev/c', + ]); + expect(mergeFound(previous, outcome(['/h/dev/c'], [], true))).toEqual(['/h/Documents/a', '/h/dev/b', '/h/dev/c']); + }); +}); + +describe('a scan file that is corrupt or in an older format', () => { + it('is no result', () => { + expect( + parseDiscoveryScan({ + version: 1, + ranAt: 1, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [], + unreadable: [], + truncated: false, + }), + ).toBeNull(); + expect(parseDiscoveryScan({ version: 2, ranAt: 1 })).toBeNull(); + expect(parseDiscoveryScan('nope')).toBeNull(); + }); +}); + +describe('runDiscoveryScanIfDue', () => { + let base: string; + const normal = [{ volume: '/', freeMb: 500_000 }]; + const low = [{ volume: '/', freeMb: 10_000 }]; + const tick = (disk = normal, env: NodeJS.ProcessEnv = {}, operation = testContext({ env })) => ({ + operation, + measured: { measurement: { disk } }, + }); + const hour = 3_600_000; + const t0 = Date.UTC(2026, 10, 14); + const options = (now: number) => ({ home: join(base, 'home'), now, attempted: { at: 0 } }); + + beforeEach(() => { + base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-run-'))); + mkdirSync(join(base, 'home', 'dev', 'app'), { recursive: true }); + writeFileSync(join(base, 'home', 'dev', 'app', 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + mkdirSync(join(base, 'machine')); + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', join(base, 'machine')); + vi.stubEnv('USERPROFILE', join(base, 'machine')); + }); + + afterEach(() => { + vi.unstubAllEnvs(); + rmSync(base, { recursive: true, force: true }); + }); + + it('stops at the first read that does not answer, starts no read while it is pending, and scans again once it settles', async () => { + let started = 0; + let release: (names: []) => void = () => {}; + const hungEntries = () => { + started++; + return new Promise<[]>((resolve) => { + release = resolve; + }); + }; + const home = join(base, 'home'); + const first = scanForCheckouts(home, marked, { entries: hungEntries, readTimeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await first).toMatchObject({ truncated: true, unreadable: [home], found: [] }); + + const later = await scanForCheckouts(home, marked, { entries: hungEntries, readTimeoutMs: 10 }); + expect(later.truncated).toBe(true); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toBeNull(); + expect(started).toBe(1); + + release([]); + await vi.advanceTimersByTimeAsync(0); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ found: [join(base, 'home', 'dev', 'app')], truncated: false }); + }); + + it('never scans for a command, only for a daemon tick', async () => { + await runDiscoveryScanIfDue(undefined, options(t0)); + expect(readDiscoveryScan()).toBeNull(); + }); + + it('never scans in a scoped home and writes nothing', async () => { + vi.stubEnv('STIM_HOME', join(base, 'state')); + await runDiscoveryScanIfDue(tick(normal, { STIM_HOME: join(base, 'state') }), options(t0)); + expect(existsSync(join(base, 'state'))).toBe(false); + }); + + it('records finds under a home dot-folder apart from the others', async () => { + mkdirSync(join(base, 'home', '.exact', 'exact2'), { recursive: true }); + writeFileSync( + join(base, 'home', '.exact', 'exact2', 'package.json'), + JSON.stringify({ dependencies: { expo: '*' } }), + ); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ + found: [join(base, 'home', 'dev', 'app')], + dotFolderFound: [join(base, 'home', '.exact', 'exact2')], + }); + }); + + it('stores the result and its schedule under the Stim home, and not again until due', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ + trigger: 'weekly', + ranAt: t0, + completedAt: t0, + lowScanAt: 0, + found: [join(base, 'home', 'dev', 'app')], + truncated: false, + }); + expect(discoveryScanFile().startsWith(join(base, 'machine'))).toBe(true); + const before = readFileSync(discoveryScanFile(), 'utf8'); + await runDiscoveryScanIfDue(tick(), options(t0 + 2 * hour)); + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + }); + + it('scans again an hour or more after a scan when the disk level is above normal, whatever a worker remembers', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + await runDiscoveryScanIfDue(tick(low), options(t0 + 2 * hour)); + expect(readDiscoveryScan()).toMatchObject({ + trigger: 'low-disk', + ranAt: t0 + 2 * hour, + lowScanAt: t0 + 2 * hour, + completedAt: t0 + 2 * hour, + }); + }); + + it('counts a scan that stopped at its time budget as finished, so it is not retried hourly at normal disk', async () => { + await runDiscoveryScanIfDue(tick(), { ...options(t0), scanOptions: { budgetMs: -1 } }); + expect(readDiscoveryScan()).toMatchObject({ truncated: true, truncation: 'budget', completedAt: t0, ranAt: t0 }); + await runDiscoveryScanIfDue(tick(), options(t0 + 2 * hour)); + expect(readDiscoveryScan()?.ranAt).toBe(t0); + }); + + it('retries a scan that ended on a read that timed out after an hour at any disk level, though a scan finished recently', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()).toMatchObject({ truncation: 'none', completedAt: t0 }); + let started = 0; + let release: (names: []) => void = () => {}; + const entries = () => { + started++; + return new Promise<[]>((resolve) => { + release = resolve; + }); + }; + const second = runDiscoveryScanIfDue(tick(low), { + ...options(t0 + 25 * hour), + scanOptions: { entries, readTimeoutMs: 10 }, + }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + await second; + expect(readDiscoveryScan()).toMatchObject({ truncation: 'read', completedAt: t0, ranAt: t0 + 25 * hour }); + release([]); + await vi.advanceTimersByTimeAsync(0); + await runDiscoveryScanIfDue(tick(), options(t0 + 27 * hour)); + expect(readDiscoveryScan()).toMatchObject({ truncation: 'none', ranAt: t0 + 27 * hour }); + }); + + it('keeps the previous finds when a scan fails and records the attempt', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + rmSync(join(base, 'home'), { recursive: true }); + await runDiscoveryScanIfDue(tick(low), options(t0 + 2 * hour)); + expect(readDiscoveryScan()).toMatchObject({ + ranAt: t0 + 2 * hour, + completedAt: t0, + truncated: true, + found: [join(base, 'home', 'dev', 'app')], + }); + }); + + it('leaves the stored result and its finds unchanged when an abort arrives during the nested checks of the last folder', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const before = readFileSync(discoveryScanFile(), 'utf8'); + mkdirSync(join(base, 'home', 'mono', 'apps', 'app'), { recursive: true }); + const directory = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + const controller = new AbortController(); + const listing = async (path: string) => { + const home = join(base, 'home'); + if (path === home) return [directory('mono')]; + if (path === join(home, 'mono')) return [directory('apps')]; + if (path === join(home, 'mono', 'apps')) return [directory('app')]; + return []; + }; + const recognition = { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path.endsWith('app')) controller.abort(); + return listing(path); + }, + }; + await expect( + runDiscoveryScanIfDue(tick(normal, {}, testContext({ signal: controller.signal })), { + ...options(t0 + 8 * 24 * hour), + scanOptions: { entries: listing, recognition }, + }), + ).rejects.toThrow(/abort/i); + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + expect(readDiscoveryScan()?.found).toEqual([join(base, 'home', 'dev', 'app')]); + }); + + test.skipIf(process.platform === 'win32' || process.getuid?.() === 0)( + 'starts no scan and writes nothing when an existing scan file cannot be read', + async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const before = readFileSync(discoveryScanFile(), 'utf8'); + chmodSync(discoveryScanFile(), 0o000); + try { + let started = 0; + await runDiscoveryScanIfDue(tick(), { + ...options(t0 + 8 * 24 * hour), + scanOptions: { + entries: async () => { + started++; + return []; + }, + }, + }); + expect(started).toBe(0); + } finally { + chmodSync(discoveryScanFile(), 0o600); + } + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + }, + ); + + describe('a read that fails for a reason other than a timeout or absence', () => { + const denied = () => Object.assign(new Error('EACCES'), { code: 'EACCES' }); + const later = () => options(t0 + 8 * 24 * hour); + + it('keeps the previous find and lists its folder unreadable when its manifest read is denied', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const app = join(base, 'home', 'dev', 'app'); + await runDiscoveryScanIfDue(tick(), { + ...later(), + scanOptions: { + recognition: { + ...REAL_RECOGNITION_IO, + readText: async () => { + throw denied(); + }, + }, + }, + }); + expect(readDiscoveryScan()).toMatchObject({ found: [app], unreadable: [app], truncation: 'none' }); + }); + + const failure = (code: string) => Object.assign(new Error(code), { code }); + + it('lists a folder unreadable and keeps the finds beneath it when its listing fails with EIO, without truncating', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const dev = join(base, 'home', 'dev'); + await runDiscoveryScanIfDue(tick(), { + ...later(), + scanOptions: { + entries: async (path: string) => { + if (path === dev) throw failure('EIO'); + return readdir(path, { withFileTypes: true }); + }, + }, + }); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [dev], truncated: false, truncation: 'none' }); + expect(readDiscoveryScan()?.found).toEqual([join(dev, 'app')]); + }); + + it('does the same when the lstat of a child fails with ESTALE, and records nothing when the child is absent', async () => { + await runDiscoveryScanIfDue(tick(), options(t0)); + const dev = join(base, 'home', 'dev'); + const failing = (code: string) => ({ + ...later(), + scanOptions: { + stat: async (path: string) => { + if (path === dev) throw failure(code); + return lstat(path); + }, + }, + }); + await runDiscoveryScanIfDue(tick(), failing('ESTALE')); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [dev], truncation: 'none' }); + expect(readDiscoveryScan()?.found).toEqual([join(dev, 'app')]); + await runDiscoveryScanIfDue(tick(), { ...failing('ENOENT'), now: t0 + 16 * 24 * hour }); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [], truncation: 'none', found: [] }); + }); + + it('keeps the previous find and lists its folder unreadable when a nested listing is denied', async () => { + const mono = join(base, 'home', 'mono'); + mkdirSync(join(mono, 'apps', 'app'), { recursive: true }); + writeFileSync(join(mono, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + await runDiscoveryScanIfDue(tick(), options(t0)); + expect(readDiscoveryScan()?.found).toContain(mono); + await runDiscoveryScanIfDue(tick(), { + ...later(), + scanOptions: { + recognition: { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === join(mono, 'apps', 'app')) throw denied(); + return REAL_RECOGNITION_IO.readEntries(path); + }, + }, + }, + }); + expect(readDiscoveryScan()).toMatchObject({ unreadable: [mono], truncation: 'none' }); + expect(readDiscoveryScan()?.found).toContain(mono); + }); + + it('rejects as an abort and leaves the file unchanged when the abort arrives during a final nested listing that also fails', async () => { + const mono = join(base, 'home', 'mono'); + mkdirSync(join(mono, 'apps', 'app'), { recursive: true }); + writeFileSync(join(mono, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + await runDiscoveryScanIfDue(tick(), options(t0)); + const before = readFileSync(discoveryScanFile(), 'utf8'); + const controller = new AbortController(); + await expect( + runDiscoveryScanIfDue(tick(normal, {}, testContext({ signal: controller.signal })), { + ...later(), + scanOptions: { + recognition: { + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === join(mono, 'apps', 'app')) { + controller.abort(); + throw denied(); + } + return REAL_RECOGNITION_IO.readEntries(path); + }, + }, + }, + }), + ).rejects.toThrow(/abort/i); + expect(readFileSync(discoveryScanFile(), 'utf8')).toBe(before); + }); + }); + + it('lets an abort escape instead of returning normally, and stores nothing', async () => { + const controller = new AbortController(); + controller.abort(); + await expect( + runDiscoveryScanIfDue(tick(normal, {}, testContext({ signal: controller.signal })), options(t0)), + ).rejects.toThrow(/abort/i); + expect(readDiscoveryScan()).toBeNull(); + }); +}); diff --git a/packages/stim-cli/src/__tests__/discovery.test.ts b/packages/stim-cli/src/__tests__/discovery.test.ts index cf2a68afa..272c80ce6 100644 --- a/packages/stim-cli/src/__tests__/discovery.test.ts +++ b/packages/stim-cli/src/__tests__/discovery.test.ts @@ -12,6 +12,7 @@ import { import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { type Config } from '@stim-cli/core/state'; +import { testContext } from './_context.ts'; import { scanSourceWrites } from '../workspace/discovered-activity.ts'; import { discoverCheckouts, @@ -21,7 +22,9 @@ import { type DiscoveryChild, } from '../workspace/discovery.ts'; -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, +})); describe('isBroadFolder', () => { it('refuses the home folder, a folder holding it and the filesystem root, and accepts a folder below it', () => { @@ -150,6 +153,45 @@ describe('discoverCheckouts', () => { expect((await discoveredOutside(config, [repo])).map(({ path }) => path)).toEqual([elsewhere]); }); + it('takes the checkouts of the last full scan as reduce-only in an unscoped home, and none in a scoped one', async () => { + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', join(base, 'home')); + vi.stubEnv('USERPROFILE', join(base, 'home')); + const docs = join(base, 'home', 'Documents', 'old-app'); + app(docs); + mkdirSync(join(base, 'home', '.stim', 'maintenance'), { recursive: true }); + const scan = { + version: 2, + completedAt: 1, + lowScanAt: 0, + ranAt: 1, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [docs, join(base, 'gone')], + dotFolderFound: [], + unreadable: [], + truncated: false, + truncation: 'none', + }; + writeFileSync(join(base, 'home', '.stim', 'maintenance', 'discovery-scan.json'), JSON.stringify(scan)); + expect( + (await discoverCheckouts({ projects: {}, repos: {} }, [])).map(({ path, origin, reduceOnly }) => [ + path, + origin, + reduceOnly, + ]), + ).toEqual([[docs, 'scan', true]]); + vi.stubEnv('STIM_HOME', join(base, 'scoped')); + expect( + await discoverCheckouts( + { projects: {}, repos: {} }, + [], + testContext({ env: { STIM_HOME: join(base, 'scoped') } }), + ), + ).toEqual([]); + }); + it('never takes the home directory as a folder that holds known projects', async () => { vi.stubEnv('STIM_HOME', ''); vi.stubEnv('HOME', join(base, 'home')); diff --git a/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts b/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts index 8af12be74..201280428 100644 --- a/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts +++ b/packages/stim-cli/src/__tests__/gc-workspaces.integration.test.ts @@ -423,7 +423,9 @@ describe('workspace directories without workspace.json', () => { }); }); -vi.mock('../workspace/protected-locations.ts', () => ({ isProtectedLocation: () => false })); +vi.mock('../workspace/protected-locations.ts', () => ({ + isProtectedLocation: () => false, +})); test('listWorkspaceDirs resolves each directory through its workspace.json', () => { const { root } = goneWorkspace('app'); diff --git a/packages/stim-cli/src/__tests__/launchers.test.ts b/packages/stim-cli/src/__tests__/launchers.test.ts new file mode 100644 index 000000000..358cff856 --- /dev/null +++ b/packages/stim-cli/src/__tests__/launchers.test.ts @@ -0,0 +1,105 @@ +import { launcherDirectories } from '../workspace/launcher-directories.ts'; +import { scanForCheckouts } from '../workspace/discovery-scan.ts'; +import { launcherReader, launcherReason } from '../workspace/launchers.ts'; + +const missing = (code: string) => Object.assign(new Error(code), { code }); + +/** A fake of the directories on PATH: `listing` maps a directory to its entries, an Error, or nothing for absent. */ +const io = (listing: Record, links: Record = {}) => ({ + readdir: async (path: string) => { + const entry = listing[path]; + if (entry instanceof Error) throw entry; + if (!entry) throw missing('ENOENT'); + return entry; + }, + isSymlink: async (path: string) => path in links, + realpath: async (path: string) => { + const target = links[path]; + if (target instanceof Error) throw target; + if (!target) throw missing('ENOENT'); + return target; + }, +}); + +const scan = (directories: string[], fake: ReturnType) => + launcherReader(undefined, { directories, io: fake })(); + +describe('the PATH launcher scan', () => { + it('finds a symlink that resolves into a checkout, and nothing for a checkout none points into', async () => { + const result = await scan(['/bin1'], io({ '/bin1': ['tool'] }, { '/bin1/tool': '/work/app/target/release/tool' })); + expect(result).toEqual({ targets: ['/work/app/target/release/tool'], incomplete: [] }); + expect(launcherReason('/work/app', result)).toContain('/work/app/target/release/tool'); + expect(launcherReason('/work/other', result)).toBeNull(); + }); + + it('treats a missing directory, a dangling symlink and a file entry as no launcher, not as unknown', async () => { + const result = await scan( + ['/absent', '/bin1'], + io({ '/bin1': ['dangling', 'plain'] }, { '/bin1/dangling': missing('ENOENT') }), + ); + expect(result).toEqual({ targets: [], incomplete: [] }); + expect(launcherReason('/work/app', result)).toBeNull(); + }); + + it('keeps every checkout when a directory or an entry could not be inspected, naming the directory', async () => { + const unreadable = await scan(['/locked', '/bin1'], io({ '/locked': missing('EACCES'), '/bin1': [] })); + expect(unreadable.incomplete).toEqual(['/locked']); + expect(launcherReason('/work/app', unreadable)).toContain('/locked'); + const unresolved = await scan(['/bin1'], io({ '/bin1': ['tool'] }, { '/bin1/tool': missing('EIO') })); + expect(unresolved.incomplete).toEqual(['/bin1']); + expect(launcherReason('/work/app', unresolved)).toContain('cannot be ruled out'); + }); + + it('ends a reading at once as incomplete when a read does not answer, and starts no read while it is pending', async () => { + let started = 0; + let release: (names: string[]) => void = () => {}; + const stuck = { + ...io({ '/bin1': ['tool'] }), + readdir: (path: string) => { + if (path !== '/hung') return Promise.resolve([]); + started++; + return new Promise((resolve) => { + release = resolve; + }); + }, + }; + const reading = () => launcherReader(undefined, { directories: ['/hung', '/bin1'], io: stuck, timeoutMs: 10 })(); + + const first = reading(); + await vi.advanceTimersByTimeAsync(10); + const result = await first; + expect(result.incomplete).toEqual(['/hung']); + expect(launcherReason('/work/app', result)).toContain('permissions'); + + const later = await Promise.all([reading(), reading(), reading()]); + expect(later.map((entry) => entry.incomplete)).toEqual([['/hung'], ['/hung'], ['/hung']]); + expect(started).toBe(1); + + let scanReads = 0; + const homeScan = await scanForCheckouts('/', () => false, { + stat: async () => ({ dev: 1 }), + entries: async () => { + scanReads++; + return []; + }, + }); + expect(homeScan.truncated).toBe(true); + expect(scanReads).toBe(0); + + release([]); + await vi.advanceTimersByTimeAsync(0); + const recovered = await launcherReader(undefined, { directories: ['/bin1'], io: stuck })(); + expect(recovered.incomplete).toEqual([]); + }); +}); + +describe('the launcher directories', () => { + afterEach(() => vi.unstubAllEnvs()); + + it('lists the PATH of the process when no environment is given, and of the given environment otherwise', () => { + vi.stubEnv('PATH', '/from/process:/also/process'); + expect(launcherDirectories()).toEqual(expect.arrayContaining(['/from/process', '/also/process'])); + expect(launcherDirectories({ PATH: '/given' })).toContain('/given'); + expect(launcherDirectories({ PATH: '/given' })).not.toContain('/from/process'); + }); +}); diff --git a/packages/stim-cli/src/__tests__/maintenance-run.test.ts b/packages/stim-cli/src/__tests__/maintenance-run.test.ts index 1a4e513d5..9e29beea5 100644 --- a/packages/stim-cli/src/__tests__/maintenance-run.test.ts +++ b/packages/stim-cli/src/__tests__/maintenance-run.test.ts @@ -12,7 +12,7 @@ import { workspaceLogsDir, sharedCompilationCache, } from '@stim-cli/core/state'; -import { runMaintenance } from '../stimd/passes/run.ts'; +import { runMaintenance, type MaintenanceTick } from '../stimd/passes/run.ts'; import { resolveMaintenanceSettings } from '../stimd/passes/settings.ts'; import * as measurements from '../stimd/passes/measure.ts'; import * as preview from '../stimd/passes/preview.ts'; @@ -25,6 +25,7 @@ import { runCli } from '../cli/define-op.ts'; import { warnCriticalDisk } from '../stimd/passes/trigger.ts'; import { resetExecutor, setExecutor } from '../exec.ts'; import { makeChildProcess } from './_factories.ts'; +import { testContext } from './_context.ts'; const disk = vi.hoisted(() => ({ full: false, reserve: '', code: 'ENOSPC', partialReserveWrite: false })); vi.mock('@stim-cli/core/ownership-claim', async (importOriginal) => { @@ -502,3 +503,47 @@ test('an invalid cache cap falls back only that cap while report mode remains en delete process.env.STIM_CACHES_BUILD_CACHE_MAX_GB; } }); + +const quietTick = (signal?: AbortSignal): MaintenanceTick => ({ + operation: testContext(signal ? { signal } : {}), + measured: { + measurement: { + disk: [{ volume: '/', freeMb: 500_000 }], + memory: { level: 'normal', availableBytes: null, totalBytes: 0 }, + load1: 0, + cores: 1, + bootedSimulators: null, + runningEmulators: null, + interaction: null, + }, + sims: null, + simsError: null, + devices: [], + }, +}); + +test('the discovery scan runs once after a successful daemon pass, with that pass tick', async () => { + const scan = vi.spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue').mockResolvedValue(); + const tick = quietTick(); + await runMaintenance('daemon', { tick }); + expect(scan).toHaveBeenCalledTimes(1); + expect(scan).toHaveBeenCalledWith(tick); +}); + +test('an unverified child cleanup in a pass surfaces as that failure, not as the abort of a due discovery scan', async () => { + const failure = Object.assign(new Error('unverified cleanup'), { + code: 'STIM_CLAIM_UNAVAILABLE', + reportCleanup: true, + }); + const controller = new AbortController(); + vi.spyOn(measurements, 'pressureFromMeasurement').mockImplementation(() => { + controller.abort(); + throw failure; + }); + const scan = vi + .spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue') + .mockImplementation(async () => controller.signal.throwIfAborted()); + const tick = quietTick(controller.signal); + await expect(runMaintenance('daemon', { tick })).rejects.toBe(failure); + expect(scan).not.toHaveBeenCalled(); +}); diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts new file mode 100644 index 000000000..cd9e29ece --- /dev/null +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -0,0 +1,240 @@ +import { existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { REAL_RECOGNITION_IO, folderSafety, nestedSafety, scanEntryValid } from '../workspace/scan-rules.ts'; + +describe('scanEntryValid with a hand-edited scan result', () => { + let home: string; + const isCheckout = (path: string) => existsSync(join(path, 'package.json')); + const rules = () => ({ home, isCheckout, skip: (path: string) => path.startsWith(join(home, '.stim')) }); + const project = (...parts: string[]) => { + mkdirSync(join(home, ...parts), { recursive: true }); + writeFileSync(join(home, ...parts, 'package.json'), '{}'); + return join(home, ...parts); + }; + + beforeEach(() => { + home = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-rules-'))); + }); + + afterEach(() => rmSync(home, { recursive: true, force: true })); + + it('accepts a real checkout the scan could have found', async () => { + expect(await scanEntryValid(project('dev', 'app'), rules())).toBe('valid'); + }); + + it('refuses entries the scan never finds: the filesystem root, another home, the Stim home, pruned and skipped folders', async () => { + const other = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-other-'))); + try { + writeFileSync(join(other, 'package.json'), '{}'); + for (const path of [ + '/', + home, + other, + project('.stim', 'x'), + project('.npm', 'x'), + project('dev', 'node_modules', 'x'), + project('Desktop', 'x'), + ]) + expect(await scanEntryValid(path, rules())).toBe('invalid'); + } finally { + rmSync(other, { recursive: true, force: true }); + } + }); + + it('refuses a path through a symlinked ancestor, a symlinked checkout and a different-case spelling of a pruned folder', async () => { + const real = project('elsewhere', 'app'); + mkdirSync(join(home, 'dev')); + symlinkSync(join(home, 'elsewhere'), join(home, 'dev', 'link')); + symlinkSync(real, join(home, 'dev', 'app-link')); + expect(await scanEntryValid(join(home, 'dev', 'link', 'app'), rules())).toBe('invalid'); + expect(await scanEntryValid(join(home, 'dev', 'app-link'), rules())).toBe('invalid'); + const cached = project('.claude', 'x'); + expect(await scanEntryValid(cached, rules())).toBe('invalid'); + const upper = join(home, '.CLAUDE', 'x'); + expect(await scanEntryValid(upper, rules())).toBe('invalid'); + }); + + describe('before project detection runs', () => { + const file = (name: string) => ({ + name, + isFile: () => true, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const fifo = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => false, + isSymbolicLink: () => false, + }); + const folder = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => true, + isSymbolicLink: () => false, + }); + + const listing = (listings: Record[]>) => ({ + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => listings[path] ?? [], + readText: async () => '{}', + }); + + it('does not call detection for a detection file that is not regular, or for an Xcode container without an app dependency', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + const path = project('dev', 'app'); + for (const entries of [ + [fifo('package.json')], + [file('package.json'), fifo('gradlew')], + [folder('App.xcodeproj')], + [file('package.json'), folder('App.xcodeproj')], + ]) + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: listing({ [path]: entries }) })).toBe( + 'invalid', + ); + expect(detect).not.toHaveBeenCalled(); + const app = { + ...listing({ [path]: [file('package.json'), folder('App.xcodeproj')] }), + readText: async () => '{"dependencies":{"expo":"*"}}', + }; + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: app })).toBe('valid'); + }); + + it('does not call detection when an ancestor, the home folder or an apps folder holds a manifest that is not a regular file', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + const path = project('mono', 'apps', 'x'); + const android = [file('settings.gradle'), file('gradlew')]; + const control = listing({ + [join(home, 'mono')]: [folder('apps')], + [join(home, 'mono', 'apps')]: [folder('x')], + [path]: android, + }); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: control })).toBe('valid'); + detect.mockClear(); + for (const unsafe of [home, join(home, 'mono'), join(home, 'mono', 'apps')]) { + const io = { + ...control, + readEntries: async (at: string) => (at === unsafe ? [fifo('package.json')] : await control.readEntries(at)), + }; + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io })).toBe('invalid'); + } + expect(detect).not.toHaveBeenCalled(); + }); + + describe('names compare case-insensitively, as a case-insensitive volume reads them', () => { + const symlink = (name: string) => ({ + name, + isFile: () => false, + isDirectory: () => false, + isSymbolicLink: () => true, + }); + const where = '/work/x'; + + it('finds a FIFO manifest, a symlinked container and a symlinked or non-directory ios or android root whatever the case', async () => { + expect(await folderSafety(where, [fifo('PACKAGE.JSON')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [fifo('App.JSON')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [symlink('iOS')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [file('Android')], 10)).toBe('unsafe'); + expect(await folderSafety(where, [folder('IOS'), folder('android')], 10)).toBe('safe'); + expect(await nestedSafety(where, [symlink('APPS')], 10)).toBe('unsafe'); + expect(await nestedSafety(where, [symlink('Packages')], 10)).toBe('unsafe'); + }); + + it('keeps the Xcode exemption only when every manifest that folds to package.json declares an app dependency', async () => { + const declaring = JSON.stringify({ dependencies: { expo: '*' } }); + const texts: Record = {}; + const io = { ...REAL_RECOGNITION_IO, readText: async (path: string) => texts[path] ?? '{}' }; + const entries = [file('package.json'), file('Package.JSON'), folder('X.xcodeproj')]; + texts['/work/x/Package.JSON'] = declaring; + expect(await folderSafety(where, entries, 10, io)).toBe('unsafe'); + texts['/work/x/package.json'] = declaring; + expect(await folderSafety(where, entries, 10, io)).toBe('safe'); + }); + + it('is denied, never safe, when a child that detection enumerates cannot be listed', async () => { + const denied = Object.assign(new Error('EACCES'), { code: 'EACCES' }); + const gone = Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); + const io = (failure: Error) => ({ + ...REAL_RECOGNITION_IO, + readEntries: async (path: string) => { + if (path === '/work/x/apps') return [folder('app')]; + if (path === '/work/x/apps/app') throw failure; + return []; + }, + }); + const entries = [folder('apps')]; + expect(await nestedSafety(where, entries, 10, io(denied))).toBe('denied'); + expect(await nestedSafety(where, entries, 10, io(gone))).toBe('safe'); + }); + + it('reads a mixed-case regular manifest through the bounded reader instead of skipping it', async () => { + const reads: string[] = []; + const io = { + ...REAL_RECOGNITION_IO, + readText: async (path: string) => { + reads.push(path); + return '{}'; + }, + }; + expect(await folderSafety(where, [file('Package.JSON')], 10, io)).toBe('safe'); + expect(reads).toEqual([join(where, 'Package.JSON')]); + expect(await folderSafety(where, [file('PACKAGE.JSON'), folder('App.XCODEPROJ')], 10, io)).toBe('unsafe'); + }); + }); + + it('treats a metadata read that does not answer as pending, before any detection', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + let release: (value: { isSymbolicLink(): boolean }) => void = () => {}; + let started = 0; + const stuck = { + ...listing({}), + lstat: () => { + started++; + return new Promise<{ isSymbolicLink(): boolean }>((resolve) => { + release = resolve; + }); + }, + }; + const path = project('dev', 'app'); + const first = scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await first).toBe('pending'); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck })).toBe('pending'); + expect(started).toBe(1); + expect(detect).not.toHaveBeenCalled(); + release({ isSymbolicLink: () => false }); + await vi.advanceTimersByTimeAsync(0); + }); + + it('does not call detection while a manifest read is pending, leaves the entry pending, and recovers once it settles', async () => { + const detect = vi.fn<(path: string) => boolean>(() => true); + let release: (text: string) => void = () => {}; + let started = 0; + const path = project('dev', 'app'); + const stuck = { + ...listing({ [path]: [file('package.json')] }), + readText: () => { + started++; + return new Promise((resolve) => { + release = resolve; + }); + }, + }; + const first = scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 }); + const waiting = { started: () => started > 0 }; + while (!waiting.started()) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(10); + expect(await first).toBe('pending'); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: stuck, timeoutMs: 10 })).toBe('pending'); + expect(detect).not.toHaveBeenCalled(); + + release('{}'); + await vi.advanceTimersByTimeAsync(0); + const ok = listing({ [path]: [file('package.json')] }); + expect(await scanEntryValid(path, { ...rules(), isCheckout: detect, io: ok })).toBe('valid'); + }); + }); +}); diff --git a/packages/stim-cli/src/commands/gc/workspaces.ts b/packages/stim-cli/src/commands/gc/workspaces.ts index 0c29fe054..5db730c07 100644 --- a/packages/stim-cli/src/commands/gc/workspaces.ts +++ b/packages/stim-cli/src/commands/gc/workspaces.ts @@ -366,7 +366,7 @@ export async function workspaceBuildReasonsAsync( async function inTreeBuildDirs(root: string, signal?: AbortSignal): Promise { const dirs: string[] = []; for (const path of await existingBuildDirsAsync(root)) - if (!(await regenerableDirProblem(root, path, signal))) dirs.push(path); + if (!(await regenerableDirProblem(root, path, signal, { launching: { reduceOnly: false } }))) dirs.push(path); return dirs; } @@ -661,7 +661,10 @@ export async function clearWorkspaceOutputs( if (again?.keptReason) return again.keptReason; const inTree: { path: string; problem: string | null }[] = []; for (const path of lighten ? await existingBuildDirsAsync(root) : []) - inTree.push({ path, problem: await regenerableDirProblem(root, path, signal) }); + inTree.push({ + path, + problem: await regenerableDirProblem(root, path, signal, { launching: { reduceOnly: false } }), + }); for (const path of [ ...workspaceOutputPaths(entry.dir), ...inTree.filter((dir) => !dir.problem).map((dir) => dir.path), diff --git a/packages/stim-cli/src/commands/gc/worktree-sizes.ts b/packages/stim-cli/src/commands/gc/worktree-sizes.ts index 03e043b41..053fede54 100644 --- a/packages/stim-cli/src/commands/gc/worktree-sizes.ts +++ b/packages/stim-cli/src/commands/gc/worktree-sizes.ts @@ -53,7 +53,13 @@ async function buildStateDirs(path: string, keys: readonly string[], operation?: }); const own = keys.filter((key) => matchWorktreeEntry(entries, key)?.path === path); const root = canonicalPath(path); - const worktree = { path, linked: true, locked: Boolean(matchWorktreeEntry(entries, path)?.locked), keys: own }; + const worktree = { + path, + linked: true, + locked: Boolean(matchWorktreeEntry(entries, path)?.locked), + keys: own, + reduceOnly: false, + }; const dirs = new Map(); const privateSized = new Set(); const reportOnly = new Set(); diff --git a/packages/stim-cli/src/commands/gc/worktrees.ts b/packages/stim-cli/src/commands/gc/worktrees.ts index f933e2b76..1c13b488b 100644 --- a/packages/stim-cli/src/commands/gc/worktrees.ts +++ b/packages/stim-cli/src/commands/gc/worktrees.ts @@ -173,6 +173,8 @@ interface WorktreeCandidate { activity: WorktreeActivity | null; /** Discovery found it; see `WorktreeFacts.discovered`. */ discovered?: boolean; + /** Discovered through a tool folder or the full scan: never removed, only reduced. */ + reduceOnly?: boolean; /** The main checkout of its repository, for a discovered worktree. */ source?: string; /** How many uncommitted or untracked entries removal archives first; absent or 0 for a clean worktree. */ @@ -599,7 +601,8 @@ async function discoveredHold(candidate: WorktreeCandidate, operation?: StimCont const hold = await discoveredRemovalHold(candidate.path, { main: candidate.source ?? null, idleMs: dormantMs(operation), - reads: discoveredReads(operation?.signal), + reads: discoveredReads(operation?.signal, operation), + reduceOnly: candidate.reduceOnly === true, operation, }); return hold ? [hold] : []; @@ -831,6 +834,7 @@ export async function collectWorktreeSweep({ const outside: WorktreeCandidate[] = []; const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); const discoveredPaths = new Set(discovered.map(({ path }) => path)); + const reduceOnlyPaths = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); const candidates = await candidateRootsAsync(operation, discovered); if (candidates.length) phase('worktrees', `checking ${plural(candidates.length, 'project')} for merged or idle worktrees`, operation); @@ -914,7 +918,13 @@ export async function collectWorktreeSweep({ skipped: null, eligibleAt: null, activity, - ...(discoveredOnly ? { discovered: true, ...('refusal' in source ? {} : { source: source.path }) } : {}), + ...(discoveredOnly + ? { + discovered: true, + ...(roots.some((root) => reduceOnlyPaths.has(root)) ? { reduceOnly: true } : {}), + ...('refusal' in source ? {} : { source: source.path }), + } + : {}), ...(archiveDirty ? { dirty } : {}), }, facts, @@ -971,12 +981,13 @@ export async function collectWorktreeSweep({ const handed = worktrees.filter((candidate) => candidate.discovered && candidate.skipped === null); if (handed.length) { const { discoveredRemovalHold } = await import('../../workspace/discovered-removal.ts'); - const reads = discoveredReads(operation?.signal); + const reads = discoveredReads(operation?.signal, operation); for (const candidate of handed) { const hold = await discoveredRemovalHold(candidate.path, { main: candidate.source ?? null, idleMs: dormantMs(operation), reads, + reduceOnly: candidate.reduceOnly === true, operation, }); if (hold) Object.assign(candidate, { skipCode: 'recent-activity', skipped: `kept: ${hold}`, eligibleAt: null }); diff --git a/packages/stim-cli/src/daemon/dependencies.ts b/packages/stim-cli/src/daemon/dependencies.ts index 7ec60b8fd..956e775b1 100644 --- a/packages/stim-cli/src/daemon/dependencies.ts +++ b/packages/stim-cli/src/daemon/dependencies.ts @@ -194,7 +194,10 @@ export async function rankSourceDependencies( }); continue; } - const verdict = await sourceCheckoutIdle(candidate.worktree, rule.thresholdMs, rule.setting, operation); + const verdict = await sourceCheckoutIdle(candidate.worktree, rule.thresholdMs, rule.setting, { + operation, + reduceOnly: candidate.reduceOnly, + }); if ('kept' in verdict) { kept.push({ target: candidate.path, workspace, reason: verdict.kept }); continue; @@ -242,15 +245,19 @@ async function stillUnused( keys: readonly string[], worktree: string, idleMs: number, + reduceOnly: boolean, operation?: StimContext, setting?: string, ): Promise { const [first] = keys; if (!source) - return first && !(await stillIdle(first, idleMs, operation, false, true, setting)) ? 'it is no longer idle' : null; + return first && !(await stillIdle(first, idleMs, { operation, setting, reduceOnly })) + ? 'it is no longer idle' + : null; const held = new Set(workspaceKeys(keys)); for (const key of new Set([...held, worktree])) - if (!(await stillIdle(key, idleMs, operation, true, held.has(key), setting))) return `${key} is no longer idle`; + if (!(await stillIdle(key, idleMs, { operation, full: true, locked: held.has(key), setting, reduceOnly }))) + return `${key} is no longer idle`; return Date.now() - warmedFromAt(keys) < idleMs ? 'a worktree warm copied from it since' : null; } @@ -284,7 +291,8 @@ export async function clearDependency( pinned: maintenancePinnedReader(operation), worktrees, requireRestorable, - recheck: (keys, worktree) => stillUnused(false, keys, worktree, idleMs, operation), + recheck: (keys, worktree) => + stillUnused(false, keys, worktree, idleMs, dependency.candidate.reduceOnly, operation), level, operation, }, @@ -341,7 +349,16 @@ export async function clearSourceDependencies( protectedRoot, pinned: maintenancePinnedReader(operation), worktrees, - recheck: (keys, worktree) => stillUnused(true, keys, worktree, idleMs, operation, setting), + recheck: (keys, worktree) => + stillUnused( + true, + keys, + worktree, + idleMs, + dependencies.some(({ candidate }) => candidate.reduceOnly), + operation, + setting, + ), level, operation, }, diff --git a/packages/stim-cli/src/daemon/idle.ts b/packages/stim-cli/src/daemon/idle.ts index 15807eb65..8e767e72e 100644 --- a/packages/stim-cli/src/daemon/idle.ts +++ b/packages/stim-cli/src/daemon/idle.ts @@ -15,6 +15,7 @@ import { type IdleStageRecord, } from '@stim-cli/core/state'; import { formatBytes, measuredDirectorySizeAsync } from '../fs-util.ts'; +import { launcherReason, type Launching } from '../workspace/launchers.ts'; import { collectOwnedDeviceActivityAsync, type OwnedDeviceActivity } from '../commands/gc/idle.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; import { workspaceOutputPaths } from '../commands/gc/workspaces.ts'; @@ -186,11 +187,15 @@ async function decide( } /** The regenerable outputs the idle stage deletes, and why each other one stays. */ -export async function idleOutputs(path: string, config: Config): Promise<{ paths: string[]; kept: string[] }> { +export async function idleOutputs( + path: string, + config: Config, + launching: Launching, +): Promise<{ paths: string[]; kept: string[] }> { const paths = workspaceOutputPaths(workspaceDir(path)); const kept: string[] = []; for (const dir of await existingBuildDirsAsync(path)) { - const problem = await regenerableDirProblem(path, dir); + const problem = await regenerableDirProblem(path, dir, undefined, { launching }); if (problem) kept.push(`kept ${dir}: ${problem}`); else paths.push(dir); } @@ -209,10 +214,20 @@ export async function idleOutputs(path: string, config: Config): Promise<{ paths return { paths, kept }; } -export async function deleteOutputs(paths: readonly string[]): Promise { +/** + * Deletes `paths`, measuring each first. For a reduce-only checkout `guard` reads the executables on PATH anew after the + * measurement, immediately before each removal, and keeps a path whose checkout one resolves into, saying so in `kept`. + */ +export async function deleteOutputs(paths: readonly string[], launching: Launching, kept: string[]): Promise { let freed = 0; for (const path of paths) { - freed += (await measuredDirectorySizeAsync(path, { timeoutMs: SIZE_TIMEOUT_MS })) ?? 0; + const bytes = (await measuredDirectorySizeAsync(path, { timeoutMs: SIZE_TIMEOUT_MS })) ?? 0; + const launcher = launching.reduceOnly ? launcherReason(launching.checkout, await launching.launchers()) : null; + if (launcher) { + kept.push(`kept ${path}: ${launcher}`); + continue; + } + freed += bytes; await rm(path, { recursive: true, force: true }); } return freed; @@ -306,8 +321,9 @@ async function enterIdleStage( const [kind, slot = 'default'] = key.split(':'); return outcome?.status === 'shut-down' && booted.has(`${kind}:${slot}`) ? [key] : []; }); - const outputs = await idleOutputs(path, config); - const freed = await deleteOutputs(outputs.paths); + const launching: Launching = { reduceOnly: false }; + const outputs = await idleOutputs(path, config, launching); + const freed = await deleteOutputs(outputs.paths, launching, outputs.kept); await recordStage(path, minutes, shutDown, freed); const idle = Math.floor(again.decision.idleMs / MINUTE_MS); const restore = revivable(path) ? 'stim revive' : 'stim ios or stim android'; diff --git a/packages/stim-cli/src/daemon/ladder.ts b/packages/stim-cli/src/daemon/ladder.ts index 5f66b3247..96c6f11ec 100644 --- a/packages/stim-cli/src/daemon/ladder.ts +++ b/packages/stim-cli/src/daemon/ladder.ts @@ -21,6 +21,7 @@ import { projectDeviceSlots } from '../devices/device-slots.ts'; import type { IosSimRecord } from '../devices/ios.ts'; import { readParked } from '../devices/sim-pool.ts'; import { restoreProblem } from '../workspace/lightened.ts'; +import { launchingOf, type Launching } from '../workspace/launchers.ts'; import { formatBytes, measuredDirectorySizeAsync } from '../fs-util.ts'; import { executeAction, type ActContext, type ActionOutcome } from '../stimd/passes/act.ts'; import { artifactWorktreesFor } from '../stimd/passes/artifacts.ts'; @@ -151,8 +152,8 @@ function recordAction( } /** The bytes of the regenerable build outputs the idle stage would delete in a workspace. */ -export async function idleOutputBytes(path: string, config: Config): Promise { - return pathsBytes((await idleOutputs(path, config)).paths); +export async function idleOutputBytes(path: string, config: Config, launching: Launching): Promise { + return pathsBytes((await idleOutputs(path, config, launching)).paths); } async function pathsBytes(paths: readonly string[]): Promise { @@ -274,7 +275,13 @@ async function idleOutputsRung(context: RungContext): Promise { ? await clearStoppedOutputs(candidate, context.config, PRESSURE_IDLE_MS, context.operation) : { outcome: 'would' as const, - bytes: context.measure ? await idleOutputBytes(candidate.path, context.config) : 0, + bytes: context.measure + ? await idleOutputBytes( + candidate.path, + context.config, + launchingOf(candidate.reduceOnly, candidate.path, context.operation), + ) + : 0, detail: wouldDetail(context), }; if (!result) continue; diff --git a/packages/stim-cli/src/daemon/passes.ts b/packages/stim-cli/src/daemon/passes.ts index 8382ff2e6..36a1477f8 100644 --- a/packages/stim-cli/src/daemon/passes.ts +++ b/packages/stim-cli/src/daemon/passes.ts @@ -12,6 +12,7 @@ import { import { processContext } from '../cli-operation.ts'; import { runMaintenance } from '../stimd/passes/run.ts'; import { createNdjsonWriter } from '../ndjson.ts'; +import { launchingOf } from '../workspace/launchers.ts'; import { loadConfig, type Config } from '../workspace/config.ts'; import { requestDaemonTick, stimdRefused } from './client.ts'; import { runBootedParkedPass } from './booted-parked.ts'; @@ -457,7 +458,9 @@ export async function runForcedLadder({ target: entry.path, workspace: entry.path, outcome: entry.outcome === 'idle' ? 'done' : entry.outcome === 'would-idle' ? 'would' : 'failed', - bytes: entry.freedBytes ?? (entry.outcome === 'would-idle' ? await idleOutputBytes(entry.path, config) : 0), + bytes: + entry.freedBytes ?? + (entry.outcome === 'would-idle' ? await idleOutputBytes(entry.path, config, { reduceOnly: false }) : 0), detail: entry.outcome === 'would-idle' ? 'stim gc would do this' : entry.detail, ...(entry.outcome === 'idle' ? { restore: 'stim revive' } : {}), }); @@ -527,7 +530,11 @@ export async function runDormantReduction({ ? await clearStoppedOutputs(candidate, config, afterMs, operation) : { outcome: 'would' as const, - bytes: await idleOutputBytes(candidate.path, config), + bytes: await idleOutputBytes( + candidate.path, + config, + launchingOf(candidate.reduceOnly, candidate.path, operation), + ), detail: 'stim gc would do this', }; if (result) diff --git a/packages/stim-cli/src/daemon/stages.ts b/packages/stim-cli/src/daemon/stages.ts index f52d5b6f6..ee90658c7 100644 --- a/packages/stim-cli/src/daemon/stages.ts +++ b/packages/stim-cli/src/daemon/stages.ts @@ -39,6 +39,7 @@ import { getProject, loadConfig, type Config, type ProjectRecord } from '../work import { checkoutRoleOf } from '../workspace/checkout-role.ts'; import { discoverCheckouts } from '../workspace/discovery.ts'; import { discoveredEvidence } from '../workspace/discovered-activity.ts'; +import { launchingOf } from '../workspace/launchers.ts'; import { discoveredReads, type DiscoveredReads } from '../workspace/discovered-reads.ts'; import { listedRoots } from '../commands/gc/worktrees.ts'; import { withIdleWorkspace } from '../workspace/in-use.ts'; @@ -64,6 +65,8 @@ export interface StageCandidate { idleMs: number; /** A source checkout is taken after every linked worktree. */ role: CheckoutRole; + /** Found through a tool folder or the full scan; a PATH launcher into it keeps its outputs. */ + reduceOnly: boolean; } /** `idle.dormantDays` in milliseconds; 0 when the dormant stage is off. */ @@ -145,9 +148,17 @@ async function idleVerdict( { locked = false, operation, - reads = discoveredReads(operation?.signal), + reads = discoveredReads(operation?.signal, operation), full = false, - }: { locked?: boolean; operation?: StimContext; reads?: DiscoveredReads; full?: boolean } = {}, + reduceOnly, + }: { + locked?: boolean; + operation?: StimContext; + reads?: DiscoveredReads; + full?: boolean; + /** Found through a tool folder or the full scan: a PATH launcher into the checkout keeps it. Every caller states it. */ + reduceOnly: boolean; + }, ): Promise<{ idleMs: number } | { kept: string }> { const now = Date.now(); const pressured = setting === PRESSURE_THRESHOLD; @@ -155,7 +166,7 @@ async function idleVerdict( const evidence = !full && (getProject(path) || existsSync(workspaceMetadataFile(path))) ? stim - : mergeEvidence(stim, await discoveredEvidence(path, { now, idleMs: thresholdMs, reads, operation })); + : mergeEvidence(stim, await discoveredEvidence(path, { now, idleMs: thresholdMs, reads, operation, reduceOnly })); const decision = decideIdleStop(evidence, { idleMs: thresholdMs, now, setting, pressured }); if (!decision.stop) return { kept: decision.reason }; try { @@ -174,7 +185,7 @@ async function idleFor( path: string, thresholdMs: number, setting: string, - options: Parameters[3] = {}, + options: Parameters[3], ): Promise { const verdict = await idleVerdict(path, thresholdMs, setting, options); return 'idleMs' in verdict ? verdict.idleMs : null; @@ -189,21 +200,22 @@ export function sourceCheckoutIdle( path: string, thresholdMs: number, setting: string, - operation?: StimContext, - locked = false, + { operation, locked = false, reduceOnly }: { operation?: StimContext; locked?: boolean; reduceOnly: boolean }, ): Promise<{ idleMs: number } | { kept: string }> { - return idleVerdict(path, thresholdMs, setting, { operation, full: true, locked }); + return idleVerdict(path, thresholdMs, setting, { operation, full: true, locked, reduceOnly }); } /** Why a discovered checkout is not idle `thresholdMs` by the activity definition, or null when it is. */ export async function discoveredKeepReason( path: string, thresholdMs: number, + reduceOnly: boolean, operation?: StimContext, reads?: DiscoveredReads, ): Promise { const verdict = await idleVerdict(path, thresholdMs, 'idle.dormantDays', { operation, + reduceOnly, ...(reads ? { reads } : {}), }); return 'kept' in verdict ? verdict.kept : null; @@ -230,13 +242,10 @@ async function idleWorkspaces( { discover = false, full = false, onKept }: Omit & { discover?: boolean } = {}, ): Promise { const readings: (Omit & { idleMs: number | null })[] = []; - const reads = discoveredReads(reading.operation?.signal); - const discovered = discover - ? (await discoverCheckouts(config, listedRoots(), reading.operation)).map(({ path }): [string, ProjectRecord] => [ - path, - {}, - ]) - : []; + const reads = discoveredReads(reading.operation?.signal, reading.operation); + const found = discover ? await discoverCheckouts(config, listedRoots(), reading.operation) : []; + const discovered = found.map(({ path }): [string, ProjectRecord] => [path, {}]); + const reduceOnly = new Set(found.filter((entry) => entry.reduceOnly).map(({ path }) => path)); for (const [path, project] of [...Object.entries(config.projects), ...discovered]) { if (sameRoot(path, reading.protectedRoot) || !(await worthIt(path, project))) continue; const used = Date.parse(String(readWorkspaceState(path)?.lastUsedAt ?? '')); @@ -248,11 +257,16 @@ async function idleWorkspaces( let idleMs: number | null = null; if (await runsSomething(path, project, reading)) onKept?.(path, 'something of it runs'); else { - const verdict = await idleVerdict(path, thresholdMs, setting, { operation: reading.operation, reads, full }); + const verdict = await idleVerdict(path, thresholdMs, setting, { + operation: reading.operation, + reads, + full, + reduceOnly: reduceOnly.has(path), + }); if ('idleMs' in verdict) idleMs = verdict.idleMs; else onKept?.(path, verdict.kept); } - readings.push({ path, project, idleMs }); + readings.push({ path, project, idleMs, reduceOnly: reduceOnly.has(path) }); } catch (error) { if (isClaimUnavailable(error) && 'reportCleanup' in error) throw error; reading.operation?.signal.throwIfAborted(); @@ -277,12 +291,16 @@ async function hasOutputs(path: string): Promise { export async function stillIdle( path: string, thresholdMs: number, - operation?: StimContext, - full = false, - locked = true, - setting: string = PRESSURE_THRESHOLD, + { + operation, + full = false, + locked = true, + setting = PRESSURE_THRESHOLD, + reduceOnly, + }: { operation?: StimContext; full?: boolean; locked?: boolean; setting?: string; reduceOnly: boolean }, ): Promise { - return (await idleFor(path, thresholdMs, setting, { locked, operation, full })) !== null; + const reads = discoveredReads(operation?.signal, operation, true); + return (await idleFor(path, thresholdMs, setting, { locked, operation, full, reads, reduceOnly })) !== null; } /** Workspaces with nothing running, idle `thresholdMs` by the activity definition: linked worktrees, then source checkouts, longest-idle first in each. */ @@ -326,11 +344,19 @@ export async function clearStoppedOutputs( workspaceDir(path), 'metro-start', async (): Promise => { - if ((await idleFor(path, thresholdMs, PRESSURE_THRESHOLD, { locked: true, operation })) === null) + if ( + (await idleFor(path, thresholdMs, PRESSURE_THRESHOLD, { + locked: true, + operation, + reads: discoveredReads(operation?.signal, operation, true), + reduceOnly: candidate.reduceOnly, + })) === null + ) return { outcome: 'kept', bytes: 0, detail: 'it is no longer idle' }; - const outputs = await idleOutputs(path, config); + const launching = launchingOf(candidate.reduceOnly, path, operation); + const outputs = await idleOutputs(path, config, launching); if (!outputs.paths.length) return null; - const bytes = await deleteOutputs(outputs.paths); + const bytes = await deleteOutputs(outputs.paths, launching, outputs.kept); return { outcome: 'done', bytes, @@ -479,7 +505,11 @@ export async function enterDormantStage(action: MaintenanceAction, operation?: S const run = await withIdleWorkspace( path, async (): Promise => { - const idleMs = await idleFor(path, thresholdMs, 'idle.dormantDays', { locked: true, operation }); + const idleMs = await idleFor(path, thresholdMs, 'idle.dormantDays', { + locked: true, + operation, + reduceOnly: false, + }); if (idleMs === null) return { status: 'kept', bytes: 0, detail: 'it is no longer idle' }; const owns = (kind: 'ios' | 'android') => projectDeviceSlots(project).some(({ platforms }) => platforms[kind]?.owned === true); diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 9c19c2221..595a65299 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -28,7 +28,8 @@ STIM_HOME home runs it like any other. The stim-daemon runs a pass in its own process on each tick when a check is due: a disk and memory reading for status (every minute), hosted idle stops (every 5 minutes), directory sizes (hourly), finished worktrees (every 15 -minutes) and the age sweep (daily). The reading plans nothing, the idle check +minutes) and the age sweep (daily); the daemon also rescans the home folder for +checkouts after the pass, weekly and also at low disk (DISCOVERED CHECKOUTS). The reading plans nothing, the idle check plans only idle stops, and the cap trims follow the size, worktree and sweep checks. Pressure itself is the daemon's: each tick classifies the machine normal, high or critical and runs the pressure ladder @@ -1208,12 +1209,80 @@ or a maintenance.discoverRoots folder; every other worktree (another tool's home, a pin checkout) is listed by stim gc --dry-run as outside the folders Stim manages and gets no action of any kind. Nothing under the system temporary directory or a dot-folder directly inside the home directory is ever touched, -except inside a maintenance.discoverRoots folder, which is always considered. +except inside a maintenance.discoverRoots folder, which is always considered, +and the ~/.codex/worktrees worktrees below, which are only reduced. A worktree nested in a registered root is left to that workspace. The home directory and / are never a folder that holds known projects. A scoped STIM_HOME discovers only from its own maintenance.discoverRoots. Discovery registers nothing, writes nothing and recomputes each run; no git fetch runs in a discovered repository and pull requests come from the cache. +Two more sources find checkouts that Stim only reduces and never removes +(reduce-only): linked worktrees of known repositories in ~/.codex/worktrees, +and the checkouts of the full scan. The scan is a pruned walk of the home +folder, run by the daemon after a maintenance pass has acted (never before its +cleanup, never by a command): weekly, and also while the disk level is above +normal when no scan has run for an hour; a scan that failed or ended on a read that timed out is retried after an hour, +one that stopped at its folder cap or time budget is not. +The schedule is kept with the result, so a restarted daemon forgets nothing. It +reads ~/Documents (macOS may ask once; a folder that does not answer within 3 +seconds, such as one waiting for that answer, is recorded as unreadable, ends +that scan early with the previous finds kept, and is read on the next scan), skips ~/Desktop, ~/Downloads, ~/Library, the media +folders, other volumes, symlinks, dependency or build folders, deeper +dot-folders, the Stim home and every dot-folder starting with .stim, the build +worker root, and the tool homes and caches among the home folder's other +dot-folders (.npm, .gradle, .cache and the like; .exact is entered), stops at a +recognised project and goes seven levels deep. The scan has a 45 second budget, +checked before every read it makes (including every read of the project checks +below); when it runs out the scan ends truncated as a budget stop and keeps the +previous finds. It can overrun the budget by at most one read's 3 second timeout, +plus the time project detection itself takes on the folder it is recognising, +which is synchronous and cannot be interrupted. Revalidation of cached finds has +no total budget: each read is bounded by the 3 second timeout, and the number of +reads by detection's own caps (50 children per container, the home folder and up to +seven below it). Before project +detection runs on a folder, the manifests detection reads are proved safe, for +the folder, each ancestor down from the home folder, its apps, examples and +packages folders and their children: a manifest (any case) that is not a regular file, an +ios or android entry that is not a real directory, a symlinked apps, examples or +packages folder, or an Xcode container with no app dependency in package.json, +makes that subtree not a scan checkout, and a manifest or file metadata read that does not answer in 3 +seconds ends the scan (cached finds stay unselected, not dropped). Not bounded: +the ancestors above the home folder, a file that becomes unreadable between that +check and detection, the ios and android directories detection resolves without +the walk entering them (checked to be real directories, not that they answer, so +a stalled mount point there is not detected), and what a pass reads inside a checkout once discovered. The result is +kept in maintenance/discovery-scan.json under the Stim home (when it ran, what +it found, what it could not read) and discovery reads that file, so a pass or gc +never walks the disk. A scan that fails or stops early keeps the previous finds, +and a finished one keeps those under a folder it could not read, which includes a +folder whose project check was denied by the system (it is listed unreadable, not +rejected). Discovery checks +each recorded checkout again before using it: still a real folder (not a +symlink), on the home volume, below no skipped folder and still a project; +anything else is dropped. Checkouts the scan finds under a dot-folder of the +home folder are recorded in the result and never acted on, because an installed +tool cannot be told from a checkout there; maintenance.discoverRoots is the way +to manage one. A reduce-only checkout that an executable on PATH (a symlink in a +PATH directory, ~/.local/bin, ~/bin, ~/.cargo/bin, ~/.bun/bin, ~/.volta/bin, +~/Library/pnpm, the active nvm bin, /usr/local/bin or /opt/homebrew/bin) +resolves into is kept whole, by every pass that deletes its outputs, +dependencies or Cargo target. A directory that cannot be read (not one that is +missing, or one that does not answer within 3 seconds) keeps every reduce-only +checkout for that pass, and the reason names it and says fixing its permissions +restores reduction. Each deletion reads the directories anew, at the moment it +reads the processes using the path (not the pass's planning reading); a +directory that hangs ends the reading as incomplete at once, and while that read +is pending (at most one per thread: the daemon's main thread and passes thread +each have one; the scan runs on the passes thread) no other launcher or scan +read starts and a scheduled scan is skipped. A scan stops at the first read that +times out, keeping the previous finds. Reduce-only checkouts have their build outputs, and +their node_modules and Pods (a linked worktree's after idle.dormantDays, a source +checkout's after idle.sourceDormantDays), reduced by the rules below. Every one is +evaluated on every pass, like a sibling-discovered checkout; the +finished-worktree check inspects every discovered worktree with git (about 0.14 +seconds each where measured). A +folder named in maintenance.discoverRoots +keeps the full rules. A scoped STIM_HOME uses neither source. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine Idle, with no Stim activity record, means the newest of its source file writes (git-ignored files included; node_modules, Pods and nested worktrees skipped, @@ -1221,8 +1290,13 @@ and build, dist, target, .cache and similar folders skipped only where git ignores them), its last commit and its git writes, for idle.dormantDays. A process running from or working in it, an agent session in it, a pin by maintenance.keep, more source files than the scan reads, or evidence that -cannot be read keeps it. Its regenerable build outputs go when it is idle; a -linked one also loses node_modules and Pods, a main checkout keeps them. A +cannot be read keeps it. Its regenerable build outputs go when it is idle, and its +node_modules and Pods go too: a linked worktree's after idle.dormantDays (default +1 day), a main (source) checkout's after idle.sourceDormantDays (default 7 days; 0 +never), by the same evidence. That holds for every kind of discovered checkout +(children of the folders holding registered checkouts, children of +maintenance.discoverRoots, reduce-only finds) even where no stim command ever ran, +and a PATH launcher keeps a reduce-only one. A target directory at a checkout root whose CACHEDIR.TAG starts with the standard CACHEDIR.TAG signature line and whose files git ignores counts as build output for every checkout. @@ -1239,8 +1313,8 @@ Any change to a tracked file, even Podfile.lock or project.pbxproj, is dirty work and keeps it, as does any tracked file marked assume-unchanged or skip-worktree, which hides edits from git status. A worktree inside a dot-folder of its main checkout other than .worktrees (.claude/worktrees) is -managed by another tool and is never removed, though its outputs are reduced. -Dirty work is never archived. +managed by another tool and is never removed, though its outputs are reduced; +so is a reduce-only checkout (above). Dirty work is never archived. stim gc --dry-run lists the discovered checkouts and what each would lose, or why it is kept; one over the scan bound is listed under Needs a person. diff --git a/packages/stim-cli/src/stimd/passes/artifacts.ts b/packages/stim-cli/src/stimd/passes/artifacts.ts index f60f86c4e..4c63ccdaf 100644 --- a/packages/stim-cli/src/stimd/passes/artifacts.ts +++ b/packages/stim-cli/src/stimd/passes/artifacts.ts @@ -20,6 +20,7 @@ import { import { reclaimKeys } from '../../commands/worktree.ts'; import { loadConfig } from '../../workspace/config.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; +import { freshLaunchers, launcherReason, launchersOf, launchingOf } from '../../workspace/launchers.ts'; import { gitQuiet, gitQuietAsync } from '../../workspace/git.ts'; import { childEnvironment } from '../../workspace/git-read.ts'; import { withIdleWorkspace, workspaceInUse, workspaceInUseAsync } from '../../workspace/in-use.ts'; @@ -67,6 +68,8 @@ export interface ArtifactWorktree { keys: string[]; /** Why a running agent session keeps the worktree, read when the worktrees were listed. */ agents?: string[]; + /** Found through a tool folder or the full scan, where an installed tool cannot be told from a checkout. */ + reduceOnly: boolean; } export interface ArtifactCandidate { @@ -77,6 +80,7 @@ export interface ArtifactCandidate { agents?: string[]; /** In a source checkout, whose dependencies no worktree warm restores. */ source?: boolean; + reduceOnly: boolean; } function isDirectory(path: string): boolean { @@ -104,7 +108,7 @@ function registeredRoots(roots: string[]): string[] { export function artifactWorktrees(): ArtifactWorktree[] { const found = new Map(); for (const root of registeredRoots(candidateRoots())) { - const entry = artifactWorktree(root, listWorktrees(root)); + const entry = artifactWorktree(root, listWorktrees(root), false); if (entry) found.set(entry.path, entry); } return [...found.values()]; @@ -113,6 +117,7 @@ export function artifactWorktrees(): ArtifactWorktree[] { function artifactWorktree( root: string, entries: WorktreeEntry[], + reduceOnly: boolean, source: SourceCheckout = sourceCheckoutOf(entries), ): ArtifactWorktree | null { const entry = matchWorktreeEntry(entries, root); @@ -124,6 +129,7 @@ function artifactWorktree( linked: entry.path !== source.path, locked: Boolean(entry.locked), keys, + reduceOnly, }; } @@ -141,6 +147,7 @@ export async function artifactCheckoutsAsync( const found = new Map(); const discovered = await discoverCheckouts(loadConfig(), listedRoots(), operation); const known = new Set([...listedRoots(), ...discovered.map(({ path }) => path)]); + const reduceOnly = new Set(discovered.filter((entry) => entry.reduceOnly).map(({ path }) => path)); for (const root of (await candidateRootsAsync(operation, discovered)).filter((entry) => known.has(entry))) { signal.throwIfAborted(); const { entries, ...source } = await resolveSourceCheckoutAsync(root, { @@ -148,8 +155,8 @@ export async function artifactCheckoutsAsync( signal, declareFor: operation, }); - const entry = artifactWorktree(root, entries, source); - if (entry) found.set(entry.path, entry); + const entry = artifactWorktree(root, entries, reduceOnly.has(root), source); + if (entry) found.set(entry.path, found.get(entry.path)?.reduceOnly ? { ...entry, reduceOnly: true } : entry); } return [...found.values()]; } @@ -252,13 +259,16 @@ type ArtifactReads = { tracked: string | null; activity: ReturnType; written: number | null; + /** Why an executable on PATH keeps a reduce-only checkout, or null. */ + launcher: string | null; }; async function artifactReads( candidate: ArtifactCandidate, kind: ArtifactKind, signal: AbortSignal, - operation?: StimContext, + operation: StimContext | undefined, + { fresh }: { fresh: boolean }, ): Promise { const { worktree, path, keys } = candidate; const tracked = await gitQuietAsync(worktree, ['ls-files', '--', relative(worktree, path)], { @@ -271,7 +281,10 @@ async function artifactReads( const windowed = tracked !== null && !tracked.trim() && kind === 'dependencies'; const activity = windowed ? await worktreeActivityOfAsync(worktree, keys, operation) : null; const written = windowed ? await newestWriteAsync(path, SCAN_DEPTH[kind], signal) : null; - return { tracked, activity, written }; + const launcher = candidate.reduceOnly + ? launcherReason(worktree, await (fresh ? freshLaunchers : launchersOf)(operation, signal)()) + : null; + return { tracked, activity, written, launcher }; } async function pinsOf( @@ -340,6 +353,7 @@ function keptReason( ): string | null { const guarded = artifactGuard(candidate, kind, pinned, locked, running, inUse); if (guarded) return guarded; + if (reads?.launcher) return reads.launcher; const { worktree, path, keys } = candidate; const tracked = reads ? reads.tracked : gitQuiet(worktree, ['ls-files', '--', relative(worktree, path)]); if (tracked === null) return 'git cannot tell whether it holds tracked files'; @@ -448,6 +462,7 @@ export async function collectArtifactsAsync( keys: worktree.keys, locked: worktree.locked, agents: worktree.agents, + reduceOnly: worktree.reduceOnly, ...(options.source ? { source: true } : {}), }; const inUse = protectedHere ? null : await inUseOf(candidate, kind, signal, operation); @@ -460,10 +475,13 @@ export async function collectArtifactsAsync( candidate, kind, { ...options, pinned, running, ...(inUse ? { inUse } : {}) }, - await artifactReads(candidate, kind, signal, operation), + await artifactReads(candidate, kind, signal, operation, { fresh: false }), ); if (!reason && candidate.source) - reason = (await regenerableDirProblem(worktree.path, path, signal)) ?? linkedWorktreeUsing(candidate, all); + reason = + (await regenerableDirProblem(worktree.path, path, signal, { + launching: launchingOf(worktree.reduceOnly, worktree.path, operation, signal), + })) ?? linkedWorktreeUsing(candidate, all); if (reason) skips.push({ target: path, reason }); else candidates.push(candidate); } @@ -562,7 +580,13 @@ export function collectArtifacts( } catch { continue; } - const candidate = { worktree: worktree.path, path, keys: worktree.keys, locked: worktree.locked }; + const candidate = { + worktree: worktree.path, + path, + keys: worktree.keys, + locked: worktree.locked, + reduceOnly: worktree.reduceOnly, + }; const reason = protectedHere ? 'the command that started this pass runs inside its worktree' : keptReason(candidate, kind, { ...options, running }); @@ -608,6 +632,7 @@ export async function clearArtifact( keys: worktree.keys, locked: worktree.locked, agents, + reduceOnly: worktree.reduceOnly, }; const running = kind === 'dependencies' ? await readRunningProcessesAsync() : null; const pinned = await pinsOf(candidate.keys, options.pinned); @@ -615,7 +640,7 @@ export async function clearArtifact( candidate, kind, { ...options, pinned, now: Date.now(), running, inUse: await inUseOf(candidate, kind, signal, operation) }, - await artifactReads(candidate, kind, signal, operation), + await artifactReads(candidate, kind, signal, operation, { fresh: true }), ); if (early) return { status: 'kept', bytes: 0, detail: early }; const unrestorable = @@ -637,7 +662,7 @@ export async function clearArtifact( inUse: await inUseOf(candidate, kind, signal, operation, key), ...(key ? { locked: key } : {}), }, - await artifactReads(candidate, kind, signal, operation), + await artifactReads(candidate, kind, signal, operation, { fresh: true }), ); if (reason) return reason; const late = key && options.recheck ? await options.recheck(candidate.keys, candidate.worktree) : null; @@ -733,6 +758,7 @@ export async function clearSourceTrees( locked: worktree.locked, agents: (await agentSessionUse([worktree.path], Date.now(), signal))[0] ?? [], source: true, + reduceOnly: worktree.reduceOnly, }; const checkTree = async (path: string, held?: readonly string[]): Promise => { const candidate = { ...base, path, ...(held ? { source: false } : {}) }; @@ -749,9 +775,11 @@ export async function clearSourceTrees( inUse: await inUseOf(candidate, 'dependencies', signal, operation, held), ...(held ? { locked: held } : {}), }, - await artifactReads(candidate, 'dependencies', signal, operation), + await artifactReads(candidate, 'dependencies', signal, operation, { fresh: true }), ) ?? - (await regenerableDirProblem(worktree.path, path, signal)) ?? + (await regenerableDirProblem(worktree.path, path, signal, { + launching: launchingOf(worktree.reduceOnly, worktree.path, operation, signal), + })) ?? linkedWorktreeUsing(candidate, options.worktrees) ); }; diff --git a/packages/stim-cli/src/stimd/passes/cargo-target.ts b/packages/stim-cli/src/stimd/passes/cargo-target.ts index e03282156..f118bda1a 100644 --- a/packages/stim-cli/src/stimd/passes/cargo-target.ts +++ b/packages/stim-cli/src/stimd/passes/cargo-target.ts @@ -9,6 +9,7 @@ import { measuredDirectorySizeAsync } from '../../fs-util.ts'; import { canonicalPath } from '../../commands/gc/paths.ts'; import { listedRoots } from '../../commands/gc/worktrees.ts'; import { discoverCheckouts } from '../../workspace/discovery.ts'; +import { launchingOf, planningLaunching } from '../../workspace/launchers.ts'; import { cargoTargetDirsAsync, isBuildDirPath, regenerableDirProblem } from '../../workspace/regenerable-dirs.ts'; import { appRoots, artifactCheckoutsAsync } from './artifacts.ts'; import { readVolumeSpaceAsync } from './budget.ts'; @@ -32,6 +33,10 @@ export interface CargoTargetCandidate { dir: string; identity: string; discovered: boolean; + /** Found through a tool folder or the full scan: a PATH launcher into the checkout keeps its target. */ + reduceOnly: boolean; + /** The checkout that owns the target, which a PATH launcher can point anywhere into; `root` is the app inside it. */ + checkout: string; measuredAt: number | null; } @@ -94,23 +99,32 @@ async function cargoTargetCandidates( ): Promise { const signal = operation?.signal ?? new AbortController().signal; const found: CargoTargetCandidate[] = []; - const add = async (base: string, discovered: boolean) => { + const add = async (base: string, discovered: boolean, reduceOnly: boolean, checkout: string) => { const root = canonicalPath(base); for (const path of await cargoTargetDirsAsync(root)) { try { await lstat(path); // Windows 8.3 aliases need native realpath. If it fails, keep the path for the existing ownership checks. const identity = process.platform === 'win32' ? await realpath(path).catch(() => path) : path; - found.push({ root, dir: path, identity, discovered, measuredAt: lastMeasured.get(path) ?? null }); + found.push({ + root, + dir: path, + identity, + discovered, + reduceOnly, + checkout, + measuredAt: lastMeasured.get(path) ?? null, + }); } catch {} } }; - for (const root of registered) await add(root, false); + for (const root of registered) await add(root, false, false, root); const discovered = new Set( (await discoverCheckouts(loadConfig(), listedRoots(), operation)).map(({ path }) => canonicalPath(path)), ); for (const checkout of await artifactCheckoutsAsync(signal, operation)) - for (const base of appRoots(checkout, 'build')) await add(base, discovered.has(canonicalPath(checkout.path))); + for (const base of appRoots(checkout, 'build')) + await add(base, discovered.has(canonicalPath(checkout.path)), checkout.reduceOnly, checkout.path); return found; } @@ -137,9 +151,12 @@ export async function measureCargoTargets( now, ); scan.unlisted = unlisted.map(({ dir }) => dir); - for (const { root, dir } of measured) { + for (const { root, dir, reduceOnly, checkout } of measured) { lastMeasured.set(dir, now); - const problem = await regenerableDirProblem(root, dir, operation?.signal, { processes: false }); + const problem = await regenerableDirProblem(root, dir, operation?.signal, { + processes: false, + launching: planningLaunching(reduceOnly, checkout, operation), + }); if (problem) { scan.refused.set(dir, { workspace: root, reason: problem }); continue; @@ -343,11 +360,14 @@ export async function planCargoTargetsAtPressure({ if (settings.cargoTargetMaxGb === 0) return plan; const registered = Object.keys(loadConfig()?.projects ?? {}); const { measured } = selectCargoTargets(await cargoTargetCandidates(registered, operation), Infinity, now); - for (const { root: workspace, dir } of measured) { + for (const { root: workspace, dir, reduceOnly, checkout } of measured) { signal?.throwIfAborted(); const tightened = level ?? (await diskLevel(dir, signal)); if (tightened === 'normal') continue; - const problem = await regenerableDirProblem(workspace, dir, signal, { processes: false }); + const problem = await regenerableDirProblem(workspace, dir, signal, { + processes: false, + launching: planningLaunching(reduceOnly, checkout, operation, signal), + }); if (problem) { plan.skips.push({ target: dir, reason: problem, workspace }); continue; @@ -383,13 +403,18 @@ export async function planCargoTargetsAtPressure({ return plan; } -async function stillManaged(root: string, operation?: StimContext): Promise { +async function managedAs( + root: string, + operation?: StimContext, +): Promise<{ reduceOnly: boolean; checkout: string } | null> { const canonical = canonicalPath(root); - if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) return true; + if (listedRoots().some((listed) => canonicalPath(listed) === canonical)) + return { reduceOnly: false, checkout: canonical }; const signal = operation?.signal ?? new AbortController().signal; - return (await artifactCheckoutsAsync(signal, operation)).some((checkout) => - appRoots(checkout, 'build').some((base) => canonicalPath(base) === canonical), + const checkout = (await artifactCheckoutsAsync(signal, operation)).find((entry) => + appRoots(entry, 'build').some((base) => canonicalPath(base) === canonical), ); + return checkout ? { reduceOnly: checkout.reduceOnly, checkout: checkout.path } : null; } async function targetVolume(target: string, signal?: AbortSignal): Promise<{ freeMb: number; device: number } | null> { @@ -407,9 +432,13 @@ export async function trimCargoTargetAction( ): Promise { const root = action.workspace; if (!root || !isBuildDirPath(action.target)) return kept('it is no longer a Cargo target/ Stim recognises'); - if (!(await stillManaged(root, operation))) - return kept('its checkout is no longer in a workspace or discovery scope'); - const problem = await regenerableDirProblem(root, action.target, operation?.signal, { processes: false }); + const managed = await managedAs(root, operation); + if (!managed) return kept('its checkout is no longer in a workspace or discovery scope'); + const launching = launchingOf(managed.reduceOnly, managed.checkout, operation); + const problem = await regenerableDirProblem(root, action.target, operation?.signal, { + processes: false, + launching, + }); if (problem) return kept(problem); const measure = action.level === undefined; const before = measure ? null : await targetVolume(action.target, operation?.signal); @@ -420,6 +449,7 @@ export async function trimCargoTargetAction( signal: operation?.signal, measure, ...(stop ? { stop } : {}), + launching, }); if (!measure) { const after = before ? await targetVolume(action.target, operation?.signal) : null; diff --git a/packages/stim-cli/src/stimd/passes/level-run.ts b/packages/stim-cli/src/stimd/passes/level-run.ts index f21e3e8fb..7dad378d3 100644 --- a/packages/stim-cli/src/stimd/passes/level-run.ts +++ b/packages/stim-cli/src/stimd/passes/level-run.ts @@ -240,7 +240,7 @@ export async function runMaintenanceAtLevel({ for (const action of ladder?.levels.flatMap((entry) => entry.actions) ?? []) take(fromLadder(action)); } const idleMs = level === 'normal' ? dormantAfterMs(loadConfig(), env) : PRESSURE_IDLE_MS; - const reads = discoveredReads(operation?.signal); + const reads = discoveredReads(operation?.signal, operation); const discovered: LevelRun['discovered'] = []; const ownerOf = (path: string) => found @@ -252,7 +252,7 @@ export async function runMaintenanceAtLevel({ const kept = taken || idleMs <= 0 || !existsSync(checkout.path) ? null - : await discoveredKeepReason(checkout.path, idleMs, operation, reads); + : await discoveredKeepReason(checkout.path, idleMs, checkout.reduceOnly === true, operation, reads); if (kept && /source entries to check|cannot be read/.test(kept)) blocked.push(`Discovered checkout ${checkout.path}: ${kept}, so it is never reduced`); discovered.push({ ...checkout, actions: mine, ...(kept ? { kept } : {}) }); diff --git a/packages/stim-cli/src/stimd/passes/run.ts b/packages/stim-cli/src/stimd/passes/run.ts index 145de9059..227bd130e 100644 --- a/packages/stim-cli/src/stimd/passes/run.ts +++ b/packages/stim-cli/src/stimd/passes/run.ts @@ -38,6 +38,7 @@ import { executeAction, type ActionOutcome } from './act.ts'; import { plannedMaintenance, triggeringRoot, type PlannedMaintenance } from './preview.ts'; import { maintenanceLogger } from './log.ts'; import { executedActionName } from '../../maintenance-kind.ts'; +import { runDiscoveryScanIfDue } from '../../workspace/discovery-scan.ts'; import { maintenanceItem, openGcLog, type GcLog } from '../../commands/gc/log.ts'; function writeState(state: MaintenanceState): void { @@ -407,12 +408,15 @@ export async function runMaintenance( crashLine(error); return; } + let scanAfter = false; + let passFailed = false; let logger: ReturnType | undefined; let gcLog: GcLog | undefined; try { const resolved = resolveMaintenanceSettings(); const settings = report && resolved.mode === 'on' ? { ...resolved, mode: 'report' as const } : resolved; if (settings.mode === 'off') return; + scanAfter = true; const state: MaintenanceState = readMaintenanceState() ?? { version: 1, lastAt: {}, @@ -729,6 +733,7 @@ export async function runMaintenance( } writeState(state); } catch (error) { + passFailed = true; throwUnverifiedCleanup(error); const message = error instanceof Error ? error.message : String(error); try { @@ -742,9 +747,21 @@ export async function runMaintenance( logger?.close(); keepReserve(); releaseClaim(claim); + if (scanAfter) await scanAfterPass(tick, passFailed); } } +/** + * The discovery scan, after the pass has released its claim. Not after a failed pass or a cancellation, and nothing it + * throws can reach the caller: the pass's own error, such as an unverified child cleanup, is the one that must surface. + */ +async function scanAfterPass(tick: MaintenanceTick | undefined, passFailed: boolean): Promise { + if (passFailed || tick?.operation.signal.aborted) return; + try { + await runDiscoveryScanIfDue(tick); + } catch {} +} + function crashLine(error: unknown): void { try { mkdirSync(maintenanceDir(), { recursive: true }); diff --git a/packages/stim-cli/src/workspace/bounded-read.ts b/packages/stim-cli/src/workspace/bounded-read.ts new file mode 100644 index 000000000..abcdfc4d3 --- /dev/null +++ b/packages/stim-cli/src/workspace/bounded-read.ts @@ -0,0 +1,44 @@ +/** + * The one filesystem read, of the launcher directories or the home scan, that timed out and has not settled. A timed-out + * native read cannot be cancelled, so while one is pending no further read of either kind starts: a hung mount holds + * exactly one libuv thread however many readings and scans are asked for. Module state, so the bound is one outstanding + * read per JavaScript isolate: the daemon's main thread and its passes thread each have their own. + */ +let hung: { label: string } | null = null; + +export class Unanswered extends Error {} + +/** The folder whose read timed out and is still pending, or null when none is. */ +export function hungRead(): string | null { + return hung?.label ?? null; +} + +/** + * `start()`'s result, or an `Unanswered` rejection when it has not answered in `timeoutMs`; the read keeps running, and + * is the pending one until it settles. It starts no read at all while one is pending, so the slot is never overwritten. + */ +export async function answered(label: string, start: () => Promise, timeoutMs: number): Promise { + if (hung) throw new Unanswered(`a read of ${hung.label} is still pending`); + const read = start(); + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + read, + new Promise((_, fail) => { + timer = setTimeout(() => fail(new Unanswered('timed out')), timeoutMs); + }), + ]); + } catch (error) { + if (error instanceof Unanswered) { + const mine = { label }; + hung = mine; + const settled = () => { + if (hung === mine) hung = null; + }; + read.then(settled, settled); + } + throw error; + } finally { + clearTimeout(timer); + } +} diff --git a/packages/stim-cli/src/workspace/cargo-target.ts b/packages/stim-cli/src/workspace/cargo-target.ts index fb7046d4d..4b1f9e9a2 100644 --- a/packages/stim-cli/src/workspace/cargo-target.ts +++ b/packages/stim-cli/src/workspace/cargo-target.ts @@ -6,6 +6,7 @@ import { releaseClaim, tryAcquireClaim, type ClaimHandle } from '../ownership-cl import { getExecutor } from '../exec.ts'; import { measuredDirectorySizeAsync } from '../fs-util.ts'; import { withCargoLocks } from './cargo-lock.ts'; +import type { LauncherScan, Launching } from './launchers.ts'; import { cargoUnitPaths, cargoUnitWasRead, idleCargoUnits, readCargoUnits, removalOrder } from './cargo-units.ts'; const CARGO_PROFILE_IDLE_MS: number = 24 * 3_600_000; @@ -299,10 +300,16 @@ function inUse(used: readonly string[], dir: string): boolean { } /** - * The paths under `target` a running process executes from or holds open, from one `lsof` and one `ps` read; null when - * either cannot be read, which keeps everything. + * The paths under `target` a running process executes from or holds open, from one `lsof` and one `ps` read, and for a + * reduce-only checkout the paths under it that an executable on PATH resolves to, from a launcher reading taken at the + * same moment; null when any of them cannot be read, which keeps everything. */ -async function readTargetUse(target: string, signal?: AbortSignal, lowPriority = false): Promise { +async function readTargetUse( + target: string, + signal?: AbortSignal, + lowPriority = false, + launchers?: () => Promise, +): Promise { const exec = getExecutor(); try { const run = (file: string, args: string[], timeoutMs: number) => @@ -311,7 +318,13 @@ async function readTargetUse(target: string, signal?: AbortSignal, lowPriority = : exec.runFileAsync(file, args, { timeoutMs, signal }); const lsof = await run('lsof', ['-nP', '-w', '-Fn'], 60_000); const ps = await run('ps', ['-axww', '-o', 'args='], 30_000); - return targetPathsInUse(target, lsof, ps); + const used = targetPathsInUse(target, lsof, ps); + if (!launchers) return used; + const scan = await launchers(); + if (scan.incomplete.length) return null; + const prefix = `${folded(target)}/`; + for (const path of scan.targets) if (folded(path).startsWith(prefix)) used.push(folded(path)); + return used; } catch { signal?.throwIfAborted(); return null; @@ -385,6 +398,11 @@ interface CargoTrimOptions { stop?: AbortSignal; /** Whether to walk what was removed to report its bytes; false reports 0 and removes at once. Default true. */ measure?: boolean; + /** + * For a reduce-only checkout, a fresh reading of the executables on PATH, taken wherever the processes using the target + * are read, so a path one of them resolves to is kept like a path a process runs from. Every caller states it. + */ + launching: Launching; } async function trimClaimed( @@ -398,9 +416,14 @@ async function trimClaimed( signal, stop, measure = true, + launching, }: CargoTrimOptions, ): Promise { const result: CargoTrimResult = { bytes: 0, removed: [], kept: [] }; + const launchers = launching.reduceOnly ? launching.launchers : undefined; + const cannotRead = launchers + ? 'the files processes hold open or the executables on PATH cannot be read' + : 'the files processes hold open cannot be read'; const stopped = (): boolean => stop?.aborted === true; const measured = (path: string): Promise => measure ? sizeOf(path, signal, lowPriority) : Promise.resolve(0); @@ -419,8 +442,8 @@ async function trimClaimed( kept: [`a leftover ${aside} cannot be deleted: ${error instanceof Error ? error.message : error}`], }; } - const first = await readTargetUse(target, signal, lowPriority); - if (first === null) return { ...result, kept: ['the files processes hold open cannot be read'] }; + const first = await readTargetUse(target, signal, lowPriority, launchers); + if (first === null) return { ...result, kept: [cannotRead] }; let used: string[] = first; const stale = staleCargoTargetPaths(target, now); if ('refusal' in stale) return { ...result, kept: [stale.refusal] }; @@ -504,8 +527,8 @@ async function trimClaimed( if (!all) return { ...none, kept: 'its units cannot be listed' }; const candidates = [...cargoUnitPaths(profile, idleCargoUnits(all, now, CARGO_PROFILE_IDLE_MS))]; if (!candidates.length) return none; - const fresh = await readTargetUse(target, signal, lowPriority); - if (fresh === null) return { ...none, kept: 'the files processes hold open cannot be read' }; + const fresh = await readTargetUse(target, signal, lowPriority, launchers); + if (fresh === null) return { ...none, kept: cannotRead }; let swept = 0; let bytes = 0; // @ref LLP 0005#never-delete-what-a-build-took-back -- Exact serializes access-time resets and ledger replacement @@ -577,11 +600,11 @@ async function trimClaimed( const refreshed = stopped() ? null : incremental === 'all' || result.bytes < needBytes - ? await readTargetUse(target, signal, lowPriority) + ? await readTargetUse(target, signal, lowPriority, launchers) : used; if (stopped()) { if (!result.kept.includes(DEADLINE_KEPT)) result.kept.push(DEADLINE_KEPT); - } else if (refreshed === null) result.kept.push('the files processes hold open cannot be read'); + } else if (refreshed === null) result.kept.push(cannotRead); else used = refreshed; if (refreshed !== null && (incremental === 'all' || result.bytes < needBytes)) { const candidates: { path: string; bytes: number }[] = []; diff --git a/packages/stim-cli/src/workspace/discovered-activity.ts b/packages/stim-cli/src/workspace/discovered-activity.ts index 42c984cee..c1fc94c3c 100644 --- a/packages/stim-cli/src/workspace/discovered-activity.ts +++ b/packages/stim-cli/src/workspace/discovered-activity.ts @@ -5,6 +5,7 @@ import { worktreeActivityOfAsync } from '../commands/gc/worktrees.ts'; import type { IdleStopEvidence } from '../policy/idle.ts'; import { gitQuietAsync } from './git.ts'; import type { DiscoveredReads } from './discovered-reads.ts'; +import { launcherReason } from './launchers.ts'; import { runningFromReason } from './running-from.ts'; /** Folders holding dependencies or other checkouts, skipped wherever they are. */ @@ -110,11 +111,14 @@ export async function discoveredEvidence( idleMs, reads, operation, + reduceOnly, }: { now: number; idleMs: number; reads: DiscoveredReads; operation?: StimContext; + /** Found through a tool folder or the full scan, where an installed tool cannot be told from a checkout. */ + reduceOnly: boolean; }, ): Promise { const signal = operation?.signal; @@ -138,5 +142,7 @@ export async function discoveredEvidence( keep.push(...(await reads.sessions(path, now))); const running = runningFromReason(path, await reads.processes()); if (running) keep.push(running); + const launcher = reduceOnly ? launcherReason(path, await reads.launchers()) : null; + if (launcher) keep.push(launcher); return { activity, keep }; } diff --git a/packages/stim-cli/src/workspace/discovered-reads.ts b/packages/stim-cli/src/workspace/discovered-reads.ts index d8d54fef3..81dfa9795 100644 --- a/packages/stim-cli/src/workspace/discovered-reads.ts +++ b/packages/stim-cli/src/workspace/discovered-reads.ts @@ -1,19 +1,25 @@ import type { AgentSession } from '@stim-cli/core/state'; import { agentKey, attributeAgentSessions, discoverAgentSessions } from '../agent-sessions.ts'; import { canonicalPath } from '../commands/gc/paths.ts'; +import { freshLaunchers, launchersOf, type LauncherScan } from './launchers.ts'; import { processReader, type RunningProcesses } from './running-from.ts'; +import type { StimContext } from '@stim-cli/core'; /** What one pass reads once for every checkout: the process table and the agent sessions. */ export interface DiscoveredReads { processes: () => Promise; + /** Where the executables on PATH that are symlinks resolve to, and what could not be read. */ + launchers: () => Promise; /** Why an agent session working in the path keeps it. */ sessions: (path: string, now: number) => Promise; } -export function discoveredReads(signal?: AbortSignal): DiscoveredReads { +/** `fresh` reads the launcher directories anew, for the check that immediately precedes a deletion. */ +export function discoveredReads(signal?: AbortSignal, operation?: StimContext, fresh = false): DiscoveredReads { let found: Promise | undefined; return { processes: processReader(signal), + launchers: fresh ? freshLaunchers(operation, signal) : launchersOf(operation, signal), sessions: async (path, now) => { found ??= discoverAgentSessions({ now }); const sessions = await found; diff --git a/packages/stim-cli/src/workspace/discovered-removal.ts b/packages/stim-cli/src/workspace/discovered-removal.ts index d00aef7f4..1db3632b0 100644 --- a/packages/stim-cli/src/workspace/discovered-removal.ts +++ b/packages/stim-cli/src/workspace/discovered-removal.ts @@ -111,15 +111,18 @@ export async function discoveredRemovalHold( main, idleMs, reads, + reduceOnly, operation, - }: { main: string | null; idleMs: number; reads: DiscoveredReads; operation?: StimContext }, + }: { main: string | null; idleMs: number; reads: DiscoveredReads; reduceOnly: boolean; operation?: StimContext }, ): Promise { + if (reduceOnly) + return 'it was found in a tool folder or by the full scan, so Stim reduces its outputs and never removes it'; const folder = main ? relative(main, path).split('/')[0] : undefined; if (folder?.startsWith('.') && folder !== '.worktrees' && !folder.startsWith('..')) return `it is managed by another tool (inside ${folder} of its main checkout)`; if (!(idleMs > 0)) return 'idle.dormantDays is 0, so its idleness cannot be established'; const now = Date.now(); - const decision = decideIdleStop(await discoveredEvidence(path, { now, idleMs, reads, operation }), { + const decision = decideIdleStop(await discoveredEvidence(path, { now, idleMs, reads, operation, reduceOnly }), { idleMs, now, setting: 'idle.dormantDays', diff --git a/packages/stim-cli/src/workspace/discovery-scan.ts b/packages/stim-cli/src/workspace/discovery-scan.ts new file mode 100644 index 000000000..9e51acd32 --- /dev/null +++ b/packages/stim-cli/src/workspace/discovery-scan.ts @@ -0,0 +1,313 @@ +import { lstat, readdir } from 'node:fs/promises'; +import { mkdirSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join, relative, sep } from 'node:path'; +import { isScopedHome, withDirLock, type StimContext } from '@stim-cli/core'; +import { + type ScanTruncation, + discoveryScanFile, + maintenanceDir, + readDiscoveryScanFile, + type DaemonMeasurement, + type DiscoveryScan, +} from '@stim-cli/core/state'; +import { canonicalPath } from '../commands/gc/paths.ts'; +import { diskPressure, diskThresholds } from '../daemon/pressure.ts'; +import { answered, hungRead, Unanswered } from './bounded-read.ts'; +import { loadConfig } from './config.ts'; +import { recognises } from './discovery.ts'; +import { + MAX_DEPTH, + REAL_RECOGNITION_IO, + folderSafety, + nestedSafety, + scanDescends, + scanSkipper, + absent, + type RecognitionIo, +} from './scan-rules.ts'; + +const HOUR_MS = 60 * 60_000; +const WEEK_MS = 7 * 24 * HOUR_MS; +const MAX_FOLDERS = 200_000; +const BUDGET_MS = 45_000; +const READ_TIMEOUT_MS = 3000; + +interface ScanOutcome { + found: string[]; + unreadable: string[]; + visited: number; + truncated: boolean; + /** Why it stopped early, for the schedule and the doctor listing. */ + truncation: Exclude; +} + +/** + * A pruned walk below `home` (`scanDescends`, `skip`): folders for which `isCheckout` holds are results and are not + * entered; symlinks and other volumes are skipped; a folder the system refuses is recorded as unreadable, never an error. + * A read that does not answer within 3 seconds stops the walk at once: the folder is recorded unreadable, the result is + * truncated, and the read stays the pending one (`bounded-read.ts`), so no further read starts, here or in a later scan, + * until it settles. It also stops at `MAX_DEPTH` levels, `MAX_FOLDERS` folders or `budgetMs`, and says so in + * `truncated`, with the reason in `truncation`. Before `isCheckout` runs on a folder, `folderSafety` has proved, for it and + * for every folder above it down from the home folder, and `nestedSafety` for what detection inspects below it, that + * project detection cannot block: an unsafe folder (a non-regular detection file, or an Xcode container without an app + * dependency) makes its whole subtree unrecognisable, and a manifest read that does not answer ends the walk like any + * other timed-out read. An abort rejects. + */ +export async function scanForCheckouts( + home: string, + isCheckout: (path: string) => boolean, + { + budgetMs = BUDGET_MS, + signal, + skip = () => false, + readTimeoutMs = READ_TIMEOUT_MS, + stat: statOf = (path) => lstat(path), + entries: entriesOf = (path) => readdir(path, { withFileTypes: true }), + recognition = REAL_RECOGNITION_IO, + }: { + budgetMs?: number; + signal?: AbortSignal; + skip?: (path: string) => boolean; + readTimeoutMs?: number; + stat?: (path: string) => Promise<{ dev: number }>; + entries?: RecognitionIo['readEntries']; + recognition?: RecognitionIo; + } = {}, +): Promise { + const outcome: ScanOutcome = { found: [], unreadable: [], visited: 0, truncated: false, truncation: 'none' }; + const deadline = Date.now() + budgetMs; + const stop = (truncation: Exclude): void => { + if (outcome.truncated) return; + outcome.truncated = true; + outcome.truncation = truncation; + }; + const exhausted = (): boolean => { + signal?.throwIfAborted(); + if (hungRead() !== null) stop('read'); + else if (Date.now() > deadline) stop('budget'); + else if (outcome.visited >= MAX_FOLDERS) stop('folders'); + return outcome.truncated; + }; + /** + * A read of `folder` failed. A timeout ends the scan (true). Anything but "it does not exist" means the scan could not + * look, so the folder is recorded unreadable and its earlier finds are kept; the walk carries on (false). + */ + const stopAt = (folder: string, error: unknown): boolean => { + signal?.throwIfAborted(); + if (error instanceof Unanswered) { + outcome.unreadable.push(folder); + stop('read'); + return true; + } + if (!absent(error)) outcome.unreadable.push(folder); + return false; + }; + if (hungRead() !== null) { + stop('read'); + return outcome; + } + let volume: number; + try { + volume = (await answered(home, () => statOf(home), readTimeoutMs)).dev; + } catch (error) { + if (!stopAt(home, error) && absent(error)) throw error; + return outcome; + } + const walk = async (folder: string, depth: number, unsafeAbove: boolean): Promise => { + if (exhausted()) return; + outcome.visited++; + let entries; + try { + entries = await answered(folder, () => entriesOf(folder), readTimeoutMs); + } catch (error) { + stopAt(folder, error); + return; + } + const safety = await folderSafety(folder, entries, readTimeoutMs, recognition, exhausted); + signal?.throwIfAborted(); + if (safety === 'stopped') return; + if (safety === 'denied') { + outcome.unreadable.push(folder); + return; + } + if (safety === 'unreadable') { + outcome.unreadable.push(folder); + stop('read'); + return; + } + const above = unsafeAbove || safety === 'unsafe'; + if (above) return; + if (depth > 0) { + const nested = await nestedSafety(folder, entries, readTimeoutMs, recognition, exhausted); + signal?.throwIfAborted(); + if (nested === 'stopped') return; + if (nested === 'denied') { + outcome.unreadable.push(folder); + return; + } + if (nested === 'unreadable') { + outcome.unreadable.push(folder); + stop('read'); + return; + } + if (nested === 'unsafe' || exhausted()) return; + if (isCheckout(folder)) { + outcome.found.push(folder); + return; + } + } + if (depth >= MAX_DEPTH) return; + for (const entry of entries) { + if (exhausted()) return; + if (!entry.isDirectory() || entry.isSymbolicLink() || !scanDescends(entry.name, depth + 1)) continue; + const path = join(folder, entry.name); + if (skip(path)) continue; + let info; + try { + info = await answered(path, () => statOf(path), readTimeoutMs); + } catch (error) { + if (stopAt(path, error)) return; + continue; + } + if (info.dev === volume) await walk(path, depth + 1, false); + } + }; + await walk(home, 0, false); + return outcome; +} + +/** + * The finds to store: a complete scan's, plus the previous ones under a folder it could not read; a truncated scan's, + * plus every previous one. A worse result never replaces a better one, and a stale entry is dropped at read by + * `scanEntryValid`, not here. + */ +export function mergeFound(previous: readonly string[], outcome: ScanOutcome): string[] { + const under = (path: string) => + outcome.unreadable.some((folder) => path === folder || path.startsWith(`${folder}${sep}`)); + const kept = outcome.truncated ? previous : previous.filter(under); + return [...new Set([...outcome.found, ...kept])].toSorted(); +} + +/** + * Whether a scan is due. Never twice within an hour, so a rise and fall of the disk level cannot start a run of scans. + * After that hour an attempt that ended on a failure or a read that timed out is retried whatever the disk level; one + * that finished, or stopped at the folder cap or the time budget (the same limit would stop a retry), follows the weekly + * schedule from the last scan that did not end on a read or failure, and runs hourly while the level is above normal + * and no scan has run during it for an hour. + */ +export function discoveryScanDue( + last: Pick | null, + now: number, + low: boolean, +): boolean { + if (!last || last.ranAt > now) return true; + if (now - last.ranAt < HOUR_MS) return false; + if (last.truncation === 'read' || last.truncation === 'failed') return true; + return now - last.completedAt >= WEEK_MS || (low && now - last.lowScanAt >= HOUR_MS); +} + +/** When this process last tried a scan, so a result that cannot be stored does not start a walk on every tick. */ +const attempts = { at: 0 }; + +function writeScan(scan: DiscoveryScan): void { + mkdirSync(maintenanceDir(), { recursive: true }); + withDirLock(join(maintenanceDir(), 'discovery-scan.lock'), () => { + const file = discoveryScanFile(); + const tmp = `${file}.${process.pid}.tmp`; + writeFileSync(tmp, `${JSON.stringify(scan)}\n`); + try { + renameSync(tmp, file); + } finally { + try { + unlinkSync(tmp); + } catch {} + } + }); +} + +/** + * Scans the home folder when the schedule says so (`discoveryScanDue`) and stores the result under the Stim home, with + * the schedule's own times, so a retired worker or a restarted daemon forgets nothing. A failed scan stores nothing but + * its attempt time, keeps the previous finds and is retried after an hour, as is a scan that ended on a read that timed + * out; one that stopped at the folder cap or the time budget counts as finished, since a retry would hit the same limit. Only a daemon tick scans, never a command, + * and a scoped STIM_HOME never does. Only an abort escapes, so a scan can never end the maintenance pass it follows. + */ +export async function runDiscoveryScanIfDue( + tick: { operation: StimContext; measured: { measurement: Pick } } | undefined, + { + home = homedir(), + now = Date.now(), + attempted = attempts, + scanOptions = {}, + }: { + home?: string; + now?: number; + attempted?: { at: number }; + /** Limits and readers of the walk, for tests. */ + scanOptions?: Parameters[2]; + } = {}, +): Promise { + if (!tick || isScopedHome(tick.operation.env)) return; + const { operation } = tick; + let last: DiscoveryScan | null; + let low: boolean; + try { + const thresholds = diskThresholds(loadConfig(), operation.env).thresholds; + low = diskPressure(tick.measured.measurement.disk, thresholds).level !== 'normal'; + const file = readDiscoveryScanFile(); + if (file.status === 'unreadable') return; + last = file.status === 'read' ? file.scan : null; + } catch { + return; + } + if (!discoveryScanDue(last, now, low) || (now >= attempted.at && now - attempted.at < HOUR_MS) || hungRead() !== null) + return; + attempted.at = now; + const attempt = { + version: 2, + ranAt: now, + lowScanAt: low ? now : (last?.lowScanAt ?? 0), + trigger: low ? 'low-disk' : 'weekly', + } as const; + const failedTruncation: ScanTruncation = 'failed'; + const failed = { + ...attempt, + completedAt: last?.completedAt ?? 0, + durationMs: 0, + visited: 0, + truncated: true, + truncation: failedTruncation, + }; + const realHome = canonicalPath(home); + const dotFolder = (path: string) => relative(realHome, path).startsWith('.'); + try { + const outcome = await scanForCheckouts(realHome, recognises, { + signal: operation.signal, + skip: scanSkipper(realHome), + ...scanOptions, + }); + operation.signal.throwIfAborted(); + writeScan({ + ...attempt, + completedAt: outcome.truncation === 'read' ? (last?.completedAt ?? 0) : now, + durationMs: Date.now() - now, + ...outcome, + found: mergeFound(last?.found ?? [], { ...outcome, found: outcome.found.filter((p) => !dotFolder(p)) }), + dotFolderFound: mergeFound(last?.dotFolderFound ?? [], { ...outcome, found: outcome.found.filter(dotFolder) }), + }); + } catch { + if (operation.signal.aborted) { + attempted.at = 0; + operation.signal.throwIfAborted(); + } + try { + writeScan({ + ...failed, + found: last?.found ?? [], + dotFolderFound: last?.dotFolderFound ?? [], + unreadable: last?.unreadable ?? [], + }); + } catch {} + } +} diff --git a/packages/stim-cli/src/workspace/discovery.ts b/packages/stim-cli/src/workspace/discovery.ts index 0af4fb2f7..82cee508d 100644 --- a/packages/stim-cli/src/workspace/discovery.ts +++ b/packages/stim-cli/src/workspace/discovery.ts @@ -3,15 +3,23 @@ import { existsSync, readdirSync, statSync } from 'node:fs'; import { homedir } from 'node:os'; import { dirname, isAbsolute, join, parse, resolve, sep } from 'node:path'; import { isScopedHome, type StimContext } from '@stim-cli/core'; -import { machineSettingValue, type Config } from '@stim-cli/core/state'; +import { machineSettingValue, readDiscoveryScan, type Config } from '@stim-cli/core/state'; import { canonicalPath } from '../commands/gc/paths.ts'; import { projectRegistry } from '../integrations/projects.ts'; import { isProtectedLocation } from './protected-locations.ts'; +import { scanEntryValid, scanSkipper } from './scan-rules.ts'; import { matchWorktreeEntry, resolveSourceCheckoutAsync } from './worktree.ts'; const DISCOVER_ROOTS_SETTING = 'maintenance.discoverRoots'; -type DiscoveryOrigin = 'worktree' | 'sibling' | 'root'; +type DiscoveryOrigin = 'worktree' | 'sibling' | 'root' | 'scan'; + +// @ref LLP 0004#tool-folders-and-the-full-scan -- reduce-only tool folders +/** + * Folders under the home folder where another tool puts linked worktrees of the user's repositories. Stim discovers + * those worktrees of known repositories and reduces their outputs, and never removes them. + */ +const TOOL_FOLDERS = ['.codex/worktrees']; export interface DiscoveredCheckout { /** The canonical path of the checkout. */ @@ -19,6 +27,8 @@ export interface DiscoveredCheckout { origin: DiscoveryOrigin; /** The repository, folder or `maintenance.discoverRoots` entry that led to it. */ from: string; + /** Found through a tool folder or the full scan: Stim reduces its outputs and never removes it. */ + reduceOnly?: boolean; } export interface DiscoveryChild { @@ -40,15 +50,17 @@ export function selectDiscovered({ children, }: { known: readonly string[]; - worktrees: readonly { path: string; from: string }[]; + worktrees: readonly { path: string; from: string; reduceOnly?: boolean }[]; children: readonly DiscoveryChild[]; }): DiscoveredCheckout[] { const found = new Map(); const covered = (path: string) => known.some((root) => path === root || path.startsWith(`${root}${sep}`) || root.startsWith(`${path}${sep}`)); - for (const { path, from } of worktrees) if (!covered(path)) found.set(path, { path, origin: 'worktree', from }); + for (const { path, from, reduceOnly } of worktrees) + if (!covered(path)) found.set(path, { path, origin: 'worktree', from, ...(reduceOnly ? { reduceOnly } : {}) }); for (const { path, origin, from, recognised } of children) - if (recognised && !found.has(path) && !covered(path)) found.set(path, { path, origin, from }); + if (recognised && !found.has(path) && !covered(path)) + found.set(path, { path, origin, from, ...(origin === 'scan' ? { reduceOnly: true } : {}) }); return [...found.values()].toSorted((a, b) => a.path.localeCompare(b.path)); } @@ -68,7 +80,7 @@ function isProject(path: string): boolean { * Whether Stim's project detection recognises the folder: it is a project itself, or a direct child of its `apps/`, * `examples/` or `packages/` is, as in a monorepo whose root holds no app. */ -function recognises(path: string): boolean { +export function recognises(path: string): boolean { if (isProject(path)) return true; return PROJECT_FOLDERS.some((folder) => { try { @@ -152,7 +164,12 @@ export function broadDiscoverRoots(config: Config | null, env: NodeJS.ProcessEnv return configuredRoots(config, env).roots.filter((root) => existsSync(root) && isBroadFolder(canonicalPath(root))); } -const inside = (path: string, folder: string) => path === folder || path.startsWith(`${folder}${sep}`); +const inside = (path: string, folder: string): boolean => path === folder || path.startsWith(`${folder}${sep}`); + +/** Whether `path` lies inside a tool folder (see `TOOL_FOLDERS`) of the home folder `home`. */ +export function inToolFolder(path: string, home: string): boolean { + return TOOL_FOLDERS.some((folder) => inside(path, join(home, folder))); +} /** * Whether Stim acts on a discovered linked worktree: it lies in its repository's own main checkout folder, in a folder @@ -181,8 +198,9 @@ interface Discovery { * projects (`git worktree list`) inside the folders Stim manages, recognised direct children of the folders holding * those checkouts, and recognised direct children of `maintenance.discoverRoots`. Nothing under the system temporary * directory or a dot-folder of the home directory is included unless it lies inside a `maintenance.discoverRoots` - * folder. A scoped STIM_HOME discovers only from that setting, so a scratch home never reaches the repositories of - * another. Nothing is registered, written or persisted. + * folder, or is a worktree in a tool folder or a checkout of the last full scan, which are reduce-only. A scoped + * STIM_HOME discovers only from that setting, so a scratch home never reaches the repositories of another. Discovery + * registers and writes nothing; the full scan's result is read from the file the daemon wrote. */ export async function discoverCheckouts( config: Config | null, @@ -210,6 +228,18 @@ function discovery(config: Config | null, known: readonly string[], operation?: return found; } +/** The checkouts of the last full scan that are still what the scan found, see `scanEntryValid`. */ +async function revalidatedScan(): Promise { + const home = canonicalPath(homedir()); + const rules = { home, isCheckout: recognises, skip: scanSkipper(home) }; + const valid: DiscoveryChild[] = []; + const ancestors = new Map(); + for (const path of readDiscoveryScan()?.found ?? []) + if (isAbsolute(path) && (await scanEntryValid(path, { ...rules, ancestors })) === 'valid') + valid.push({ path, origin: 'scan', from: 'full scan', recognised: true }); + return valid; +} + async function computeDiscovered( config: Config | null, known: readonly string[], @@ -265,7 +295,7 @@ async function computeDiscovered( repositories.push(repository); } } - const worktrees: { path: string; from: string }[] = []; + const worktrees: { path: string; from: string; reduceOnly?: boolean }[] = []; const outside: Discovery['outside'] = []; for (const { entries, main } of repositories) { for (const entry of entries) { @@ -273,18 +303,20 @@ async function computeDiscovered( if (path === canonicalPath(main)) continue; if (worktreeInScope(path, { main: canonicalPath(main), folders: [...folders.keys()], roots: canonicalRoots })) worktrees.push({ path, from: main }); + else if (inToolFolder(path, canonicalPath(homedir()))) worktrees.push({ path, from: main, reduceOnly: true }); else outside.push({ path, from: main }); } } - const allowed = children.filter( + const scanned: DiscoveryChild[] = scoped ? [] : await revalidatedScan(); + const allowed = [...children, ...scanned].filter( (child) => canonicalRoots.some((root) => inside(child.path, root)) || !isProtectedLocation(child.path), ); const knownRoots = known.map(canonicalPath); - const selected = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); - const listed = new Set(selected.map(({ path }) => path)); + const everything = selectDiscovered({ known: knownRoots, worktrees, children: allowed }); + const listed = new Set(everything.map(({ path }) => path)); const coveredByKnown = (path: string) => knownRoots.some((root) => inside(path, root) || inside(root, path)); return { - found: selected, + found: everything, outside: outside.filter(({ path }) => !listed.has(path) && !coveredByKnown(path)), }; } diff --git a/packages/stim-cli/src/workspace/launcher-directories.ts b/packages/stim-cli/src/workspace/launcher-directories.ts new file mode 100644 index 000000000..64a2fae71 --- /dev/null +++ b/packages/stim-cli/src/workspace/launcher-directories.ts @@ -0,0 +1,24 @@ +import { homedir } from 'node:os'; +import { delimiter, join } from 'node:path'; + +/** + * Where the executables a person installs by linking them live: the PATH directories of the process, and the per-user + * bin folders a daemon started by launchd, with its minimal PATH, does not list: `~/.local/bin` and `~/bin` (user + * links), `~/.cargo/bin` (cargo install), `~/.bun/bin` (bun), `~/.volta/bin` (Volta), `~/Library/pnpm` (pnpm's global + * bin), the active nvm bin (`NVM_BIN`), and `/usr/local/bin` and `/opt/homebrew/bin` (Homebrew). + */ +export function launcherDirectories(env: NodeJS.ProcessEnv = process.env): string[] { + const home = homedir(); + const extra = [ + join(home, '.local', 'bin'), + join(home, 'bin'), + join(home, '.cargo', 'bin'), + join(home, '.bun', 'bin'), + join(home, '.volta', 'bin'), + join(home, 'Library', 'pnpm'), + ...(env.NVM_BIN ? [env.NVM_BIN] : []), + '/usr/local/bin', + '/opt/homebrew/bin', + ]; + return [...new Set([...(env.PATH ?? '').split(delimiter).filter(Boolean), ...extra])]; +} diff --git a/packages/stim-cli/src/workspace/launchers.ts b/packages/stim-cli/src/workspace/launchers.ts new file mode 100644 index 000000000..184695d5e --- /dev/null +++ b/packages/stim-cli/src/workspace/launchers.ts @@ -0,0 +1,161 @@ +import { lstat, readdir, realpath } from 'node:fs/promises'; +import { join, sep } from 'node:path'; +import type { StimContext } from '@stim-cli/core'; +import { canonicalPath } from '../commands/gc/paths.ts'; +import { answered, hungRead, Unanswered } from './bounded-read.ts'; +import { launcherDirectories } from './launcher-directories.ts'; + +/** + * Where the executables on PATH that are symlinks resolve to, and the directories whose inspection could not be + * completed. A directory or entry that does not exist, and a dangling symlink, is no launcher. Anything else that + * fails (permission, I/O, a timeout) is incomplete, and nothing is known about that directory. + */ +export interface LauncherScan { + targets: string[]; + incomplete: string[]; +} + +interface LauncherIo { + readdir: (path: string) => Promise; + isSymlink: (path: string) => Promise; + realpath: (path: string) => Promise; +} + +const REAL_IO: LauncherIo = { + readdir: (path) => readdir(path), + isSymlink: async (path) => (await lstat(path)).isSymbolicLink(), + realpath: (path) => realpath(path), +}; + +const ABSENT = ['ENOENT', 'ENOTDIR']; +const absent = (error: unknown) => error instanceof Error && 'code' in error && ABSENT.includes(String(error.code)); + +const READ_TIMEOUT_MS = 3000; + +/** Readings run one at a time, so a reading never adds a second read in flight. */ +let queue: Promise = Promise.resolve(); + +/** + * Reads the entries of each launcher directory and resolves the symlinks among them, once per reader. A wrapper + * script that merely names a path is not followed. Every call to the directory has 3 seconds to answer; one that does + * not ends the reading at once as incomplete, and no read of any kind starts while an earlier one is still pending + * (`bounded-read.ts`). + */ +export function launcherReader( + signal?: AbortSignal, + { + directories, + io = REAL_IO, + timeoutMs = READ_TIMEOUT_MS, + }: { directories: readonly string[]; io?: LauncherIo; timeoutMs?: number } = { directories: [] }, +): () => Promise { + let scan: Promise | undefined; + const read = async (): Promise => { + const found: LauncherScan = { targets: [], incomplete: [] }; + for (const directory of directories) { + signal?.throwIfAborted(); + const stuck = hungRead(); + if (stuck) return { targets: found.targets, incomplete: [...found.incomplete, stuck] }; + try { + for (const name of await answered(directory, () => io.readdir(directory), timeoutMs)) { + const entry = join(directory, name); + try { + if (!(await answered(directory, () => io.isSymlink(entry), timeoutMs))) continue; + found.targets.push(await answered(directory, () => io.realpath(entry), timeoutMs)); + } catch (error) { + if (!absent(error)) throw error; + } + } + } catch (error) { + signal?.throwIfAborted(); + if (absent(error)) continue; + found.incomplete.push(directory); + if (error instanceof Unanswered) return found; + } + } + return found; + }; + return () => { + if (!scan) { + const run = queue.then(read, read); + queue = run; + scan = run; + } + return scan; + }; +} + +const readers = new WeakMap Promise>(); + +const newReader = ( + operation: StimContext | undefined, + signal: AbortSignal | undefined, + directories?: readonly string[], +) => launcherReader(signal ?? operation?.signal, { directories: directories ?? launcherDirectories(operation?.env) }); + +/** The launcher reader of an operation for planning: every route of one pass shares one reading. */ +export function launchersOf( + operation: StimContext | undefined, + signal?: AbortSignal, + directories?: readonly string[], +): () => Promise { + if (!operation) return newReader(operation, signal, directories); + let reader = readers.get(operation); + if (!reader) readers.set(operation, (reader = newReader(operation, signal, directories))); + return reader; +} + +/** + * A launcher reader that reads anew on every call, never the operation's planning reading, which can be minutes old. A + * deletion route calls it at the moment it reads the processes that use the path, and no earlier, so a launcher is + * treated like a running process: the window that remains is the one after that read. + */ +export function freshLaunchers( + operation: StimContext | undefined, + signal?: AbortSignal, + directories?: readonly string[], +): () => Promise { + return () => newReader(operation, signal, directories)(); +} + +/** + * Whether a PATH launcher can keep a checkout's outputs, which every caller on a deletion path states: `reduceOnly: false` + * for a registered workspace, which has no launcher keep, or a fresh launcher reading and the checkout it is checked + * against. + */ +export type Launching = + | { reduceOnly: false } + | { reduceOnly: true; launchers: () => Promise; checkout: string }; + +/** The `Launching` of a checkout for planning: the operation's shared reading for a reduce-only one, none otherwise. */ +export function planningLaunching( + reduceOnly: boolean, + checkout: string, + operation: StimContext | undefined, + signal?: AbortSignal, +): Launching { + return reduceOnly ? { reduceOnly, launchers: launchersOf(operation, signal), checkout } : { reduceOnly }; +} + +/** The `Launching` of a checkout for a deletion: a fresh launcher reading for a reduce-only one, none otherwise. */ +export function launchingOf( + reduceOnly: boolean, + checkout: string, + operation: StimContext | undefined, + signal?: AbortSignal, +): Launching { + return reduceOnly ? { reduceOnly, launchers: freshLaunchers(operation, signal), checkout } : { reduceOnly }; +} + +/** + * Why `folder` is kept for an executable on PATH: one resolves to a path inside it, or a launcher directory could + * not be inspected, so none can be ruled out. Null when every directory was read and none points into it. + */ +export function launcherReason(folder: string, scan: LauncherScan): string | null { + const root = canonicalPath(folder); + const target = scan.targets.find((path) => path.startsWith(`${root}${sep}`)); + if (target) return `an executable on PATH runs from it (${target})`; + if (scan.incomplete.length) + return `an executable on PATH cannot be ruled out: ${scan.incomplete.join(', ')} could not be read; fixing its permissions or access restores reduction`; + return null; +} diff --git a/packages/stim-cli/src/workspace/protected-locations.ts b/packages/stim-cli/src/workspace/protected-locations.ts index 6a4c903f3..76d2ae185 100644 --- a/packages/stim-cli/src/workspace/protected-locations.ts +++ b/packages/stim-cli/src/workspace/protected-locations.ts @@ -25,7 +25,7 @@ function real(path: string): string | null { /** * Whether a path lies under the system temporary directory, or under a dot-folder directly inside the home directory * (`~/.codex`, `~/.config`, `~/.exact`): other tools own what is there, so Stim never acts on it. A path that does not - * resolve counts as protected. + * resolve counts as protected. Discovery lists the worktrees in `~/.codex/worktrees` anyway, as reduce-only. */ export function isProtectedLocation(path: string, roots: Roots = defaultRoots()): boolean { const target = real(path); diff --git a/packages/stim-cli/src/workspace/regenerable-dirs.ts b/packages/stim-cli/src/workspace/regenerable-dirs.ts index b0921cf60..a3669cdf8 100644 --- a/packages/stim-cli/src/workspace/regenerable-dirs.ts +++ b/packages/stim-cli/src/workspace/regenerable-dirs.ts @@ -3,6 +3,7 @@ import { lstat, readdir, readFile } from 'node:fs/promises'; import { basename, dirname, join, relative, sep } from 'node:path'; import { canonicalPath } from '../commands/gc/paths.ts'; import { gitQuietAsync } from './git.ts'; +import { launcherReason, type Launching } from './launchers.ts'; import { othersRunningAsync, readRunningProcessesAsync, runningFromReason } from './running-from.ts'; export type ArtifactKind = 'build' | 'dependencies'; @@ -218,8 +219,8 @@ export function isBuildDirPath(path: string): boolean { export async function regenerableDirProblem( root: string, path: string, - signal?: AbortSignal, - { processes = true }: { processes?: boolean } = {}, + signal: AbortSignal | undefined, + { processes = true, launching }: { processes?: boolean; launching: Launching }, ): Promise { const rel = relative(root, path); try { @@ -240,6 +241,10 @@ export async function regenerableDirProblem( const unignored = await gitQuietAsync(root, ['ls-files', '--others', '--exclude-standard', '--', rel], { signal }); if (unignored === null) return 'git cannot tell whether it ignores every file in it'; if (unignored.trim()) return 'it holds files git does not ignore'; + if (launching.reduceOnly) { + const launcher = launcherReason(launching.checkout, await launching.launchers()); + if (launcher) return launcher; + } if ( processes && basename(path) === CARGO_TARGET && diff --git a/packages/stim-cli/src/workspace/scan-rules.ts b/packages/stim-cli/src/workspace/scan-rules.ts new file mode 100644 index 000000000..f0564096d --- /dev/null +++ b/packages/stim-cli/src/workspace/scan-rules.ts @@ -0,0 +1,358 @@ +import { lstat, readdir, readFile, realpath, stat } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, relative, sep } from 'node:path'; +import { configDir } from '@stim-cli/core'; +import { buildWorkerRoot } from '@stim-cli/core/state'; +import { canonicalPath } from '../commands/gc/paths.ts'; +import { answered, hungRead, Unanswered } from './bounded-read.ts'; +import { declaresAppDependency } from './project-files.ts'; +import { loadConfig } from './config.ts'; + +export const MAX_DEPTH = 7; + +const within = (path: string, folder: string): boolean => path === folder || path.startsWith(`${folder}${sep}`); + +/** Folders directly in the home folder the scan never reads: system, media and the two the maintainer excluded. */ +const HOME_SKIPPED = new Set([ + 'Library', + 'Applications', + 'Desktop', + 'Downloads', + 'Movies', + 'Music', + 'Pictures', + 'Public', +]); + +// @ref LLP 0004#tool-folders-and-the-full-scan -- the home dot-folders the scan does not enter +/** + * Dot-folders directly in the home folder that hold many files and no projects: tool homes, package and build caches, + * runtimes, editor state and other tools' own state. Counted on one machine (entries to depth four): .gradle 67k, + * .agent-device 45k, .member-app-metro-cache 25k, .dartwords-metro-cache 17k, .npm 16k, .bun 16k, .rvm 15k, .yarn 15k, + * .android 7k, .vscode 6k, .cursor 5k, .claude 3k, .cache 2k, .ivy2 2k, .cargo 2k. `.codex` is reached through its + * worktrees folder. `.volta`, `.asdf`, `.mise`, `.fnm`, `.pyenv`, `.rbenv`, `.sdkman`, `.deno`, `.pub-cache` and `.fvm` are + * version managers and package caches whose installs hold thousands of files and no checkouts of the user's. Every other home dot-folder is entered, such as `.exact`, except those starting with `.stim`: the + * Stim home and every other Stim state folder (`.stim-agent`, `.stim-release`, `.stim-npm`) hold no checkouts the user owns, + * and the build worker's client checkouts must never become reduce-only. + */ +const HOME_DOT_PRUNED = new Set([ + '.git', + '.Trash', + '.codex', + '.claude', + '.gradle', + '.agent-device', + '.npm', + '.pnpm-store', + '.yarn', + '.bun', + '.cargo', + '.rustup', + '.nvm', + '.rvm', + '.cache', + '.android', + '.cocoapods', + '.vscode', + '.cursor', + '.antigravity', + '.local', + '.config', + '.docker', + '.m2', + '.ivy2', + '.gem', + '.expo', + '.zplug', + '.virtualenvs', + '.wrangler', + '.foundry', + '.pw-profiles', + '.swiftpm', + '.volta', + '.asdf', + '.mise', + '.fnm', + '.pyenv', + '.rbenv', + '.sdkman', + '.deno', + '.pub-cache', + '.fvm', +]); + +/** Folders at any depth that hold many files and no projects: dependencies, build output and tool caches. */ +const PRUNED = new Set([ + 'node_modules', + 'Pods', + 'Carthage', + 'build', + 'dist', + 'out', + 'target', + 'vendor', + 'DerivedData', + 'venv', + '__pycache__', + 'site-packages', + 'coverage', + 'Library', +]); + +// @ref LLP 0004#tool-folders-and-the-full-scan -- what the scan prunes +/** + * Whether the scan descends into a folder named `name`, `depth` levels below the home folder (1 is a child of it). + * Dot-folders are entered only directly in the home folder and not when they are in `HOME_DOT_PRUNED` or named like a + * cache; deeper ones, such as `.worktrees`, are skipped because git reaches their worktrees. + */ +export function scanDescends(name: string, depth: number): boolean { + if (name.startsWith('.')) + return depth === 1 && !name.startsWith('.stim') && !HOME_DOT_PRUNED.has(name) && !name.endsWith('-cache'); + if (PRUNED.has(name)) return false; + return depth !== 1 || !HOME_SKIPPED.has(name); +} + +/** + * The paths the scan never enters, by rule: the Stim home, the build worker root, and the system temporary directory + * unless the home folder itself is under it. Computed once, with no per-folder realpath: the walk never follows a + * symlink, so a path below the canonical home is already canonical. + */ +export function scanSkipper(home: string): (path: string) => boolean { + const stim = [canonicalPath(configDir()), canonicalPath(buildWorkerRoot(loadConfig()))]; + const temps = [tmpdir(), '/tmp', '/var/tmp', '/var/folders'].map(canonicalPath); + const homeIsTemporary = temps.some((temp) => within(home, temp)); + return (path) => + stim.some((folder) => within(path, folder)) || (!homeIsTemporary && temps.some((temp) => within(path, temp))); +} + +/** The files of a folder that project detection reads or stats: the manifests every integration's `inspect` looks at. */ +/** + * Compared case-insensitively on every volume: a case-insensitive one reads `Package.JSON` as `package.json`. On a case- + * sensitive one detection reads only the exact `package.json`, so this can only reject a folder it could have accepted, + * except that the Xcode exemption below must hold for every manifest that folds to `package.json`, since which one + * detection reads depends on the volume. + */ +const DETECTION_FILES = new Set([ + 'package.json', + 'settings.gradle', + 'settings.gradle.kts', + 'gradlew', + 'gradlew.bat', + 'package.swift', + 'exact.mjs', + 'app.json', +]); +const PROJECT_FOLDERS = ['apps', 'examples', 'packages']; +const PROJECT_FOLDER_ENTRIES = 50; +const XCODE_CONTAINER = /\.(xcodeproj|xcworkspace)$/i; +/** React Native's owned roots, which detection resolves with `realpath` when it walks up from a native layout. */ +const OWNED_ROOTS = new Set(['ios', 'android']); +const folded = (name: string): string => name.toLowerCase(); +/** The entry is not there: the only read failure that is not unsafe, and only for an entry a successful listing showed. */ +export const absent = (error: unknown): boolean => + error instanceof Error && 'code' in error && (error.code === 'ENOENT' || error.code === 'ENOTDIR'); + +interface Entry { + name: string; + isFile(): boolean; + isDirectory(): boolean; + isSymbolicLink(): boolean; +} + +/** How the recognition gate reads the disk; injected by tests. */ +export interface RecognitionIo { + readEntries: (path: string) => Promise; + readText: (path: string) => Promise; + /** Metadata reads of a cached entry; `realpath` is the native one, which folds case on macOS. */ + lstat: (path: string) => Promise<{ isSymbolicLink(): boolean }>; + stat: (path: string) => Promise<{ isDirectory(): boolean; dev: number }>; + realpath: (path: string) => Promise; +} + +export const REAL_RECOGNITION_IO: RecognitionIo = { + readEntries: (path) => readdir(path, { withFileTypes: true }), + readText: (path) => readFile(path, 'utf8'), + lstat: (path) => lstat(path), + stat: (path) => stat(path), + realpath: (path) => realpath(path), +}; + +/** + * `unsafe`: a structural fact, the folder is not a checkout and there is nothing to record. `denied`: a read failed for a + * reason that is neither a timeout nor "absent" (EACCES, EPERM, EIO), so the scan could not look, as when a folder's own + * listing fails: it is recorded unreadable and its finds are kept. `unreadable`: a read timed out or another is pending. + * `stopped`: the scan's limit ran out. + */ +type Safety = 'safe' | 'unsafe' | 'denied' | 'unreadable' | 'stopped'; + +/** Asked before every read the gate makes: true when the scan's own limit (abort, time budget, folder cap) has run out. */ +type Stop = () => boolean; + +/** + * Whether project detection can read in `folder` without blocking, from the folder's own listing: `unsafe` when a + * detection file is not a regular file (a FIFO, or a symlink into a stalled mount) or the folder holds an Xcode + * container with no app dependency, whose project files `containers()` follows by the paths they name; `unreadable` + * when a read did not answer in time or another read is still pending (`bounded-read.ts`). A regular `package.json`, + * the one detection file detection reads rather than stats, is read here first through the bounded reader, so an + * evicted or stalled file is found out before the synchronous read. Detection also inspects every ancestor of a folder, + * so a folder is recognisable only when this holds for it and for each ancestor from the home folder down, with no + * condition: the walk and the revalidation both apply it to every folder on the way. + */ +export async function folderSafety( + folder: string, + entries: readonly Entry[], + timeoutMs: number, + io: RecognitionIo = REAL_RECOGNITION_IO, + stop?: Stop, +): Promise { + if (hungRead() !== null) return 'unreadable'; + const detection = entries.filter((entry) => DETECTION_FILES.has(folded(entry.name))); + if (detection.some((entry) => !entry.isFile())) return 'unsafe'; + if (entries.some((entry) => OWNED_ROOTS.has(folded(entry.name)) && !entry.isDirectory())) return 'unsafe'; + const hasContainer = entries.some((entry) => XCODE_CONTAINER.test(entry.name)); + const manifests = detection.filter((entry) => folded(entry.name) === 'package.json'); + if (manifests.length === 0) return hasContainer ? 'unsafe' : 'safe'; + let declares = true; + for (const manifest of manifests) { + if (stop?.()) return 'stopped'; + let text: string; + try { + text = await answered(folder, () => io.readText(join(folder, manifest.name)), timeoutMs); + } catch (error) { + return error instanceof Unanswered ? 'unreadable' : 'denied'; + } + try { + declares &&= declaresAppDependency(JSON.parse(text)); + } catch { + declares = false; + } + } + return hasContainer && !declares ? 'unsafe' : 'safe'; +} + +/** + * `folderSafety` for what detection inspects below `folder` itself: each of `apps`, `examples` and `packages` (the + * container folder, with its own `package.json`) and up to 50 children of each. A symlinked container, or any unsafe or + * unreadable member, makes the result that. The ancestors of `folder` are the caller's: they have passed `folderSafety`. + * `stop` is asked before every read, so a scan's budget is honoured between them (`stopped`); revalidation passes none, + * and is bounded per read and by detection's own caps (50 children per container; the chain is the home folder and up to seven below it). + */ +export async function nestedSafety( + folder: string, + entries: readonly Entry[], + timeoutMs: number, + io: RecognitionIo = REAL_RECOGNITION_IO, + stop?: Stop, +): Promise { + /** The listing, or why it cannot be trusted; `stop` is never asked inside it, so an abort is never caught as a read error. */ + const list = async (path: string): Promise => { + try { + return await answered(path, () => io.readEntries(path), timeoutMs); + } catch (error) { + return error instanceof Unanswered ? 'unreadable' : 'denied'; + } + }; + for (const entry of entries.filter((candidate) => PROJECT_FOLDERS.includes(folded(candidate.name)))) { + if (entry.isSymbolicLink()) return 'unsafe'; + if (!entry.isDirectory()) continue; + const container = join(folder, entry.name); + if (stop?.()) return 'stopped'; + const members = await list(container); + if (typeof members === 'string') return members; + const own = await folderSafety(container, members, timeoutMs, io, stop); + if (own !== 'safe') return own; + for (const child of members + .filter((member) => member.isDirectory() && !member.isSymbolicLink()) + .slice(0, PROJECT_FOLDER_ENTRIES)) { + const path = join(container, child.name); + if (stop?.()) return 'stopped'; + let inner: Entry[]; + try { + inner = await answered(path, () => io.readEntries(path), timeoutMs); + } catch (error) { + if (error instanceof Unanswered) return 'unreadable'; + if (absent(error)) continue; + return 'denied'; + } + const result = await folderSafety(path, inner, timeoutMs, io, stop); + if (result !== 'safe') return result; + } + } + return 'safe'; +} + +/** A folder's listing and `folderSafety`, for the chain of folders above a cached entry. */ +async function gateOf( + folder: string, + timeoutMs: number, + io: RecognitionIo, +): Promise<{ listing: Entry[]; safety: Safety }> { + try { + const listing = await answered(folder, () => io.readEntries(folder), timeoutMs); + return { listing, safety: await folderSafety(folder, listing, timeoutMs, io) }; + } catch (error) { + return { listing: [], safety: error instanceof Unanswered ? 'unreadable' : 'denied' }; + } +} + +/** + * Whether a checkout a scan recorded may still be used: it is a real directory (not a symlink), its canonical path is + * the recorded one, it lies below the home folder on the home volume, no folder between the two is one the scan skips + * (`scanDescends`, `skip`), every folder from the home folder down to it passes `folderSafety` (memoised in `ancestors`), + * what it inspects below itself passes `nestedSafety`, and `isCheckout` still recognises it. + * `invalid` drops it, such as a checkout moved away with a symlink left behind; `pending` means a read did not answer, + * so it is not selected this time and stays in the scan file. + */ +export async function scanEntryValid( + path: string, + { + home, + isCheckout, + skip, + timeoutMs = 3000, + io = REAL_RECOGNITION_IO, + ancestors = new Map(), + }: { + home: string; + isCheckout: (path: string) => boolean; + skip: (path: string) => boolean; + timeoutMs?: number; + io?: RecognitionIo; + /** The listing and safety of folders on the way to a checkout, shared by the entries of one revalidation. */ + ancestors?: Map>; + }, +): Promise<'valid' | 'invalid' | 'pending'> { + if (hungRead() !== null) return 'pending'; + if (!within(path, home) || path === home || skip(path)) return 'invalid'; + const names = relative(home, path).split(sep); + if (names.length > MAX_DEPTH || !names.every((name, i) => scanDescends(name, i + 1))) return 'invalid'; + try { + if ((await answered(path, () => io.lstat(path), timeoutMs)).isSymbolicLink()) return 'invalid'; + const info = await answered(path, () => io.stat(path), timeoutMs); + if (!info.isDirectory()) return 'invalid'; + if ((await answered(path, () => io.realpath(path), timeoutMs)) !== path) return 'invalid'; + if ((await answered(home, () => io.stat(home), timeoutMs)).dev !== info.dev) return 'invalid'; + } catch (error) { + return error instanceof Unanswered ? 'pending' : 'invalid'; + } + const chain = [ + home, + ...relative(home, path) + .split(sep) + .map((_, i, parts) => join(home, ...parts.slice(0, i + 1))), + ]; + let own: Entry[] = []; + for (const folder of chain) { + let gate = ancestors.get(folder); + if (!gate) { + gate = gateOf(folder, timeoutMs, io); + ancestors.set(folder, gate); + } + const { listing, safety } = await gate; + if (safety !== 'safe') return safety === 'unsafe' ? 'invalid' : 'pending'; + if (folder === path) own = listing; + } + const nested = await nestedSafety(path, own, timeoutMs, io); + if (nested !== 'safe') return nested === 'unsafe' ? 'invalid' : 'pending'; + return isCheckout(path) ? 'valid' : 'invalid'; +} diff --git a/website/docs/worktrees.md b/website/docs/worktrees.md index 27001477c..06e47ce4e 100644 --- a/website/docs/worktrees.md +++ b/website/docs/worktrees.md @@ -84,11 +84,13 @@ A finished worktree with uncommitted changes can be removed after Stim saves a r Maintenance and `stim gc` also manage checkouts no `stim` command ever ran in. Discovery finds direct children of the folders that hold the checkouts of your registered projects, direct children of the folders in `maintenance.discoverRoots`, and the linked worktrees of their repositories. Use absolute paths for discovery roots. A leading `~` expands to your home folder. Stim refuses the home folder, its ancestors, and filesystem roots. `stim doctor` reports a configured path that is missing or is not a folder. A child counts only when Stim's project detection recognises it as a project it handles, itself or through a direct child of its `apps/`, `examples/` or `packages/` folder. -Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`). A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. +Stim acts on a discovered linked worktree only inside its repository's main checkout folder, a folder holding a known project, or a `maintenance.discoverRoots` folder. Worktrees elsewhere, such as another tool's home or a pin checkout, are listed by `stim gc --dry-run` as outside the folders Stim manages and are never touched. Nothing under the system temporary directory or a dot-folder directly inside your home directory is ever touched, except inside a `maintenance.discoverRoots` folder, which is always considered (for example `~/.exact/worktrees`), and the `~/.codex/worktrees` worktrees described below, whose outputs are only reduced. A worktree nested in a registered project is left to that workspace. A `STIM_HOME` scoped home discovers only from its own `maintenance.discoverRoots`. -With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, a linked worktree also loses `node_modules` and Pods, and a `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. +Two more sources find checkouts that Stim only reduces and never removes. Linked worktrees of your known repositories in `~/.codex/worktrees` are discovered by default. A full scan of your home folder finds other checkouts that Stim's project detection recognises. The daemon runs the scan after a maintenance pass has acted, never before its cleanup and never from a command: weekly, and also while the disk level is above normal when no scan has run for an hour. A scan that failed or ended on a read that timed out is retried after an hour; one that stopped at its folder or time limit is not, since a retry would stop at the same place. The scan reads `~/Documents`, so macOS may ask for permission. A folder that does not answer within 3 seconds, such as one waiting for your answer, is recorded as unreadable and read on the next scan. It skips `~/Desktop`, `~/Downloads`, `~/Library`, the media folders, other volumes, symlinks, dependency or build folders, dot-folders below the first level, the Stim home and every dot-folder starting with `.stim`, and the tool homes and caches among your home folder's other dot-folders (`.npm`, `.gradle`, `.cache` and similar). It enters other dot-folders of your home folder, such as `~/.exact`, and records the checkouts it finds there, but never acts on them: Stim cannot tell an installed tool from a checkout in a dot-folder, so add the folder to `maintenance.discoverRoots` to have it managed. It stops at a recognised project. The scan has a 45 second budget that it checks before every read, including the project checks below, so it can overrun it by at most one read's 3 second timeout plus the time the project check itself takes on the folder in hand; when the budget runs out the scan ends early and keeps the previous finds. Checking the saved finds again has no total budget; each read has the 3 second timeout and the number of reads is capped. Before it checks a folder for a project it makes sure the manifests detection reads can be read without hanging, for that folder, every folder above it down from your home folder, and the `apps`, `examples` and `packages` folders inside it with their children. A folder whose manifest (in any letter case) is not a regular file, whose `ios` or `android` entry is not a real folder, whose `apps`, `examples` or `packages` is a link, or that holds an Xcode project or workspace while its `package.json` does not depend on React Native or Expo, makes that part of the tree not scan checkouts. A manifest or file that does not answer within 3 seconds, such as an evicted iCloud file, ends the scan early. This does not cover the folders above your home folder, a file that becomes unreadable between the check and the project check, an `ios` or `android` folder that is itself a stalled mount point (Stim checks that it is a real folder, not that it answers), or what a cleanup pass reads inside a checkout once it is found, which is the same for every checkout. Discovery uses the last scan's saved result and checks each saved checkout again before using it, so `stim gc` never walks your disk itself, and a checkout that moved or stopped being a project is dropped. A scan that fails or stops early keeps the previous finds. So does a folder whose manifest or listing the system denies during the project checks: it is recorded as unreadable, not rejected, and its earlier finds stay. A checkout found this way that an executable on your `PATH` (or in a user bin folder such as `~/.cargo/bin`) links into is kept whole by every pass that deletes its build output, dependencies or Cargo target. Each deletion reads those folders anew, at the moment it checks which processes use the path, so the only remaining window is the one for a process that starts after that check. A folder that exists but cannot be read, or does not answer within 3 seconds, keeps such checkouts, and while a hung read is pending no other launcher or scan read starts (at most one hung read per daemon thread, so up to two) and a scheduled scan is skipped; fixing the folder restores reduction. A scan stops at the first read that does not answer and keeps the previous finds; the first read of `~/Documents` while macOS waits for your answer can end a scan early, and the next scan reads it. For these checkouts Stim reduces build outputs, `node_modules` and Pods (after `idle.dormantDays` in a linked worktree, `idle.sourceDormantDays` in a main checkout) by the rules below, and keeps the checkout. Every one is evaluated on every cleanup pass, like a checkout found next to your registered ones; the check for finished worktrees inspects every discovered worktree with git, which took about 0.14 seconds each where it was measured. Add the folder to `maintenance.discoverRoots` to give it the full rules, finished-worktree removal included. A scoped `STIM_HOME` uses neither source. Discovery registers nothing and creates no workspace directory; Stim runs no `git fetch` in a discovered repository. -A clean linked worktree whose branch is merged or whose pull request ended is removed only when it is idle by that same evidence at every pressure level, with no grace shortcut, and every git-ignored file also exists, byte for byte, in the main checkout. The exception is regenerable folders: directories git reports as ignored that are named `node_modules` or like build output (`build`, `dist`, `target`, `.next`, `.turbo`, `.expo` and similar), whose files are deleted with the worktree. An ignored file elsewhere that the main checkout lacks, even inside a tracked `dist/` folder, keeps the worktree. Any change to a tracked file, even `Podfile.lock` or `project.pbxproj`, is dirty work and keeps it, and so does a tracked file marked assume-unchanged or skip-worktree, which hides edits from `git status`. A worktree inside a dot-folder of its main checkout other than `.worktrees` (for example `.claude/worktrees`) belongs to another tool and is never removed, although its build output and dependencies follow the normal idle rules. Dirty work is never archived, where a registered finished worktree's dirty work is archived first. +With no Stim activity record, idle means the newest of its source file writes (git-ignored files included; `node_modules`, Pods and nested worktrees are skipped, and `build`, `dist`, `target`, `.cache` and similar folders only where git ignores them), its last commit and its git writes, for `idle.dormantDays`. A process running from it, an agent session working in it, a pin with `maintenance.keep`, more source files than the scan reads, or anything Stim cannot read keeps it. An idle discovered checkout loses its regenerable build outputs, and its `node_modules` and Pods too: a linked worktree's after `idle.dormantDays` (default 1 day), a main (source) checkout's after `idle.sourceDormantDays` (default 7 days; 0 never), by the same evidence, whichever way it was discovered and even if no `stim` command ever ran there. A `PATH` launcher keeps a checkout found through `~/.codex/worktrees` or the full scan. A `target/` at a checkout root whose `CACHEDIR.TAG` starts with the standard signature line and whose files git ignores counts as build output. + +A clean linked worktree whose branch is merged or whose pull request ended is removed only when it is idle by that same evidence at every pressure level, with no grace shortcut, and every git-ignored file also exists, byte for byte, in the main checkout. The exception is regenerable folders: directories git reports as ignored that are named `node_modules` or like build output (`build`, `dist`, `target`, `.next`, `.turbo`, `.expo` and similar), whose files are deleted with the worktree. An ignored file elsewhere that the main checkout lacks, even inside a tracked `dist/` folder, keeps the worktree. Any change to a tracked file, even `Podfile.lock` or `project.pbxproj`, is dirty work and keeps it, and so does a tracked file marked assume-unchanged or skip-worktree, which hides edits from `git status`. A worktree inside a dot-folder of its main checkout other than `.worktrees` (for example `.claude/worktrees`) belongs to another tool and is never removed, although its build output and dependencies follow the normal idle rules. The same holds for a checkout found through `~/.codex/worktrees` or the full scan. Dirty work is never archived, where a registered finished worktree's dirty work is archived first.