diff --git a/llp/0004-discovered-checkouts.decision.md b/llp/0004-discovered-checkouts.decision.md index 77dee7a40..dcdf9c0ca 100644 --- a/llp/0004-discovered-checkouts.decision.md +++ b/llp/0004-discovered-checkouts.decision.md @@ -142,8 +142,8 @@ list. Reason: Stim cannot tell an installed tool from a checkout there, for example a Swift CLI cloned under `~/.mytool` whose launcher on PATH points into its ignored `.build/release`; clearing that output once idle would break the installed command. The record lets doctor report what the folder holds and the -command that adds it to `maintenance.discoverRoots`; that listing is not part of -this change. +command that adds it to `maintenance.discoverRoots`; the listing is described in +"The doctor listing and the low-disk line". The scan has a 45 second budget (and a 200 thousand folder cap, and an abort), checked before every read it makes, including every read of the safety checks @@ -288,8 +288,154 @@ retry after an hour of a scan that failed or ended on a read that timed out, whi after the pass, and the PATH launcher keep. The scan is not deferred under host load; it takes about a second and a half on one MacBook. -Not part of this change: points 4, 5 and 7, which belong to the doctor listing -and the low-disk line. +Points 4, 5 and 7 are implemented in "The doctor listing and the low-disk line". + +## The doctor listing and the low-disk line + +Maintainer direction, 2026-11-14 (#5762), verbatim, on the line Stim commands +print when the disk is low: "do stim commands when low on disk space already log +smt", then "could be part of that", then "smt like low disk space, found projects +not managed by stim taking x space , command to add to discover roots". On the +first of the two points it records for doctor, the decision above says: "doctor +lists them and suggest adding as managed path or delete if idle for long time". +Maintainer statement, 2026-11-14, as the issue records it (comment of 2026-11-15), +verbatim: "like maybe not part of doctor since it could be slow". It was said about the +scan itself, so doctor never walks the disk. + +What these words state: doctor lists the finds Stim does not fully manage and +suggests adding the folder as a managed path, or deleting what has been idle a +long time; the low-disk line carries how much they hold and the command that adds +them; the scan stays out of doctor. + +Chosen by the implementing lane and the coordinator's brief, not by the +maintainer: + +- The daemon writes the result. After a scan it classifies the scan's finds from + the complete discovery list and measures their recognised build output into + `maintenance/discovery-census.json`. Doctor and the low-disk line read that + file, the scan file and the `maintenance.discoverRoots` setting, and walk nothing. + Only the classification drops an entry, and only with evidence that a route + manages it; the report never hides a census entry, it chooses how to present it. + A group whose folder is a root discovery uses, or the setting names now, is shown + with no adopt command and words saying why its entries are listed. A group in a + root discovery ignores (a broad one such as `~`) is an ordinary group. +- Two kinds, worded differently because they are different facts: reduce-only + scan finds (Stim reduces their outputs and never removes them), and finds under + a home dot-folder (Stim does nothing with them). Both are grouped by the + folder that holds them, the only folder `maintenance.discoverRoots` can name to + adopt them, with a command built from the current value plus that folder and + quoted for the shell. The home folder and a folder holding it cannot be named, so + such a group gets words and no command. The command adds the folder to the full + discovered-checkout rules, finished-worktree removal included, and for a + dot-folder find lets Stim act where it did not; the listing says so. +- Size is the recognised build output only (the build directories maintenance + clears, a Cargo target with `CACHEDIR.TAG` or Exact's), not `node_modules` or + Pods, and not proof that a directory is deletable. It goes through + `refreshMeasure` and the disk-usage cache: one walk at a time on the machine, 10 + seconds per directory, at most 25 checkouts and 90 seconds per run; a checkout + measured within a day is carried over. The checkouts due are taken in the order + they have waited longest (never tried, then the oldest attempt, `attemptedAt`, which + a run cut short by the budget also sets without touching the measurement), and what + a run leaves keeps the census pending, so the next one, an hour later, continues; + a measurement older than a day when the census was made is never read as current. + A walk that failed or timed out is "could not be + measured" and a directory not reached is "not measured yet", never zero; a total + with either is "at least". +- Every read of a checkout's layout goes through the shared bounded reader + (`answered`: 3 seconds, one outstanding timed-out read, none while one is + pending) with the abort and the run's budget checked first, driving the same + directory rules the deleting passes use. A path that does not exist is absent; a + denied or timed-out read is "could not be measured", never zero. Walks and their + bounds are the ones above. Idle evidence is not read: it walks a checkout's + source with no bound the census could apply without changing code the deleting + passes use, so doctor says Stim has not checked whether a checkout is in use or + holds uncommitted or unpushed work, and leaves deleting one to the person. +- What is and is not bounded on the census path. Bounded (`answered`): the scan's + safety gate for each folder (`folderSafety`, `nestedSafety`), the census's own + layout reads and the git activity reads. Not bounded by the shared + reader, found by reading `computeDiscovered`, `unmanagedScanFinds`, `scanSkipper` and + `childrenOf` (the callees of project recognition and of the git helpers were not + traced): synchronous `existsSync` on each configured `maintenance.discoverRoots` + entry, each registered project root and each worktree entry of a repository; + synchronous `realpathSync` (through `canonicalPath`) on those, on the repository + folders, on the home folder, on every child `childrenOf` lists, on the known roots + and, in `scanSkipper`, on the Stim home, the build worker root and the temporary + directories (with a config read under the Stim home); `isProtectedLocation` + (`realpathSync`) on every worktree path and on every candidate child outside a + configured root, scan finds included; project recognition (`recognises`: + `projectRegistry.selectDoctor(path)` and a synchronous `readdirSync` of the folder's + `apps`, `examples` and `packages`) in `childrenOf` and at the end of + `scanEntryValid`; the `childrenOf` listings, which are raced against 3 seconds + rather than going through the shared reader; and git subprocesses + (`git worktree list` and `git symbolic-ref`, through the exec wrapper, abortable + and bounded by their own timeouts) in `resolveSourceCheckoutAsync`. The passes run + discovery the same way; what this change adds is that project recognition now also + runs on finds under home dot-folders. +- Instead of idle, the census records one bounded fact per checkout, the last git + activity: the newest modification time of `HEAD`, `index` and `logs/HEAD` in the + checkout's git directory. That is `/.git` for a main checkout, and for a + linked worktree the directory its `.git` file names after `gitdir:`, accepted + only when absolute and ending in `.git/worktrees/`. The reads are `lstat` + and a 512 byte prefix of a regular `.git` file, through `answered`, under the + run's budget and abort checks; no symlink is followed and no git process runs. + Commits, checkouts, staging and reflog writes change the date; editing files + without touching git does not, so it never proves a checkout unused. A missing + or unusual git directory, a failed or timed-out read or a spent budget is + unknown, shown as unknown and never as old. Doctor lists each checkout with the + date, oldest first, unknown last, in text and as an ISO string or null in the + JSON, with no threshold and without the word idle: the date is the fact and the + person judges. Chosen by the implementing lane and the coordinator, not by the + maintainer. +- The git directory is tied to the checkout the way git records it: a linked + worktree's `.git` file names a directory (a relative name resolves against the + checkout) ending in `.git/worktrees/` that holds a regular `gitdir` file + naming this checkout's `.git` file, and its canonical path (bounded + `realpath`) equals its normalised path, so no component is a symlink; a `logs` + that is not a real directory gives unknown. +- A scan find whose revalidation did not answer is neither managed nor gone: the + census carries it as `unchecked` with any earlier measurement and its age, + stays pending (retried within the hour, as is a checkout whose layout read + failed), and doctor prints "every checkout it found is managed" only for a + finished scan, a census made from it and no unchecked find. A project check + that could not read what decides it (a manifest that does not parse, a listing + that fails other than by absence) leaves the find unchecked in the same way, + while only proof that it is no longer a project drops it. The deleting passes + select only a find that is proven a project, as before. +- The measuring walk is told the directory exists (`knownToExist`), because the + census established that through the bounded reader; `refreshMeasure` makes no + other synchronous call on the measured path (its cache, failure and claim files + are under the Stim home). Discovery's own reads of configured roots and + registered projects (`existsSync`, `realpath`) are the ones the passes already + make. +- The per-command critical-disk hook makes no filesystem call on a configured + root or a find. It reads the daemon snapshot, the scan file, the census file and + the config (all under the Stim home), compares strings with the canonical roots + the census recorded, and adds no line once `maintenance.discoverRoots` changed + since the census. `stim doctor`, which a person runs on purpose and which already + canonicalises roots elsewhere, resolves symlinks of the configured roots with + `realpath`. The census writes its file under a directory lock in the state + directory. +- The census classifies a discovery computed from the scan result it records + (not the per-operation memo the passes share) and records that `ranAt`; a census + from an older scan than the scan file shows its checkouts as not measured yet. + An unverified child cleanup propagates out of the pass unchanged (LLP 0007); + any other census error is contained and logged once as `maintenance_failure`. +- The low-disk line is a second stderr line after the existing one, at critical + only, never on stdout or in JSON. It gives the total (with "at least" while some + is unmeasured) and the command for the largest reduce-only folder; a dot-folder + group counts in the total and is never the one named. +- Doctor shows it as a section outside the findings, so it never changes a PASS or + the count, and carries it in `--json` as `discovery`, an internal field. Doctor + runs only inside a project, so the section does too. +- The worktrees in `~/.codex/worktrees` reach discovery through git as reduce-only + worktrees and are not scan finds, so they are not listed. + +Open for the maintainer: whether the last git activity date is enough for "delete if +idle for long time", or Stim should offer more (it states no number and does not +read source activity); whether +the low-disk line also appears at the high level; whether listing the +`~/.codex/worktrees` worktrees belongs here; whether `stim status` and Stim +Desktop show the listing. ## Removing a discovered worktree needs more evidence @@ -323,11 +469,45 @@ hidden from `git status`. Chosen by the coordinator and the reviews of #5519, not by the maintainer. +## Known limits of the listing + +A find that contains, or lies inside, a registered project or a discovered checkout +is treated as managed and is not listed, although only that project or checkout is +managed. This is the symmetric coverage rule `selectDiscovered` applies to the +deleting passes; a monorepo root with a registered app inside it is the common case. + +A find whose layout read is permanently denied is re-measured each run, after the +checkouts that have waited longer, so it takes a slot of the 25 (`looked`) but never +starves the others, until it is fixed or leaves the scan result. +A find whose metadata read is denied, or whose project check could not read a +manifest (one that does not parse, as a merge conflict leaves it) or a listing, +is carried as unchecked without taking a slot, and the full scan records the folder +of such a manifest in its `unreadable` list, so `mergeFound` keeps the earlier finds +at or below it (without descending less or finding more), +but the census stays pending and reruns discovery every hour. A find that stays +unreadable or unchecked keeps the list from ever being called complete, and doctor +says so each time: it prints that the list may be incomplete and never that +everything is managed. After `maintenance.discoverRoots` changes, or a new scan, the +census predates it until the daemon lists again (about an hour): doctor says so, and +offers no command and no reason that discovery did not select the listed folders. +After a project is registered the census is not rebuilt for up to a day and doctor +cannot tell, since the census does not record the registered projects: the listing +may then include checkouts that are managed now, with no notice. A non-absolute path +in the scan file is dropped by revalidation and by the coverage test without +notice; it cannot be a checkout the scanner found, and rejecting it when the file +is read would make the file unusable for the scheduler, which then keeps none of the +previous finds of a truncated scan. The census +file is read without a lock (it is replaced atomically under a lock, so a reader +sees the old or the new file). A `du` that exits non-zero (denied files) is recorded +as a measured lower bound: its bytes count and the totals read "at least". + ## Links - Issues: #5512, #5515, #5757, #5762. Pull request: #5519. - Code: `packages/stim-cli/src/workspace/discovery.ts`, `packages/stim-cli/src/workspace/discovery-scan.ts`, + `packages/stim-cli/src/workspace/discovery-census.ts`, + `packages/stim-cli/src/workspace/discovery-report.ts`, `packages/stim-cli/src/workspace/discovered-removal.ts`, `packages/stim-cli/src/workspace/protected-locations.ts`, `packages/stim-cli/src/workspace/hidden-edits.ts`, diff --git a/packages/core/state/discovery-census.ts b/packages/core/state/discovery-census.ts new file mode 100644 index 000000000..b176243b3 --- /dev/null +++ b/packages/core/state/discovery-census.ts @@ -0,0 +1,153 @@ +import { isJsonObject, readStateFile } from './json-file.ts'; +import { discoveryCensusFile } from './paths.ts'; + +/** + * A checkout the last full scan found that Stim does not fully manage: `reduce-only` ones are reduced and never removed, + * `dot-folder` ones lie under a home dot-folder, where Stim does nothing with them, and `unchecked` ones could not be + * revalidated this time because a read did not answer (neither managed nor gone; any measurement is from an earlier run). + */ +export interface CensusFind { + path: string; + kind: 'reduce-only' | 'dot-folder' | 'unchecked'; + /** The folder whose direct children include the checkout, the one `maintenance.discoverRoots` would take. */ + folder: string; + /** Whether `maintenance.discoverRoots` can name `folder`: the home folder and the folders holding it cannot. */ + adoptable: boolean; + /** When its build output was last measured, epoch ms; 0 when it was not measured yet. */ + evaluatedAt: number; + /** + * When the daemon last tried to measure it, epoch ms, whether or not the run got far enough to measure anything; it + * only orders the work, so a checkout whose run is cut short does not stay first. Absent in a file written before it + * existed, which reads as never tried. + */ + attemptedAt?: number; + /** The recognised build directories in it, the ones Stim would clear. */ + dirs: number; + /** How many of them were measured, and how many were tried and failed or timed out (the rest are not measured yet). */ + measuredDirs: number; + failedDirs: number; + /** How many of the measured directories were walked by a `du` that exited non-zero: their bytes are a lower bound. */ + partialDirs: number; + /** The bytes of the measured directories; never an estimate of the others. */ + bytes: number; + /** Listing its build directories failed or did not answer, so nothing about its output is known. */ + unreadable: boolean; + /** + * When git last wrote in the checkout (epoch ms): the newest modification time of the git directory's HEAD, index and + * logs/HEAD. Null is unknown, never old. It says nothing about edits made without touching git. + */ + gitActivityAt: number | null; +} + +/** + * What the daemon derived from the last full scan for the doctor listing and the low-disk line: which finds Stim does + * not fully manage, and the size of their build output. Commands read it and never walk the disk. + */ +export interface DiscoveryCensus { + version: 1; + /** When the daemon last wrote it, epoch ms. */ + at: number; + /** The `ranAt` of the scan it was derived from. */ + scanRanAt: number; + /** The `maintenance.discoverRoots` entries it was classified under, as absolute paths. */ + roots: string[]; + /** The existing, non-broad folders of `roots` with symlinks resolved, as the classification used; commands compare strings only. */ + canonicalRoots: string[]; + /** Some find has a directory not measured yet, so another run continues. */ + pending: boolean; + finds: CensusFind[]; +} + +const finite = (value: unknown): value is number => typeof value === 'number' && Number.isFinite(value); +const strings = (value: unknown): value is string[] => + Array.isArray(value) && value.every((entry) => typeof entry === 'string'); + +function parseFind(value: unknown): CensusFind | null { + if ( + !isJsonObject(value) || + typeof value.path !== 'string' || + (value.kind !== 'reduce-only' && value.kind !== 'dot-folder' && value.kind !== 'unchecked') || + typeof value.folder !== 'string' || + typeof value.adoptable !== 'boolean' || + !finite(value.evaluatedAt) || + !finite(value.dirs) || + !finite(value.measuredDirs) || + !finite(value.failedDirs) || + !finite(value.partialDirs) || + !finite(value.bytes) || + typeof value.unreadable !== 'boolean' || + (value.gitActivityAt !== null && !finite(value.gitActivityAt)) || + (value.attemptedAt !== undefined && !finite(value.attemptedAt)) + ) + return null; + return { + path: value.path, + kind: value.kind, + folder: value.folder, + adoptable: value.adoptable, + evaluatedAt: value.evaluatedAt, + dirs: value.dirs, + measuredDirs: value.measuredDirs, + failedDirs: value.failedDirs, + partialDirs: value.partialDirs, + bytes: value.bytes, + unreadable: value.unreadable, + gitActivityAt: value.gitActivityAt, + ...(value.attemptedAt === undefined ? {} : { attemptedAt: value.attemptedAt }), + }; +} + +/** Rejects missing, corrupt or unsupported payloads without throwing. */ +export function parseDiscoveryCensus(value: unknown): DiscoveryCensus | null { + if ( + !isJsonObject(value) || + value.version !== 1 || + !finite(value.at) || + !finite(value.scanRanAt) || + !strings(value.roots) || + !strings(value.canonicalRoots) || + typeof value.pending !== 'boolean' || + !Array.isArray(value.finds) + ) + return null; + const finds: CensusFind[] = []; + for (const entry of value.finds) { + const find = parseFind(entry); + if (!find) return null; + finds.push(find); + } + return { + version: 1, + at: value.at, + scanRanAt: value.scanRanAt, + roots: value.roots, + canonicalRoots: value.canonicalRoots, + pending: value.pending, + finds, + }; +} + +/** + * The census file: `absent` (the daemon has not written one), `unreadable` (it exists and could not be read), or `read` + * with the census, which is null when the contents do not parse or are another version. + */ +export type DiscoveryCensusFile = + | { status: 'absent' } + | { status: 'unreadable' } + | { status: 'read'; census: DiscoveryCensus | null }; + +export function readDiscoveryCensusFile(): DiscoveryCensusFile { + let text: string; + try { + text = readStateFile(discoveryCensusFile()); + } catch (error) { + return error instanceof Error && 'code' in error && error.code === 'ENOENT' + ? { status: 'absent' } + : { status: 'unreadable' }; + } + try { + return { status: 'read', census: parseDiscoveryCensus(JSON.parse(text)) }; + } catch { + return { status: 'read', census: null }; + } +} diff --git a/packages/core/state/index.ts b/packages/core/state/index.ts index a710b757f..67c3f74ee 100644 --- a/packages/core/state/index.ts +++ b/packages/core/state/index.ts @@ -53,6 +53,7 @@ export * from './server-service.ts'; export * from './hosted-logs.ts'; export * from './maintenance.ts'; export * from './discovery-scan.ts'; +export * from './discovery-census.ts'; export * from './gc-log.ts'; export * from './build-ledger.ts'; export * from './agent-device.ts'; diff --git a/packages/core/state/paths.ts b/packages/core/state/paths.ts index 1e2a4996a..cd8ade398 100644 --- a/packages/core/state/paths.ts +++ b/packages/core/state/paths.ts @@ -250,6 +250,10 @@ export function discoveryScanFile(): string { return join(maintenanceDir(), 'discovery-scan.json'); } +export function discoveryCensusFile(): string { + return join(maintenanceDir(), 'discovery-census.json'); +} + export function gcLogDir(): string { return join(configDir(), 'gc'); } diff --git a/packages/core/state/status-measures.ts b/packages/core/state/status-measures.ts index 5bef371c3..c8fd8c837 100644 --- a/packages/core/state/status-measures.ts +++ b/packages/core/state/status-measures.ts @@ -212,6 +212,15 @@ export function readDiskUsage(path: string): DiskMeasure | null { }; } +/** + * Whether the cached walk of `path` ended with `du` exiting non-zero, so its bytes are a lower bound. A separate reader: + * `DiskMeasure` and everything that reads it stay as they were, and an entry without the field reads as complete. + */ +export function readDiskUsagePartial(path: string): boolean { + const entry = readJsonObject(diskUsageCacheFile(path)); + return entry !== null && entry.path === path && entry.partial === true; +} + /** When the last walk of `path` failed (an ISO time), or null when none is recorded. */ export function readDiskUsageFailure(path: string): string | null { const entry = readJsonObject(diskUsageFailureFile(path)); diff --git a/packages/stim-cli/api.api.txt b/packages/stim-cli/api.api.txt index 10df1fb61..e27239c85 100644 --- a/packages/stim-cli/api.api.txt +++ b/packages/stim-cli/api.api.txt @@ -361,8 +361,17 @@ [*] interface DiscoveredCheckout { @@1 origin: DiscoveryOrigin; [*] interface DiscoveredCheckout { @@1 path: string; [*] interface DiscoveredCheckout { @@1 reduceOnly?: boolean; +[*] interface DiscoveryReport { +[*] interface DiscoveryReport { @@1 census: "read" | "absent" | "unreadable" | "unusable" | "none"; +[*] interface DiscoveryReport { @@1 censusAt: number | null; +[*] interface DiscoveryReport { @@1 censusCurrent: boolean; +[*] interface DiscoveryReport { @@1 complete: boolean; +[*] interface DiscoveryReport { @@1 groups: UnmanagedGroup[]; +[*] interface DiscoveryReport { @@1 rootsChangedSinceCensus: boolean; +[*] interface DiscoveryReport { @@1 scan: ScanSummary; [*] interface DoctorReport { [*] interface DoctorReport { @@1 @internal daemon: DaemonStatus & { +[*] interface DoctorReport { @@1 @internal discovery: DiscoveryReport; [*] interface DoctorReport { @@1 @internal maintenance: ReturnType & { [*] interface DoctorReport { @@1 deviceHosts: Array>["machines"][number] & { [*] interface DoctorReport { @@1 findings: Finding[]; @@ -1410,6 +1419,21 @@ [*] interface TunnelOutcome { @@1 provider?: string; [*] interface TunnelOutcome { @@1 reason?: string; [*] interface TunnelOutcome { @@1 status: string; +[*] interface UnmanagedGroup { +[*] interface UnmanagedGroup { @@1 bytes: number; +[*] interface UnmanagedGroup { @@1 checkouts: number; +[*] interface UnmanagedGroup { @@1 command: string | null; +[*] interface UnmanagedGroup { @@1 commandProblem: "too-broad" | "already-named" | "setting-unreadable" | "unchecked" | null; +[*] interface UnmanagedGroup { @@1 failed: number; +[*] interface UnmanagedGroup { @@1 folder: string; +[*] interface UnmanagedGroup { @@1 kind: "reduce-only" | "dot-folder" | "unchecked"; +[*] interface UnmanagedGroup { @@1 measured: number; +[*] interface UnmanagedGroup { @@1 members: Array<{ +[*] interface UnmanagedGroup { @@1 notMeasuredYet: number; +[*] interface UnmanagedGroup { @@1 oldestMeasuredAt: number | null; +[*] interface UnmanagedGroup { @@1 partial: number; +[*] interface UnmanagedGroup { @@2 gitActivityAt: string | null; +[*] interface UnmanagedGroup { @@2 path: string; [*] interface UnverifiedCleanup { [*] interface UnverifiedCleanup { @@1 child: number; [*] interface UnverifiedCleanup { @@1 message: string; @@ -1662,6 +1686,16 @@ [*] type RunSelectors = PlatformRunOptions extends (infer Options) ? Options extends { @branch1 @@0 } ? Omit & { [*] type RunSelectors = PlatformRunOptions extends (infer Options) ? Options extends { @branch1 @@1 waitSeconds?: number | false; [*] type RunSelectors = PlatformRunOptions extends (infer Options) ? Options extends { @branch2 @@0 } : Options : never; +[*] type ScanSummary = { +[*] type ScanSummary = { @@1 ranAt: number; +[*] type ScanSummary = { @@1 status: "read"; +[*] type ScanSummary = { @@1 trigger: "weekly" | "low-disk"; +[*] type ScanSummary = { @@1 truncation: ScanTruncation; +[*] type ScanSummary = { @@1 unreadable: string[]; +[*] type ScanSummary = { @@1 visited: number; +[*] type ScanSummary = { @branch1 @@0 } | { +[*] type ScanSummary = { @branch1 @@1 status: "absent" | "unreadable" | "unusable"; +[*] type ScanTruncation = "none" | "budget" | "folders" | "read" | "failed"; [*] type SettingScope = "machine" | "workspace" | "repo" | "committed"; [*] type SettingsPayload = { [*] type SettingsPayload = { @@1 files: Partial>; diff --git a/packages/stim-cli/src/__tests__/discovery-census.test.ts b/packages/stim-cli/src/__tests__/discovery-census.test.ts new file mode 100644 index 000000000..6dad943eb --- /dev/null +++ b/packages/stim-cli/src/__tests__/discovery-census.test.ts @@ -0,0 +1,675 @@ +import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, utimesSync, writeFileSync } from 'node:fs'; +import { lstat, readdir, readFile, realpath } from 'node:fs/promises'; +import { homedir, tmpdir } from 'node:os'; +import { join, relative } from 'node:path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { StimContext } from '@stim-cli/core'; +import { readDiscoveryCensusFile, readDiscoveryScanFile, readDiskUsage } from '@stim-cli/core/state'; +import * as diskUsage from '../disk-usage.ts'; +import { getExecutor, resetExecutor, setExecutor } from '../exec.ts'; +import { discoveryLines, discoveryReport, lowDiskLine } from '../workspace/discovery-report.ts'; +import { censusDue, runDiscoveryCensusIfDue } from '../workspace/discovery-census.ts'; +import { runDiscoveryScanIfDue } from '../workspace/discovery-scan.ts'; +import { hungRead } from '../workspace/bounded-read.ts'; +import { loadConfig, saveConfig } from '../workspace/config.ts'; +import { discoverCheckouts } from '../workspace/discovery.ts'; +import { testContext } from './_context.ts'; + +vi.mock('../workspace/protected-locations.ts', async (original) => { + const real = await original(); + return { + ...real, + isProtectedLocation: (path: string) => real.isProtectedLocation(path, { home: homedir(), temps: [] }), + }; +}); + +const hour = 3_600_000; +const day = 24 * hour; +const t0 = Date.UTC(2026, 10, 14); + +describe('censusDue', () => { + const last = { at: t0, scanRanAt: 5, roots: ['/a'], pending: false }; + it('is due for no census, another scan, other roots, a day, or an hour while something waits', () => { + expect(censusDue(null, 5, ['/a'], t0)).toBe(true); + expect(censusDue(last, 6, ['/a'], t0 + hour)).toBe(true); + expect(censusDue(last, 5, ['/a', '/b'], t0 + hour)).toBe(true); + expect(censusDue(last, 5, ['/a'], t0 + hour)).toBe(false); + expect(censusDue(last, 5, ['/a'], t0 + day)).toBe(true); + expect(censusDue({ ...last, pending: true }, 5, ['/a'], t0 + hour)).toBe(true); + expect(censusDue({ ...last, pending: true }, 5, ['/a'], t0 + hour - 1)).toBe(false); + }); +}); + +describe('runDiscoveryCensusIfDue', () => { + let base: string; + let home: string; + const tick = (env: NodeJS.ProcessEnv = {}, operation = testContext({ env })) => ({ + operation, + measured: { measurement: { disk: [{ volume: '/', freeMb: 500_000 }] } }, + }); + + function checkout(path: string, output = 4096) { + mkdirSync(join(path, 'android', 'build'), { recursive: true }); + writeFileSync(join(path, 'package.json'), JSON.stringify({ dependencies: { expo: '*' } })); + writeFileSync(join(path, 'android', 'build', 'out.bin'), Buffer.alloc(output)); + } + + async function run( + now: number, + options: Parameters[1] = {}, + operation?: StimContext, + ) { + const result = await runDiscoveryCensusIfDue(tick({}, operation), { now, attempted: { at: 0 }, ...options }); + const file = readDiscoveryCensusFile(); + return { result, census: file.status === 'read' ? file.census : null }; + } + + const scanAt = (now: number) => runDiscoveryScanIfDue(tick(), { home, now, attempted: { at: 0 } }); + + beforeEach(async () => { + base = realpathSync(mkdtempSync(join(tmpdir(), 'stim-census-'))); + home = join(base, 'home'); + mkdirSync(home); + vi.stubEnv('STIM_HOME', ''); + vi.stubEnv('HOME', home); + vi.stubEnv('USERPROFILE', home); + saveConfig({ version: 2, projects: {}, repos: {}, maintenance: { discoverRoots: [join(home, 'dev', 'tools')] } }); + checkout(join(home, 'dev', 'old'), 8192); + checkout(join(home, 'dev', 'fresh')); + checkout(join(home, 'dev', 'tools', 'managed')); + checkout(join(home, '.exact', 'exact2', 'a'), 16384); + checkout(join(home, '.exact', 'exact2', 'b')); + checkout(join(home, '.mytool')); + await scanAt(t0); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + rmSync(base, { recursive: true, force: true }); + }); + + it('lists what no other route manages, with each folder, and measures the build output of both kinds', async () => { + const { census } = await run(t0); + expect(census?.scanRanAt).toBe(t0); + const found = (census?.finds ?? []).map(({ path, kind, folder, adoptable }) => [ + path.slice(home.length), + kind, + folder.slice(home.length), + adoptable, + ]); + expect(found).toEqual([ + ['/dev/fresh', 'reduce-only', '/dev', true], + ['/dev/old', 'reduce-only', '/dev', true], + ['/.exact/exact2/a', 'dot-folder', '/.exact/exact2', true], + ['/.exact/exact2/b', 'dot-folder', '/.exact/exact2', true], + ['/.mytool', 'dot-folder', '', false], + ]); + for (const find of census?.finds ?? []) + expect(find).toMatchObject({ dirs: 1, measuredDirs: 1, failedDirs: 0, unreadable: false }); + expect(census?.finds.find(({ path }) => path.endsWith('/a'))?.bytes).toBeGreaterThanOrEqual(16384); + expect(census?.pending).toBe(false); + }); + + it('never reports a walk that failed as zero bytes, and a walk that waits as not measured yet', async () => { + vi.spyOn(diskUsage, 'refreshMeasure').mockImplementation(async (path) => + path.includes('/old/') ? { outcome: 'failed', measure: null } : { outcome: 'busy', measure: null }, + ); + const { census } = await run(t0); + const of = (name: string) => census?.finds.find(({ path }) => path.endsWith(name)); + expect(of('/dev/old')).toMatchObject({ dirs: 1, measuredDirs: 0, failedDirs: 1, bytes: 0 }); + expect(of('/dev/fresh')).toMatchObject({ dirs: 1, measuredDirs: 0, failedDirs: 0, bytes: 0 }); + expect(census?.pending).toBe(true); + }); + + it('measures at most the bound per run, leaves the rest not measured, and the next run continues without repeating', async () => { + const { census: first } = await run(t0, { maxCheckouts: 2 }); + expect(first?.finds.filter(({ evaluatedAt }) => evaluatedAt === 0)).toHaveLength(3); + expect(first?.pending).toBe(true); + const refresh = vi.spyOn(diskUsage, 'refreshMeasure'); + const { census: second } = await run(t0 + hour, { maxCheckouts: 2 }); + expect(second?.finds.filter(({ evaluatedAt }) => evaluatedAt === 0)).toHaveLength(1); + expect(refresh).toHaveBeenCalledTimes(2); + const { census: third } = await run(t0 + 2 * hour, { maxCheckouts: 2 }); + expect(third?.pending).toBe(false); + }); + + it('keeps the census pending while the bound leaves expired measurements, refreshes the longest-waiting first, and finishes the rest in the next runs', async () => { + const complete = () => + discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: { value: [join(home, 'dev', 'tools')], error: null }, + home, + canonical: null, + }).complete; + const { census: first } = await run(t0, { maxCheckouts: 3 }); + const measuredFirst = (first?.finds ?? []).filter(({ evaluatedAt }) => evaluatedAt === t0).map(({ path }) => path); + expect(measuredFirst).toHaveLength(3); + const { census: second } = await run(t0 + day, { maxCheckouts: 3 }); + const at = (census: typeof second, when: number) => + (census?.finds ?? []).filter(({ evaluatedAt }) => evaluatedAt === when).map(({ path }) => path); + expect(at(second, t0 + day).filter((path) => measuredFirst.includes(path))).toHaveLength(1); + expect(at(second, t0 + day)).toHaveLength(3); + expect(second?.pending).toBe(true); + expect(complete()).toBe(false); + const { census: third } = await run(t0 + day + hour, { maxCheckouts: 3 }); + expect(third?.finds.every(({ evaluatedAt }) => evaluatedAt >= t0 + day)).toBe(true); + expect(third?.pending).toBe(false); + expect(complete()).toBe(true); + }); + + it('measures the others when one checkout exhausts the budget every run, and leaves that one not measured', async () => { + const slow = join(home, 'dev', 'fresh'); + const real = { + readNames: (path: string) => readdir(path), + isDirectory: async (path: string) => (await lstat(path)).isDirectory(), + realpath: (path: string) => realpath(path), + lstat: async (path: string) => { + const info = await lstat(path); + return { + kind: info.isDirectory() ? 'directory' : info.isFile() ? 'file' : 'other', + mtimeMs: info.mtimeMs, + } as const; + }, + head: async (path: string, bytes: number) => + (await lstat(path)).isFile() ? (await readFile(path, 'utf8')).slice(0, bytes) : null, + }; + const io = { + ...real, + readNames: async (path: string) => { + if (path === join(slow, 'apps')) vi.setSystemTime(Date.now() + 100_000); + return real.readNames(path); + }, + }; + const complete = () => + discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: { value: [join(home, 'dev', 'tools')], error: null }, + home, + canonical: null, + }).complete; + await run(t0, { io }); + const { census } = await run(t0 + hour, { io }); + const of = (name: string) => census?.finds.find(({ path }) => path.endsWith(name)); + expect(of('/dev/old')?.evaluatedAt).toBe(t0 + hour); + expect(of('/dev/fresh')).toMatchObject({ evaluatedAt: 0, measuredDirs: 0 }); + expect(census?.pending).toBe(true); + expect(complete()).toBe(false); + }); + + it('starts no read once the budget is spent, and keeps every checkout not measured yet', async () => { + const { census } = await run(t0, { budgetMs: -1 }); + expect(census?.finds).toHaveLength(5); + expect(census?.finds.every(({ evaluatedAt, dirs }) => evaluatedAt === 0 && dirs === 0)).toBe(true); + }); + + it('classifies again after the roots change, so a folder just added is no longer listed', async () => { + await run(t0); + saveConfig({ + version: 2, + projects: {}, + repos: {}, + maintenance: { discoverRoots: [join(home, 'dev', 'tools'), join(home, 'dev')] }, + }); + const { census } = await run(t0 + hour); + expect(census?.finds.filter(({ kind }) => kind === 'reduce-only')).toEqual([]); + }); + + it('classifies the scan it records, not a discovery the same operation computed from an older scan', async () => { + const operation = testContext(); + await discoverCheckouts(loadConfig(), [], operation); + checkout(join(home, 'dev', 'added')); + await scanAt(t0 + 8 * day); + const { census } = await run(t0 + 8 * day, {}, operation); + expect(census?.scanRanAt).toBe(t0 + 8 * day); + expect(census?.finds.map(({ path }) => path.slice(home.length))).toContain('/dev/added'); + }); + + it('is never run in a scoped home and writes nothing', async () => { + await runDiscoveryCensusIfDue(tick({ STIM_HOME: join(base, 'state') }), { now: t0, attempted: { at: 0 } }); + expect(readDiscoveryCensusFile()).toEqual({ status: 'absent' }); + }); + + describe('reads of a checkout', () => { + const real = { + readNames: (path: string) => readdir(path), + isDirectory: async (path: string) => (await lstat(path)).isDirectory(), + realpath: (path: string) => realpath(path), + lstat: async (path: string) => { + const info = await lstat(path); + return { + kind: info.isDirectory() ? 'directory' : info.isFile() ? 'file' : 'other', + mtimeMs: info.mtimeMs, + } as const; + }, + head: async (path: string, bytes: number) => + (await lstat(path)).isFile() ? (await readFile(path, 'utf8')).slice(0, bytes) : null, + }; + const denied = () => Object.assign(new Error('EACCES: permission denied'), { code: 'EACCES' }); + + it('reads a denied listing as a checkout that could not be measured, never as a measured zero', async () => { + const { census } = await run(t0, { + io: { + ...real, + readNames: async (path) => { + if (path.endsWith(join('dev', 'old', 'android'))) throw denied(); + return real.readNames(path); + }, + }, + }); + const old = census?.finds.find(({ path }) => path.endsWith('/dev/old')); + expect(old).toMatchObject({ unreadable: true, bytes: 0, measuredDirs: 0 }); + expect(census?.finds.find(({ path }) => path.endsWith('/dev/fresh'))).toMatchObject({ + unreadable: false, + measuredDirs: 1, + }); + }); + + it('takes a path that does not exist as absent, not as a failure', async () => { + const { census } = await run(t0, { + io: { + ...real, + readNames: async (path) => { + if (path.endsWith(join('dev', 'old', 'apps'))) throw Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); + return real.readNames(path); + }, + }, + }); + expect(census?.finds.find(({ path }) => path.endsWith('/dev/old'))).toMatchObject({ + unreadable: false, + measuredDirs: 1, + }); + }); + + describe.each([ + ['a listing that never answers', 'readNames'], + ['a tag file read that never answers', 'head'], + ] as const)('%s', (_name, which) => { + let release: () => void = () => {}; + afterEach(() => release()); + + it('ends the read at its timeout, leaves that checkout could not be measured and starts no further read', async () => { + const never = new Promise((_, reject) => { + release = () => reject(denied()); + }); + const started: string[] = []; + if (which === 'head') { + const target = join(home, 'dev', 'old', 'target'); + mkdirSync(target); + writeFileSync(join(target, 'CACHEDIR.TAG'), 'Signature: 8a477f597d28d172789f06886806bc55'); + } + const running = run(t0, { + readTimeoutMs: 20, + io: { + ...real, + readNames: (path) => { + if (which === 'readNames' && path.endsWith(join('dev', 'fresh', 'apps'))) { + started.push(which); + return never; + } + return real.readNames(path); + }, + head: () => { + started.push(which); + return never; + }, + }, + }); + while (started.length === 0) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(20); + const { result, census } = await running; + expect(result).toBeUndefined(); + expect(started).toHaveLength(1); + const failed = census?.finds.filter(({ unreadable }) => unreadable) ?? []; + expect(failed).toHaveLength(1); + expect(failed[0]).toMatchObject({ bytes: 0, measuredDirs: 0 }); + expect(census?.finds.filter(({ evaluatedAt }) => evaluatedAt === 0).length).toBeGreaterThan(0); + expect(hungRead()).not.toBeNull(); + }); + }); + }); + + describe('last git activity', () => { + const real = { + readNames: (path: string) => readdir(path), + isDirectory: async (path: string) => (await lstat(path)).isDirectory(), + realpath: (path: string) => realpath(path), + lstat: async (path: string) => { + const info = await lstat(path); + return { + kind: info.isDirectory() ? 'directory' : info.isFile() ? 'file' : 'other', + mtimeMs: info.mtimeMs, + } as const; + }, + head: async (path: string, bytes: number) => + (await lstat(path)).isFile() ? (await readFile(path, 'utf8')).slice(0, bytes) : null, + }; + const at = (iso: string) => new Date(iso); + const touch = (path: string, when: Date) => { + mkdirSync(join(path, '..'), { recursive: true }); + writeFileSync(path, 'x'); + utimesSync(path, when, when); + }; + const git = async (io = real) => { + const { census } = await run(t0, { io }); + return Object.fromEntries( + (census?.finds ?? []).map(({ path, gitActivityAt }) => [path.slice(home.length), gitActivityAt]), + ); + }; + + it('is the newest of HEAD, index and logs/HEAD of a main checkout, and unknown without a git directory', async () => { + const dir = join(home, 'dev', 'old', '.git'); + touch(join(dir, 'HEAD'), at('2026-01-01T00:00:00Z')); + touch(join(dir, 'index'), at('2026-03-01T00:00:00Z')); + touch(join(dir, 'logs', 'HEAD'), at('2026-02-01T00:00:00Z')); + const found = await git(); + expect(found['/dev/old']).toBe(at('2026-03-01T00:00:00Z').getTime()); + expect(found['/dev/fresh']).toBeNull(); + }); + + it('follows a linked worktree to its git directory under .git/worktrees', async () => { + const gitDir = join(base, 'main', '.git', 'worktrees', 'fresh'); + touch(join(gitDir, 'HEAD'), at('2026-04-01T00:00:00Z')); + writeFileSync(join(gitDir, 'gitdir'), `${join(home, 'dev', 'fresh', '.git')}\n`); + writeFileSync(join(home, 'dev', 'fresh', '.git'), `gitdir: ${gitDir}\n`); + expect((await git())['/dev/fresh']).toBe(at('2026-04-01T00:00:00Z').getTime()); + }); + + it('accepts a relative gitdir that validates', async () => { + const gitDir = join(base, 'main', '.git', 'worktrees', 'fresh'); + touch(join(gitDir, 'HEAD'), at('2026-05-01T00:00:00Z')); + writeFileSync(join(gitDir, 'gitdir'), `${join(home, 'dev', 'fresh', '.git')}\n`); + writeFileSync(join(home, 'dev', 'fresh', '.git'), `gitdir: ${relative(join(home, 'dev', 'fresh'), gitDir)}\n`); + expect((await git())['/dev/fresh']).toBe(at('2026-05-01T00:00:00Z').getTime()); + }); + + it('is unknown for a .git/worktrees directory that does not name this checkout back, however it is named', async () => { + const foreign = join(base, 'main', '.git', 'worktrees', 'foreign'); + touch(join(foreign, 'HEAD'), at('2026-04-01T00:00:00Z')); + writeFileSync(join(foreign, 'gitdir'), `${join(base, 'somewhere', 'else', '.git')}\n`); + writeFileSync(join(home, 'dev', 'fresh', '.git'), `gitdir: ${foreign}\n`); + expect((await git())['/dev/fresh']).toBeNull(); + }); + + it('is unknown when a component of the git directory is a symlink', async () => { + const target = join(base, 'real', '.git', 'worktrees', 'fresh'); + touch(join(target, 'HEAD'), at('2026-04-01T00:00:00Z')); + writeFileSync(join(target, 'gitdir'), `${join(home, 'dev', 'fresh', '.git')}\n`); + symlinkSync(join(base, 'real'), join(base, 'link')); + const viaLink = join(base, 'link', '.git', 'worktrees', 'fresh'); + writeFileSync(join(home, 'dev', 'fresh', '.git'), `gitdir: ${viaLink}\n`); + expect((await git())['/dev/fresh']).toBeNull(); + }); + + it('is unknown for a .git file that names a directory outside a .git/worktrees directory', async () => { + const elsewhere = join(base, 'elsewhere', 'gitdir'); + touch(join(elsewhere, 'HEAD'), at('2026-04-01T00:00:00Z')); + writeFileSync(join(home, 'dev', 'fresh', '.git'), `gitdir: ${elsewhere}\n`); + writeFileSync(join(home, 'dev', 'old', '.git'), 'gitdir: relative/.git/worktrees/x\n'); + const found = await git(); + expect(found['/dev/fresh']).toBeNull(); + expect(found['/dev/old']).toBeNull(); + }); + + it('is unknown, not old, when a git file is a symlink', async () => { + const dir = join(home, 'dev', 'old', '.git'); + touch(join(base, 'target-file'), at('2020-01-01T00:00:00Z')); + mkdirSync(dir, { recursive: true }); + symlinkSync(join(base, 'target-file'), join(dir, 'HEAD')); + expect((await git())['/dev/old']).toBeNull(); + }); + + it('is unknown when the read never answers, and the run does not wait on it', async () => { + touch(join(home, 'dev', 'fresh', '.git', 'HEAD'), at('2026-01-01T00:00:00Z')); + let release: () => void = () => {}; + const never = new Promise((_, reject) => { + release = () => reject(new Error('released')); + }); + const started: string[] = []; + const running = run(t0, { + readTimeoutMs: 20, + io: { + ...real, + lstat: (path) => { + if (path.endsWith(join('dev', 'fresh', '.git'))) { + started.push(path); + return never; + } + return real.lstat(path); + }, + }, + }); + while (started.length === 0) await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(20); + const { census } = await running; + release(); + expect(started).toHaveLength(1); + expect(census?.finds.find(({ path }) => path.endsWith('/dev/fresh'))?.gitActivityAt).toBeNull(); + }); + }); + + describe('finds whose validity could not be established', () => { + it('are kept as unchecked with their earlier measurement and age, and make the census retry within the hour', async () => { + const { census: first } = await run(t0); + expect(first?.pending).toBe(false); + const scanRules = await import('../workspace/scan-rules.ts'); + const real = scanRules.scanEntryValid; + vi.spyOn(scanRules, 'scanEntryValid').mockImplementation(async (path, options) => + path.endsWith(join('dev', 'fresh')) || path.endsWith('.mytool') ? 'pending' : real(path, options), + ); + const { census } = await run(t0 + day); + const unchecked = census?.finds.filter(({ kind }) => kind === 'unchecked') ?? []; + expect(unchecked.map(({ path }) => path.slice(home.length)).toSorted()).toEqual(['/.mytool', '/dev/fresh']); + for (const find of unchecked) expect(find).toMatchObject({ evaluatedAt: t0, measuredDirs: 1 }); + expect(census?.pending).toBe(true); + expect(census?.finds.some(({ path }) => path.endsWith('/dev/old'))).toBe(true); + }); + + it('list a find whose metadata read was denied as unchecked, keep the census pending, and never print that everything is managed', async () => { + const { REAL_RECOGNITION_IO } = await import('../workspace/scan-rules.ts'); + const real = REAL_RECOGNITION_IO.lstat; + vi.spyOn(REAL_RECOGNITION_IO, 'lstat').mockImplementation(async (path) => { + if (path.endsWith('.mytool') || path.endsWith(join('dev', 'fresh'))) + throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + return real(path); + }); + const { census } = await run(t0); + expect( + census?.finds + .filter(({ kind }) => kind === 'unchecked') + .map(({ path }) => path.slice(home.length)) + .toSorted(), + ).toEqual(['/.mytool', '/dev/fresh']); + expect(census?.pending).toBe(true); + const report = discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: { value: [join(home, 'dev', 'tools')], error: null }, + home, + canonical: null, + }); + expect(report.groups.filter(({ kind }) => kind === 'unchecked').map(({ command }) => command)).toEqual([ + null, + null, + ]); + expect(report.complete).toBe(false); + }); + + it('list a find whose manifest no longer parses as unchecked, keep the census pending, and never print that everything is managed', async () => { + writeFileSync(join(home, 'dev', 'fresh', 'package.json'), '<<<<<<< HEAD\n{}\n=======\n{}\n>>>>>>> main\n'); + const { census } = await run(t0); + expect( + census?.finds.filter(({ kind }) => kind === 'unchecked').map(({ path }) => path.slice(home.length)), + ).toEqual(['/dev/fresh']); + expect(census?.pending).toBe(true); + const report = discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: { value: [join(home, 'dev', 'tools')], error: null }, + home, + canonical: null, + }); + expect(report.complete).toBe(false); + }); + + it('drop a find that is no longer a project, as only that proves it gone', async () => { + writeFileSync(join(home, 'dev', 'fresh', 'package.json'), JSON.stringify({ dependencies: {} })); + const { census } = await run(t0); + expect(census?.finds.some(({ path }) => path.endsWith('/dev/fresh'))).toBe(false); + }); + + it('keep a find whose manifest stops parsing at the next full scan, record its folder unreadable, and list it as unchecked', async () => { + writeFileSync(join(home, 'dev', 'fresh', 'package.json'), '<<<<<<< HEAD\n{}\n=======\n{}\n>>>>>>> main\n'); + await scanAt(t0 + 8 * day); + const scanned = readDiscoveryScanFile(); + const fresh = join(home, 'dev', 'fresh'); + expect(scanned.status === 'read' && scanned.scan?.found).toContain(fresh); + expect(scanned.status === 'read' && scanned.scan?.unreadable).toContain(fresh); + const { census } = await run(t0 + 8 * day); + expect(census?.finds.find(({ path }) => path === fresh)).toMatchObject({ kind: 'unchecked' }); + expect(census?.pending).toBe(true); + expect( + discoveryReport({ + scan: scanned, + census: readDiscoveryCensusFile(), + setting: { value: [join(home, 'dev', 'tools')], error: null }, + home, + canonical: null, + }).complete, + ).toBe(false); + }); + + it('make a checkout whose layout could not be read retried within the hour, not the day', async () => { + const denied = Object.assign(new Error('EACCES'), { code: 'EACCES' }); + const { census } = await run(t0, { + io: { + readNames: async (path) => { + if (path.endsWith(join('dev', 'old', 'android'))) throw denied; + return readdir(path); + }, + isDirectory: async (path) => (await lstat(path)).isDirectory(), + realpath: (path) => realpath(path), + lstat: async (path) => { + const info = await lstat(path); + return { kind: info.isDirectory() ? 'directory' : info.isFile() ? 'file' : 'other', mtimeMs: info.mtimeMs }; + }, + head: async () => null, + }, + }); + expect(census?.finds.find(({ path }) => path.endsWith('/dev/old'))?.unreadable).toBe(true); + expect(census?.pending).toBe(true); + expect(censusDue(census, census?.scanRanAt ?? 0, census?.roots ?? [], t0 + hour)).toBe(true); + }); + }); + + describe('a walk whose du exited with status 1', () => { + afterEach(() => resetExecutor()); + + it('is recorded as measured with a lower bound, read back as partial, and worded as at least', async () => { + const real = getExecutor(); + setExecutor({ + ...real, + runFileAsync: async (file, args, options) => { + if (file === 'du' || args.includes('du')) + throw Object.assign(new Error('du: Permission denied'), { status: 1, stdout: '12\t/walked\n' }); + return real.runFileAsync(file, args, options); + }, + }); + const { census } = await run(t0); + const find = census?.finds.find(({ path }) => path.endsWith('/dev/old')); + expect(find).toMatchObject({ measuredDirs: 1, failedDirs: 0, partialDirs: 1, bytes: 12 * 1024 }); + const dir = join(home, 'dev', 'old', 'android', 'build'); + expect( + await diskUsage.refreshMeasure(dir, { timeoutMs: 1000, maxAgeMs: 86_400_000, knownToExist: true }), + ).toMatchObject({ + outcome: 'fresh', + partial: true, + }); + expect(readDiskUsage(dir)).not.toHaveProperty('partial'); + const report = discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: { value: [join(home, 'dev', 'tools')], error: null }, + home, + canonical: null, + }); + expect(report.groups[0]).toMatchObject({ partial: 2 }); + expect(discoveryLines(report, home).join('\n')).toContain('at least'); + expect(lowDiskLine(report, home)).toContain('at least'); + }); + }); + + describe('below a configured root', () => { + const under = (root: string) => + saveConfig({ version: 2, projects: {}, repos: {}, maintenance: { discoverRoots: [root] } }); + const reportOf = (root: string) => + discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: { value: [root], error: null }, + home, + canonical: null, + }); + + it('lists a find two levels down with an adopt command for its own parent, and not a direct child discovery selected', async () => { + const root = join(home, '.exact'); + checkout(join(root, 'direct')); + await scanAt(t0 + hour); + under(root); + const { census } = await run(t0 + hour); + const names = (census?.finds ?? []).map(({ path }) => path.slice(home.length)).toSorted(); + expect(names).toContain('/.exact/exact2/a'); + expect(names).not.toContain('/.exact/direct'); + const group = reportOf(root).groups.find(({ folder }) => folder === join(root, 'exact2')); + expect(group?.command).toContain(JSON.stringify([root, join(root, 'exact2')])); + }); + + it('keeps a find whose metadata read was denied as unchecked', async () => { + const root = join(home, '.exact'); + under(root); + const { REAL_RECOGNITION_IO } = await import('../workspace/scan-rules.ts'); + const real = REAL_RECOGNITION_IO.lstat; + vi.spyOn(REAL_RECOGNITION_IO, 'lstat').mockImplementation(async (path) => { + if (path.endsWith(join('exact2', 'b'))) throw Object.assign(new Error('EACCES'), { code: 'EACCES' }); + return real(path); + }); + const { census } = await run(t0); + expect(census?.finds.find(({ path }) => path.endsWith(join('exact2', 'b')))).toMatchObject({ kind: 'unchecked' }); + expect(census?.pending).toBe(true); + }); + }); + + describe('errors from measuring', () => { + it('lets an unverified child cleanup propagate as the same object', async () => { + const failure = Object.assign(new Error('unverified'), { reportCleanup: true }); + vi.spyOn(diskUsage, 'refreshMeasure').mockRejectedValue(failure); + await expect(run(t0)).rejects.toBe(failure); + }); + + it('lets an unverified child cleanup propagate even when the operation was aborted, as the same object', async () => { + const failure = Object.assign(new Error('unverified'), { reportCleanup: true }); + const controller = new AbortController(); + vi.spyOn(diskUsage, 'refreshMeasure').mockImplementation(async () => { + controller.abort(); + throw failure; + }); + await expect(run(t0, {}, testContext({ signal: controller.signal }))).rejects.toBe(failure); + }); + + it('passes every measured directory to refreshMeasure as known to exist, so no synchronous probe runs on it', async () => { + const refresh = vi.spyOn(diskUsage, 'refreshMeasure'); + await run(t0); + expect(refresh).toHaveBeenCalled(); + for (const [, options] of refresh.mock.calls) expect(options.knownToExist).toBe(true); + }); + + it('contains any other error, returns its message and writes nothing', async () => { + vi.spyOn(diskUsage, 'refreshMeasure').mockRejectedValue(new Error('disk exploded')); + const { result, census } = await run(t0); + expect(result).toBe('disk exploded'); + expect(census).toBeNull(); + }); + }); +}); diff --git a/packages/stim-cli/src/__tests__/discovery-report.test.ts b/packages/stim-cli/src/__tests__/discovery-report.test.ts new file mode 100644 index 000000000..a160ca960 --- /dev/null +++ b/packages/stim-cli/src/__tests__/discovery-report.test.ts @@ -0,0 +1,536 @@ +import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import type { CensusFind, DiscoveryCensusFile, DiscoveryScan, DiscoveryScanFile } from '@stim-cli/core/state'; +import { + adoptCommand, + discoveryLines, + discoveryReport, + lowDiskLine, + type DiscoverRootsValue, +} from '../workspace/discovery-report.ts'; + +const home = '/Users/me'; +const now = Date.UTC(2026, 10, 20); +const day = 24 * 60 * 60_000; +const GB = 1024 ** 3; + +const scan: DiscoveryScan = { + version: 2, + ranAt: Date.UTC(2026, 10, 14), + completedAt: Date.UTC(2026, 10, 14), + lowScanAt: 0, + durationMs: 1000, + trigger: 'weekly', + visited: 4200, + found: [], + dotFolderFound: [], + unreadable: [], + truncated: false, + truncation: 'none', +}; + +function find(over: Partial & Pick): CensusFind { + return { + kind: 'reduce-only', + folder: over.path.slice(0, over.path.lastIndexOf('/')), + adoptable: true, + evaluatedAt: now, + dirs: 1, + measuredDirs: 1, + failedDirs: 0, + partialDirs: 0, + bytes: GB, + unreadable: false, + gitActivityAt: null, + ...over, + }; +} + +const census = (finds: CensusFind[]): DiscoveryCensusFile => ({ + status: 'read', + census: { + version: 1, + at: now, + scanRanAt: scan.ranAt, + roots: [], + canonicalRoots: [], + pending: false, + finds, + }, +}); + +const realpathOrSelf = (path: string): string => { + try { + return realpathSync(path); + } catch { + return path; + } +}; +const same = (path: string) => path; +const noRoots: DiscoverRootsValue = { value: [], error: null }; + +function report( + finds: CensusFind[], + setting: DiscoverRootsValue = noRoots, + scanFile: DiscoveryScanFile = { status: 'read', scan }, +) { + return discoveryReport({ + scan: scanFile, + census: census(finds), + setting, + home, + canonical: realpathOrSelf, + }); +} + +describe('adoptCommand', () => { + it('builds the pasteable command from the current value, quoting a path with a single quote for the shell', () => { + expect(adoptCommand({ value: ['/a'], error: null }, "/Users/me/it's here")).toBe( + `stim settings set maintenance.discoverRoots '["/a","/Users/me/it'\\''s here"]' --scope machine`, + ); + }); + + it('offers none when the current value does not read', () => { + expect(adoptCommand({ value: undefined, error: 'maintenance is not an object' }, '/a')).toBeNull(); + expect(adoptCommand({ value: 'x', error: null }, '/a')).toBeNull(); + }); +}); + +describe('discoveryReport', () => { + it('groups by containing folder, reduce-only first, each with its own sum and adopt command', () => { + const result = report([ + find({ + path: '/Users/me/.exact/exact2/a', + kind: 'dot-folder', + folder: '/Users/me/.exact/exact2', + bytes: 80 * GB, + }), + find({ path: '/Users/me/dev/app1', bytes: 2 * GB }), + find({ path: '/Users/me/dev/app2', bytes: 3 * GB }), + ]); + expect(result.groups.map(({ folder, kind, checkouts, bytes }) => ({ folder, kind, checkouts, bytes }))).toEqual([ + { folder: '/Users/me/dev', kind: 'reduce-only', checkouts: 2, bytes: 5 * GB }, + { folder: '/Users/me/.exact/exact2', kind: 'dot-folder', checkouts: 1, bytes: 80 * GB }, + ]); + expect(result.groups[1]?.command).toBe( + `stim settings set maintenance.discoverRoots '["/Users/me/.exact/exact2"]' --scope machine`, + ); + }); + + it('shows a folder that maintenance.discoverRoots names since the daemon classified, with no command and the reason', () => { + const result = report([find({ path: '/Users/me/dev/app1' }), find({ path: '/Users/me/work/app2' })], { + value: ['~/dev'], + error: null, + }); + const byFolder = Object.fromEntries(result.groups.map((group) => [group.folder, group])); + expect(byFolder['/Users/me/dev']).toMatchObject({ command: null, commandProblem: 'already-named' }); + expect(byFolder['/Users/me/work']?.command).toContain(`'["~/dev","/Users/me/work"]'`); + }); + + it('offers no command for the home folder, and none when the setting does not read, and says which', () => { + const [direct] = report([ + find({ path: '/Users/me/.mytool', kind: 'dot-folder', folder: '/Users/me', adoptable: false }), + ]).groups; + expect(direct).toMatchObject({ command: null, commandProblem: 'too-broad' }); + const [broken] = report([find({ path: '/Users/me/dev/a' })], { value: undefined, error: 'bad' }).groups; + expect(broken).toMatchObject({ command: null, commandProblem: 'setting-unreadable' }); + }); + + it('counts a checkout that failed and one not measured yet apart from the measured, never as zero bytes', () => { + const [group] = report([ + find({ path: '/Users/me/dev/a', bytes: GB }), + find({ path: '/Users/me/dev/b', bytes: 0, measuredDirs: 0, failedDirs: 1 }), + find({ path: '/Users/me/dev/c', bytes: 0, measuredDirs: 0, failedDirs: 0, evaluatedAt: 0, dirs: 0 }), + ]).groups; + expect(group).toMatchObject({ measured: 1, failed: 1, notMeasuredYet: 1, bytes: GB }); + expect( + discoveryLines(report([find({ path: '/Users/me/dev/b', bytes: 0, measuredDirs: 0, failedDirs: 1 })]), home).join( + '\n', + ), + ).toContain('build output not measured'); + }); + + it('counts a checkout whose layout could not be read as could not be measured, never as a measured zero', () => { + const [group] = report([ + find({ path: '/Users/me/dev/a', bytes: 0, dirs: 0, measuredDirs: 0, unreadable: true }), + ]).groups; + expect(group).toMatchObject({ measured: 0, failed: 1, notMeasuredYet: 0 }); + }); + + it('shows no measurement of a census made from an older scan than the scan file holds', () => { + const older = { ...scan, ranAt: scan.ranAt + 1 }; + const result = discoveryReport({ + scan: { status: 'read', scan: older }, + census: census([find({ path: '/Users/me/dev/a', bytes: 5 * GB })]), + setting: noRoots, + home, + canonical: same, + }); + expect(result.censusCurrent).toBe(false); + expect(result.groups[0]).toMatchObject({ bytes: 0, measured: 0, notMeasuredYet: 1 }); + expect(lowDiskLine(result, home)).toBeNull(); + }); + + it('shows the age of the oldest measurement, and a folder the setting names through a symlink as already named', () => { + const [group] = report([ + find({ path: '/Users/me/dev/a', evaluatedAt: now - 5 * day }), + find({ path: '/Users/me/dev/b' }), + ]).groups; + expect(group?.oldestMeasuredAt).toBe(now - 5 * day); + const dir = mkdtempSync(join(tmpdir(), 'stim-report-')); + try { + const real = realpathSync(dir); + mkdirSync(join(real, 'target')); + symlinkSync(join(real, 'target'), join(real, 'link')); + const result = report([find({ path: `${real}/target/app`, folder: join(real, 'target') })], { + value: [join(real, 'link')], + error: null, + }); + expect(result.groups).toMatchObject([{ command: null, commandProblem: 'already-named' }]); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('keeps an unreadable, absent and unusable scan or census apart', () => { + const base = { setting: noRoots, home, canonical: same }; + const missing: DiscoveryCensusFile = { status: 'absent' }; + expect(discoveryReport({ ...base, scan: { status: 'absent' }, census: missing }).scan).toEqual({ + status: 'absent', + }); + expect(discoveryReport({ ...base, scan: { status: 'unreadable' }, census: missing }).scan).toEqual({ + status: 'unreadable', + }); + expect(discoveryReport({ ...base, scan: { status: 'read', scan: null }, census: missing }).scan).toEqual({ + status: 'unusable', + }); + const scanFile: DiscoveryScanFile = { status: 'read', scan }; + const withScan = { ...base, scan: scanFile }; + expect(discoveryReport({ ...withScan, census: missing }).census).toBe('absent'); + expect(discoveryReport({ ...withScan, census: { status: 'unreadable' } }).census).toBe('unreadable'); + expect(discoveryReport({ ...withScan, census: { status: 'read', census: null } }).census).toBe('unusable'); + }); +}); + +describe('sizes', () => { + it('prints a group measured with no build output as zero, not as a megabyte', () => { + const text = discoveryLines(report([find({ path: '/Users/me/dev/a', bytes: 0 })]), home).join('\n'); + expect(text).toContain('0 B of build output'); + }); + + it('reads a checkout whose du was partial as at least, never as exact', () => { + const result = report([find({ path: '/Users/me/dev/a', partialDirs: 1 })]); + expect(result.groups[0]).toMatchObject({ measured: 1, partial: 1 }); + expect(discoveryLines(result, home).join('\n')).toContain('at least 1.0 GB'); + expect(lowDiskLine(result, home)).toContain('at least'); + }); +}); + +describe('the largest folder on the low-disk line', () => { + it('reads as at least when that folder holds a partial, failed or not yet measured checkout, and exact otherwise', () => { + const exact = lowDiskLine(report([find({ path: '/Users/me/dev/a', bytes: 2 * GB })]), home); + expect(exact).toContain('(2.0 GB)'); + for (const over of [{ partialDirs: 1 }, { failedDirs: 1, measuredDirs: 0 }, { evaluatedAt: 0 }]) { + const line = lowDiskLine(report([find({ path: '/Users/me/dev/a', bytes: 2 * GB, ...over })]), home); + expect(line).toContain('(at least 2.0 GB)'); + } + }); + + it('lists a group below a configured root with its own command, and the root itself as already named', () => { + const below = find({ path: '/Users/me/.exact/exact2/a', folder: '/Users/me/.exact/exact2', kind: 'dot-folder' }); + const direct = find({ path: '/Users/me/.exact/b', folder: '/Users/me/.exact', kind: 'dot-folder' }); + const result = report([below, direct], { value: ['/Users/me/.exact'], error: null }); + const byFolder = Object.fromEntries(result.groups.map((group) => [group.folder, group])); + expect(byFolder['/Users/me/.exact/exact2']?.command).toContain('/Users/me/.exact/exact2'); + expect(byFolder['/Users/me/.exact']).toMatchObject({ command: null, commandProblem: 'already-named' }); + }); +}); + +describe('completeness', () => { + const finished = (over: Partial = {}) => ({ status: 'read', scan: { ...scan, ...over } }) as const; + + it('is one value that the totals, the figures and the all-managed sentence follow: a truncated scan with measured finds reads at least', () => { + const complete = report([find({ path: '/Users/me/dev/a', bytes: GB })]); + expect(complete.complete).toBe(true); + expect(lowDiskLine(complete, home)).toContain('hold 1.0 GB'); + for (const over of [ + { truncated: true, truncation: 'budget' as const }, + { truncated: true, truncation: 'failed' as const }, + { unreadable: ['/Users/me/Documents/x'] }, + ]) { + const result = report([find({ path: '/Users/me/dev/a', bytes: GB })], noRoots, finished(over)); + expect(result.complete).toBe(false); + expect(lowDiskLine(result, home)).toContain('hold at least 1.0 GB'); + expect(discoveryLines(result, home).join('\n')).toContain('at least 1.0 GB'); + } + }); + + it('is false for a pending census or one made from another scan, and true only when nothing is unknown', () => { + const pending: DiscoveryCensusFile = { + status: 'read', + census: { version: 1, at: now, scanRanAt: scan.ranAt, roots: [], canonicalRoots: [], pending: true, finds: [] }, + }; + const input = { scan: finished(), setting: noRoots, home, canonical: same }; + expect(discoveryReport({ ...input, census: pending }).complete).toBe(false); + expect(discoveryReport({ ...input, census: census([]) }).complete).toBe(true); + expect(discoveryReport({ ...input, scan: finished({ ranAt: scan.ranAt + 1 }), census: census([]) }).complete).toBe( + false, + ); + }); + + it('lists a find under a broad configured root with the home-folder wording, and the line follows', () => { + const file: DiscoveryCensusFile = { + status: 'read', + census: { + version: 1, + at: now, + scanRanAt: scan.ranAt, + roots: ['/Users/me'], + canonicalRoots: [], + pending: false, + finds: [find({ path: '/Users/me/app', folder: '/Users/me', adoptable: false })], + }, + }; + const result = discoveryReport({ + scan: finished(), + census: file, + setting: { value: ['~'], error: null }, + home, + canonical: same, + }); + expect(result.groups).toMatchObject([{ folder: '/Users/me', command: null, commandProblem: 'too-broad' }]); + expect(result.complete).toBe(true); + expect(lowDiskLine(result, home)).toContain('hold 1.0 GB'); + }); + + it('lists an unchecked direct child of a configured root with no adopt command', () => { + const result = report([find({ path: '/Users/me/dev/a', kind: 'unchecked' })], { + value: ['/Users/me/dev'], + error: null, + }); + expect(result.groups).toMatchObject([{ kind: 'unchecked', command: null }]); + expect(result.complete).toBe(false); + }); +}); + +describe('what the listing may claim', () => { + it('says every checkout is managed only for a finished scan, a census made from it and no unchecked find', () => { + const lines = (scanOver: Partial, finds: CensusFind[] = []) => + discoveryLines(report(finds, noRoots, { status: 'read', scan: { ...scan, ...scanOver } }), home); + expect(lines({}).join('\n')).toContain('Every checkout it found is managed.'); + expect(lines({ truncated: true, truncation: 'budget' }).join('\n')).not.toContain( + 'Every checkout it found is managed.', + ); + expect(lines({ unreadable: ['/Users/me/Documents/x'] }).join('\n')).not.toContain( + 'Every checkout it found is managed.', + ); + expect(lines({ ranAt: scan.ranAt + 1 }).join('\n')).not.toContain('Every checkout it found is managed.'); + }); + + it('lists a find that could not be checked as its own group, with no command, never as managed', () => { + const result = report([ + find({ path: '/Users/me/dev/a', kind: 'unchecked', evaluatedAt: now - 3 * day, gitActivityAt: now - 9 * day }), + ]); + expect(result.groups).toMatchObject([ + { kind: 'unchecked', checkouts: 1, command: null, commandProblem: 'unchecked', oldestMeasuredAt: now - 3 * day }, + ]); + expect(lowDiskLine(result, home)).toContain('at least'); + }); + + it('makes no filesystem call without a canonicaliser: it names a folder by the recorded roots, and says the setting changed', () => { + const recorded = (roots: string[]): DiscoveryCensusFile => ({ + status: 'read', + census: { + version: 1, + at: now, + scanRanAt: scan.ranAt, + roots, + canonicalRoots: ['/real/dev'], + pending: false, + finds: [find({ path: '/real/dev/a', folder: '/real/dev' })], + }, + }); + const through = (file: DiscoveryCensusFile, value: string[]) => + discoveryReport({ + scan: { status: 'read', scan }, + census: file, + setting: { value, error: null }, + home, + canonical: null, + }); + expect(through(recorded(['/Users/me/link']), ['/Users/me/link']).groups).toMatchObject([ + { folder: '/real/dev', commandProblem: 'already-named' }, + ]); + const changed = through(recorded(['/Users/me/link']), ['/Users/me/other']); + expect(changed.rootsChangedSinceCensus).toBe(true); + expect(changed.groups).toHaveLength(1); + expect(changed.complete).toBe(false); + expect(lowDiskLine(changed, home)).toBeNull(); + }); +}); + +describe('members', () => { + it('lists the checkouts oldest git activity first, unknown last, as ISO times or null', () => { + const [group] = report([ + find({ path: '/Users/me/dev/unknown' }), + find({ path: '/Users/me/dev/new', gitActivityAt: now - day }), + find({ path: '/Users/me/dev/old', gitActivityAt: now - 40 * day }), + ]).groups; + expect(group?.members).toEqual([ + { path: '/Users/me/dev/old', gitActivityAt: new Date(now - 40 * day).toISOString() }, + { path: '/Users/me/dev/new', gitActivityAt: new Date(now - day).toISOString() }, + { path: '/Users/me/dev/unknown', gitActivityAt: null }, + ]); + }); + + it('prints the date or unknown for each, and shows no date for a census made from an older scan', () => { + const finds = [ + find({ path: '/Users/me/dev/old', gitActivityAt: now - 40 * day }), + find({ path: '/Users/me/dev/x' }), + ]; + const text = discoveryLines(report(finds), home).join('\n'); + expect(text).toContain('~/dev/old: last git activity 2026-10-11'); + expect(text).toContain('~/dev/x: last git activity unknown'); + const stale = discoveryReport({ + scan: { status: 'read', scan: { ...scan, ranAt: scan.ranAt + 1 } }, + census: census(finds), + setting: noRoots, + home, + canonical: same, + }); + expect(stale.groups[0]?.members.every(({ gitActivityAt }) => gitActivityAt === null)).toBe(true); + }); +}); + +describe('discoveryLines', () => { + it('words the two kinds differently, shows the command, when the scan ran and what it could not read', () => { + const lines = discoveryLines( + discoveryReport({ + scan: { + status: 'read', + scan: { ...scan, truncated: true, truncation: 'budget', unreadable: ['/Users/me/Documents/x'] }, + }, + census: census([ + find({ + path: '/Users/me/.exact/exact2/a', + kind: 'dot-folder', + folder: '/Users/me/.exact/exact2', + bytes: 80 * GB, + }), + find({ path: '/Users/me/dev/old', bytes: 2 * GB }), + ]), + setting: noRoots, + home, + canonical: same, + }), + home, + ).join('\n'); + expect(lines).toContain('last scanned the home folder on 2026-11-14'); + expect(lines).toContain('stopped early (time budget)'); + expect(lines).toContain('could not read ~/Documents/x'); + expect(lines).toContain('~/dev: 1 checkout, at least 2.0 GB of build output'); + expect(lines).toContain('~/.exact/exact2: 1 checkout, at least 80.0 GB of build output'); + expect(lines).toContain('Stim reduces their build output when idle and never removes them'); + expect(lines).toContain('Stim cannot tell a checkout from an installed tool, so it does nothing with them'); + expect(lines).toContain(`'["/Users/me/.exact/exact2"]'`); + expect(lines).toContain('Stim has not checked whether they are in use or hold uncommitted or unpushed work'); + }); +}); + +describe('lowDiskLine', () => { + it('names the total and the largest reduce-only folder, though a dot-folder group is larger', () => { + const line = lowDiskLine( + report([ + find({ + path: '/Users/me/.exact/exact2/a', + kind: 'dot-folder', + folder: '/Users/me/.exact/exact2', + bytes: 80 * GB, + }), + find({ path: '/Users/me/dev/app1', bytes: 2 * GB }), + ]), + home, + ); + expect(line).toContain('hold 82.0 GB'); + expect(line?.endsWith(`stim settings set maintenance.discoverRoots '["/Users/me/dev"]' --scope machine`)).toBe( + true, + ); + }); + + it('says at least when some output is unmeasured, and adds no command when only a dot-folder holds output', () => { + const line = lowDiskLine( + report([ + find({ path: '/Users/me/.t/a', kind: 'dot-folder', folder: '/Users/me/.t', bytes: 5 * GB }), + find({ + path: '/Users/me/.t/b', + kind: 'dot-folder', + folder: '/Users/me/.t', + bytes: 0, + measuredDirs: 0, + failedDirs: 1, + }), + ]), + home, + ); + expect(line).toBe( + 'Stim: checkouts it does not fully manage hold at least 5.0 GB of build output; stim doctor lists them.', + ); + }); + + it('adds nothing when nothing is measured or nothing is unmanaged', () => { + expect(lowDiskLine(report([]), home)).toBeNull(); + expect( + lowDiskLine( + report([find({ path: '/Users/me/dev/a', bytes: 0, measuredDirs: 0, evaluatedAt: 0, dirs: 0 })]), + home, + ), + ).toBeNull(); + }); +}); + +describe('a list older than the state it is read against', () => { + const lines = (file: DiscoveryCensusFile, value: string[]) => + discoveryLines( + discoveryReport({ + scan: { status: 'read', scan }, + census: file, + setting: { value, error: null }, + home, + canonical: same, + }), + home, + ); + const recorded = (roots: string[], at = now, finds = [find({ path: '/Users/me/dev/a' })]): DiscoveryCensusFile => ({ + status: 'read', + census: { version: 1, at, scanRanAt: scan.ranAt, roots, canonicalRoots: [], pending: false, finds }, + }); + + it('says the list predates the setting before the group, and offers no command or reason the old state decides', () => { + const changed = lines(recorded([]), ['/Users/me/dev']); + const unchanged = lines(recorded(['/Users/me/dev']), ['/Users/me/dev']); + const group = changed.findIndex((line) => line.startsWith('~/dev:')); + expect(changed.findIndex((line) => line.includes('maintenance.discoverRoots'))).toBeLessThan(group); + expect(changed.slice(group + 1).some((line) => line.includes('maintenance.discoverRoots'))).toBe(false); + expect(changed.some((line) => line.includes('stim settings set'))).toBe(false); + expect(unchanged.some((line) => line.includes('discovery did not select'))).toBe(true); + expect(changed.some((line) => line.includes('discovery did not select'))).toBe(false); + }); + + it('never reads a measurement older than a day when the census was made as a current exact figure', () => { + const old = recorded([], now, [find({ path: '/Users/me/dev/a', evaluatedAt: now - 3 * day })]); + const result = discoveryReport({ + scan: { status: 'read', scan }, + census: old, + setting: noRoots, + home, + canonical: same, + }); + expect(result.complete).toBe(false); + const text = discoveryLines(result, home).join('\n'); + expect(text).toContain('at least 1.0 GB'); + expect(text).toContain('oldest measurement 2026-11-17'); + expect(lowDiskLine(result, home)).toContain('at least'); + }); +}); diff --git a/packages/stim-cli/src/__tests__/disk-usage-known-path.test.ts b/packages/stim-cli/src/__tests__/disk-usage-known-path.test.ts new file mode 100644 index 000000000..0e6f30db8 --- /dev/null +++ b/packages/stim-cli/src/__tests__/disk-usage-known-path.test.ts @@ -0,0 +1,44 @@ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, beforeEach, expect, it, vi } from 'vitest'; +import { refreshMeasure } from '../disk-usage.ts'; + +const probed = vi.hoisted(() => ({ paths: new Array() })); +vi.mock('node:fs', async (original) => { + const real = await original(); + return { + ...real, + existsSync: (path: string) => { + probed.paths.push(String(path)); + return real.existsSync(path); + }, + }; +}); + +let base: string; +let dir: string; + +beforeEach(() => { + base = mkdtempSync(join(tmpdir(), 'stim-known-path-')); + vi.stubEnv('STIM_HOME', join(base, 'home')); + dir = join(base, 'out'); + mkdirSync(dir); + writeFileSync(join(dir, 'x'), 'x'); + probed.paths.length = 0; +}); + +afterEach(() => { + vi.unstubAllEnvs(); + rmSync(base, { recursive: true, force: true }); +}); + +it('probes the folder with existsSync unless the caller established that it exists', async () => { + await refreshMeasure(dir, { timeoutMs: 10_000, private: false }); + expect(probed.paths).toContain(dir); + probed.paths.length = 0; + rmSync(join(base, 'home'), { recursive: true, force: true }); + const { outcome } = await refreshMeasure(dir, { timeoutMs: 10_000, private: false, knownToExist: true }); + expect(outcome).toBe('measured'); + expect(probed.paths).not.toContain(dir); +}); diff --git a/packages/stim-cli/src/__tests__/disk-usage.integration.test.ts b/packages/stim-cli/src/__tests__/disk-usage.integration.test.ts index 942866d27..966bf447a 100644 --- a/packages/stim-cli/src/__tests__/disk-usage.integration.test.ts +++ b/packages/stim-cli/src/__tests__/disk-usage.integration.test.ts @@ -163,3 +163,41 @@ test('a walk records how long it took, and an entry written without it still rea expect(readDiskUsage(plain)).toEqual(walked.measure); expect((await refreshMeasure(plain, { ...options, now: () => clock })).outcome).toBe('fresh'); }); + +test('a walk whose du exited with status 1 is recorded as partial, a later complete walk clears it, and an entry without the field reads as complete', async () => { + const plain = join(scratch, 'partial'); + mkdirSync(plain); + writeFileSync(join(plain, 'blob'), Buffer.alloc(10_000)); + let clock = Date.parse('2026-11-04T10:00:00.000Z'); + let denied = true; + const real = getExecutor(); + setExecutor({ + ...real, + runFileAsync: (file, args, opts) => { + if (denied && (file === 'du' || args.includes('du'))) + return Promise.reject(Object.assign(new Error('du: Permission denied'), { status: 1, stdout: '8\t/walked\n' })); + return real.runFileAsync(file, args, opts); + }, + }); + const walk = (extra = {}) => + refreshMeasure(plain, { ...options, private: false, maxAgeMs: 60_000, now: () => clock, ...extra }); + + const first = await walk(); + expect(first).toMatchObject({ outcome: 'measured', partial: true }); + expect(readJsonObject(diskUsageCacheFile(plain))?.partial).toBe(true); + expect(await walk()).toMatchObject({ outcome: 'fresh', partial: true }); + + clock += 120_000; + denied = false; + const second = await walk(); + expect(second.outcome).toBe('measured'); + expect(second).not.toHaveProperty('partial'); + expect(readJsonObject(diskUsageCacheFile(plain))?.partial).toBeUndefined(); + + const { partial: _partial, ...legacy } = { + ...JSON.parse(readFileSync(diskUsageCacheFile(plain), 'utf8')), + partial: true, + }; + writeFileSync(diskUsageCacheFile(plain), JSON.stringify(legacy)); + expect(await walk()).not.toHaveProperty('partial'); +}); diff --git a/packages/stim-cli/src/__tests__/doctor.integration.test.ts b/packages/stim-cli/src/__tests__/doctor.integration.test.ts index e38a6f5fa..78273c383 100644 --- a/packages/stim-cli/src/__tests__/doctor.integration.test.ts +++ b/packages/stim-cli/src/__tests__/doctor.integration.test.ts @@ -2044,3 +2044,95 @@ test.skipIf(process.platform === 'win32')( } }, ); + +test('doctor lists the checkouts the scan found that Stim does not fully manage, in the JSON payload and in the text', async () => { + const project = mkdtempSync(join(tmpdir(), 'stim-doctor-discovery-')); + const home = mkdtempSync(join(tmpdir(), 'stim-doctor-discovery-home-')); + process.env.STIM_HOME = home; + const cwd = process.cwd(); + const logs: string[] = []; + const originalLog = console.log; + writeFileSync(join(project, 'package.json'), JSON.stringify({ name: 'app' })); + mkdirSync(join(home, 'maintenance'), { recursive: true }); + const ranAt = Date.UTC(2026, 10, 14); + writeFileSync( + join(home, 'maintenance', 'discovery-scan.json'), + JSON.stringify({ + version: 2, + ranAt, + completedAt: ranAt, + lowScanAt: 0, + durationMs: 1, + trigger: 'weekly', + visited: 10, + found: [], + dotFolderFound: [], + unreadable: [], + truncated: false, + truncation: 'none', + }), + ); + const gitAt = Date.UTC(2026, 9, 1); + const find = (name: string, at: number | null) => ({ + path: `/work/dev/${name}`, + kind: 'reduce-only', + folder: '/work/dev', + adoptable: true, + evaluatedAt: ranAt, + dirs: 1, + measuredDirs: 1, + failedDirs: 0, + partialDirs: 0, + bytes: 2 * 1024 ** 3, + unreadable: false, + gitActivityAt: at, + }); + writeFileSync( + join(home, 'maintenance', 'discovery-census.json'), + JSON.stringify({ + version: 1, + at: ranAt, + scanRanAt: ranAt, + roots: [], + canonicalRoots: [], + pending: false, + finds: [find('b', null), find('a', gitAt)], + }), + ); + const doctor = doctorOp(() => testStimVersions); + console.log = (msg) => logs.push(String(msg)); + process.chdir(project); + try { + await runCli(doctor, ['--json'], { version: '1.2.3' }); + await runCli(doctor, [], { version: '1.2.3' }); + } finally { + process.chdir(cwd); + console.log = originalLog; + delete process.env.STIM_HOME; + rmSync(home, { recursive: true, force: true }); + rmSync(project, { recursive: true, force: true }); + } + const payload = JSON.parse(logs[0] ?? ''); + expect(payload.discovery).toMatchObject({ + census: 'read', + censusCurrent: true, + groups: [ + { + folder: '/work/dev', + kind: 'reduce-only', + checkouts: 2, + bytes: 4 * 1024 ** 3, + members: [ + { path: '/work/dev/a', gitActivityAt: new Date(gitAt).toISOString() }, + { path: '/work/dev/b', gitActivityAt: null }, + ], + }, + ], + }); + const { command } = payload.discovery.groups[0]; + expect(command).toContain('maintenance.discoverRoots'); + const text = logs.slice(1).join('\n'); + expect(text).toContain('/work/dev'); + expect(text).toContain(command); + expect(text).toContain('2026-10-01'); +}); diff --git a/packages/stim-cli/src/__tests__/maintenance-run.test.ts b/packages/stim-cli/src/__tests__/maintenance-run.test.ts index 9e29beea5..f7d5533f4 100644 --- a/packages/stim-cli/src/__tests__/maintenance-run.test.ts +++ b/packages/stim-cli/src/__tests__/maintenance-run.test.ts @@ -547,3 +547,35 @@ test('an unverified child cleanup in a pass surfaces as that failure, not as the await expect(runMaintenance('daemon', { tick })).rejects.toBe(failure); expect(scan).not.toHaveBeenCalled(); }); + +test('an unverified child cleanup from the census reaches the caller of the pass as the same object', async () => { + const failure = Object.assign(new Error('unverified cleanup'), { reportCleanup: true }); + vi.spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue').mockResolvedValue(); + vi.spyOn(await import('../workspace/discovery-census.ts'), 'runDiscoveryCensusIfDue').mockRejectedValue(failure); + await expect(runMaintenance('daemon', { tick: quietTick() })).rejects.toBe(failure); +}); + +test('a census failure that was contained is recorded in the maintenance log', async () => { + vi.spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue').mockResolvedValue(); + vi.spyOn(await import('../workspace/discovery-census.ts'), 'runDiscoveryCensusIfDue').mockResolvedValue( + 'disk exploded', + ); + await runMaintenance('daemon', { tick: quietTick() }); + const failures = readFileSync(gcLogFile(), 'utf8') + .trim() + .split('\n') + .map((line) => JSON.parse(line)) + .filter((record) => record.event === 'maintenance_failure' && record.target === 'discovery census'); + expect(failures).toEqual([expect.objectContaining({ error: 'disk exploded' })]); +}); + +test('an unverified child cleanup from the census reaches the caller even when the pass was aborted meanwhile', async () => { + const failure = Object.assign(new Error('unverified cleanup'), { reportCleanup: true }); + const controller = new AbortController(); + vi.spyOn(await import('../workspace/discovery-scan.ts'), 'runDiscoveryScanIfDue').mockResolvedValue(); + vi.spyOn(await import('../workspace/discovery-census.ts'), 'runDiscoveryCensusIfDue').mockImplementation(async () => { + controller.abort(); + throw failure; + }); + await expect(runMaintenance('daemon', { tick: quietTick(controller.signal) })).rejects.toBe(failure); +}); diff --git a/packages/stim-cli/src/__tests__/maintenance-rung6.test.ts b/packages/stim-cli/src/__tests__/maintenance-rung6.test.ts index 0d31bcf0f..5bba61863 100644 --- a/packages/stim-cli/src/__tests__/maintenance-rung6.test.ts +++ b/packages/stim-cli/src/__tests__/maintenance-rung6.test.ts @@ -130,6 +130,170 @@ test('the next command, report-only gc included, warns on stderr while the daemo expect(errors).toEqual([expect.stringContaining('disk pressure is critical (4.0 GB free on /')]); }); +test('the critical-disk warning adds one line with what unmanaged checkouts hold and the command for the largest folder, only from a readable census', () => { + mkdirSync(join(home, 'daemon'), { recursive: true }); + mkdirSync(join(home, 'maintenance'), { recursive: true }); + writeFileSync( + join(home, 'daemon', 'snapshot.json'), + JSON.stringify({ + version: 1, + pid: 1, + startedAt: new Date().toISOString(), + mode: 'on', + tickAt: new Date().toISOString(), + tickMs: 1, + intervalMs: 60_000, + measurement: {}, + workspaces: [], + pressure: { + level: 'critical', + disk: 'critical', + memory: 'normal', + reasons: ['4.0 GB free on /'], + memoryWarningSince: null, + }, + }), + ); + const ranAt = Date.now(); + writeFileSync( + join(home, 'maintenance', 'discovery-scan.json'), + JSON.stringify({ + version: 2, + ranAt, + completedAt: ranAt, + lowScanAt: 0, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [], + dotFolderFound: [], + unreadable: [], + truncated: false, + truncation: 'none', + }), + ); + const find = { + path: '/Users/x/dev/app', + kind: 'reduce-only', + folder: '/Users/x/dev', + adoptable: true, + evaluatedAt: ranAt, + dirs: 1, + measuredDirs: 1, + failedDirs: 0, + partialDirs: 0, + bytes: 3 * 1024 ** 3, + unreadable: false, + gitActivityAt: null, + }; + const census = join(home, 'maintenance', 'discovery-census.json'); + const errors: string[] = []; + vi.spyOn(console, 'error').mockImplementation((...args) => void errors.push(args.join(' '))); + + writeFileSync(census, '{'); + warnCriticalDisk(['ios']); + expect(errors).toHaveLength(1); + + errors.length = 0; + writeFileSync( + census, + JSON.stringify({ + version: 1, + at: ranAt, + scanRanAt: ranAt, + roots: [], + canonicalRoots: [], + pending: false, + finds: [find], + }), + ); + warnCriticalDisk(['ios']); + expect(errors).toHaveLength(2); + expect(errors[1]).toContain('hold 3.0 GB'); + expect(errors[1]?.endsWith(`stim settings set maintenance.discoverRoots '["/Users/x/dev"]' --scope machine`)).toBe( + true, + ); +}); + +test('the critical-disk warning lists nothing once maintenance.discoverRoots changed since the census, and reads no folder to decide', () => { + mkdirSync(join(home, 'daemon'), { recursive: true }); + mkdirSync(join(home, 'maintenance'), { recursive: true }); + writeFileSync( + join(home, 'daemon', 'snapshot.json'), + JSON.stringify({ + version: 1, + pid: 1, + startedAt: new Date().toISOString(), + mode: 'on', + tickAt: new Date().toISOString(), + tickMs: 1, + intervalMs: 60_000, + measurement: {}, + workspaces: [], + pressure: { + level: 'critical', + disk: 'critical', + memory: 'normal', + reasons: ['4.0 GB free on /'], + memoryWarningSince: null, + }, + }), + ); + const ranAt = Date.now(); + writeFileSync( + join(home, 'maintenance', 'discovery-scan.json'), + JSON.stringify({ + version: 2, + ranAt, + completedAt: ranAt, + lowScanAt: 0, + durationMs: 1, + trigger: 'weekly', + visited: 1, + found: [], + dotFolderFound: [], + unreadable: [], + truncated: false, + truncation: 'none', + }), + ); + writeFileSync( + join(home, 'maintenance', 'discovery-census.json'), + JSON.stringify({ + version: 1, + at: ranAt, + scanRanAt: ranAt, + roots: [], + canonicalRoots: [], + pending: false, + finds: [ + { + path: '/nowhere/dev/app', + kind: 'reduce-only', + folder: '/nowhere/dev', + adoptable: true, + evaluatedAt: ranAt, + dirs: 1, + measuredDirs: 1, + failedDirs: 0, + partialDirs: 0, + bytes: 1024 ** 3, + unreadable: false, + gitActivityAt: null, + }, + ], + }), + ); + const errors: string[] = []; + vi.spyOn(console, 'error').mockImplementation((...args) => void errors.push(args.join(' '))); + warnCriticalDisk(['ios']); + expect(errors).toHaveLength(2); + errors.length = 0; + saveConfig({ version: 2, projects: {}, repos: {}, maintenance: { discoverRoots: ['/nowhere/stalled-mount'] } }); + warnCriticalDisk(['ios']); + expect(errors).toHaveLength(1); +}); + const DAEMON_ROW = ' 101 /opt/jdk/bin/java -cp /opt/gradle-9.4.1/lib/gradle-daemon-main-9.4.1.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 9.4.1\n'; const WRAPPER_ROW = ' 202 /opt/jdk/bin/java -jar /app/gradle/wrapper/gradle-wrapper.jar assembleDebug\n'; diff --git a/packages/stim-cli/src/__tests__/scan-rules.test.ts b/packages/stim-cli/src/__tests__/scan-rules.test.ts index cd9e29ece..ad76e4638 100644 --- a/packages/stim-cli/src/__tests__/scan-rules.test.ts +++ b/packages/stim-cli/src/__tests__/scan-rules.test.ts @@ -23,6 +23,21 @@ describe('scanEntryValid with a hand-edited scan result', () => { expect(await scanEntryValid(project('dev', 'app'), rules())).toBe('valid'); }); + it('keeps an entry whose metadata read fails for anything but proven absence as pending, and drops one that is absent', async () => { + const path = project('dev', 'app'); + const failing = (code: string) => ({ + ...REAL_RECOGNITION_IO, + lstat: async (target: string) => { + if (target === path) throw Object.assign(new Error(code), { code }); + return REAL_RECOGNITION_IO.lstat(target); + }, + }); + for (const code of ['EACCES', 'EPERM', 'EIO']) + expect(await scanEntryValid(path, { ...rules(), io: failing(code) })).toBe('pending'); + for (const code of ['ENOENT', 'ENOTDIR']) + expect(await scanEntryValid(path, { ...rules(), io: failing(code) })).toBe('invalid'); + }); + it('refuses entries the scan never finds: the filesystem root, another home, the Stim home, pruned and skipped folders', async () => { const other = realpathSync(mkdtempSync(join(tmpdir(), 'stim-scan-other-'))); try { diff --git a/packages/stim-cli/src/commands/doctor.ts b/packages/stim-cli/src/commands/doctor.ts index c27b1ab9f..09f89a475 100644 --- a/packages/stim-cli/src/commands/doctor.ts +++ b/packages/stim-cli/src/commands/doctor.ts @@ -1,9 +1,13 @@ +import { realpathSync } from 'node:fs'; import { homedir } from 'node:os'; import { daemonNdjsonFile, daemonSnapshotFile, gcLogFile, + machineSettingValue, readDaemonSnapshot, + readDiscoveryCensusFile, + readDiscoveryScanFile, type DaemonPressure, type DaemonStatus, } from '@stim-cli/core/state'; @@ -20,6 +24,12 @@ import type { ProjectRegistry } from '../integrations/project-registry.ts'; import type { ProjectDoctor } from '../integrations/project-doctor.ts'; import { getProject, loadConfig } from '../workspace/config.ts'; import { broadDiscoverRoots, missingDiscoverRoots } from '../workspace/discovery.ts'; +import { + DISCOVER_ROOTS_KEY, + discoveryLines, + discoveryReport, + type DiscoveryReport, +} from '../workspace/discovery-report.ts'; import { projectSettingsContextAsync, resolveAndroidLayout, @@ -27,7 +37,7 @@ import { type ProjectSettingsContext, type SettingsObject, } from '../workspace/settings.ts'; -import type { StimContext } from '@stim-cli/core'; +import { isScopedHome, type StimContext } from '@stim-cli/core'; import { readMachineSettings, type MachineSettings } from '../diagnostics/doctor-config.ts'; import { checkWorktreeStorageLayout } from '../diagnostics/doctor-storage.ts'; import { @@ -219,6 +229,24 @@ function toolchainLines(report: ToolchainReport): string[] { ]; } +function canonicalOrSelf(path: string): string { + try { + return realpathSync(path); + } catch { + return path; + } +} + +const DISCOVERY_TITLE = 'Checkouts Stim does not fully manage'; + +function discoverySection(report: DiscoveryReport, env: NodeJS.ProcessEnv): string[] { + const lines = + report.scan.status === 'absent' && isScopedHome(env) + ? ['A scoped Stim home never scans the home folder, so it lists nothing.'] + : discoveryLines(report, homedir()); + return lines.map((line) => (line ? ` ${line}` : line)); +} + function pressureLines(pressure: DaemonPressure | null): string[] { return pressure ? [phaseLine('pressure', `${pressure.level}${pressure.reasons.length ? `: ${pressure.reasons.join('; ')}` : ''}`)] @@ -398,6 +426,11 @@ export interface DoctorReport { maintenance: ReturnType & { logPath: string }; /** @internal Carries no compatibility: the stim-daemon state, as `status --json` prints it. */ daemon: DaemonStatus & { logPath: string }; + /** + * @internal Carries no compatibility: what the last home folder scan found that Stim does not fully manage, grouped + * by folder with the size of the build output, from the daemon's cached result. + */ + discovery: DiscoveryReport; remoteMachines: Array; deviceHosts: Array>['machines'][number] & { priority?: number }>; toolchain: ToolchainReport | null; @@ -758,6 +791,13 @@ async function assembleDoctorReport( pressure: pressure.pressure, maintenance: { ...maintenance, logPath: gcLogFile() }, daemon: { ...daemon, logPath: daemonNdjsonFile() }, + discovery: discoveryReport({ + scan: readDiscoveryScanFile(), + census: readDiscoveryCensusFile(), + setting: machineSettingValue(DISCOVER_ROOTS_KEY, { config: loadConfig(), env }), + home: homedir(), + canonical: canonicalOrSelf, + }), remoteMachines: withPriority(remoteMachines.machines), deviceHosts: withPriority(deviceHostMachines.machines), toolchain, @@ -842,6 +882,9 @@ export function doctorOp( toolchain ? toolchainLines(toolchain) : [], ), ...maintenanceLines, + '', + DISCOVERY_TITLE, + ...discoverySection(report.discovery, process.env), ]; for (const [index, line] of lines.entries()) { if (index === 1) console.log(chalk.green(line)); @@ -872,6 +915,9 @@ export function doctorOp( if (f.fix) console.log(chalk.dim(` -> ${f.fix}`)); } + console.log(chalk.bold(`\n${DISCOVERY_TITLE}`)); + for (const line of discoverySection(report.discovery, process.env)) console.log(chalk.dim(line)); + const serverOnly = findings.every(isServerFinding); console.log( chalk.dim( diff --git a/packages/stim-cli/src/disk-usage.ts b/packages/stim-cli/src/disk-usage.ts index 15c1ce683..d8396fe4a 100644 --- a/packages/stim-cli/src/disk-usage.ts +++ b/packages/stim-cli/src/disk-usage.ts @@ -16,6 +16,7 @@ import { diskUsageWalkClaims, readDiskUsage, readDiskUsageFailure, + readDiskUsagePartial, type DiskMeasure, } from '@stim-cli/core/state'; import { writeCacheFile } from './cache-file.ts'; @@ -119,6 +120,11 @@ export interface RefreshOptions { valid?: (measure: DiskMeasure) => boolean; /** Records a failed walk, and leaves the folder alone for `maxAgeMs` after it. */ failures?: boolean; + /** + * The caller has established, through a bounded read, that `path` exists: do not probe it. Without it the folder is + * probed with a synchronous `existsSync`, which a stalled mount can block. + */ + knownToExist?: true; } export type RefreshOutcome = 'fresh' | 'measured' | 'failed' | 'busy'; @@ -133,7 +139,7 @@ export type RefreshOutcome = 'fresh' | 'measured' | 'failed' | 'busy'; export async function refreshMeasure( path: string, options: RefreshOptions, -): Promise<{ outcome: RefreshOutcome; measure: DiskMeasure | null }> { +): Promise<{ outcome: RefreshOutcome; measure: DiskMeasure | null; partial?: true }> { const { operation } = options; operation?.signal.throwIfAborted(); const now = options.now ?? Date.now; @@ -144,7 +150,14 @@ export async function refreshMeasure( return failedAt !== null && now() - Date.parse(failedAt) < maxAgeMs; }; const skip = () => freshMeasure(path, options) !== null || failedRecently(); - const fresh = () => ({ outcome: 'fresh' as const, measure: freshMeasure(path, options) }); + const fresh = () => { + const measure = freshMeasure(path, options); + return { + outcome: 'fresh' as const, + measure, + ...(measure && readDiskUsagePartial(path) ? { partial: true as const } : {}), + }; + }; if (skip()) return fresh(); const record = ( bytes: number, @@ -152,6 +165,7 @@ export async function refreshMeasure( signature: string | null, at: number, walkMs: number | null, + partial?: true, ): DiskMeasure => { const measure: DiskMeasure = { bytes, @@ -159,10 +173,15 @@ export async function refreshMeasure( ...(privateBytes !== null ? { privateBytes } : {}), ...(signature ? { signature } : {}), }; - writeCacheFile(diskUsageCacheFile(path), { path, ...measure, ...(walkMs !== null ? { walkMs } : {}) }); + writeCacheFile(diskUsageCacheFile(path), { + path, + ...measure, + ...(walkMs !== null ? { walkMs } : {}), + ...(partial ? { partial } : {}), + }); return measure; }; - if (!existsSync(path)) { + if (options.knownToExist !== true && !existsSync(path)) { if (!dependency) return { outcome: 'fresh', measure: null }; return { outcome: 'measured', measure: record(0, 0, ABSENT, now(), null) }; } @@ -174,7 +193,7 @@ export async function refreshMeasure( const startedAt = now(); const signature = dependency ? treeSignature(path) : null; const targets = options.targets?.(); - let size: { bytes: number; privateBytes: number | null } | null; + let size: { bytes: number; privateBytes: number | null; partial?: true } | null; if (targets && !targets.length) size = { bytes: 0, privateBytes: null }; else { if (claim) { @@ -198,7 +217,15 @@ export async function refreshMeasure( } return { outcome: 'measured', - measure: record(size.bytes, size.privateBytes, signature, startedAt, Math.max(0, now() - startedAt)), + measure: record( + size.bytes, + size.privateBytes, + signature, + startedAt, + Math.max(0, now() - startedAt), + size.partial, + ), + ...(size.partial ? { partial: true as const } : {}), }; } catch (error) { cleanupUnverified = error instanceof Error && 'reportCleanup' in error; diff --git a/packages/stim-cli/src/guide/cleanup.ts b/packages/stim-cli/src/guide/cleanup.ts index 595a65299..f8416c42d 100644 --- a/packages/stim-cli/src/guide/cleanup.ts +++ b/packages/stim-cli/src/guide/cleanup.ts @@ -259,7 +259,14 @@ DISK PRESSURE tool's own command, only at critical and only while the earlier rungs left the disk short (\`stim guide cleanup gc\`); Xcode DerivedData outside Stim is never deleted. Every command prints one stderr line while the daemon's last tick, at most 10 - minutes old, found the disk critical. + minutes old, found the disk critical. When the daemon's census (DISCOVERED + CHECKOUTS NOT FULLY MANAGED) holds checkouts with measured build output, a + second stderr line says how much they hold ("at least" while some are not + measured), that adopting a folder gives it the full rules (finished linked + worktree removal included), and the command that adds the largest reduce-only + folder to maintenance.discoverRoots; a dot-folder group counts in the total and is never + the one named. It reads the daemon snapshot, the scan file, the census file and the config (all under the Stim home) and makes no call on a configured folder or a checkout (it compares strings with the roots the census recorded, and lists nothing once the setting changed since the census), never walks the disk, and goes to + stderr only, never into a JSON payload. It appears at critical only. stim status last checks, plan and running pass stim status --json maintenance observations and recent records @@ -1283,6 +1290,118 @@ finished-worktree check inspects every discovered worktree with git (about 0.14 seconds each where measured). A folder named in maintenance.discoverRoots keeps the full rules. A scoped STIM_HOME uses neither source. + +DISCOVERED CHECKOUTS NOT FULLY MANAGED (the census, the doctor listing) + After a scan the daemon (also on the passes thread, after the scan) writes + maintenance/discovery-census.json: the finds Stim does not fully manage, + classified from the complete discovery list, and the size of their build + output. Reduce-only: scan finds that discovery selects with origin scan. + Dot-folder: scan finds under a home dot-folder that no route reaches (a + registered project or a discovered checkout that touches it; lying below a + maintenance.discoverRoots folder is not coverage, since a root reaches only its + direct children) and that pass the same revalidation as a scan + find; Stim does nothing with them. A find that contains, or lies inside, a + registered project or a discovered checkout is treated as managed and not listed, + although only that project or checkout is managed. And the worktrees in ~/.codex/worktrees are not scan finds, so + they are not listed either. + Size is the recognised build output only: the directories maintenance clears + as build output for the checkout and for each folder directly in its apps and + packages (Android build and .gradle, .build, android .cxx, a .dd laid out as + DerivedData, a Cargo target that Exact owns or that carries CACHEDIR.TAG). + node_modules and Pods are not counted, and the size is not proof that the + directory could be deleted. Every read of a checkout's layout (listings, a + directory check, the first 64 bytes of a CACHEDIR.TAG that the listing shows as + a regular file) is asynchronous, goes through the shared bounded reader (3 + seconds, one timed-out read outstanding per thread, none started while one is + pending) and checks the abort and the run's budget first; the directory rules + are the ones the deleting passes use, driven by that reader. A path that does + not exist is absent; a denied or timed-out read makes the checkout "could not + be measured", never a measured zero. Not bounded on this path (found by + reading computeDiscovered, unmanagedScanFinds, scanSkipper and childrenOf; the + callees of project recognition and of the git helpers were not traced): existsSync + on each configured discover root, registered project root and worktree entry; + realpathSync (canonicalPath) on those, the repository folders, the home folder, + every child listed, the known roots and, in scanSkipper, the Stim home, the build + worker root and the temporary directories; isProtectedLocation (realpathSync) on + every worktree path and candidate child outside a configured root; project + recognition (selectDoctor and a synchronous readdirSync of apps, examples and + packages) for each child and at the end of the revalidation of every scan find; + the child listings, which are raced against 3 seconds, not read through the shared + reader; and git subprocesses (git worktree list and git symbolic-ref through the + exec wrapper, abortable, with their own timeouts). The passes run them the same + way; the census adds that recognition now also runs on finds under home + dot-folders. A revalidation metadata read that fails for anything but proven + absence (a denied lstat, for one) leaves the find pending, never dropped. A du that + exits non-zero is a measured lower bound: the bytes count and the totals read "at + least"; the cache entry records it, and no other reader of the cache sees it. Sizes go through the disk-usage cache, one + walk at a time on the machine, 10 seconds per directory (a build directory the + gc worktree report sizes with private bytes is measured with them too), at most + 25 checkouts and 90 seconds per run; a checkout measured within a day is + carried over without a read; the others are taken longest-waiting first (by last attempt), and what + a run leaves keeps its earlier entry (or is "not measured yet") and keeps the + census pending, so the next run, an hour later, continues; so does a + checkout whose layout could not be read. A find whose revalidation did not + answer (a read pending) is kept as its own kind, unchecked: neither listed as + managed nor dropped, with any earlier measurement and its age, and the census + stays pending so it is retried within the hour; so is a find whose project + check could not read a manifest or listing. Doctor prints that every + checkout it found is managed only for a finished scan, a census made from it + and no unchecked find. The measuring walk is told the directory exists + (refreshMeasure knownToExist), so it makes no synchronous probe of the + measured path. A walk that + fails or times out is "could not be measured", never zero; a total with any + unknown is "at least"; a group shows its oldest measurement date. The census + is written again after a new scan, when the machine's maintenance.discoverRoots + changes, and after a day. It classifies a discovery computed from the scan + result it records, not the per-operation discovery the passes share, and + records that result's ranAt; doctor shows no size from a census made from an + older scan than the scan file holds (those checkouts read "not measured yet"). + An error that carries reportCleanup (a measuring child whose cleanup could not + be verified, or the STIM_CLAIM_UNAVAILABLE errors the operation's spawn claims + raise; a walk claim held elsewhere is only "busy", not an error) propagates out + of the pass unchanged for the daemon's cleanup containment (stim guide + lifecycle), and does so before an abort is considered; any other census error is + contained, writes nothing and is one maintenance_failure record (target + discovery census) in the cleanup log. The census reads no idle evidence: that + evidence walks the source of a checkout with no bound it could apply. It + records one cheap fact instead, the last git activity: the newest + modification time of HEAD, index and logs/HEAD in the checkout's git + directory (its .git directory, or for a linked worktree the .git/worktrees/ + directory its .git file names, relative names resolved against the checkout, + which must hold a gitdir file naming this checkout's .git file and whose + canonical path must equal its path, so no component is a symlink), by bounded + reads that never follow a symlink and run no git. Commits, checkouts, staging + and reflog writes change it; editing files without touching git does not. A + missing git directory, a name outside .git/worktrees, a read that fails or + times out, or a spent budget gives unknown, never old. Doctor lists each + checkout with that date, oldest first, unknown last, with no threshold, and + says Stim has not checked whether a checkout is in use or holds uncommitted or + unpushed work; deleting one is the person's call. + stim doctor prints it as the section "Checkouts Stim does not fully manage" + and carries it in --json as discovery (internal, no compatibility promise): + scan { status: read | absent | unreadable | unusable, ranAt, trigger, + truncation, visited, unreadable }, census (read | absent | unreadable | + unusable | none), censusAt, censusCurrent, and groups, one per containing + folder and kind, each { folder, kind, checkouts, bytes, measured, failed, + partial, notMeasuredYet, command, commandProblem (too-broad | already-named | + setting-unreadable | unchecked), oldestMeasuredAt, members: [{ path, + gitActivityAt (ISO time or null) }] }, plus complete and rootsChangedSinceCensus. + complete is one value, false when the scan stopped early, failed or could not + read a path, the census is pending, from another scan or older than a roots + change, or any group is unchecked or has a failed, partial or not yet measured + checkout; every figure ("at least") and the sentence that every checkout is + managed follow it. Doctor reads the two files and the setting and walks nothing, + and never hides a census entry: a group whose folder is a root discovery uses, or + that the setting names now (symlinks resolved), is shown with no command and the + reason, and a group in a root discovery ignores (a broad one such as ~) is an + ordinary group. The section shows only where doctor runs, + which is inside a project. command is stim settings set + maintenance.discoverRoots '' --scope machine with the current value plus + the folder, quoted for the shell; it is null for the home folder or a folder + holding it (discoverRoots refuses those) and when the current value does not + read. Adding a folder gives its checkouts the full discovered-checkout rules, + finished-worktree removal included; for a dot-folder find it also lets Stim act + where it did not. The critical-disk line says so before the command. stim settings set maintenance.discoverRoots '["~/Developer"]' --scope machine Idle, with no Stim activity record, means the newest of its source file writes (git-ignored files included; node_modules, Pods and nested worktrees skipped, diff --git a/packages/stim-cli/src/guide/facts.ts b/packages/stim-cli/src/guide/facts.ts index b22692995..4ef877a0e 100644 --- a/packages/stim-cli/src/guide/facts.ts +++ b/packages/stim-cli/src/guide/facts.ts @@ -562,6 +562,18 @@ leased until