Repository navigation
feat: list unmanaged checkouts in doctor and on the critical disk line - #5767
janicduplessis wants to merge 11 commits into
Conversation
… not fully manage
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: no blocking findings. Seven non-blocking findings, one of them a product-decision list. Read-only review at a4f7b08; I did not build or run tests.
Report-only (verified by reading)
unmanagedScanFinds(discovery.ts:340-373) returns two arrays torunDiscoveryCensusIfDue, which is its only caller. Nothing from it reaches discovery's selected list, the stages, artifacts, Cargo passes, the ladder or gc.dotFolderFoundhas exactly one reader, and it is this one. Each dot-folder path goes throughscanEntryValidwith the same rules asrevalidatedScan. No project detection is called on it outside that gate.refreshMeasurewrites the disk-usage cache for the build dirs.ARTIFACT_DIRS.buildand the discovered build dirs are notisDependencyTreepaths, sotrustedMeasureindaemon/dependencies.ts(node_modules/Pods only) does not see them. The caps (plan.ts, fed bymeasure.tsthroughcollectWorkspaceOutputs, andcargo-target.tsthroughmeasuredDirectorySizeAsync) do not read this cache. The readers arestatus-measures.tsand the gc worktree sweep, which showbuildStateand decide nothing. Dot-folder checkouts are not registered workspaces or discovered worktrees, so neither reader reaches them. Measuring them changes nothing Stim later does to any checkout.- The census runs in
scanAfterPass, after the claim is released. It does not run after a failed or aborted pass. Its own errors are swallowed, so it cannot mask the pass error. Only an abort escapes the function. A scoped home never runs it. An unreadable or unusable scan returns without a write. An unreadable census file returns without a write and is never repaired. The write is a temp file plus rename underwithDirLock. Core owns the path and the reader. The reader separates absent, unreadable and unparsable. Unmeasured is kept apart from failed and from zero. - Doctor and
warnCriticalDiskread two files plus config. There is no walk, git or process read. JSON mode prints one payload. The section is outsidefindings, so PASS and the exit code are unchanged. The low-disk line goes to stderr only and sits inside a try/catch. shellQuote(JSON.stringify([...]))in single quotes is correct POSIX quoting for spaces,',$, backticks and non-ASCII. The home folder, a folder holding it, and/get words and no command (isBroadFolder). Groups whose folder is a configured root are dropped.
Findings
- NON-BLOCKING, discovery-census.ts:177-181. The 90 s budget is checked only before each
measureBuildOutput.idleSincehas no budget check. It runsdiscoveredEvidence, which walks up to 20,000 source entries, runsgit check-ignoreper skipped-where-ignored entry, and runsgit logand worktree activity, for each of up to 25 checkouts.unmanagedScanFindsalso re-runs full discovery. The census is awaited inscanAfterPass, so the passes thread can run well past 90 s. All of it is async and abortable, so nothing blocks the event loop. Suggest checking the deadline beforeidleSince, or documenting that the bound covers walks only. Separately,appRoots(reached throughmeasureBuildOutput) does syncreaddirSyncandisDirectoryonapps/andpackages/of the checkout, on the daemon thread. The artifacts stage already does this, but a dot-folder path is only validated once, earlier in the run. - NON-BLOCKING, discovery-census.ts:157-160 and discovery-report.ts:158-160. The idle verdict is stale.
lastActivityAtis carried over for up to a day (earlierreused without re-evaluating). The process table, launcher scan and agent sessions are read once per run (the launcher reader is cached per operation and can be minutes old). Doctor prints "Idle for at least N days and nothing uses it" as present fact. If someone starts using the checkout after the evaluation, doctor still says this until the next census. Suggest "as of ", or re-evaluating idle on every run. It is words only, but it tells a person to delete.discoveredEvidencehas no check for uncommitted or unpushed work in the checkout. Old uncommitted edits count as idle. The text says "delete it yourself if you no longer need it" and does not mention that risk. - NON-BLOCKING, discovery-report.ts:253-259. The low-disk line says "to what Stim manages with: ". It does not say the command grants removal of finished linked worktrees. The doctor text and the guide do say so. The line is meant for an agent to run, and it is an LLP-listed consequence. Suggest one clause.
- NON-BLOCKING, discovery-report.ts.
discoveryReportnever comparescensus.scanRanAttoscan.ranAt, or the census age. After a later scan, or a failed daemon, doctor shows an older census next to the new scan date ("Listed and measured by the daemon on ..." does carry its own date). AlsorootsOfusesresolve, not canonical paths, while census folders are canonical. A symlinked adopted root can keep its group, andadoptCommanddoes not dedupe, so it re-proposes the folder until the next census. This is transient. Over-budget entries with an older complete measurement keep that stale value as "measured" with no age. - NON-BLOCKING, discovery-census.ts catch block. If
unmanagedScanFindsthrows, nothing is written and nothing is logged. Doctor keeps saying "has not yet listed" with no reason. The attempt is retried hourly. - NON-BLOCKING, minor and not verified. Census measures with
private: falseand writes entries withoutprivateBytes. The gc sweep reuses a fresh cache entry for dirs inprivateSized, which could then showprivateBytesnull in the report. That is display only. I did not traceprivateSized. - NON-BLOCKING, docs. LLP 0004 says "like maybe not part of doctor since it could be slow" was "Relayed by the coordinator, not on the issue". It is on #5762 as a comment, verbatim. The other two quotes match the issue word for word. The
warnCriticalDiskcomment and guide say "two small files". It also reads config and the setting.
Tests (read, not run)
discovery-census.test.ts redirects HOME, USERPROFILE and STIM_HOME to a temp dir and injects the process and session readers. The report tests use hand-written files. The maintenance-rung6 test covers the extra line. The assertions can disagree with the code: failed vs zero, the bound, carry-over, scoped home, roots change. The lowDiskLine / adoptCommand tests check exact strings. I did not check whether they fail when each behaviour is removed.
Product decisions the maintainer did not state (list for review: needs-human)
- Doctor prints an adopt command for dot-folder groups (e.g.
~/.exact/exact2). The low-disk line withholds it. Doctor discloses that adoption "lets Stim act where it did not". Needs a maintainer decision. - Idle threshold reuses
maintenance.olderThanDays(3 days). - The line appears at critical only, not at high.
- Sizes cover build output only, not node_modules or Pods.
- The 25-checkout / 90 s / 10 s / one-day carry-over bounds.
- The
discoveryJSON field on doctor, and doctor refusing outside a project, so the listing shows only inside one. - The census file, and the doctor output section title and wording.
- Total "at least" wording.
Not checked: CI results, api.api.txt, knip, a live daemon run, privateSized semantics, and whether any other command prints the low-disk line in --json mode (stderr only here).
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: clear. No blocking or actionable defect at 2655135. Read-only review (read the diff, the callers and the tests; I did not build or run tests). CI on this head is green, including merge gate. Seven non-blocking notes and a product-decision list below.
1. Report-only (verified by reading)
unmanagedScanFinds(discovery.ts:340-382) is called only byrunDiscoveryCensusIfDue;dotFolderFoundhas one reader (discovery.ts:369) and one writer (the scan). Nothing from either reaches discovery's selected list, stages, artifacts, Cargo passes, the ladder or gc. It callscomputeDiscoveredwith the scan and discards everything but the classification.regenerable-dirs.tsdiff is fourexportkeywords (DirRead,DirReads,signatureHeads,existingBuildDirsReads). No behaviour change for the deleting passes.- Readers of the disk-usage cache:
status-measures.ts(display),gc/worktree-sizes.ts(display ofbuildState),daemon/dependencies.tsandladder.tsthroughknownPrivateBytes/trustedMeasure. Those take dependency-tree paths (node_modules,ios/Pods) from the pass's own candidates; the census writes only recognised build directories, never those paths, so the 256MREAL_BYTES_THRESHOLDrule never reads a census entry. The census'sprivate: !FIXED_BUILD...matches thehow !== 'fixed'rule inworktree-sizes.ts:71, anddirSizeonly reports it. No decision, cap or deletion reads an entry the census wrote. - The passes' memo is unchanged:
discovery()passesreadDiscoveryScan()whererevalidatedScanused to read it itself.
2. Reads the census makes
Through answered with signal.throwIfAborted() and the deadline checked first (boundedReader.read, discovery-census.ts:119-130): readdir of apps, packages and each build-dir parent; the directory check (lstat); the 64-byte tag head; the generators from regenerable-dirs.ts (driven by drive); lstat of .git, of the gitdir and of HEAD, index, logs/HEAD; the 512-byte .git file head. Dot-folder revalidation (scanEntryValid) uses answered too.
Not through it, and the lane's list is incomplete in two items that I find acceptable: (a) refreshMeasure's du, its synchronous existsSync and cache/claim reads under $STIM_HOME (the path was listed through answered moments before; the gc sweep and status-measures make the same call); (b) the discovery work computeDiscovered does: git via resolveSourceCheckoutAsync, sync existsSync/realpathSync/readdirSync in canonicalPath, childrenOf, read and recognises. recognises runs on a dot-folder path only after scanEntryValid's bounded safety gates, as for scan finds. Not on the lane's list: writeCensus's sync withDirLock (state dir). Reads of dot-folder paths by existsSync were not made before this change on that lane, but only after the bounded gate above.
3. lastGitActivity
Only lstat and a prefix read of a regular .git file (head lstats first). .git as a symlink, HEAD/index/logs/HEAD as symlinks, or a symlinked gitdir end in unknown. resolve normalises .. before the /.git/worktrees/<name> test. A crafted .git file can make it lstat HEAD, index and logs/HEAD in any real directory on the machine whose path ends in .git/worktrees/<name> (intermediate components are followed by the OS). The only disclosure is the date of the newest mtime, printed to the user who runs doctor; nothing is read. Windows paths: the regex takes both separators and isAbsolute is platform-correct. Unknown prints "unknown" and sorts last (discovery-report.ts:127-130).
4. Errors
reportCleanup and STIM_CLAIM_UNAVAILABLE rethrow as the same object from the census catch (discovery-census.ts:~403) and scanAfterPass (run.ts:~772); scanAfterPass runs only when !passFailed and not aborted, so it cannot replace a pass error from the finally. Other errors are returned and logged once. Tested in maintenance-run.test.ts and discovery-census.test.ts.
5. Classification, 6. Failed is never zero
Census computes discovery fresh from the scan it records and stores finds.scanRanAt. Report zeroes measurements when scanRanAt differs, so the total is 0 and lowDiskLine returns null. Denied or timed-out layout read gives unreadable; a failed walk gives failedDirs; not reached gives not measured yet; sums use "at least".
7. Doctor and the critical line
Doctor text is added in both print paths, JSON discovery, outside findings, exit code unchanged. The critical line is stderr only inside the existing try/catch. The command is shellQuote(JSON.stringify(...)), home and its ancestors get words only, and a group whose folder is a root (symlinks resolved) is dropped.
9. Docs
The three quotes in LLP 0004 match the issue word for word. Lane and coordinator choices are labelled as such.
Non-blocking notes
- discovery-report.ts:84-86.
canonicalRootscallsrealpathSyncon each configured root in doctor and on every command's critical-disk path. A discover root on a hung mount would stall that. Doctor already doesexistsSyncon the roots; the critical-disk path did not. - discovery-census.ts:~370. The census runs a second full discovery (git processes, sync reads) before its 90 s deadline starts, so the run's time is unbounded by the budget. It is hourly at most and abortable.
- A checkout whose read timed out is
unreadable, is notpending, and is retried after a day, not an hour. - A linked worktree
.gitwith a relativegitdir:(git 2.48worktree.useRelativePaths) shows unknown. Acceptable and shown as unknown. guide/cleanup.tssaysSTIM_CLAIM_UNAVAILABLE"from a walk" propagates;takeWalkClaimturns it intobusy, so onlyreportCleanupcan arrive from there. Wording only.- No test covers doctor's text paths or the
discoveryJSON field;discovery-report.test.tscovers the report. - Real tool calls (invariant 9): the census runs no new tool.
Not checked: a live daemon run, api.api.txt.
|
Human review: choices the maintainer did not state, all at commit 2655135.
|
…ist unchecked finds, make the hook string-only
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head 6ff4865 (merge of main into 439da3e)
Verdict: changes needed
I read the diff, the two LLPs, and the merged run.ts and disk-usage.ts, and ran discovery-census, discovery-report, disk-usage-known-path, maintenance-rung6 and maintenance-run tests (all pass). The four earlier defects are fixed except that item 2 (no synchronous call on a find) is still not true on the census path.
Actionable
- Synchronous filesystem calls on a scan find's path remain on the census path, and the PR body, LLP 0004 and
guide cleanupsay there are none beyond "configured roots and registered projects". The census callsunmanagedScanFinds->computeDiscovered/scanEntryValid(workspace/discovery.ts:241-253,:380-395;workspace/scan-rules.ts:306-352):scanEntryValidends withisCheckout(path), which isrecognises(discovery.ts:62-77):projectRegistry.selectDoctor(path)plusreaddirSync(join(path, 'apps'|'examples'|'packages')). It runs on every scan find and, in the new loop atdiscovery.ts:389-395, on every dot-folder find, inside the tool-owned folder the PR says the census reads only through the bounded reader. A stalled mount blocks the daemon thread here; the 3 s bound does not apply.allowed = ... isProtectedLocation(child.path)(discovery.ts:323-325,protected-locations.ts:14-20) callsrealpathSyncon every scan find not under a configured root.canonicalPath(join(folder, entry.name))(discovery.ts:211,realpathSync) runs for each child of a registered-repo sibling folder or configured root, which can be a find's path.
Fix by bounding or skipping these for the census, or at least state them in the PR body, LLP 0004 ("Discovery's own reads ... are the ones the passes already make" understates it) andguide cleanup("Every read of a checkout's layout ... is asynchronous"). The sentence in the PR body, "first code that reads inside a home dot-folder find ... every read ... is asynchronous", is false as written.
Notes
discovery-report.ts:216:gb(0)returns "1 MB", so a group whose checkouts are all measured with no build directories reads "1 MB of build output". Not a zero-as-unknown bug, but it states a size that is not there.discovery-census.tsquota:looked++counts every incomplete find, including permanentlyunreadableones, in fixed order. 25 or more permanently unreadable finds are re-read each hour and later finds are never reached. Acceptable if intended; not documented.refreshMeasurerecordsdu -skstdout on exit status 1 (partial permission denial) as a measured size, so a partly unreadable build directory is counted as complete. Pre-existing behaviour, now surfaced in a "measured" figure.writeCensustakes the lock only for the write;lastis read unlocked earlier in the run. A pass in another process after the claim is released could write a fresher file and be overwritten. Benign (last writer wins, both valid). The scoped home is skipped (isScopedHome).adoptCommandwrites the valuemachineSettingValuereturns, which can come from an env override rather than the machine config; the command would persist that override. Edge case.
The five checks
- Cancelled operation hides unverified cleanup: verified.
isUnverifiedCleanupis tested first in the catch ofrunDiscoveryCensusIfDueand inscanAfterPass, then the abort is checked and the same object rethrown.refreshMeasure's error propagates uncaught throughmeasureBuildOutput; the bounded reader's catch handles read errors only, andbuildDirsOfandlastGitActivityrethrow anything exceptUnreadableandOutOfBudget.daemon/passes.tsrecords it asfailureandsettled()rethrows. Tests atdiscovery-census.test.ts:427-434cover the same-object and aborted cases. - Synchronous calls: not met, see finding 1. Everything inside
discovery-census.ts(layout reads, git directory,bounded-read, the regenerable-dirs generators) is asynchronous.knownToExistonly changes theexistsSyncguard (disk-usage.ts:170); the default path is unchanged for other callers. A directory removed after the bounded read makesdufail, which records "failed", not zero or a size. Dependency trees (treeSignature, sync) are never passed because only build directories are listed. - Unchecked finds: verified.
uncheckedfinds become akind: 'unchecked'group (discovery-report.ts:204), setpending, and the sentence "Every checkout it found is managed." (:270-274) needs no groups,truncation === 'none', no unreadable folders andcensusCurrent. An unchecked find that survivesroots.hasalways yields a group. The low-disk line says "at least" while any group is unchecked and never claims completeness; a census from an older scan gives zero bytes and no line. - Git directory: verified. A linked worktree needs
.git/worktrees/<name>suffix, a regulargitdirfile that resolves to this checkout's.git, andrealpath(gitDir) === gitDir..git,HEAD,indexandlogsarelstated, so a symlink gives kindotherand unknown. I could not construct a layout reading metadata from an unrelated place other than a genuine worktree of another repository whose owngitdirfile names this checkout, which is the real git layout. Submodule and bare-repo layouts give unknown. - Critical-disk hook: verified.
warnCriticalDiskreads only the snapshot, scan file, census file and config under the Stim home, callsdiscoveryReportwithcanonical: null, and writes withconsole.erroronly. Doctor's JSON branch adds onediscoveryfield to the single payload.
Other
- Merge of main:
run.tsanddisk-usage.tsread correctly together; thetimedStepwrappers live indaemon/*and do not touchscanAfterPass;record(..., walkMs)is unchanged byknownToExist. - Nothing on these paths deletes, reduces, registers or writes outside
maintenance/discovery-census.json(plus the existing disk-usage cache and failure filesrefreshMeasurealways wrote). - Adopt command: canonical paths, single-quote escaping, never for the home folder or a folder holding it; the low-disk line names only reduce-only groups.
- Tests do not echo mocks or copy prose into regexes; documentation otherwise matches the code apart from finding 1.
Human review: the maintainer did not state these on #5762: the last-git-activity date in place of an idle threshold (LLP 0004 "Instead of idle"); the second line at critical only; sizes counting build output only; the 25-checkout and 90 s bounds; doctor offering the adopt command for dot-folder groups while the low-disk line does not; the separate "unchecked" kind. The PR body lists these already.
…what is bounded, print zero as zero
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head acab0d8
Verdict: changes needed (two small items, no deleting-path risk found)
Scope: the last push (6ff4865..acab0d8). Re-spot-checked the earlier points (cancellation not hiding an unverified cleanup, pending finds carried as unchecked, linked-worktree git directory validation, the per-command hook reading only Stim-home files): unchanged and still hold. Ran scan-rules, discovery-census and discovery-report tests: 61 pass. I did not run the full suites.
Proved: no deleting or reducing pass acts on more because of pending.
- Callers of
scanEntryValid:revalidatedScan(workspace/discovery.ts:250) and the dot-folder loop inunmanagedScanFinds(discovery.ts:392). Nothing else. revalidatedScankeepsvalidasDiscoveryChildand collectspendingintoscanPending(discovery.ts:251-252).scanPendingis read only at discovery.ts:372-388 (unmanagedScanFinds). The passes usefound/outsidefromcomputeDiscoveredand ignorescanPending. Sopendingandinvalidare the same set for every pass: not selected.- Nothing rewrites or prunes the scan file from revalidation:
mergeFound(discovery-scan.ts:183) drops stale entries "at read by scanEntryValid, not here", and stores nothing from it. - Retry:
discoveryScanDue(discovery-scan.ts, ~line 195) reads onlyranAt,completedAt,lowScanAt,truncation. A permanently denied find does NOT cause the full home scan to retry hourly. It only affects the census: anuncheckedentry setspending: true(discovery-census.ts, writeCensus call), socensusDue(discovery-census.ts:55) reruns the census hourly. That rerun redoescomputeDiscovered(reads, revalidation, git worktree lists) but not the scan and not theduwalks of unchanged finds. Cost is bounded but permanent per find, and doctor never prints "Every checkout it found is managed." and the low-disk line says "at least" for as long as the find stays denied. Note for the maintainer.
Proved: partial du reaches no other reader.
measureTreeaddspartialon exit 1 only (private-size.ts:94). Other callers read named fields only: other-homes.ts:148, machine-cache-usage.ts:338.recordwrites the explicitDiskMeasurefields;partialgoes into the cache file only (disk-usage.ts, record).readDiskUsage(core status-measures.ts:200) copies named fields, soDiskMeasure, the phone type, status, gc sizes, the caps and the ladder get the same objects as before. An older Stim ignores the extra key. gcstatus-cachessees only file bytes (a few bytes larger).- A fresh hit stays partial via
readDiskUsagePartial; a later complete walk rewrites the whole file without the flag, which clears it (not covered by a test, see below).
Findings
- Actionable, minor.
lowDiskLine(workspace/discovery-report.ts, thelargestbranch of the return) prints(${gb(largest.bytes)})for the largest folder with no "at least", even when that group has partial, failed or not-yet-measured checkouts. The total gets the prefix; the per-folder size does not. Item 3 of the brief (no exact figure including a partial or unmeasured directory) is violated. Prefix it with "at least" when that group'sfailed + notMeasuredYet + partial > 0, and add a test. - Actionable, doc. LLP 0004 "Known limits": "a find whose read is permanently denied or unreadable takes one of the 25 slots of each census run, every run". True for a find whose layout read is denied (
unreadable: trueis re-measured, discovery-census.ts,completecheck thenlooked++). False for a find whose metadata read is denied, which is the case this push creates:uncheckedentries are pushed andcontinuebeforelooked++. Say which. - Note, doc. The "not bounded" lists (PR body, LLP 0004, guide/cleanup.ts, website/docs/worktrees.md) read as complete, but
computeDiscovered(which the census now runs each time) also makes synchronousexistsSynccalls on registered project roots, configured discover roots and each worktree entry (discovery.ts,readandcanonicalDiscoverRoots), callsrecognisesinchildrenOf, and appliesisProtectedLocationto all children, not only "ordinary scan finds". Either add them or say "among others". - Note, tests. Missing: a later complete walk clears
partial; an entry without the field reads as complete.not.toContain('Every checkout it found is managed.')copies a sentence of prose; assert on report data (groupscontainingunchecked, which the same test already does) instead.toContain('at least')in the du test is loose but acceptable. Other new tests (zero as0 B, EACCES/EPERM/EIO vs ENOENT/ENOTDIR, du status 1 through realmeasureTreeparsing) have independent expectations and are not mock echoes. - Note, perf.
refreshMeasure'sfresh()now does a second cache file read (readDiskUsagePartial) for every fresh hit, for all callers (status, gc, daemon), though only the census uses the flag. Behaviour is unchanged. Compute it only on request if that matters.
Documentation otherwise agrees with the code: the bounded vs not-bounded split for scanEntryValid and recognises, "pending, never dropped" for a denied lstat, and "no other reader of the cache sees it" (strictly, readDiskUsagePartial does).
Human review: a permanently denied find now stays "could not be checked" forever and keeps doctor from ever saying the list is complete. The maintainer did not state whether that is wanted over dropping it.
…e largest folder size, complete the unbounded-call list
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head 4e016d9
Verdict: changes needed (one small code defect, one documentation sentence that is false; no deleting-path risk found)
I read the whole diff against origin/main, the last push, the two LLPs, computeDiscovered and what it calls, and the report and census code. I ran discovery-report.test.ts (23 pass) and ran discoveryReport/discoveryLines/lowDiskLine directly on hand-written files. I did not run a daemon. The earlier items (cancellation, linked-worktree git directory, per-command hook reading only Stim-home files, pending for denied metadata, the partial flag) I spot-checked and found as the earlier reviews describe.
Actionable
- discovery-report.ts:187 and :191. The report drops a group when
roots.has(listed.folder), androotsis built from every configured root lexically (rootsOf), including roots discovery ignores: a root that is the home folder or holds it (isBroadFolder) or does not exist.canonicalDiscoverRootsfilters those out, so discovery does not reach children of such a root, and the census records the find as reduce-only withfolder= home. Repro: scan finds~/app(a recognised checkout directly in home),maintenance.discoverRoots = ["~"], census holds the find.discoveryLinesprints "Every checkout it found is managed." andlowDiskLinereturns null; the find is reduce-only, unmanaged and unlisted. Doctor reports the broad root elsewhere, but the sentence is wrong and the checkout is hidden. Fix: drop a group only when its folder is incensus.canonicalRoots(what the classification used), or in the canonical form of a configured root that is also not broad; keep the lexical set only for the changed-roots comparison. Add a test with a broad root. - website/docs/worktrees.md:95: "The census does not walk source files or run git." The census runs
unmanagedScanFinds->computeDiscovered->resolveSourceCheckoutAsync(git worktree list, andgit symbolic-reffor a bare repository) for every registered project root and every recognised child. Only the git-activity reads run no git. Scope the sentence to those reads.
Notes
- The "every filesystem call
computeDiscoveredand what it calls make, read top to bottom" claim (LLP 0004 :358, guide/cleanup.ts, PR body) still omits: the git subprocesses above (async, no bound visible on the census path), andscanSkipper's synchronouscanonicalPathof the Stim config dir, the build worker root and the temp folders (run once perrevalidatedScanand once inunmanagedScanFinds), plusloadConfig. Either add them or say "the filesystem calls on checkouts". - discovery.ts:383-386 and guide/cleanup.ts:1303 ("A checkout another route already manages is not listed").
touchesis true in both directions, asselectDiscovered.coveredis. A registered project that is a descendant of a find (a registered app inside a scanned monorepo or a dot-folder checkout) hides the whole find, though Stim manages only that app's folder. This matches what the passes do on main, so I do not call it a defect, but the sentence overstates it. A discovered worktree inside a dot-folder find is not possible today (such worktrees are protected and land inoutside). - Coverage answers otherwise hold. Every scan find ends in
reduceOnly(origin scan),dotFolder(valid dot find no route touches),unchecked, or is intentionally absent: covered by a known root, selected by another origin, or dropped byisProtectedLocation(non-dot finds are protected only under a temp directory, or when the syncrealpathfails after the bounded one succeeded; that race leaves the find in none of the lists and the sentence can print). Nothing under~/.codex/worktreescan come from the scan (scanDescendsprunes.codex; stale entries die inscanEntryValid), so removing the tool-folder clause is correct. Direct children of a configured root are selected with origin root, so they are not listed. A child whose 3 s listing timed out inchildrenOfis listed with folder = the root and then dropped by the root test, so it is not over-reported either. - The JSON
discoveryfield has no completeness field. A consumer must derive "at least" fromfailed,partialandnotMeasuredYetper group and fromcensusCurrentandscan.truncation. The comment onUnmanagedGroup.bytes(discovery-report.ts:16) names only failed and notMeasuredYet and omitspartial; the guide's JSON shape (cleanup.ts, "each { folder, kind, checkouts, bytes, measured, failed, notMeasuredYet ...") also omitspartial. Either add a boolean (for examplelowerBound) or fix both lists. - Byte figures. I could not construct an exact figure that should read "at least": group text, total and largest folder all use the same condition (
failed + notMeasuredYet + partial, plus unchecked for the total), a census from an older scan is zeroed and prints no line, andgb(0)prints "0 B". Residual: a failed or not-yet-measured entry that carries an earlier measurement may be larger than it now is, so "at least" is a statement about the earlier size. Doctor text does not say sizes cover recognised build output only (the website and guide do). - Complete-list sentence prints only when: scan file read and usable, census file read and parsed, no group left after the root test,
scan.truncation === 'none',scan.unreadableempty, and the census made from this scan (scanRanAt === ranAt). The guide's wording ("a finished scan, a census made from it and no unchecked find") leaves out the emptyunreadablelist and the empty-group condition. The only wrong print I found is finding 1 (and the protected-location race in note 5). - Adopt command. The folder is
dirname(find); the test is exact string equality against configured, canonical-recorded and canonicalised roots; a root reaches direct children only, so adding the parent makes the find a direct child thatchildrenOfselects (dot-folder roots included, since inside a root is not protected). The home folder and its ancestors get words, no command. The only exception is finding 1.
Tests in the last push
- discovery-census.test.ts: the assertion that the sentence is absent for unchecked finds was replaced by
command === null || !command.includes('unchecked'), which cannot fail (an unchecked group's command is always null). The sentence is still covered by "what the listing may claim" in discovery-report.test.ts, so nothing is lost, but delete the vacuous line. - The "below a configured root" tests and the largest-folder test assert observable output against hand-written inputs and could disagree with the code. No test covers a broad or non-existent configured root (finding 1) or an ancestor/descendant registered project (note 4).
- The disk-usage partial test goes through the real parsing with a faked
durejection;_partialis an unused binding used to drop a key.
Human review: product choices the maintainer did not state on #5762: a registered or discovered checkout that is an ancestor or descendant of a find counts as managing it (note 4); doctor offers the adopt command for dot-folder groups, the low-disk line does not; the line at critical only; sizes of recognised build output only; the 25-checkout and 90 s bounds; the date of last git activity instead of an idle threshold; ~/.codex/worktrees worktrees not listed.
…verns every figure
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head 1eb9299
Verdict: changes needed
Scope: reviewed the whole diff against origin/main and the last push, read unmanagedScanFinds, all of discovery-census.ts and discovery-report.ts, tried to break the author's drop audit and the single complete value with hand-written scan and census files run through the report functions, and checked the LLP, guide, website and PR body against the code. Review only: nothing was edited, no suite was run. CI on this head is green (12 success, rest skipped).
Findings
-
Medium, actionable.
packages/stim-cli/src/workspace/discovery-report.ts:339and:227. After a person runs the adopt command, the census still predates the setting.rootsChangedSinceCensusis true and only feedscompleteand the low-disk line; the doctor text never says the list is older than the setting. I reproduced it (census withroots: [], setting now['/Users/me/dev']): doctor prints "~/dev: 1 checkout, at least 4.7 GB ..." then the lead-in "Adding the folder to maintenance.discoverRoots applies the full discovered-checkout rules...:" followed by "This folder is already named in maintenance.discoverRoots, so there is nothing to add; these are listed because discovery did not select them or could not check them yet." In this window the stated reason is untrue: discovery most likely selects them now, they are fully managed, and they are listed only because the daemon has not listed again. "At least" appears with no explanation. Same window opens when a project is registered (no census trigger; up to a day) and for a root that was removed. The "Known limits" sentence that doctor "says so each time" the list is incomplete is not true for this state (nothing is printed except "at least"). Fix: whenrootsChangedSinceCensus, print one sentence that the list was made before the last change of maintenance.discoverRoots and may include checkouts that are managed now, and give thealready-namedtext a reason that does not claim discovery skipped them (or drop the lead-in for that group). Add a test on the printed text for this state; the current tests assert onlycommand: nullandcommandProblem. -
Low, actionable (docs or one-line code).
discovery-census.ts:~395-433. A checkout that was fully measured earlier, is a day or more old, and is left over by the 25-checkout cap or the 90 s budget is pushed askeptwith its old measurement and does not setpending. The next run is then a day later, not an hour. LLP 0004, the guide and the website say what a run leaves is measured by the next one an hour later; that holds for never-measured and incomplete entries only. The figure is shown as exact (with "oldest measurement "). Either setpendingwhen a stale entry is left over, or correct the three documents. -
Low, note. Audit gaps (all drop or hide without being in the 17 items):
a.discovery.ts:249(revalidatedScan): a non-absolute path inscan.foundis skipped silently. Item 4 sayscovered()does this, butcovered()(:384) only seesdotFolderFoundand pending paths; thefoundcase is this other line. The justification is the same.
b.discovery.ts:322-323(computeDiscovered,allowed): a valid scan find in a protected location (temp dir, or a path whose realpath fails) is dropped fromfound, so it is neither reduce-only nor dot-folder. Not reachable from a real scan (it skips temp, the scan file partitions dot-folders intodotFolderFound,scanEntryValidalready took the realpath), so low, but it is a silent drop and belongs in the audit.
c.covered()counts reduce-only scan finds as coverage for a pending or dot-folder path that contains or lies inside them. Documented known limit, not reported.
Item 4 answer: a non-absolute entry cannot hide a real checkout. The scanner writes canonical absolute paths and the deleting passes ignore such entries too. If the maintainer wants it surfaced, rejecting it inparseDiscoveryScan(scan "unusable") is cleaner than "incomplete" or "unchecked" ("checks again within the hour" would be false for bad data). I would not block on it.
Items 1-3, 5-17 hold in the code as stated. Item 9 says an unchecked find is pushed unchanged; it is{...earlier, ...placement}, which keeps the earlier measurement (the docs say so). -
Low, note.
discovery-report.ts:366: the low-disk line ends the command with; stim doctor lists the rest.... A person who copies the whole sentence tail to a shell runsstim settings set ...; stim doctor lists .... Put the command last or on its own line. -
Note. A permanently unreadable scan path, a permanent partial
duor a permanently unreadable layout meanscompleteis never true on that machine, so the "every checkout it found is managed" sentence never prints and the low-disk line always says "at least". That is truthful and documented as a known limit; no change asked.
Item results
- Single value: every consumer uses
report.complete(sizeText, the empty-list sentence,lowDiskLine, JSON). I could not construct an exact figure, the complete-list sentence ordiscovery.complete === truewith a truncated or failed scan, unreadable scan paths, a non-current or pending census, changed roots, an unchecked group, or a failed, partial or not-measured checkout. The one gap is finding 2 (stale left-over entry reads as complete, dated). Reverse nuisance: only the permanent states in finding 5. - Removed filter wording: the home group gets "home folder or holds it" with no command (
adoptablefalse beforenamed); an already-named group gets no command;lowDiskLinepicks only areduce-onlygroup with a command and bytes, so never a dot-folder, already-named, too-broad or unchecked group. Wording is truthful except in the window of finding 1. - Over-reporting: yes, in the window of finding 1 (and after project registration). The low-disk line is silent then; doctor is not explanatory.
- Documents: the claims that go beyond the code are findings 1 (known limits "says so each time") and 2 (an hour). The website sentence "For each folder doctor prints ... the command" omits the home, already-named and unchecked cases (minor).
- Spot checks: cancellation order (
isUnverifiedCleanupbefore the abort check), the per-command hook (Stim-home files only,canonical: null, stderr only) and no delete, reduce, register or set all hold. - Tests in the last push: they follow AGENTS.md (observable report output, expectations independent of the code). The gap is the missing text assertion for finding 1.
Human review: whether doctor should keep listing a folder the person just adopted (marked as older than the setting) or hide it until the daemon lists again; the issue does not say.
Description
The home folder scan from #5764 records which checkouts it found and what Stim does with each kind: finds outside home dot-folders are reduce-only, finds under a home dot-folder are recorded and never acted on. Nothing shows that to a person or an agent. On the machine where this started, 80 GB of build output under
~/.exact/exact2would now be found by the scan and still be invisible, with no hint that one setting brings it under management.The maintainer asked for it in two places (verbatim on #5762): "doctor lists them and suggest adding as managed path or delete if idle for long time", and, for the line Stim commands print on low disk, "smt like low disk space, found projects not managed by stim taking x space , command to add to discover roots". And about the scan itself: "like maybe not part of doctor since it could be slow".
This change only reports. It deletes, reduces, registers and sets nothing.
Solution
The daemon measures after a scan. Doctor and the command line must not walk the disk, so after the scan the daemon classifies the finds and measures the recognised build directories of the ones Stim does not fully manage, through
refreshMeasure(the walk claim, one walk at a time, 10 s per directory). It handles at most 25 checkouts and 90 s per run, carries over a checkout measured within a day, and continues an hour later with what was left. The result is its own file,maintenance/discovery-census.json, so the scan file is untouched.This is the first code that reads inside a home dot-folder find, so the census's own reads of a checkout's layout go through the bounded reader the scan uses (asynchronous, 3 s, one outstanding read, fail closed), with the abort and the run budget checked before each. The directory logic in
regenerable-dirs.tsis driven by a bounded answerer, without changing it for the passes that delete; a cache tag is read only if it is a regular file, and only its first 64 bytes.refreshMeasuregets an option,knownToExist, that skips its synchronous existence probe; the census passes it because the bounded read has already established the directory, and every other caller is unchanged. Bounded throughanswered: the scan's safety gate for each folder, the census's own layout reads and the git-activity reads. Not bounded by the shared reader: theduwalks insiderefreshMeasure, which have their own time limit and run in a child process; and what discovery already does for the passes on main, found by readingcomputeDiscovered,unmanagedScanFinds,scanSkipperandchildrenOf(the callees of project recognition and of the git helpers were not traced): synchronousexistsSyncon each configured discover root, registered project root and worktree entry; synchronousrealpathSync(canonicalPath) on those, on the repository folders, the home folder, every listed child, the known roots and, inscanSkipper, the Stim home, build worker root and temporary directories (with a config read);isProtectedLocation(realpathSync) on every worktree path and candidate child outside a configured root; project recognition (selectDoctorand a synchronousreaddirSyncofapps,examplesandpackages) inchildrenOfand at the end ofscanEntryValid;childrenOflistings raced against 3 seconds; and git subprocesses (git worktree list,git symbolic-ref) through the exec wrapper. What this change adds is that project recognition now also runs on finds under home dot-folders.Four things a reader may not confuse are kept apart: a checkout whose layout could not be read or whose walk failed is "could not be measured", one not reached is "not measured yet", one whose scan entry could not be re-checked this run is "could not be checked" (kept with its earlier measurement and that measurement's date, and retried within the hour), and none of them is zero; totals read "at least". A
duthat exits with status 1 after printing a size (a walk it could only partly read) is recorded as a lower bound:measureTreeandrefreshMeasurecarry an optionalpartialflag, stored in the cache entry only when set, the census counts the bytes and marks the group "at least", and no other reader of the disk-usage cache sees a different value. A scan find whose metadata read fails with anything other than proven absence (ENOENT, ENOTDIR) is nowpending, notinvalid, inscanEntryValid; the deleting passes select onlyvalidentries, so they select the same set as before, and the census lists such a find as unchecked. Doctor says every checkout the scan found is managed only when the scan finished, the census was made from that scan, and nothing is unchecked. The census classifies from a discovery computed after the scan it records, not from the list a pass memoised earlier in the same operation, and records that scan's time; when the census belongs to an older scan than the scan file, the report shows "not measured yet" and the low-disk line is not printed. A measuring child whose cleanup cannot be verified is not swallowed, including when the operation was aborted at the same moment: that error, and a claim-unavailable error, leave the census and the pass unchanged, as LLP 0007 requires; any other census error is contained and logged as onemaintenance_failurerecord.stim doctorlists them, in text and in--json, from the two cached files and the setting only. Groups are by containing folder, reduce-only finds first and dot-folder finds separately, because Stim reduces the first and does nothing with the second. Each group carries the exactstim settings set maintenance.discoverRoots '[...]' --scope machinecommand, built from the current value plus the folder (canonical paths, shell-quoted), and says that adopting a folder applies the full discovered-checkout rules, which include removing finished linked worktrees. The report never hides a census entry. A find leaves the listing only in classification, when a registered project or a discovered checkout reaches it; a find that contains, or lies inside, one of those is treated as managed, although only that project or checkout is (the ruleselectDiscoveredalready uses for the passes). A find deeper under a configured root is listed, because a root reaches only its direct children, and its command adds that find's own parent folder. A group whose folder is a root discovery already uses is shown with no command and says so: its entries are there because discovery did not select them or could not check them. The home folder gets words and no command, since it cannot be a root.One completeness value drives every figure and the complete-list sentence, in text, in
--json(discovery.complete) and on the low-disk line. It is false when the scan was not read, was truncated or has unreadable paths; when the census was not read, is for another scan, is pending or predates a change ofmaintenance.discoverRoots; or when any group is unchecked or holds a failed, partial or not-yet-measured checkout. Every figure then reads "at least". The section is not a finding, so a passing doctor still passes.The date of last git activity, not an idle verdict. For each listed checkout the census records the newest modification time of
HEAD,indexandlogs/HEADin its git directory, with boundedlstatreads, no git process and no walk of the source tree; a linked worktree's git directory is taken from its.gitfile (absolute, or relative as git 2.48 and later can write it) only when it ends in.git/worktrees/<name>, holds a regulargitdirfile that names this checkout's.gitfile, and has a canonical path equal to its written path, so no component is a symlink. Doctor shows the date per checkout, oldest first, and "unknown" when it could not be read, never as old. It states that Stim has not checked whether a checkout is in use or holds uncommitted or unpushed work, and that the date says nothing about edits made without git. There is no threshold: the maintainer asked for a suggestion to delete what has been idle a long time, gave no number, and the existing idle evidence cannot be bounded on these paths without changing code the deleting passes share.The critical-disk line gains a second line on stderr when the census measured anything: the total these checkouts hold, that adopting a folder applies the full rules, and the command for the largest folder. Its trigger is unchanged: the daemon's last tick, at most 10 minutes old, found the disk critical. This runs on every command while the disk is critical, so it reads only files under the Stim home (the daemon snapshot, the scan file, the census file and the config) and compares against the canonical roots the census recorded; when
maintenance.discoverRootshas changed since the census, the extra line is left out.stim doctorstill resolves symlinks of configured roots itself. It never names a dot-folder group as the folder to add, because adopting a tool's home gives it the full removal rules; the bytes still count in the total.Choices the maintainer did not state, for review: the date in place of an idle threshold; the line at critical only, not at high; sizes counting build output, not
node_modulesor Pods; worktrees under~/.codex/worktreesnot being listed, since they are not scan finds; the per-run bound of 25 checkouts and 90 s; doctor offering the adopt command for dot-folder groups while the low-disk line does not. Doctor refuses outside a project, so the listing shows only inside one.Not changed:
stim statusand Desktop. A later change can add the same data from the two files.Test plan
reportCleanupreaches the caller ofrunMaintenanceas the same object, also when the operation is aborted.refreshMeasureprobes the path by default and not withknownToExist; the census passes it for every directory.scanEntryValidwith a failinglstat: EACCES, EPERM and EIO pending, ENOENT and ENOTDIR invalid; through the census and doctor, an EACCES find is unchecked and the complete-list sentence is not printed.duresult of exit status 1 with output, through the realmeasureTreeparsing, to the census record and the "at least" wording in doctor and on the low-disk line;readDiskUsagereturns nopartialproperty.stim doctor --jsoncarries thediscoveryfield with the groups and their facts; the text path prints the same groups.gitdir:that validates (date shown); a.gitfile pointing outside a.git/worktrees/directory, an unrelated.git/worktrees/directory, a symlinked ancestor, a symlinkedHEADand a never-settling read (all unknown).~/.exact,~/.exact/exact2/ais listed and its command adds~/.exact/exact2; a direct child discovery selects is not listed; an EACCES find under the root is unchecked.~with a find at~/app: listed with the home-folder wording, complete-list sentence absent, low-disk line present. An unchecked direct child of a configured root: listed, no command (report level only, not through the census with a fixture). A truncated scan with fully measured finds: total and largest figure read "at least",completefalse; likewise for failed, unreadable, pending and other-scan states.stim doctorandstim doctor --jsonin a scratchSTIM_HOMEwith hand-written scan and census files: a measured group, one with a failed measurement, an 18-checkout dot-folder group of 79.2 GB and a home-folder group with no command were listed; the printed command for a folder containing'was run in the scratch home and stored the exact path, after which the group left the listing; the JSON parsed; with a critical daemon snapshot both stderr lines appeared.dutiming on large Cargo targets.Part of #5762.