Skip to content

feat: list unmanaged checkouts in doctor and on the critical disk line - #5767

Draft
janicduplessis wants to merge 11 commits into
mainfrom
feat/5762-doctor-and-low-disk-line
Draft

janicduplessis wants to merge 11 commits into
mainfrom
feat/5762-doctor-and-low-disk-line

Conversation

@janicduplessis

@janicduplessis janicduplessis commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The home folder scan from #5764 records which checkouts it found and what Stim does with each kind: finds outside home dot-folders are reduce-only, finds under a home dot-folder are recorded and never acted on. Nothing shows that to a person or an agent. On the machine where this started, 80 GB of build output under ~/.exact/exact2 would now be found by the scan and still be invisible, with no hint that one setting brings it under management.

The maintainer asked for it in two places (verbatim on #5762): "doctor lists them and suggest adding as managed path or delete if idle for long time", and, for the line Stim commands print on low disk, "smt like low disk space, found projects not managed by stim taking x space , command to add to discover roots". And about the scan itself: "like maybe not part of doctor since it could be slow".

This change only reports. It deletes, reduces, registers and sets nothing.

Solution

The daemon measures after a scan. Doctor and the command line must not walk the disk, so after the scan the daemon classifies the finds and measures the recognised build directories of the ones Stim does not fully manage, through refreshMeasure (the walk claim, one walk at a time, 10 s per directory). It handles at most 25 checkouts and 90 s per run, carries over a checkout measured within a day, and continues an hour later with what was left. The result is its own file, maintenance/discovery-census.json, so the scan file is untouched.

This is the first code that reads inside a home dot-folder find, so the census's own reads of a checkout's layout go through the bounded reader the scan uses (asynchronous, 3 s, one outstanding read, fail closed), with the abort and the run budget checked before each. The directory logic in regenerable-dirs.ts is driven by a bounded answerer, without changing it for the passes that delete; a cache tag is read only if it is a regular file, and only its first 64 bytes. refreshMeasure gets an option, knownToExist, that skips its synchronous existence probe; the census passes it because the bounded read has already established the directory, and every other caller is unchanged. Bounded through answered: the scan's safety gate for each folder, the census's own layout reads and the git-activity reads. Not bounded by the shared reader: the du walks inside refreshMeasure, which have their own time limit and run in a child process; and what discovery already does for the passes on main, found by reading computeDiscovered, unmanagedScanFinds, scanSkipper and childrenOf (the callees of project recognition and of the git helpers were not traced): synchronous existsSync on each configured discover root, registered project root and worktree entry; synchronous realpathSync (canonicalPath) on those, on the repository folders, the home folder, every listed child, the known roots and, in scanSkipper, the Stim home, build worker root and temporary directories (with a config read); isProtectedLocation (realpathSync) on every worktree path and candidate child outside a configured root; project recognition (selectDoctor and a synchronous readdirSync of apps, examples and packages) in childrenOf and at the end of scanEntryValid; childrenOf listings raced against 3 seconds; and git subprocesses (git worktree list, git symbolic-ref) through the exec wrapper. What this change adds is that project recognition now also runs on finds under home dot-folders.

Four things a reader may not confuse are kept apart: a checkout whose layout could not be read or whose walk failed is "could not be measured", one not reached is "not measured yet", one whose scan entry could not be re-checked this run is "could not be checked" (kept with its earlier measurement and that measurement's date, and retried within the hour), and none of them is zero; totals read "at least". A du that exits with status 1 after printing a size (a walk it could only partly read) is recorded as a lower bound: measureTree and refreshMeasure carry an optional partial flag, stored in the cache entry only when set, the census counts the bytes and marks the group "at least", and no other reader of the disk-usage cache sees a different value. A scan find whose metadata read fails with anything other than proven absence (ENOENT, ENOTDIR) is now pending, not invalid, in scanEntryValid; the deleting passes select only valid entries, so they select the same set as before, and the census lists such a find as unchecked. Doctor says every checkout the scan found is managed only when the scan finished, the census was made from that scan, and nothing is unchecked. The census classifies from a discovery computed after the scan it records, not from the list a pass memoised earlier in the same operation, and records that scan's time; when the census belongs to an older scan than the scan file, the report shows "not measured yet" and the low-disk line is not printed. A measuring child whose cleanup cannot be verified is not swallowed, including when the operation was aborted at the same moment: that error, and a claim-unavailable error, leave the census and the pass unchanged, as LLP 0007 requires; any other census error is contained and logged as one maintenance_failure record.

stim doctor lists them, in text and in --json, from the two cached files and the setting only. Groups are by containing folder, reduce-only finds first and dot-folder finds separately, because Stim reduces the first and does nothing with the second. Each group carries the exact stim settings set maintenance.discoverRoots '[...]' --scope machine command, built from the current value plus the folder (canonical paths, shell-quoted), and says that adopting a folder applies the full discovered-checkout rules, which include removing finished linked worktrees. The report never hides a census entry. A find leaves the listing only in classification, when a registered project or a discovered checkout reaches it; a find that contains, or lies inside, one of those is treated as managed, although only that project or checkout is (the rule selectDiscovered already uses for the passes). A find deeper under a configured root is listed, because a root reaches only its direct children, and its command adds that find's own parent folder. A group whose folder is a root discovery already uses is shown with no command and says so: its entries are there because discovery did not select them or could not check them. The home folder gets words and no command, since it cannot be a root.

One completeness value drives every figure and the complete-list sentence, in text, in --json (discovery.complete) and on the low-disk line. It is false when the scan was not read, was truncated or has unreadable paths; when the census was not read, is for another scan, is pending or predates a change of maintenance.discoverRoots; or when any group is unchecked or holds a failed, partial or not-yet-measured checkout. Every figure then reads "at least". The section is not a finding, so a passing doctor still passes.

The date of last git activity, not an idle verdict. For each listed checkout the census records the newest modification time of HEAD, index and logs/HEAD in its git directory, with bounded lstat reads, no git process and no walk of the source tree; a linked worktree's git directory is taken from its .git file (absolute, or relative as git 2.48 and later can write it) only when it ends in .git/worktrees/<name>, holds a regular gitdir file that names this checkout's .git file, and has a canonical path equal to its written path, so no component is a symlink. Doctor shows the date per checkout, oldest first, and "unknown" when it could not be read, never as old. It states that Stim has not checked whether a checkout is in use or holds uncommitted or unpushed work, and that the date says nothing about edits made without git. There is no threshold: the maintainer asked for a suggestion to delete what has been idle a long time, gave no number, and the existing idle evidence cannot be bounded on these paths without changing code the deleting passes share.

The critical-disk line gains a second line on stderr when the census measured anything: the total these checkouts hold, that adopting a folder applies the full rules, and the command for the largest folder. Its trigger is unchanged: the daemon's last tick, at most 10 minutes old, found the disk critical. This runs on every command while the disk is critical, so it reads only files under the Stim home (the daemon snapshot, the scan file, the census file and the config) and compares against the canonical roots the census recorded; when maintenance.discoverRoots has changed since the census, the extra line is left out. stim doctor still resolves symlinks of configured roots itself. It never names a dot-folder group as the folder to add, because adopting a tool's home gives it the full removal rules; the bytes still count in the total.

Choices the maintainer did not state, for review: the date in place of an idle threshold; the line at critical only, not at high; sizes counting build output, not node_modules or Pods; worktrees under ~/.codex/worktrees not being listed, since they are not scan finds; the per-run bound of 25 checkouts and 90 s; doctor offering the adopt command for dot-folder groups while the low-disk line does not. Doctor refuses outside a project, so the listing shows only inside one.

Not changed: stim status and Desktop. A later change can add the same data from the two files.

Test plan

  • Classification and report from hand-written scan and census files and temp directories: reduce-only and dot-folder groups, a folder the setting names dropped, a path containing a quote producing a command that round-trips, an unreadable scan or census file adding nothing, unmeasured kept apart from zero.
  • The census with injected readers and measuring: the 25-checkout and time bounds, carry-over within a day, a never-settling listing and a never-settling tag read stopping at the budget with the checkout "could not be measured", a denied listing not being zero, the same-operation case, a census from an older scan, and nothing written in a scoped home.
  • A measuring error that carries reportCleanup reaches the caller of runMaintenance as the same object, also when the operation is aborted.
  • A find whose re-check did not answer: listed as unchecked with its earlier measurement, the census left pending, and neither doctor nor the low-disk line claiming a complete list.
  • refreshMeasure probes the path by default and not with knownToExist; the census passes it for every directory.
  • The low-disk line with the setting changed since the census: no extra line.
  • scanEntryValid with a failing lstat: EACCES, EPERM and EIO pending, ENOENT and ENOTDIR invalid; through the census and doctor, an EACCES find is unchecked and the complete-list sentence is not printed.
  • A fake du result of exit status 1 with output, through the real measureTree parsing, to the census record and the "at least" wording in doctor and on the low-disk line; readDiskUsage returns no partial property.
  • A group with no build output prints 0 B.
  • stim doctor --json carries the discovery field with the groups and their facts; the text path prints the same groups.
  • Last git activity: a main checkout, a linked worktree, a relative gitdir: that validates (date shown); a .git file pointing outside a .git/worktrees/ directory, an unrelated .git/worktrees/ directory, a symlinked ancestor, a symlinked HEAD and a never-settling read (all unknown).
  • The critical-disk line: one extra stderr line, nothing on stdout.
  • A find is managed only when a registered project or a discovered checkout reaches it: with root ~/.exact, ~/.exact/exact2/a is listed and its command adds ~/.exact/exact2; a direct child discovery selects is not listed; an EACCES find under the root is unchecked.
  • The largest folder on the low-disk line reads "at least" when its group has partial, failed or not-yet-measured checkouts.
  • The cache flag: a partial walk re-read as partial, a later complete walk clearing it, and an entry without the field reading as complete.
  • The broad root ~ with a find at ~/app: listed with the home-folder wording, complete-list sentence absent, low-disk line present. An unchecked direct child of a configured root: listed, no command (report level only, not through the census with a fixture). A truncated scan with fully measured finds: total and largest figure read "at least", complete false; likewise for failed, unreadable, pending and other-scan states.
  • Full CI workflow on 1eb9299, all four test jobs green with no rerun: https://github.com/appandflow/stim/actions/runs/43740069815
  • Real run, on an earlier head of this branch, of the built stim doctor and stim doctor --json in a scratch STIM_HOME with hand-written scan and census files: a measured group, one with a failed measurement, an 18-checkout dot-folder group of 79.2 GB and a home-folder group with no command were listed; the printed command for a folder containing ' was run in the scratch home and stored the exact path, after which the group left the listing; the JSON parsed; with a critical daemon snapshot both stderr lines appeared.
  • Not exercised: the census inside a live daemon or against the real Stim home; du timing on large Cargo targets.

Part of #5762.

@janicduplessis janicduplessis added area: cli packages/stim-cli, including its README and guide area: core packages/core area: website website/ labels Nov 15, 2026

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking findings. Seven non-blocking findings, one of them a product-decision list. Read-only review at a4f7b08; I did not build or run tests.

Report-only (verified by reading)

  • unmanagedScanFinds (discovery.ts:340-373) returns two arrays to runDiscoveryCensusIfDue, which is its only caller. Nothing from it reaches discovery's selected list, the stages, artifacts, Cargo passes, the ladder or gc. dotFolderFound has exactly one reader, and it is this one. Each dot-folder path goes through scanEntryValid with the same rules as revalidatedScan. No project detection is called on it outside that gate.
  • refreshMeasure writes the disk-usage cache for the build dirs. ARTIFACT_DIRS.build and the discovered build dirs are not isDependencyTree paths, so trustedMeasure in daemon/dependencies.ts (node_modules/Pods only) does not see them. The caps (plan.ts, fed by measure.ts through collectWorkspaceOutputs, and cargo-target.ts through measuredDirectorySizeAsync) do not read this cache. The readers are status-measures.ts and the gc worktree sweep, which show buildState and decide nothing. Dot-folder checkouts are not registered workspaces or discovered worktrees, so neither reader reaches them. Measuring them changes nothing Stim later does to any checkout.
  • The census runs in scanAfterPass, after the claim is released. It does not run after a failed or aborted pass. Its own errors are swallowed, so it cannot mask the pass error. Only an abort escapes the function. A scoped home never runs it. An unreadable or unusable scan returns without a write. An unreadable census file returns without a write and is never repaired. The write is a temp file plus rename under withDirLock. Core owns the path and the reader. The reader separates absent, unreadable and unparsable. Unmeasured is kept apart from failed and from zero.
  • Doctor and warnCriticalDisk read two files plus config. There is no walk, git or process read. JSON mode prints one payload. The section is outside findings, so PASS and the exit code are unchanged. The low-disk line goes to stderr only and sits inside a try/catch.
  • shellQuote(JSON.stringify([...])) in single quotes is correct POSIX quoting for spaces, ', $, backticks and non-ASCII. The home folder, a folder holding it, and / get words and no command (isBroadFolder). Groups whose folder is a configured root are dropped.

Findings

  1. NON-BLOCKING, discovery-census.ts:177-181. The 90 s budget is checked only before each measureBuildOutput. idleSince has no budget check. It runs discoveredEvidence, which walks up to 20,000 source entries, runs git check-ignore per skipped-where-ignored entry, and runs git log and worktree activity, for each of up to 25 checkouts. unmanagedScanFinds also re-runs full discovery. The census is awaited in scanAfterPass, so the passes thread can run well past 90 s. All of it is async and abortable, so nothing blocks the event loop. Suggest checking the deadline before idleSince, or documenting that the bound covers walks only. Separately, appRoots (reached through measureBuildOutput) does sync readdirSync and isDirectory on apps/ and packages/ of the checkout, on the daemon thread. The artifacts stage already does this, but a dot-folder path is only validated once, earlier in the run.
  2. NON-BLOCKING, discovery-census.ts:157-160 and discovery-report.ts:158-160. The idle verdict is stale. lastActivityAt is carried over for up to a day (earlier reused without re-evaluating). The process table, launcher scan and agent sessions are read once per run (the launcher reader is cached per operation and can be minutes old). Doctor prints "Idle for at least N days and nothing uses it" as present fact. If someone starts using the checkout after the evaluation, doctor still says this until the next census. Suggest "as of ", or re-evaluating idle on every run. It is words only, but it tells a person to delete. discoveredEvidence has no check for uncommitted or unpushed work in the checkout. Old uncommitted edits count as idle. The text says "delete it yourself if you no longer need it" and does not mention that risk.
  3. NON-BLOCKING, discovery-report.ts:253-259. The low-disk line says "to what Stim manages with: ". It does not say the command grants removal of finished linked worktrees. The doctor text and the guide do say so. The line is meant for an agent to run, and it is an LLP-listed consequence. Suggest one clause.
  4. NON-BLOCKING, discovery-report.ts. discoveryReport never compares census.scanRanAt to scan.ranAt, or the census age. After a later scan, or a failed daemon, doctor shows an older census next to the new scan date ("Listed and measured by the daemon on ..." does carry its own date). Also rootsOf uses resolve, not canonical paths, while census folders are canonical. A symlinked adopted root can keep its group, and adoptCommand does not dedupe, so it re-proposes the folder until the next census. This is transient. Over-budget entries with an older complete measurement keep that stale value as "measured" with no age.
  5. NON-BLOCKING, discovery-census.ts catch block. If unmanagedScanFinds throws, nothing is written and nothing is logged. Doctor keeps saying "has not yet listed" with no reason. The attempt is retried hourly.
  6. NON-BLOCKING, minor and not verified. Census measures with private: false and writes entries without privateBytes. The gc sweep reuses a fresh cache entry for dirs in privateSized, which could then show privateBytes null in the report. That is display only. I did not trace privateSized.
  7. NON-BLOCKING, docs. LLP 0004 says "like maybe not part of doctor since it could be slow" was "Relayed by the coordinator, not on the issue". It is on #5762 as a comment, verbatim. The other two quotes match the issue word for word. The warnCriticalDisk comment and guide say "two small files". It also reads config and the setting.

Tests (read, not run)
discovery-census.test.ts redirects HOME, USERPROFILE and STIM_HOME to a temp dir and injects the process and session readers. The report tests use hand-written files. The maintenance-rung6 test covers the extra line. The assertions can disagree with the code: failed vs zero, the bound, carry-over, scoped home, roots change. The lowDiskLine / adoptCommand tests check exact strings. I did not check whether they fail when each behaviour is removed.

Product decisions the maintainer did not state (list for review: needs-human)

  • Doctor prints an adopt command for dot-folder groups (e.g. ~/.exact/exact2). The low-disk line withholds it. Doctor discloses that adoption "lets Stim act where it did not". Needs a maintainer decision.
  • Idle threshold reuses maintenance.olderThanDays (3 days).
  • The line appears at critical only, not at high.
  • Sizes cover build output only, not node_modules or Pods.
  • The 25-checkout / 90 s / 10 s / one-day carry-over bounds.
  • The discovery JSON field on doctor, and doctor refusing outside a project, so the listing shows only inside one.
  • The census file, and the doctor output section title and wording.
  • Total "at least" wording.

Not checked: CI results, api.api.txt, knip, a live daemon run, privateSized semantics, and whether any other command prints the low-disk line in --json mode (stderr only here).

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: clear. No blocking or actionable defect at 2655135. Read-only review (read the diff, the callers and the tests; I did not build or run tests). CI on this head is green, including merge gate. Seven non-blocking notes and a product-decision list below.

1. Report-only (verified by reading)

  • unmanagedScanFinds (discovery.ts:340-382) is called only by runDiscoveryCensusIfDue; dotFolderFound has one reader (discovery.ts:369) and one writer (the scan). Nothing from either reaches discovery's selected list, stages, artifacts, Cargo passes, the ladder or gc. It calls computeDiscovered with the scan and discards everything but the classification.
  • regenerable-dirs.ts diff is four export keywords (DirRead, DirReads, signatureHeads, existingBuildDirsReads). No behaviour change for the deleting passes.
  • Readers of the disk-usage cache: status-measures.ts (display), gc/worktree-sizes.ts (display of buildState), daemon/dependencies.ts and ladder.ts through knownPrivateBytes/trustedMeasure. Those take dependency-tree paths (node_modules, ios/Pods) from the pass's own candidates; the census writes only recognised build directories, never those paths, so the 256M REAL_BYTES_THRESHOLD rule never reads a census entry. The census's private: !FIXED_BUILD... matches the how !== 'fixed' rule in worktree-sizes.ts:71, and dirSize only reports it. No decision, cap or deletion reads an entry the census wrote.
  • The passes' memo is unchanged: discovery() passes readDiscoveryScan() where revalidatedScan used to read it itself.

2. Reads the census makes
Through answered with signal.throwIfAborted() and the deadline checked first (boundedReader.read, discovery-census.ts:119-130): readdir of apps, packages and each build-dir parent; the directory check (lstat); the 64-byte tag head; the generators from regenerable-dirs.ts (driven by drive); lstat of .git, of the gitdir and of HEAD, index, logs/HEAD; the 512-byte .git file head. Dot-folder revalidation (scanEntryValid) uses answered too.
Not through it, and the lane's list is incomplete in two items that I find acceptable: (a) refreshMeasure's du, its synchronous existsSync and cache/claim reads under $STIM_HOME (the path was listed through answered moments before; the gc sweep and status-measures make the same call); (b) the discovery work computeDiscovered does: git via resolveSourceCheckoutAsync, sync existsSync/realpathSync/readdirSync in canonicalPath, childrenOf, read and recognises. recognises runs on a dot-folder path only after scanEntryValid's bounded safety gates, as for scan finds. Not on the lane's list: writeCensus's sync withDirLock (state dir). Reads of dot-folder paths by existsSync were not made before this change on that lane, but only after the bounded gate above.

3. lastGitActivity
Only lstat and a prefix read of a regular .git file (head lstats first). .git as a symlink, HEAD/index/logs/HEAD as symlinks, or a symlinked gitdir end in unknown. resolve normalises .. before the /.git/worktrees/<name> test. A crafted .git file can make it lstat HEAD, index and logs/HEAD in any real directory on the machine whose path ends in .git/worktrees/<name> (intermediate components are followed by the OS). The only disclosure is the date of the newest mtime, printed to the user who runs doctor; nothing is read. Windows paths: the regex takes both separators and isAbsolute is platform-correct. Unknown prints "unknown" and sorts last (discovery-report.ts:127-130).

4. Errors
reportCleanup and STIM_CLAIM_UNAVAILABLE rethrow as the same object from the census catch (discovery-census.ts:~403) and scanAfterPass (run.ts:~772); scanAfterPass runs only when !passFailed and not aborted, so it cannot replace a pass error from the finally. Other errors are returned and logged once. Tested in maintenance-run.test.ts and discovery-census.test.ts.

5. Classification, 6. Failed is never zero
Census computes discovery fresh from the scan it records and stores finds.scanRanAt. Report zeroes measurements when scanRanAt differs, so the total is 0 and lowDiskLine returns null. Denied or timed-out layout read gives unreadable; a failed walk gives failedDirs; not reached gives not measured yet; sums use "at least".

7. Doctor and the critical line
Doctor text is added in both print paths, JSON discovery, outside findings, exit code unchanged. The critical line is stderr only inside the existing try/catch. The command is shellQuote(JSON.stringify(...)), home and its ancestors get words only, and a group whose folder is a root (symlinks resolved) is dropped.

9. Docs
The three quotes in LLP 0004 match the issue word for word. Lane and coordinator choices are labelled as such.

Non-blocking notes

  1. discovery-report.ts:84-86. canonicalRoots calls realpathSync on each configured root in doctor and on every command's critical-disk path. A discover root on a hung mount would stall that. Doctor already does existsSync on the roots; the critical-disk path did not.
  2. discovery-census.ts:~370. The census runs a second full discovery (git processes, sync reads) before its 90 s deadline starts, so the run's time is unbounded by the budget. It is hourly at most and abortable.
  3. A checkout whose read timed out is unreadable, is not pending, and is retried after a day, not an hour.
  4. A linked worktree .git with a relative gitdir: (git 2.48 worktree.useRelativePaths) shows unknown. Acceptable and shown as unknown.
  5. guide/cleanup.ts says STIM_CLAIM_UNAVAILABLE "from a walk" propagates; takeWalkClaim turns it into busy, so only reportCleanup can arrive from there. Wording only.
  6. No test covers doctor's text paths or the discovery JSON field; discovery-report.test.ts covers the report.
  7. Real tool calls (invariant 9): the census runs no new tool.

Not checked: a live daemon run, api.api.txt.

@janicduplessis

Copy link
Copy Markdown
Collaborator Author

Human review: choices the maintainer did not state, all at commit 2655135.

@janicduplessis janicduplessis added review: ai-approved A fresh AI review found nothing left to fix review: needs-human Holds merge until a maintainer reviews the lines named in the Human review: comment labels Nov 15, 2026
@janicduplessis
janicduplessis marked this pull request as ready for review November 15, 2026 15:37
@janicduplessis
janicduplessis marked this pull request as draft November 15, 2026 15:38
@janicduplessis janicduplessis removed the review: ai-approved A fresh AI review found nothing left to fix label Nov 15, 2026

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: head 6ff4865 (merge of main into 439da3e)

Verdict: changes needed

I read the diff, the two LLPs, and the merged run.ts and disk-usage.ts, and ran discovery-census, discovery-report, disk-usage-known-path, maintenance-rung6 and maintenance-run tests (all pass). The four earlier defects are fixed except that item 2 (no synchronous call on a find) is still not true on the census path.

Actionable

  1. Synchronous filesystem calls on a scan find's path remain on the census path, and the PR body, LLP 0004 and guide cleanup say there are none beyond "configured roots and registered projects". The census calls unmanagedScanFinds -> computeDiscovered / scanEntryValid (workspace/discovery.ts:241-253, :380-395; workspace/scan-rules.ts:306-352):
    • scanEntryValid ends with isCheckout(path), which is recognises (discovery.ts:62-77): projectRegistry.selectDoctor(path) plus readdirSync(join(path, 'apps'|'examples'|'packages')). It runs on every scan find and, in the new loop at discovery.ts:389-395, on every dot-folder find, inside the tool-owned folder the PR says the census reads only through the bounded reader. A stalled mount blocks the daemon thread here; the 3 s bound does not apply.
    • allowed = ... isProtectedLocation(child.path) (discovery.ts:323-325, protected-locations.ts:14-20) calls realpathSync on every scan find not under a configured root.
    • canonicalPath(join(folder, entry.name)) (discovery.ts:211, realpathSync) runs for each child of a registered-repo sibling folder or configured root, which can be a find's path.
      Fix by bounding or skipping these for the census, or at least state them in the PR body, LLP 0004 ("Discovery's own reads ... are the ones the passes already make" understates it) and guide cleanup ("Every read of a checkout's layout ... is asynchronous"). The sentence in the PR body, "first code that reads inside a home dot-folder find ... every read ... is asynchronous", is false as written.

Notes

  1. discovery-report.ts:216: gb(0) returns "1 MB", so a group whose checkouts are all measured with no build directories reads "1 MB of build output". Not a zero-as-unknown bug, but it states a size that is not there.
  2. discovery-census.ts quota: looked++ counts every incomplete find, including permanently unreadable ones, in fixed order. 25 or more permanently unreadable finds are re-read each hour and later finds are never reached. Acceptable if intended; not documented.
  3. refreshMeasure records du -sk stdout on exit status 1 (partial permission denial) as a measured size, so a partly unreadable build directory is counted as complete. Pre-existing behaviour, now surfaced in a "measured" figure.
  4. writeCensus takes the lock only for the write; last is read unlocked earlier in the run. A pass in another process after the claim is released could write a fresher file and be overwritten. Benign (last writer wins, both valid). The scoped home is skipped (isScopedHome).
  5. adoptCommand writes the value machineSettingValue returns, which can come from an env override rather than the machine config; the command would persist that override. Edge case.

The five checks

  1. Cancelled operation hides unverified cleanup: verified. isUnverifiedCleanup is tested first in the catch of runDiscoveryCensusIfDue and in scanAfterPass, then the abort is checked and the same object rethrown. refreshMeasure's error propagates uncaught through measureBuildOutput; the bounded reader's catch handles read errors only, and buildDirsOf and lastGitActivity rethrow anything except Unreadable and OutOfBudget. daemon/passes.ts records it as failure and settled() rethrows. Tests at discovery-census.test.ts:427-434 cover the same-object and aborted cases.
  2. Synchronous calls: not met, see finding 1. Everything inside discovery-census.ts (layout reads, git directory, bounded-read, the regenerable-dirs generators) is asynchronous. knownToExist only changes the existsSync guard (disk-usage.ts:170); the default path is unchanged for other callers. A directory removed after the bounded read makes du fail, which records "failed", not zero or a size. Dependency trees (treeSignature, sync) are never passed because only build directories are listed.
  3. Unchecked finds: verified. unchecked finds become a kind: 'unchecked' group (discovery-report.ts:204), set pending, and the sentence "Every checkout it found is managed." (:270-274) needs no groups, truncation === 'none', no unreadable folders and censusCurrent. An unchecked find that survives roots.has always yields a group. The low-disk line says "at least" while any group is unchecked and never claims completeness; a census from an older scan gives zero bytes and no line.
  4. Git directory: verified. A linked worktree needs .git/worktrees/<name> suffix, a regular gitdir file that resolves to this checkout's .git, and realpath(gitDir) === gitDir. .git, HEAD, index and logs are lstated, so a symlink gives kind other and unknown. I could not construct a layout reading metadata from an unrelated place other than a genuine worktree of another repository whose own gitdir file names this checkout, which is the real git layout. Submodule and bare-repo layouts give unknown.
  5. Critical-disk hook: verified. warnCriticalDisk reads only the snapshot, scan file, census file and config under the Stim home, calls discoveryReport with canonical: null, and writes with console.error only. Doctor's JSON branch adds one discovery field to the single payload.

Other

  • Merge of main: run.ts and disk-usage.ts read correctly together; the timedStep wrappers live in daemon/* and do not touch scanAfterPass; record(..., walkMs) is unchanged by knownToExist.
  • Nothing on these paths deletes, reduces, registers or writes outside maintenance/discovery-census.json (plus the existing disk-usage cache and failure files refreshMeasure always wrote).
  • Adopt command: canonical paths, single-quote escaping, never for the home folder or a folder holding it; the low-disk line names only reduce-only groups.
  • Tests do not echo mocks or copy prose into regexes; documentation otherwise matches the code apart from finding 1.

Human review: the maintainer did not state these on #5762: the last-git-activity date in place of an idle threshold (LLP 0004 "Instead of idle"); the second line at critical only; sizes counting build output only; the 25-checkout and 90 s bounds; doctor offering the adopt command for dot-folder groups while the low-disk line does not; the separate "unchecked" kind. The PR body lists these already.

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: head acab0d8

Verdict: changes needed (two small items, no deleting-path risk found)

Scope: the last push (6ff4865..acab0d8). Re-spot-checked the earlier points (cancellation not hiding an unverified cleanup, pending finds carried as unchecked, linked-worktree git directory validation, the per-command hook reading only Stim-home files): unchanged and still hold. Ran scan-rules, discovery-census and discovery-report tests: 61 pass. I did not run the full suites.

Proved: no deleting or reducing pass acts on more because of pending.

  • Callers of scanEntryValid: revalidatedScan (workspace/discovery.ts:250) and the dot-folder loop in unmanagedScanFinds (discovery.ts:392). Nothing else.
  • revalidatedScan keeps valid as DiscoveryChild and collects pending into scanPending (discovery.ts:251-252). scanPending is read only at discovery.ts:372-388 (unmanagedScanFinds). The passes use found/outside from computeDiscovered and ignore scanPending. So pending and invalid are the same set for every pass: not selected.
  • Nothing rewrites or prunes the scan file from revalidation: mergeFound (discovery-scan.ts:183) drops stale entries "at read by scanEntryValid, not here", and stores nothing from it.
  • Retry: discoveryScanDue (discovery-scan.ts, ~line 195) reads only ranAt, completedAt, lowScanAt, truncation. A permanently denied find does NOT cause the full home scan to retry hourly. It only affects the census: an unchecked entry sets pending: true (discovery-census.ts, writeCensus call), so censusDue (discovery-census.ts:55) reruns the census hourly. That rerun redoes computeDiscovered (reads, revalidation, git worktree lists) but not the scan and not the du walks of unchanged finds. Cost is bounded but permanent per find, and doctor never prints "Every checkout it found is managed." and the low-disk line says "at least" for as long as the find stays denied. Note for the maintainer.

Proved: partial du reaches no other reader.

  • measureTree adds partial on exit 1 only (private-size.ts:94). Other callers read named fields only: other-homes.ts:148, machine-cache-usage.ts:338.
  • record writes the explicit DiskMeasure fields; partial goes into the cache file only (disk-usage.ts, record). readDiskUsage (core status-measures.ts:200) copies named fields, so DiskMeasure, the phone type, status, gc sizes, the caps and the ladder get the same objects as before. An older Stim ignores the extra key. gc status-caches sees only file bytes (a few bytes larger).
  • A fresh hit stays partial via readDiskUsagePartial; a later complete walk rewrites the whole file without the flag, which clears it (not covered by a test, see below).

Findings

  1. Actionable, minor. lowDiskLine (workspace/discovery-report.ts, the largest branch of the return) prints (${gb(largest.bytes)}) for the largest folder with no "at least", even when that group has partial, failed or not-yet-measured checkouts. The total gets the prefix; the per-folder size does not. Item 3 of the brief (no exact figure including a partial or unmeasured directory) is violated. Prefix it with "at least" when that group's failed + notMeasuredYet + partial > 0, and add a test.
  2. Actionable, doc. LLP 0004 "Known limits": "a find whose read is permanently denied or unreadable takes one of the 25 slots of each census run, every run". True for a find whose layout read is denied (unreadable: true is re-measured, discovery-census.ts, complete check then looked++). False for a find whose metadata read is denied, which is the case this push creates: unchecked entries are pushed and continue before looked++. Say which.
  3. Note, doc. The "not bounded" lists (PR body, LLP 0004, guide/cleanup.ts, website/docs/worktrees.md) read as complete, but computeDiscovered (which the census now runs each time) also makes synchronous existsSync calls on registered project roots, configured discover roots and each worktree entry (discovery.ts, read and canonicalDiscoverRoots), calls recognises in childrenOf, and applies isProtectedLocation to all children, not only "ordinary scan finds". Either add them or say "among others".
  4. Note, tests. Missing: a later complete walk clears partial; an entry without the field reads as complete. not.toContain('Every checkout it found is managed.') copies a sentence of prose; assert on report data (groups containing unchecked, which the same test already does) instead. toContain('at least') in the du test is loose but acceptable. Other new tests (zero as 0 B, EACCES/EPERM/EIO vs ENOENT/ENOTDIR, du status 1 through real measureTree parsing) have independent expectations and are not mock echoes.
  5. Note, perf. refreshMeasure's fresh() now does a second cache file read (readDiskUsagePartial) for every fresh hit, for all callers (status, gc, daemon), though only the census uses the flag. Behaviour is unchanged. Compute it only on request if that matters.

Documentation otherwise agrees with the code: the bounded vs not-bounded split for scanEntryValid and recognises, "pending, never dropped" for a denied lstat, and "no other reader of the cache sees it" (strictly, readDiskUsagePartial does).

Human review: a permanently denied find now stays "could not be checked" forever and keeps doctor from ever saying the list is complete. The maintainer did not state whether that is wanted over dropping it.

…e largest folder size, complete the unbounded-call list

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: head 4e016d9

Verdict: changes needed (one small code defect, one documentation sentence that is false; no deleting-path risk found)

I read the whole diff against origin/main, the last push, the two LLPs, computeDiscovered and what it calls, and the report and census code. I ran discovery-report.test.ts (23 pass) and ran discoveryReport/discoveryLines/lowDiskLine directly on hand-written files. I did not run a daemon. The earlier items (cancellation, linked-worktree git directory, per-command hook reading only Stim-home files, pending for denied metadata, the partial flag) I spot-checked and found as the earlier reviews describe.

Actionable

  1. discovery-report.ts:187 and :191. The report drops a group when roots.has(listed.folder), and roots is built from every configured root lexically (rootsOf), including roots discovery ignores: a root that is the home folder or holds it (isBroadFolder) or does not exist. canonicalDiscoverRoots filters those out, so discovery does not reach children of such a root, and the census records the find as reduce-only with folder = home. Repro: scan finds ~/app (a recognised checkout directly in home), maintenance.discoverRoots = ["~"], census holds the find. discoveryLines prints "Every checkout it found is managed." and lowDiskLine returns null; the find is reduce-only, unmanaged and unlisted. Doctor reports the broad root elsewhere, but the sentence is wrong and the checkout is hidden. Fix: drop a group only when its folder is in census.canonicalRoots (what the classification used), or in the canonical form of a configured root that is also not broad; keep the lexical set only for the changed-roots comparison. Add a test with a broad root.
  2. website/docs/worktrees.md:95: "The census does not walk source files or run git." The census runs unmanagedScanFinds -> computeDiscovered -> resolveSourceCheckoutAsync (git worktree list, and git symbolic-ref for a bare repository) for every registered project root and every recognised child. Only the git-activity reads run no git. Scope the sentence to those reads.

Notes

  1. The "every filesystem call computeDiscovered and what it calls make, read top to bottom" claim (LLP 0004 :358, guide/cleanup.ts, PR body) still omits: the git subprocesses above (async, no bound visible on the census path), and scanSkipper's synchronous canonicalPath of the Stim config dir, the build worker root and the temp folders (run once per revalidatedScan and once in unmanagedScanFinds), plus loadConfig. Either add them or say "the filesystem calls on checkouts".
  2. discovery.ts:383-386 and guide/cleanup.ts:1303 ("A checkout another route already manages is not listed"). touches is true in both directions, as selectDiscovered.covered is. A registered project that is a descendant of a find (a registered app inside a scanned monorepo or a dot-folder checkout) hides the whole find, though Stim manages only that app's folder. This matches what the passes do on main, so I do not call it a defect, but the sentence overstates it. A discovered worktree inside a dot-folder find is not possible today (such worktrees are protected and land in outside).
  3. Coverage answers otherwise hold. Every scan find ends in reduceOnly (origin scan), dotFolder (valid dot find no route touches), unchecked, or is intentionally absent: covered by a known root, selected by another origin, or dropped by isProtectedLocation (non-dot finds are protected only under a temp directory, or when the sync realpath fails after the bounded one succeeded; that race leaves the find in none of the lists and the sentence can print). Nothing under ~/.codex/worktrees can come from the scan (scanDescends prunes .codex; stale entries die in scanEntryValid), so removing the tool-folder clause is correct. Direct children of a configured root are selected with origin root, so they are not listed. A child whose 3 s listing timed out in childrenOf is listed with folder = the root and then dropped by the root test, so it is not over-reported either.
  4. The JSON discovery field has no completeness field. A consumer must derive "at least" from failed, partial and notMeasuredYet per group and from censusCurrent and scan.truncation. The comment on UnmanagedGroup.bytes (discovery-report.ts:16) names only failed and notMeasuredYet and omits partial; the guide's JSON shape (cleanup.ts, "each { folder, kind, checkouts, bytes, measured, failed, notMeasuredYet ...") also omits partial. Either add a boolean (for example lowerBound) or fix both lists.
  5. Byte figures. I could not construct an exact figure that should read "at least": group text, total and largest folder all use the same condition (failed + notMeasuredYet + partial, plus unchecked for the total), a census from an older scan is zeroed and prints no line, and gb(0) prints "0 B". Residual: a failed or not-yet-measured entry that carries an earlier measurement may be larger than it now is, so "at least" is a statement about the earlier size. Doctor text does not say sizes cover recognised build output only (the website and guide do).
  6. Complete-list sentence prints only when: scan file read and usable, census file read and parsed, no group left after the root test, scan.truncation === 'none', scan.unreadable empty, and the census made from this scan (scanRanAt === ranAt). The guide's wording ("a finished scan, a census made from it and no unchecked find") leaves out the empty unreadable list and the empty-group condition. The only wrong print I found is finding 1 (and the protected-location race in note 5).
  7. Adopt command. The folder is dirname(find); the test is exact string equality against configured, canonical-recorded and canonicalised roots; a root reaches direct children only, so adding the parent makes the find a direct child that childrenOf selects (dot-folder roots included, since inside a root is not protected). The home folder and its ancestors get words, no command. The only exception is finding 1.

Tests in the last push

  • discovery-census.test.ts: the assertion that the sentence is absent for unchecked finds was replaced by command === null || !command.includes('unchecked'), which cannot fail (an unchecked group's command is always null). The sentence is still covered by "what the listing may claim" in discovery-report.test.ts, so nothing is lost, but delete the vacuous line.
  • The "below a configured root" tests and the largest-folder test assert observable output against hand-written inputs and could disagree with the code. No test covers a broad or non-existent configured root (finding 1) or an ancestor/descendant registered project (note 4).
  • The disk-usage partial test goes through the real parsing with a faked du rejection; _partial is an unused binding used to drop a key.

Human review: product choices the maintainer did not state on #5762: a registered or discovered checkout that is an ancestor or descendant of a find counts as managing it (note 4); doctor offers the adopt command for dot-folder groups, the low-disk line does not; the line at critical only; sizes of recognised build output only; the 25-checkout and 90 s bounds; the date of last git activity instead of an idle threshold; ~/.codex/worktrees worktrees not listed.

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: head 1eb9299

Verdict: changes needed

Scope: reviewed the whole diff against origin/main and the last push, read unmanagedScanFinds, all of discovery-census.ts and discovery-report.ts, tried to break the author's drop audit and the single complete value with hand-written scan and census files run through the report functions, and checked the LLP, guide, website and PR body against the code. Review only: nothing was edited, no suite was run. CI on this head is green (12 success, rest skipped).

Findings

  1. Medium, actionable. packages/stim-cli/src/workspace/discovery-report.ts:339 and :227. After a person runs the adopt command, the census still predates the setting. rootsChangedSinceCensus is true and only feeds complete and the low-disk line; the doctor text never says the list is older than the setting. I reproduced it (census with roots: [], setting now ['/Users/me/dev']): doctor prints "~/dev: 1 checkout, at least 4.7 GB ..." then the lead-in "Adding the folder to maintenance.discoverRoots applies the full discovered-checkout rules...:" followed by "This folder is already named in maintenance.discoverRoots, so there is nothing to add; these are listed because discovery did not select them or could not check them yet." In this window the stated reason is untrue: discovery most likely selects them now, they are fully managed, and they are listed only because the daemon has not listed again. "At least" appears with no explanation. Same window opens when a project is registered (no census trigger; up to a day) and for a root that was removed. The "Known limits" sentence that doctor "says so each time" the list is incomplete is not true for this state (nothing is printed except "at least"). Fix: when rootsChangedSinceCensus, print one sentence that the list was made before the last change of maintenance.discoverRoots and may include checkouts that are managed now, and give the already-named text a reason that does not claim discovery skipped them (or drop the lead-in for that group). Add a test on the printed text for this state; the current tests assert only command: null and commandProblem.

  2. Low, actionable (docs or one-line code). discovery-census.ts:~395-433. A checkout that was fully measured earlier, is a day or more old, and is left over by the 25-checkout cap or the 90 s budget is pushed as kept with its old measurement and does not set pending. The next run is then a day later, not an hour. LLP 0004, the guide and the website say what a run leaves is measured by the next one an hour later; that holds for never-measured and incomplete entries only. The figure is shown as exact (with "oldest measurement "). Either set pending when a stale entry is left over, or correct the three documents.

  3. Low, note. Audit gaps (all drop or hide without being in the 17 items):
    a. discovery.ts:249 (revalidatedScan): a non-absolute path in scan.found is skipped silently. Item 4 says covered() does this, but covered() (:384) only sees dotFolderFound and pending paths; the found case is this other line. The justification is the same.
    b. discovery.ts:322-323 (computeDiscovered, allowed): a valid scan find in a protected location (temp dir, or a path whose realpath fails) is dropped from found, so it is neither reduce-only nor dot-folder. Not reachable from a real scan (it skips temp, the scan file partitions dot-folders into dotFolderFound, scanEntryValid already took the realpath), so low, but it is a silent drop and belongs in the audit.
    c. covered() counts reduce-only scan finds as coverage for a pending or dot-folder path that contains or lies inside them. Documented known limit, not reported.
    Item 4 answer: a non-absolute entry cannot hide a real checkout. The scanner writes canonical absolute paths and the deleting passes ignore such entries too. If the maintainer wants it surfaced, rejecting it in parseDiscoveryScan (scan "unusable") is cleaner than "incomplete" or "unchecked" ("checks again within the hour" would be false for bad data). I would not block on it.
    Items 1-3, 5-17 hold in the code as stated. Item 9 says an unchecked find is pushed unchanged; it is {...earlier, ...placement}, which keeps the earlier measurement (the docs say so).

  4. Low, note. discovery-report.ts:366: the low-disk line ends the command with ; stim doctor lists the rest.... A person who copies the whole sentence tail to a shell runs stim settings set ...; stim doctor lists .... Put the command last or on its own line.

  5. Note. A permanently unreadable scan path, a permanent partial du or a permanently unreadable layout means complete is never true on that machine, so the "every checkout it found is managed" sentence never prints and the low-disk line always says "at least". That is truthful and documented as a known limit; no change asked.

Item results

  • Single value: every consumer uses report.complete (sizeText, the empty-list sentence, lowDiskLine, JSON). I could not construct an exact figure, the complete-list sentence or discovery.complete === true with a truncated or failed scan, unreadable scan paths, a non-current or pending census, changed roots, an unchecked group, or a failed, partial or not-measured checkout. The one gap is finding 2 (stale left-over entry reads as complete, dated). Reverse nuisance: only the permanent states in finding 5.
  • Removed filter wording: the home group gets "home folder or holds it" with no command (adoptable false before named); an already-named group gets no command; lowDiskLine picks only a reduce-only group with a command and bytes, so never a dot-folder, already-named, too-broad or unchecked group. Wording is truthful except in the window of finding 1.
  • Over-reporting: yes, in the window of finding 1 (and after project registration). The low-disk line is silent then; doctor is not explanatory.
  • Documents: the claims that go beyond the code are findings 1 (known limits "says so each time") and 2 (an hour). The website sentence "For each folder doctor prints ... the command" omits the home, already-named and unchecked cases (minor).
  • Spot checks: cancellation order (isUnverifiedCleanup before the abort check), the per-command hook (Stim-home files only, canonical: null, stderr only) and no delete, reduce, register or set all hold.
  • Tests in the last push: they follow AGENTS.md (observable report output, expectations independent of the code). The gap is the missing text assertion for finding 1.

Human review: whether doctor should keep listing a folder the person just adopted (marked as older than the setting) or hide it until the daemon lists again; the issue does not say.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli packages/stim-cli, including its README and guide area: core packages/core area: website website/ review: needs-human Holds merge until a maintainer reviews the lines named in the Human review: comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant