Repository navigation
feat: discover tool folders and scan the home folder for checkouts - #5764
Conversation
…duce-only checkouts
…can never end a maintenance pass
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head 1153971, read only (nothing built or run; the lane's uncommitted dot-folder edits in the worktree are covered only in the last section).
No blocking finding on deletion safety. 1 to 9 are NON-BLOCKING; 10 and 11 are for the maintainer and the dot-folder change.
What I checked on reduce-only (verified by reading)
- The only automated callers that remove a worktree are
removeWorktrees(viaremoveWorktreeTarget,commands/gc/worktrees.ts:1157) andact.ts removeWorktree, which callsremoveWorktrees.reduceOnlyis honoured at the plan step (worktrees.ts:981-996) and in the lock guard (discoveredHold,:1088), and the flag travels on the candidate. Orphan pruning acts on state directories, and doctor and parity only remove their own worktrees. The dormant stage, ladder rungs, artifacts and cargo-target consumediscoverCheckoutsbut only clear dependencies and outputs, whichregenerableDirProblemgates on git tracking and ignoring. - The hold returns before the idle evidence, the same position as the
.claude/worktreeshold, so it matches that rule. A reduce-only checkout loses build outputs, tagged caches and, in a linked worktree,node_modulesand Pods. It does not lose the worktree, uncommitted files or other ignored files. - Precedence:
selectDiscoveredputsworktreesbeforechildrenandchildrenbeforescan. AdiscoverRootsworktree is full-rule becauseworktreeInScopechecks roots first. A scan path covered by a registered root is dropped.reduceOnlycan only over-apply: inworktrees.ts:924, any reduce-only root marks the whole worktree. - A scoped STIM_HOME gets neither source:
repositoriesis empty when scoped,scannedis[], andrunDiscoveryScanIfDuereturns early.
Findings
- NON-BLOCKING,
stimd/passes/run.ts:420. The scan runs inside the held pass claim, beforedue(), and delays that pass by up to 2 minutes (BUDGET_MS). That is longest on the low-disk rise tick, when the pass is wanted, and #5531 is about discovery slowing maintenance.lstat(path)in the walk (discovery-scan.ts~line 119) has no timeout, so a hung mount can exceed the budget (the deadline is only checked between folders). Run it after the checks, or as its own step. - NON-BLOCKING,
discovery-scan.tsprocessMemory. The state lives in thepassesworker, whichthreads.ts(IDLE_MS2 minutes,lane ??=) retires and recreates, as the #5531 comments describe.wasLowresets to false, soroseis true again after each replacement during sustained low disk. That makes the scan hourly, not once per rise. The reverse also happens: a rise within an hour of the weekly scan is swallowed, and the next scan is 7 days away. Persist the last level, for example in the scan file or maintenance state. - NON-BLOCKING,
discovery.tsscanned. A cached path is checked only withexistsSync, neverrecognises, and results can be 7 days old (longer after a failed scan, which stampsranAtwith the old list). A folder that stopped being a project stays reduce-only. Deletion is still guarded by the git-ignore checks, so impact is low; re-runningrecogniseson about 100 paths is cheap. - NON-BLOCKING,
discovery.tsboundReduceOnly. When a tool-folder worktree falls outside the window, it is dropped fromfoundand fromoutside. If its repository is registered,candidateRootsAsyncstill lists it as a linked worktree, so gc reportsnot-registered: no discovery source names it(worktrees.ts:324), which is wrong. At more than 40 entries,gc --dry-runoutput changes every 15 minutes and the guide never says the rest are unlisted. Today there are about 29, so it does not trigger yet. - NON-BLOCKING, cost (#5531). Each added checkout costs a source-tree write scan, 3 git calls and a session read per dormant pass (sequential per #5531), plus a worktree classification per sweep. Per-pass cost with the scan file present is not measured (the PR says so). Please attach discovery, per-checkout and event-loop-delay numbers; the lane already logs
eventLoopDelay. The scan itself runs in the passes worker, not the stimd main loop (threads.ts:242), so RPC is not blocked. The synchronousrecognisescalls (6,300 folders) block only that worker, in small chunks between awaits. - NON-BLOCKING, TCC and timeouts. The first
readdirof~/Documentsis raced at 3 s, which is shorter than a human answering the prompt. The first scan records Documents as unreadable and misses it for a week (or an hour under low disk). The abandonedreaddiralso holds a libuv thread. A truncated or failed scan replacesfound, so checkouts drop in and out under load. InrunDiscoveryScanIfDue, an abort returns instead of rethrowing, contrary to its docblock, and the consumedroseis lost. - NON-BLOCKING,
inToolFolder. It compares canonical paths. If~/.codexis a symlink to another volume, its worktrees fall tooutsideand are never reduced. That fails safe, but it is untested and undocumented. - NON-BLOCKING, tests.
- HEAD's
discovered-reduce-only.test.ts:27-30mocksisProtectedLocationto always false, although the PR body says the real rule is used. The working-tree copy now usesisHomeDotFolder, which is better. - Not covered: a window-excluded worktree in the sweep, a scan-found linked worktree through plan and removal, a corrupt or old-format scan file, a stale entry, a reduce-only worktree reduced by the dormant stage, and a scoped tool folder.
computeDiscoveredcallsDate.now()directly, so the window cannot be tested throughdiscoverCheckoutswith a fake clock.boundReduceOnlytakesnow, which is fine.runDiscoveryScanIfDuetests backdate the file instead of using fake timers.
- HEAD's
- NON-BLOCKING, docs.
guide/cleanup.ts(~line 31) says "weekly, or hourly at low disk", but the code, the LLP and the other guide paragraph say once per rise, at most hourly. The LLP quotes the maintainer decision verbatim, matching the #5762 comment, and labels the coordinator's reading and the lane's choices separately. The api report change (reduceOnly, originscan) matches the exportedDiscoveredCheckoutandDiscoveryOrigin.
Product decisions the maintainer did not state, beyond the PR body's list
- Scan-found main checkouts (for example in
~/Documents) losenode_modulesand Pods afteridle.sourceDormantDays(7). Codex worktrees lose outputs and dependencies afteridle.dormantDays(1 day). "In use" there means only a Codex thread updated in the last 30 minutes (agent-sessions.ts), so an older but resumable thread counts as idle. For an unregistered checkout, the "restore with" text assumes a later Stim command runs there. - A failed scan suppresses retry for a week. Truncation replaces the previous result. The scan is skipped entirely for scoped homes.
- The coordinator's reading is that reduce-only means never removed. That is a changed default for scan finds even though the maintainer said "existing safer rules".
What the lane's dot-folder change must not break
- Keep scan results reduce-only, and keep the
isProtectedLocationrelaxation limited to scan origin.worktreeInScopeand.other/wtstayingoutsidemust not change. - Exclude
$STIM_HOMEand anyserver.workerRootoutside it, otherwise the build worker's client checkouts become reduce-only (CLAUDE.md, Build access). The uncommittedHOME_DOT_PRUNEDis a blacklist and lacks.stim-agentand.stim-release. Prefer an allowlist, or a.stim*prefix rule. - Dot-folders share the 200k-folder and 2-minute budget with Documents. One expensive tree truncates the walk and overwrites the previous
found. Walk non-dot folders first, or cap each subtree. - The new
skipdoes two sync realpaths per folder. Measure it with the 6,300-folder run. - The HEAD test asserts that dot-folders are skipped; update it deliberately.
… scan after the pass and window reduce-only checkouts without hiding them
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head 0f08652, read only (nothing built or run). No BLOCKING finding. All findings below are NON-BLOCKING.
Checked and found sound (verified by reading)
- Protection relaxation (
discovery.ts:344,protected-locations.ts):origin === 'scan'is created in one place,revalidatedScan(discovery.ts~line 290).sibling,rootandworktreeentries still go throughisProtectedLocation/worktreeInScopeunchanged. InselectDiscoveredworktrees andchildrencome before scan entries, so a path that is also a sibling or root child keeps its origin. Everyscanentry getsreduceOnly: truewith no exception (selectDiscovered); the sweep marks the candidate reduce-only (worktrees.ts:924) anddiscoveredRemovalHoldreturns before any idle evidence (discovered-removal.ts:118).deferredentries stay infound, so they stay inreduceOnlyPaths. scanEntryValid: rejects outside home, home itself, symlink, non-directory, canonical path different from the stored one (so a symlinked ancestor fails), other volume, the Stim home and the build worker root (skip), more than 7 levels, any pruned or skipped component (home dot-folders.codex,.claude,.config,.stim*,*-cache, any dot-folder below level 1,node_modules,Library,buildand so on), and anythingrecognisesno longer accepts./, another user's home, a path in$STIM_HOMEand a path inside a pruned folder all fail. Entries are re-read from the file on every discovery, so a stale or hand-edited entry is dropped, not trusted.- Deletion for a scan find is still gated by
regenerableDirProblem: a non-git folder returns "git cannot tell", so build outputs andnode_modulesof a non-git installed tool are kept. A process running from the folder keeps it (runningFromReason). A worktree is never removed. - Merge (
mergeFound) and persistence: a stale find that stays in the file is harmless because of revalidation. A failed scan stampsranAtand keepscompletedAt, so it retries hourly.scopedhomes and mode off skip the scan. Two processes can scan at once; the write is under a dir lock and the last writer wins. - Maintainer decision in the LLP matches the #5762 comment verbatim. Coordinator readings and the lane's choices are labelled separately.
Findings
-
NON-BLOCKING,
scan-rules.ts:128.realpathSync(JS) does not normalise case on macOS's case-insensitive volume. A hand-edited or corrupt scan file entry such as~/.CLAUDE/x,~/LIBRARY/xor~/.STIM/home/apppassesscanDescends,skipand the canonical check, because all of them compare strings, and is then reduced. It needs a recognised project at that path and a bad file under$STIM_HOME, so the risk is low.realpathSync.nativefor the canonical comparison closes it. -
NON-BLOCKING,
stimd/passes/run.ts:742. Theawait runDiscoveryScanIfDue(tick)sits in thefinally. (a) Its abort rethrow replaces whatever thetryorcatchwas already throwing, such asthrowUnverifiedCleanup's error. (b) When that catch rethrows, a scan of up to 45 s still runs before the error propagates. (c)loadConfig()anddiskThresholds(...)atdiscovery-scan.ts:166are outside itstry, so a throw there masks the pass's own result too. Suggest: skip the scan when an error is in flight, and put lines 166-168 inside thetry. -
NON-BLOCKING,
scan-rules.tsHOME_DOT_PRUNED. Not pruned and likely large or tool-owned:.volta,.asdf,.mise,.fnm,.pyenv,.rbenv,.sdkman,.deno,.pub-cache,.fvm,.oh-my-zsh,.maestro,.fastlane. I found no deletion harm: a scan find is reduced only if git tracks nothing and ignores everything in the folder to be cleared, and a tool install that is not a git clone is kept. The residual harm case is a git-cloned tool in an unlisted dot-folder with an Expo, RN, Xcode or Gradle project at its root and untouched source for 7 days: it losesnode_modulesand the tool breaks until reinstall. Unmeasured cost: the walk shares one 200k-folder, 45 s budget with~/Documents, in readdir order. A tree like.asdf/installscan truncate it every time, so later folders are never reached, andmergeFoundkeeps the old finds. Suggest walking non-dot folders first, or capping each subtree. -
NON-BLOCKING, docs.
llp/0004...:45says "The one other exception is~/.codex/worktrees", andguide/cleanup.tsandwebsite/docs/worktrees.mdsay nothing under a home dot-folder is touched except discover roots and~/.codex/worktrees. The same pages now say the scan enters~/.exactand other unlisted dot-folders. A linked worktree of a known repository under an unpruned home dot-folder, such as another agent tool's worktree folder, used to be listed asoutside. It is now found by the scan and reduced (build outputs,node_modules, Pods after 1 idle day). That is consistent with the decision text but contradicts those sentences, and the test title "no other home dot-folder worktree" only holds with no scan file. Reword the exceptions and say it in the LLP. -
NON-BLOCKING, tests.
scanEntryValidhas no direct test. Missing cases: a pruned dot-folder,.stim*, the Stim home and worker root, depth over 7, outside home, home itself, a hand-edited case variant. Only the symlink case is covered.discovered-reduce-only.test.ts:31mocksisTemporaryLocation: () => false, so the scan-origin temporary rule atdiscovery.ts:344is not exercised by any test. The dot-folder part is, via the real rule with the home stubbed.discovery-scan.test.tsscanDescendsasserts the prune list by name (.npm,.gradleand so on), which duplicates the constant; only the.stim*and*-cacherules and the depth rule are logic.scanForCheckoutsalready covers.npmby behaviour.runDiscoveryScanIfDuetests depend on module-levelattemptedHerecarrying across tests; they pass in file order.- The two new files stub
STIM_HOMEto''andHOMEto a temp dir. State lands in the temp dir and is removed; nothing reads the real home.
-
NON-BLOCKING, cost (not measured, as the PR says).
revalidatedScanrunsrecognisessynchronously for every stored find on eachdiscoverCheckoutscall. The window bounds only the acting consumers:collectWorktreeSweepstill runs git calls for every found checkout, deferred ones included. Worth attaching numbers for a home with a few hundred finds. -
NON-BLOCKING, product (maintainer's call, not a defect). Scan-found main checkouts lose
node_modulesand Pods afteridle.sourceDormantDays(7), and nothing marks an installed app as "used" except processes and source writes. The coordinator reading and the maintainer's "safer rules we have already" allow it; the LLP does not name this case.
Not checked: real behaviour of the walk on a loaded home, TCC prompts, the gc --dry-run output for deferred entries, and the API report beyond the diff.
…n, keep checkouts that PATH launchers run from, and keep a scan failure from masking a pass failure
…ool-folders-and-scan
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head 6a9fc02, read only (nothing built or run). One BLOCKING finding (the PATH launcher keep does not cover every route that deletes outputs, and the docs say it does). Everything else is NON-BLOCKING. No label added and not marked ready.
1. BLOCKING, workspace/running-from.ts:170 and workspace/discovered-activity.ts:144 against guide/cleanup.ts and website/docs/worktrees.md ("is kept whole") and LLP 0004 ("Discovery keeps a reduce-only checkout ... and so its outputs").
launcherReason is consulted only through discoveredEvidence, which only idleVerdict (dormant stage, daemon/stages.ts:165) and discoveredKeepReason (level-run.ts:256, a report annotation skipped when an action was already taken) reach. The routes that actually clear outputs of a discovered checkout do not read it:
collectArtifactsAsync(stimd/passes/artifacts.ts:397-470) iteratesartifactWorktreesAsync, which includes every non-deferred discovered checkout, reduce-only ones too.artifactGuardchecks pin, lock, agents,workspaceInUseand, for dependencies only,runningFromReason(processes andstimon PATH). No launcher check. It feeds the pressure ladder and the dependency passes indaemon/dependencies.ts:116,172.cargo-target.ts:107-114adds the same checkouts'target/withregenerableDirProblem(regenerable-dirs.ts:225-262), which checks git tracking/ignore and a running process, not launchers.
Scenario:~/Developer/tool(scan find, not a dot-folder),~/.local/bin/tool -> ~/Developer/tool/target/release/tool, tool not running. The dormant stage keeps it. At high or critical disk the Cargo or artifact route clears the ignoredtarget/orbuild/output and the installed command breaks until rebuilt. That is the exact case the keep was added for. The only test (discovered-reduce-only.test.ts:134) goes throughstoppedOutputCandidates.
Fix either way: passreduceOnlyand the launcher targets intoartifactGuardand the Cargo/regenerable check, or reword the guide, website and LLP to say the keep applies to the idle-based dormant reduction only, and say what other passes can still clear.
Checked and sound
- Dot-folder finds:
dotFolderFoundis read by nothing (grep ofdotFolderFound,readDiscoveryScan,discoveryScanFile: only the parser, the writer,revalidatedScanwhich readsfoundonly, and tests).protected-locations.tsdiffers from main by one comment line. Routes into a home dot-folder remaindiscoverRootsandTOOL_FOLDERS. - Classification:
scanDescendsadmits dot-names only at depth 1, so a find's first relative segment starting with.is exactly the dot-folder case. Symlinked entries are skipped (entry.isSymbolicLink()), canonical home is used. A hand-edited~/.exact/xinfoundpassesscanEntryValid(scanDescendsallows.exactat depth 1) and is dropped only byisProtectedLocationindiscovery.ts:344; case variants fail onrealpathSync.native. Tested (discovered-reduce-only.test.ts"never selects"). Single barrier: see 3. - Scan finds outside dot-folders are always
reduceOnly: true(selectDiscovered), held at plan and removal (worktrees.ts:924,:990,discovered-removal.ts:118). - After the pass:
run.ts:411-420,736,750,758.passFailedis set beforethrowUnverifiedCleanup, so the cleanup error is the one that propagates and no scan runs; abort skips it;scanAfterPassswallows anything else. Earlyreturns inside the try still reach it. It composes with #5763:deferCheck/recordedDiskLevel(run.ts:324,341,429) are untouched and the scan takes its level fromtick.measured. - Limits:
exhausted()runs before every readdir and every sibling stat, and rethrows abort;answeredtimes each read out at 3 s. - LLP 0004 quotes the maintainer decision verbatim as in the issue comment; the coordinator's points are labelled; the dot-folder decision is labelled the coordinator's. The guide and website match the code (hourly while above normal, retry after an hour).
NON-BLOCKING
2. running-from.ts:152-165. Fails open: an unreadable PATH directory gives [] (no launcher) and a broken link is skipped. Keep would be the safe direction; the process check fails closed ("cannot be listed"). Coverage: a launchd daemon's PATH is /usr/bin:/bin:/usr/sbin:/sbin plus the four extra directories, so npm link under nvm, ~/.bun/bin, ~/.cargo/bin, ~/Library/pnpm and ~/.volta/bin are not read. Copied (non-symlink) executables and wrapper scripts are not followed (documented). Say in the docs that the list is partial, or record the user's PATH.
3. scan-rules.ts:130. scanEntryValid accepts a depth-1 dot-folder path; the protection against it rests on isProtectedLocation alone. Reject it there too (defence in depth); the .exact case is then not reachable from a hand-edited file.
4. running-from.ts:152 with discovered-reduce-only.test.ts:134. With no operation the extras still read the real /usr/local/bin and /opt/homebrew/bin. Not the home folder or PATH, but the test is not hermetic. The scan-after-pass positive path (the scan does run after a successful pass) has no test; only the failure case does.
5. The scan is not deferred by #5763's host-load check; it can run up to 45 s plus synchronous recognises on a loaded host. Cost not measured (also raised in earlier rounds).
6. LLP 0004: the PATH keep is not listed under "Chosen by the implementing lane". It is not attributed to the maintainer either; add it to that list. The "Reason for entering home dot-folders" paragraph still argues the scan must enter ~/.exact to manage it, while the next paragraph says it is never acted on.
Not checked: real behaviour of the walk or the PATH read on a populated machine, TCC prompts, gc --dry-run output, the api report beyond the diff, CI.
When this is fixed, review: needs-human applies by the rule (changed defaults and user-visible behaviour the maintainer did not state: ~/.codex/worktrees default folder with its worktrees reduced after one idle day, scan finds reduce-only, dot-folder finds never reduced, the PATH keep, weekly/hourly schedule, limits and prune list, the 40-checkout window, source checkouts losing node_modules and Pods after 7 idle days, the scan running in the background daemon so the Documents prompt comes from it). I will list the path:lines in a Human review: comment at the next clear review.
…ckout a PATH launcher runs from, and an unreadable launcher directory keeps them all
…ct and dependency route
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: head b8df58d, read only (nothing built or run). Clear: no BLOCKING finding. The earlier BLOCKING defect (the PATH launcher keep applied only in the idle stage) is fixed. Findings below are NON-BLOCKING.
Routes enumerated from the code (every consumer of the discovered list and of reduceOnly)
- Idle-stage outputs (
stoppedOutputCandidatesthenclearStoppedOutputs, ladder rung 1,daemon/passes.ts):reduceOnlyis set on theStageCandidate(stages.tsreadings), passed toidleVerdictat selection and again under the metro-start lock. Covered. - Dependencies, linked worktree and source tree (
rankDependencies,rankSourceDependencies,clearDependency,clearSourceCheckouts, ladder rungs 4/5, dormant dependencies):collectArtifactsAsyncandartifactReadsread the launcher intokeptReason;clearArtifactandclearSourceTreesre-run it before and under the claims.reduceOnlyis rebuilt from the freshartifactWorktreesForlist by path (containing), not from the candidate, so it is not dropped. With no operation the list is registered-only and the target is kept. Covered. - Cargo target: measurement,
planCargoTargetsAtPressure,trimCargoTargetAction(viamanagedAs) all pass the launcher reader.planCargoTargetsuses only the measured and refused set. Covered. - Worktree removal,
gcsweep, orphan and stale-record pruning: a reduce-only checkout is held at plan and at removal (discoveredRemovalHold); never removed.gcworkspace outputs (lighten) touch only checkouts with a Stim workspace record;listedRoots()includes recorded roots, so such a checkout is "known" and is never discovered as reduce-only. Not a gap. level-run.tsonly annotates;act.tswould-clear-outputsclears state under$STIM_HOME, not the checkout.
Fail closed. launchers.ts: ENOENT and ENOTDIR on a directory or an entry, and a dangling symlink, mean absent; every other error (EACCES, EIO, ELOOP) marks the directory incomplete, and launcherReason then keeps every reduce-only checkout on all routes through the one function. Earlier-confirmed properties unchanged: dotFolderFound has no acting reader, protected-locations.ts differs only by a comment, scan finds are reduce-only, revalidation and scan-after-pass code are untouched since the last review; the new maintenance-run test covers the scan running after a successful pass. The maintainer quote in LLP 0004 matches the issue comment word for word; coordinator and lane choices are labelled separately and the PATH keep is in the lane list.
NON-BLOCKING
launchers.ts:43-62, LLP 0004 and the guide say a "timeout" keeps every checkout. No timeout exists:readdir,lstatandrealpathhave none, and the abort signal is checked only between directories. A stalled mount on PATH hangs the pass on its first reduce-only checkout, including at critical disk. Safe for deletion, but the docs promise something the code does not do. Wrap each directory read in thewithin-style race and record a timeout as incomplete, or drop "a timeout" from the text.launchers.ts:68-80. In a daemon pass the reading is one snapshot per operation, so "again under the locks" re-runs the check on the same reading.discovered-reduce-only.test.ts"at the check under the lock" passes only becauseclearArtifactis called without an operation, which is not the production shape. A link made during a long pass is missed. Low risk (needs a link into a checkout idle for days during one pass); a fresh read under the claim is cheap. At minimum say "per pass" in LLP 0004.launchers.ts:72. With no operation,launcherDirectories({})reads no PATH at all, only the home and Homebrew folders. Useprocess.envas the fallback, aslevel-run.tsdoes.- An unreadable PATH directory (for example one under a TCC-protected folder) disables reduction of every scan-found checkout until fixed; the reason names the directory. Fine as fail-closed; doctor should list it later.
Not checked: real behaviour on a populated PATH, hung mounts, TCC prompts, CI beyond the status list (green apart from skipped lanes).
|
Human review: product decisions the maintainer did not state in the #5762 decision. The decision covers discovery as current logic plus common paths plus a periodic full scan (more often on low disk) that prunes and reads
|
…gainst its checkout, time out hung launcher directories and read the process PATH without an operation
janicduplessis
left a comment
There was a problem hiding this comment.
AI review at 60fdb10: clear. No blocking defects. Fresh reviewer, did not write the change.
Checked, deletion safety first:
- Fresh reading before each deletion. Every check that immediately precedes a deletion of a reduce-only checkout reads fresh:
clearArtifactearly check and under the lock (artifacts.ts:638, 660),clearSourceTrees/checkTree(artifacts.ts:773),clearStoppedOutputs(stages.ts:348),trimCargoTargetAction(cargo-target.ts:439). The remaininglaunchersOfuses are planning only (cargo-target.ts:159, 370; level-run, idle-stop and gc planning reads). Reduce-only checkouts are never removed whole (discoveredRemovalHold). The 5-second cache is aWeakMapkeyed by theStimContext, so one operation cannot reuse another's reading; without an operation every call reads anew. An incomplete reading is cached as incomplete andlauncherReasonkeeps on it, so it is never returned as complete. A cached rejection (abort) fails the caller. The cache keys on the reading's start time, so a 5-second-old reading is the longest window a deletion check can rely on. I accept that window: the check holds the lock and the next deletion starts within 5 s of a read that postdates planning. - Nested Cargo.
checkoutis set fromcheckout.pathofartifactCheckoutsAsyncfor every reduce-only route (registered roots are never reduce-only), andlauncherReasoncanonicalizes it. The regression places the launcher outside the app but inside the checkout, so it fails against an app-root check. When the app root is the checkout,checkoutequalsroot. A nested git repository is its own checkout entry, and a launcher into the outer repo outside it is not inside it, the same as for other routes. - Timeout.
answeredclears its timer infinally, and the losingreadhas a handler fromPromise.race, so no dangling timer or unhandled rejection. A timeout is incomplete and keeps. process.envdefault plus the.oxlintrc.jsonentry is an allowlist for a rule, not a suppression, and matchesconfig.tsandgit-read.tsbeside it. Acceptable.- No regression found in what the commit touches.
protected-locations.tsdiffers from main by a comment only. - Tests: the under-lock regression passes one
StimContextto the plan and toclearArtifact, adds the launcher after planning and would fail on the planning memo. Timeout test uses the global fake timers. No test reads the real PATH (the launcher directories are mocked). - LLP 0004 matches the issue comment's quote verbatim; the coordinator's choices stay labelled as such.
NON-BLOCKING:
- launchers.ts:66-72, 115-129: each directory that hangs costs 3 s per fresh reading, so N slow directories cost up to 3N s, and a reading that takes longer than 5 s is never reused. It fails closed (keeps). Each hung readdir also holds a libuv thread until the mount answers; repeated fresh reads on a dead mount can occupy all 4 threads and stall unrelated fs calls in the daemon. Unchanged in kind from before (a hung read already blocked), but the timeout hides it.
- No test covers the 5-second reuse window or its per-operation key.
- website/docs/worktrees.md:89 says an unreadable folder keeps the checkouts; it does not mention the 3-second answer limit or the read before each deletion (the guide and LLP do).
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: clear at 1c22881. No finding.
Read the whole diff, deletion safety first, then 1c22881 line by line. No code ran (read-only).
Reads in discovery-scan.ts and scan-rules.ts:
| Read | ENOENT/ENOTDIR | Timeout | Other error (EIO, ESTALE, EACCES) | Abort |
|---|---|---|---|---|
home lstat |
throws, so a failed attempt is recorded and the previous finds stay | unreadable + truncated read, finds kept |
home recorded unreadable, so every earlier find is under it and kept | stopAt rethrows |
folder readdir |
folder gone, nothing recorded | unreadable + truncated, stops | unreadable, walk goes on, finds in and under it kept | rethrown |
child lstat |
child gone, nothing recorded | unreadable, ends walk | unreadable, continue |
rethrown |
gate manifest read (folderSafety) |
denied, so unreadable |
unreadable + truncated |
denied, so unreadable, kept |
exhausted() throws before each read, checked again after |
gate listing (nestedSafety) |
container: denied; child: skipped |
unreadable |
denied |
same |
revalidation metadata (lstat/stat/realpath) |
invalid |
pending |
invalid (excluded this time; no write, the file keeps the entry) |
none (no signal) |
| revalidation gate | pending |
pending |
pending |
none |
| scan file read/write | n/a | n/a | see below | n/a |
After this commit absent() is the only place that names error codes in the scan and the gate. No error is swallowed into a normal result. In scanForCheckouts no catch yields "safe" or "valid". The only drop of earlier finds in a complete scan is a folder that is absent. Revalidation is read-only and fails closed.
Vanished folder (item 2): ENOENT/ENOTDIR can only come from a path that a successful listing of its parent just showed. The walk never follows symlinks and the home path is canonical, so a path through a file or a dangling symlinked ancestor does not occur. A folder that has become a mount point of another volume is skipped by device id with nothing recorded. Finds that were beneath it are dropped in a complete scan. scanEntryValid already returns invalid for them (dev !== info.dev), so discovery would not have used them. An unmounted volume leaves an empty directory whose listing succeeds, and its finds go, correctly. Same as before this commit.
The three catches in runDiscoveryScanIfDue (item 3) are harmless. A failed config, threshold or scan-file read returns without scanning and changes nothing. A failed write of the failed record keeps the in-process one-hour throttle (attempted.at) and the earlier file. Temporary-file cleanup runs after the rename.
Scope (item 4): the commit touches discovery-scan.ts, scan-rules.ts (export of absent) and the scan test. protected-locations.ts still differs from main by a comment. No cleanup route changed.
Tests (item 5): the EIO listing, ESTALE lstat and ENOENT lstat cases assert on the stored file through runDiscoveryScanIfDue with injected entries and stat only. ENOENT yields no unreadable record and drops the find.
Docs (item 6): the 0004 section and the guide still read accurately. "denies" understates "any failure that is not absent" but is not false. The Maintainer decision block in LLP 0004 matches the issue comment word for word. Coordinator and lane choices are labelled as theirs.
Notes, not findings:
- A non-absent failure on the home
lstatstores a complete scan (completedAt= now) with the previous finds kept. The next scan is then due in a week, not an hour. Nothing is dropped. - There is no test for the home
lstatfailing with a non-absent code.
|
Human review: lines that carry choices the maintainer has not stated, at commit 1c22881.
|
…t for a later pass
janicduplessis
left a comment
There was a problem hiding this comment.
AI review of f3e3d27: one finding on deletion safety.
BLOCKING: packages/stim-cli/src/commands/gc/worktrees.ts:845-849 can remove a reduce-only worktree that was held before.
The filter drops deferred roots before grouping. Sweep candidates are grouped by the worktree entry that matchWorktreeEntry finds, and a path prefix is enough to match (worktree.ts:883). candidate.path is the entry path, not a root, and reduceOnly is set only when some root in the group is reduce-only (line 936).
Scenario:
- The user sets
maintenance.discoverRootsto a folder inside a Codex worktree, for example~/.codex/worktrees/aa11/main/packages. - The worktree
~/.codex/worktrees/aa11/mainis not inside that root, so discovery makes it a reduce-only tool-folder entry. - A recognised child
packages/appis discovered as a non-reduce-onlyrootentry. Both roots match the same worktree entry. - Before this commit, the group held both roots,
reduceOnlywas true, anddiscoveredRemovalHoldreturned the reduce-only hold at plan time and under the locks. - Now the worktree root is deferred and dropped. The group holds only
packages/app, soreduceOnlyis false,candidate.pathis still~/.codex/worktrees/aa11/main, and the worktree is judged by the full discovered rules (merged, idle, dormant). That can remove it.
This is narrow, because it needs discoverRoots inside a tool-folder worktree. It still changes what is removed, and only for deferred entries, which is the case the commit's claim ("leaving one out cannot change what is removed") rules out.
Suggested fix: apply the deferred filter after grouping, and drop a group only when every root in it is deferred. A mixed group keeps all its roots and its reduceOnly flag. Add a regression with a nested non-reduce-only root.
Verified, no finding
- Deferred is set only on
reduceOnlyentries (discovery.ts:197). Discovered paths are canonical, andselectDiscoveredexcludes any path a known root covers. A non-reduce-only checkout is never dropped by path collision, because the filter canonicalizes every root. A linked worktree of a registered repo sitting under a tool folder was already classed reduce-only (discoveredOnly), so the classification does not change. skipDeferredis set only byplannedMaintenancewhenbackgroundis true.stim gc, its dry run andworktreesRungdo not set it. The docs statement about the ladder rung is true.- Nothing downstream treats "absent from the sweep" as "gone".
finishedWorkspacesexcludesrecent-activityholds. Dropping a deferred entry delays its finished-device action to a later window, which matches the deferral. NON-BLOCKING note. - The test asserts on recorded git
statuscalls and the listing with and without the option. It does not cover the nested-root group above. - LLP 0004, the guide and the website text on the 40-checkout window match the code.
Not done: other items in the 4 to 6 list beyond the f3e3d27 diff were not re-reviewed in full this round. No label added, not marked ready.
…kout left for a later pass" This reverts commit f3e3d27.
…hat a nested discover root cannot release a held tool-folder worktree
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: clear at cc647e1. No findings.
Fresh reviewer, did not implement this change.
Diff 1c22881..cc647e1 (verified myself): llp/0004-discovered-checkouts.decision.md, packages/stim-cli/src/guide/cleanup.ts (guide text inside a template string), website/docs/worktrees.md, and discovered-reduce-only.test.ts. No other production code differs.
- Revert complete.
git diff 1c22881c8 cc647e11d -- packages/stim-cli/src/commands/gc/worktrees.tsis empty;git grep skipDeferredfinds nothing;stimd/passes/preview.tsandsweep.tsare not in the diff (4b5265a restored them). - Window statements checked against code. The passes that skip
deferredcheckouts areartifactCheckoutsAsync(artifacts.ts:149),idleWorkspaces(daemon/stages.ts:247) andcargoTargetCandidates(cargo-target.ts:124);level-run.ts:252marks them "left for a later pass" forgc --dry-run.collectWorktreeSweep(worktrees.ts:835-837) callsdiscoverCheckoutswith no deferred filter and inspects every group with git, so it is unbounded. Grouping is by the worktree path (groups.set(entry.path, ...)), andreduceOnlyis set whenroots.some(...)is reduce-only (worktrees.ts:924). The LLP's account of the reverted skip matches. The website and guide wording is consistent with this. - New test (discovered-reduce-only.test.ts:542) runs the real
discoverCheckoutsandcollectWorktreeSweep, asserts both roots are discovered with the right reduce-only flags, and that the codex worktree isreduceOnly: truewith a "never removes" skip. It would fail for aneveryor last-root decision. Note only: roots sort with the worktree root first, so a first-root-only decision would still pass. - Final deletion-safety pass: production code equals the head the previous reviewer cleared; CI shows no failing check;
protected-locations.tsdiffers from main by one comment sentence. - LLP 0004 line 249-254 matches the issue's 2026-11-14 maintainer decision verbatim; coordinator and lane choices are labelled as such.
Not run: tests, builds (read-only worktree).
janicduplessis
left a comment
There was a problem hiding this comment.
AI review: clear at 938fee2. No findings.
Checked:
- Window gone: no
windowReduceOnly,REDUCE_ONLY_WINDOW_MS,MAX_REDUCE_ONLY,deferred(discovery sense) or "left for a later pass" in source, tests,api.api.txt, guide or website.discoverCheckoutsreturns the complete list; thenowparameter is removed and no caller passes it. LLP 0004 mentions the window only in the sentence recording that one was built and removed, and in the unrelated "window for a process that starts after the last evidence read". - Reduce-only classification is made from the complete list and does not depend on root order.
artifactCheckoutsAsynckeepsreduceOnly: truewhichever of the worktree root or the nested child is visited first (the map write keeps true if the previous entry was true, and a later true entry replaces a false one).collectWorktreeSweepgroups by worktree and usesroots.some. Cargo candidates takecheckout.reduceOnlyfrom that artifact entry. The idle stage and level-run report per discovered path; the nested child is a separate candidate with full rules, as the person asked, and nothing belonging to the worktree as a whole (root node_modules, Cargo target, the worktree itself) loses reduce-only on any route. - Regressions: the removal-hold test uses the real
discoverCheckoutsandcollectWorktreeSweep; the artifact/Cargo test runsartifactCheckoutsAsync,collectArtifactsAsyncandplanCargoTargetsAtPressure, and would fail if the worktree were classified not reduce-only (the PATH launcher keep would not apply and node_modules and target would stay candidates). - Scan file:
readDiscoveryScanFileseparates absent (ENOENT), unreadable (any other error) and read.runDiscoveryScanIfDuereturns before any scan or write on unreadable; revalidation usesreadDiscoveryScan, which returns null on unreadable, so no scan finds are selected that tick. The test checks no scan starts and the file bytes are unchanged. Note (non-blocking, documented): a file that reads but does not parse counts as no previous result. - Deletion safety over the diff against main: reduce-only hold at plan and under removal locks (
discoveredRemovalHold), requiredreduceOnlyargument ondiscoveredEvidence, launcher evidence read beside process evidence on idle, artifact and Cargo routes and failing closed,dotFolderFoundhas no acting reader,protected-locations.tsdiffers from main by a comment only, the scan merges earlier finds and is skipped after a failed or aborted pass. - Docs: LLP 0004, the cleanup guide and
website/docs/worktrees.mdstate no limit that no longer exists; the maintainer decision in LLP 0004 matches the issue comment word for word, and the coordinator and lane choices are not attributed to the maintainer.
Not run: tests, build and lint (read-only review).
|
Coordinator note on the last Codex pass at 938fee2. It reported one finding, which is recorded here and not acted on in this pull request:
This is the documented rule, not a gap: a folder named in Whether the PATH launcher keep should apply to every discovered checkout, not only reduce-only ones, is a separate product question and is filed as its own issue. |
Description
Stim learns about a checkout when a stim command runs in it, when it sits beside a known project, or when its folder is named in
maintenance.discoverRoots. Build output anywhere else is invisible to status, gc and the pressure ladder. On 2026-11-14 that was 80 GB under a home dot-folder (#5757), and~/.codex/worktreeson the same Mac holds month-old worktrees Stim lists as "outside the folders Stim manages" and never touches.The maintainer decided (verbatim on #5762, and in LLP 0004): discovery is today's logic plus common paths plus a full scan once in a while, more often on low disk; the scan prunes to stay fast and reads
~/Documents; what is found is cleaned with the existing discovered-checkout rules; doctor lists it. This change is the discovery half. The doctor listing and the low-disk command line are a separate change.Solution
Reduce-only checkouts.
discoveredRemovalHoldalready keeps a worktree that lives in a dot-folder of its main checkout ("managed by another tool": outputs reduced, worktree never removed). Checkouts found through a tool folder or the scan carryreduceOnlyand take that same hold, at the plan step and again under the removal locks. No new deletion path. A folder named inmaintenance.discoverRootsis checked first and keeps the full discovered rules.Tool folders. Linked worktrees of known repositories in
~/.codex/worktreesare discovered by default. The list is one constant with that one entry.The scan. A pruned walk below the home folder looks for folders Stim's existing project detection recognises. It stops at a recognised checkout, at 7 levels, 200,000 folders or 45 seconds. The limits are checked before every read the scan makes, the project checks included, so it can overrun the budget by at most one read's 3-second timeout plus the time detection takes on the one folder in hand. It never follows a symlink or leaves the home volume, and records what it could not read instead of failing. It does not enter dependency, build-output or tool-cache folders. It skips
Library,Applications,Desktop,Downloads,Movies,Music,PicturesandPublic, and readsDocuments.Recognition cannot block the scan. Project detection reads manifests synchronously, and the scan reaches folders where such a read may never return, for example a manifest in an iCloud-synced
Documentsthat macOS has evicted. The walk checks every folder once on the way down from the home folder: each file detection would read or stat there must be a regular file in the listing the scan holds, names compared without regard to case because detection's own lookups fold case on a case-insensitive volume; aniosorandroidentry must be a real directory; and a regularpackage.jsonis read once through the bounded reader. A folder that fails is not entered, so nothing below it is recognised, and detection only runs on a folder whose every ancestor below home passed, with no condition that switches the check on or off. Theapps,examplesandpackagescontainers and the children detection inspects in them get the same check, and a container that fails it makes the folder's subtree unrecognisable too. The recheck of a cached find walks the same chain from home down, memoised per discovery call, and reads the entry's type and canonical path through the bounded reader too. A read that times out records the folder as unreadable and cuts the scan short, or leaves a cached find unselected for that call with the file untouched. A folder with an Xcode project or workspace is not a scan find unless itspackage.jsondepends on React Native or Expo, because detecting a native Xcode project follows paths named inside project files; sibling and root discovery still recognise it.Every failure in the check fails closed, and the kinds are kept apart: a structural fact (a special file where a manifest should be, a symlinked container) means the folder is not a checkout; a read the system denies records the folder as unreadable, keeps the earlier finds in and under it and lets the scan go on; a read that times out does the same and cuts the scan short; and an entry counts as absent only when the listing that would show it was read. An abort is checked after every result of the check and again before the result is stored.
Bounded: the reads detection makes when the scan or the recheck decides whether a folder is a checkout. Not bounded: the ancestors above the home folder; an
iosorandroiddirectory that is itself a stalled mount point, which detection resolves without the walk entering it; a file that becomes unreadable between the bounded read and detection; and everything a pass later reads synchronously inside a checkout once it is discovered, which is the same for registered and sibling-discovered checkouts today.Home dot-folders are read and never acted on. The scan enters dot-folders directly in the home folder, except a named list of tool homes, version managers and caches (in LLP 0004, with the entry counts behind it), anything named
.stim*or*-cache, the Stim home and the build worker root. What it finds there goes into a separate list in the result that discovery never reads. Stim cannot tell an installed tool from a checkout in those folders: a Swift command-line tool cloned to~/.mytoolwith a launcher pointing into its.build/releaseis recognised as a project, and clearing its build output would break the command.isProtectedLocationis unchanged, and the ways into a home dot-folder staymaintenance.discoverRootsand the tool-folder list. The separate list exists so the doctor and low-disk change can report what was found, its size and the command that adds it.A checkout a command on PATH runs from is kept. For reduce-only checkouts, an executable whose symlink resolves into the checkout keeps the whole checkout on every route that can delete from it: the idle stage when it selects and again under the locks before it deletes; the artifact and dependency passes through their one shared keep check, before clearing and under the lock; and the Cargo target passes at measurement, in the low-disk plan and again immediately before the trim. The directories read are PATH plus the per-user ones a launchd-started daemon does not have (
~/.local/bin,~/bin,/usr/local/bin,/opt/homebrew/bin,~/.cargo/bin,~/.bun/bin,~/.volta/bin,~/Library/pnpm, the active nvm bin). A directory that does not exist means no launcher there; one that could not be read keeps every reduce-only checkout for that operation, naming the directory. Launcher evidence is read at the same moments and by the same code as running-process evidence: planning uses one reading per operation, and each route reads again where it last reads the process list before deleting. In the Cargo trim that is everyreadTargetUse(at the start, before the unit sweep and before the incremental data), where launcher targets under the target join the paths in use, so a profile holding a PATH-linked executable is kept like one a process runs from.clearArtifactandclearSourceTreesread it after their process read, early and under the lock; the idle stage reads it after its process read and once more after it has measured sizes, immediately before each removal. The window that remains is the one Stim already accepts for a process that starts executing from a path after the last evidence read. Whether a checkout is reduce-only, and the launcher reader that goes with it, is a required argument with no default on every function of the evidence path, so a caller cannot omit it; an earlier head had one deletion recheck that silently used the default. Each directory read has a 3-second timeout. Launcher reads and the scan share one guard: a read that times out ends that reading or scan as incomplete, and until it settles no launcher read and no scan starts another, so a hung mount holds one filesystem thread per JavaScript isolate (up to two in the daemon: the main thread and the passes thread) and reduction of reduce-only checkouts stays off meanwhile. For a Cargo target nested in a monorepo the launcher is checked against the owning checkout, not the app directory. A wrapper script that only names a path is not followed.The cached result is rechecked before use. The scan writes
maintenance/discovery-scan.json(type and reader in core); discovery reads that file and never walks. Before a cached entry is used it must still be a real directory at the same canonical path, on the home volume, under the home folder, outside every skipped folder, and still recognised. A checkout that was moved and left a symlink behind is dropped. A scan cut short or failed keeps the previous finds; a complete one keeps previous finds under folders it could not read.Schedule. After a maintenance pass has released its claim, never before its cleanup, and not at all when the pass failed or was aborted, so nothing the scan throws can replace a pass's own failure: weekly, hourly while a volume is above normal, and never twice within an hour. The result records how the last attempt ended: one cut short by a failed or timed-out read is retried after an hour whatever the disk level; one that stopped at the folder cap or the time budget counts as finished. The schedule is stored with the result, because the worker that runs passes retires when idle. Never in a scoped home.
No per-pass limit. Every reduce-only checkout is evaluated on every pass, as a sibling-discovered checkout is today, and whether a worktree is reduce-only is always decided from the complete discovery list. An earlier head limited the reduction passes to 40 such checkouts at a time; it was removed, because filtering entries out before classification changed what was protected in two places. The finished-worktree check inspects every discovered worktree with git, about 0.14 s each where it was measured; bounding the cost per pass belongs to #5531. Two regressions pin the grouping: a tool-folder worktree with a
maintenance.discoverRootsentry inside it is still held from removal, and its dependencies and Cargo target are still kept by a PATH launcher.Choices the maintainer did not state, for review: reduce-only for tool-folder and scan finds;
~/.codex/worktreesas the only tool folder; the weekly and hourly schedule; the scan's limits; the dot-folder prune list; dot-folder finds being recorded and never reduced; the PATH launcher keep; no setting to turn the scan off. Consequences of the existing rules now reaching more folders: a scan-found source checkout losesnode_modulesand Pods afteridle.sourceDormantDays(7) and a Codex worktree afteridle.dormantDays(1). The guide said a discovered main checkout keeps them, which was already untrue onmainfor every kind of checkout; it now states the rule. Also, and a Codex thread counts as in use for 30 minutes, so an older thread that is resumed finds its dependencies gone. The scan runs in the daemon, so the macOS prompt forDocumentscomes from a background process, and the first read times out before a person can answer it; the folder is read on the next scan.Test plan
/, another home, the Stim home, pruned and skipped folders, a different-case spelling, a symlinked ancestor.appscontainer oriosentry is a symlink; in the recheck a pending manifest read leaves the entry unselected and the scan file unchanged, and it is selected once the read settles.maintenance.discoverRootsmakes it discoverable under the full rules.~/.codex/worktrees-style linked worktree is discovered and held at the plan step and under the removal locks; a scan-found linked worktree is held; the dormant stage clears a reduce-only worktree's output and keeps the worktree; the PATH launcher keep at each route: the idle stage, the Cargo low-disk plan and the acting trim (a stale release profile holding a PATH-linked executable), and the dependency pass before clearing and under the lock, each a candidate without the symlink; a launcher created, or a launcher directory made unreadable, between planning and deletion in the same operation keeps the checkout; a nested Cargo target in a monorepo is kept by a launcher elsewhere in the checkout; a launcher that appears after the trim's first reading keeps the directory it points into; a launcher that appears during the artifact route's last process read keeps the tree; a launcher that appears during a dependency clear's final idleness recheck, or while the idle stage measures sizes, keeps the tree, and each of those guards fails its test when removed; a read that never settles starts exactly one filesystem read across several readings and scans, all incomplete or truncated, and both work again once it settles; absent, dangling, EACCES and EIO at the reader with an injected directory list; a scoped home gets neither tool folders nor the scan.main, all jobs green: https://github.com/appandflow/stim/actions/runs/43720287243. Earlier runs on this branch needed reruns oftest (node 24)for worker crashes recorded on exec integration test for a cancelled child after a stalled event loop fails intermittently in the merge queue #5521.Documents.Part of #5762.