Skip to content

Discovery makes the daemon's maintenance lane and gc dry runs slow with many checkouts #5531

Closed as not planned

Description

@janicduplessis

Report

With discovery (#5519) live, the daemon's maintenance lane spends much more time in the dormant dependency step, and stim gc --dry-run takes several minutes on a machine with many checkouts.

Read-only measurements on one MacBook, 2026-11-11 23:57 to 2026-11-12 00:26, discovery build 16a2751ef, under heavy load from other work (so partly load, not only code):

  • The dependency phase took 331 s, 251 s and 277 s in 31 minutes: about 46% of the maintenance lane's time, against 2.7% in the six hours before. The lane runs one thing at a time, so a busy lane cannot start the pass that would run the size scan.
  • That step reaches discovery through stimd/passes/artifacts.ts (artifactWorktreesAsync: discoverCheckouts, then a source-checkout resolve and walks per candidate) and daemon/stages.ts (discover: true).
  • discoverCheckouts is memoised per operation only, so every pass recomputes it. Its listing is cheap (git worktree list over 41 roots took 1.3 s in total); the cost is the per-checkout evaluation: 41 registered plus about 112 discovered checkouts, each checked for merged, idle, dirty, unpushed and ignored files.
  • recognises() and isProject() in workspace/discovery.ts use synchronous directory reads on the event loop; event-loop max delay in lane passes was 6.5 s and 11 s after the discovery build, 3.2 s before.
  • The dependency step is gated to once per hour (daemon/dependencies.ts, dormantCheckedAt) yet ran three times in 30 minutes in one daemon run; the cause was not found (possibly stim gc --dry-run forcing it through the daemon).
  • stim gc --dry-run sat for minutes at "checking 153 projects for merged or idle worktrees".

Expected

  • One discovery result reused across ticks for a bounded time.
  • Discovered-checkout evaluation under a time budget per pass, resuming where it stopped, so it cannot occupy the lane for minutes.
  • No synchronous filesystem walks on the daemon's event loop.
  • The reason the hourly gate did not hold, found and fixed.
  • A dry run that reports progress per checkout or finishes in seconds on 150 checkouts.

Parent: #5512.

Activity

  1. added
    type: bugBreaks documented or expected behavior
    priority: P1Next up: user-facing defect or unblocks other work
    area: clipackages/stim-cli, including its README and guide
    on Nov 12, 2026
  2. janicduplessis commented on Nov 14, 2026

    @janicduplessis
    CollaboratorAuthor

    Current-main triage at 7f822024d: the original numbers came from a loaded Mac, and the report distinguishes the cheap checkout listing from the slower per-checkout evaluation. Keep this open for a matched dry-run plus maintenance tick against the same read-only inventory. Record discovery, per-checkout evaluation, and event-loop delay separately, and distinguish scheduled dependency runs from manual gc requests before changing their cadence.

    #5163 is related on the serving side. Reusing a fresh preview can avoid repeat traversal by Desktop, while this issue still owns the cost and scheduling of producing that preview. Preserve the per-checkout evidence and pre-deletion rechecks described by LLP 0004.

  3. janicduplessis commented on Nov 14, 2026

    @janicduplessis
    CollaboratorAuthor

    Found a path that resets the hourly dependency check without restarting stimd. On main df9bfb200, dormantCheckedAt belongs to the module loaded in the passes worker. The worker exits after two minutes without work, so its replacement starts with -Infinity. The same code was present in the reported 16a2751ef build. Hourly gate · Worker lifetime

    A read-only probe called the built dependency pass with an empty inventory and a fixed clock in real Node workers. The same worker inspected the inventory at +0s, skipped +60s, ran the explicit force at +61s, and skipped +62s and +180s. A replacement worker inspected the inventory at that same +180s. The original log has three matching gaps of 136 s, 327 s and 151 s between the end of the last pass and the estimated start of the next expensive pass, all in one daemon run. The log does not carry a worker ID, so those older worker retirements remain an inference. Manual GC, including dry-run, intentionally forces a separate pass and does not emit daemon_maintenance for it. Manual path

    Current dry-run worktree classification already overlaps some Git reads when child claims permit it and reports an aggregate project count. The dormant pass's candidate loop remains sequential, discovery is cached only within one operation, and folder recognition still uses synchronous reads. The scheduled dependency pass runs in its own worker, while other foreground idle-stage paths also call discovery. This isolates the cadence defect from the remaining traversal, timing and progress work. Dry-run classification · Dormant candidates · Discovery memo

  4. janicduplessis commented on Nov 14, 2026

    @janicduplessis
    CollaboratorAuthor

    Claimed: fix/5531-dormant-dependency-cadence

  5. janicduplessis commented on Nov 14, 2026

    @janicduplessis
    CollaboratorAuthor

    #5745 landed as 715d8588a. The dependency schedule now survives retirement of an idle passes worker. The focused test rejects the first inspection, replaces its worker, skips the premature automatic pass, runs a manual forced scan, and permits a scheduled scan at the hour boundary. Independent review, CI and the merge gate passed.

    This leaves the cost and reporting of individual discovery scans open. The next measurement should distinguish scheduled scan frequency from the end-to-end latency of one large scan or manual dry run.

  6. added this to the 2.0 milestone on Nov 14, 2026
  7. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    The reproduced hourly scheduling defect was fixed in #5745. Closing the broader resumable-scan and shared-cache redesign. The original timings predate that fix and were collected under heavy host load. Reopen a focused performance report with current large-checkout timings. The separately reproduced configured-root issue remains tracked in #5757.

  8. janicduplessis commented on Nov 15, 2026

    @janicduplessis
    CollaboratorAuthor

    Measurement for this issue, from the work on #5762 (pull request #5764), read-only on one MacBook at high load, 2026-11-15:

    • A real collectWorktreeSweep took 23 s for 162 candidates, about 0.14 s per discovered worktree (git status, activity, HEAD, unpushed).
    • feat: discover tool folders and scan the home folder for checkouts #5764 adds checkouts found through ~/.codex/worktrees and a home folder scan (17 more on that machine, an estimated 2 to 3 s). Its 40-checkout window bounds the reduction passes only. The finished-worktree check, in the background pass and in the pressure ladder, still inspects every discovered worktree.
    • Skipping deferred reduce-only checkouts in the sweep was tried and reverted there: worktrees are grouped by path prefix and a group is held when any root in it is reduce-only, so dropping a root could release a held worktree. Any bound on the sweep has to keep the reduce-only classification from the complete discovery set before it filters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: clipackages/stim-cli, including its README and guidepriority: P1Next up: user-facing defect or unblocks other worktype: bugBreaks documented or expected behavior

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions