Repository navigation
Description
Problem
Under sustained host load, the Cargo target/ trim that high and critical disk levels call for waits on the size scan, and the size scan waits on load. On 2026-11-14 a Mac spent the evening at high and critical, reached 0.2 GB free three times, and Stim emptied its own shared caches before it trimmed 40 GB of Cargo data it was allowed to remove.
Maintainer, 2026-11-14:
- "we def should not have to wait on full disk measurement to be able to remove caches"
- "could caches we know are fast to measure be measured with pretty sort cache ttl"
- "Can we also make sure measurement doesnt get delayed by load, or at least not too much"
Evidence
From ~/.stim/maintenance/state.json and ~/.stim/daemon/daemon.ndjson, Stim 2.0.0-next.20261114220335.g84b6915, 14 cores:
| Time | Event |
|---|---|
| 18:21 | last size scan |
| 18:42 | level high, 34.5 GB free |
| 18:45 | worktree and idle checks deferred for load |
| 19:09 | 11.2 GB free |
| 20:06, 20:25, 20:37 | critical, 0.2 GB free; rung 8 deletes recordings and archives |
| 21:24 to 21:27 | rung 3 empties the build, Metro and Swift compilation caches, 6.1 GB |
| 21:47 | first Cargo measurement since 18:21: one pass, 9 actions, 39.9 GB freed, 747 s |
- One-minute load was 190 to 410 all evening, about 14 to 29 per core. The gate is 4 per core (
SIZE_SCAN_MAX_LOAD_PER_CORE,stimd/passes/measure.ts), so on a machine where agents build, deferred is the normal state and the ceilings are the real schedule. du -sktimings at load 194 to 410, metadata possibly warm:
| Folder | Size | Files | Time |
|---|---|---|---|
Cargo target/ |
4.5 GB | 18,654 | 0.8 s |
Cargo target/ |
8.9 GB | not counted | 1.0 s |
node_modules |
1.8 GB | 79,910 | 5.8 s |
node_modules |
2.9 GB | 163,209 | 8.4 s |
node_modules |
4.6 GB | 165,610 | 13.9 s |
Cost follows file count, roughly 12,000 to 19,000 files a second here. LLP 0005 records that a walk of a large target/ can take its full 120 seconds, so a target is not always cheap.
stim gc --dry-runon the real home took over 20 minutes under that load (146 projects).
Cause
deferIfLoaded(stimd/passes/run.ts) defers the size scan while load is above 4 per core, for up to 3 hours (DEFERRAL_CEILING_MS,stimd/passes/defer.ts).- The Cargo trim is planned from measured sizes. While the scan is deferred,
measureCapOnlymeasures the targets on their own, at most hourly (CAP_CHECK_MS). LLP 0005 records both as coordinator choices, not maintainer decisions. - Unknown: why 3 h 26 min passed between Cargo measurements when the fallback is hourly. stimd started four times in that window (
daemon_startedx4), and fourdaemon_maintenance_failedrecords name an unverified report child. Whether a restart or a failed pass resets or skipscapAtis not traced. - The pressure rule itself needs no size: at high every recognised
target/loses all stale profiles and allincremental/data whatever the cap (LLP 0005, Pressure tightens the cap). Only the cap at normal compares bytes.
Fix idea
- Acting at pressure never waits for a measurement. When a volume's level rises to high or critical, plan the Cargo trim for the targets on it at once, from folder structure, lock files and Exact's ledger, with no
dufirst and no load gate. Report the bytes from the free-space change afterwards. Check every other rung for the same measure-then-decide dependency. - Run that trim before the rungs that empty shared caches:
incremental/is cheaper to lose than the Swift compilation cache. - Freshness by cost. Record how long each walk took with its measurement, and let a folder's measurement age out in proportion to that cost, with a floor and a ceiling, so a one-second walk is repeated every few minutes and a two-minute walk every few hours. No list of "fast" folders to maintain, and a target that grows slow to walk slows itself down.
- Replace the on/off load gate with a budget: one walk at a time at low priority (already the case), measuring for at most a small share of wall time, with a short maximum deferral for cheap folders. While any volume is above normal, load defers nothing: a full disk fails the builds the gate protects.
- Trace and fix the 3 h 26 min gap.
Preserves: the ownership, lock and running-process checks before every deletion; the dependency-tree signature cache; one walk at a time. Trades away: some IO during loaded periods, bounded by the budget.
LLP 0005 (The size cap) and LLP 0001 need the maintainer's words above.
Activity
janicduplessis commentedon Nov 15, 2026
Maintainer decision, 2026-11-14: "Lets make sure that if disk space is low we do any needed measurement right away since disk space is prio over cpu use".
Scope this adds to the first pull request: while any volume is above the normal level, host load defers no measurement that a cleanup step at that level needs. The load gate stays for the normal level. The walk-cost freshness and the load budget for the normal level remain the second pull request.
janicduplessis commentedon Nov 15, 2026
Claimed: fix/5759-cargo-trim-at-pressure
janicduplessis commentedon Nov 15, 2026
Maintainer decision, 2026-11-14: "Yes, add it", in answer to the proposal to add a size measurement at the end of each build to the second pull request for this issue.
Scope for the second pull request (measurement freshness), in addition to Fix idea items 3 and 4:
target/for an Exact app) are marked stale and measured promptly by the daemon, not by the command, so the command is not slowed.packages/stim-cli/src/commands,native,integrationsorjobwhen a build finishes. Sizes are refreshed only by the maintenance size scan, the status watcher (status-measures.ts: a live workspace's folders at most every 5 minutes whilestatus --watchruns, hourly otherwise), the daemon ladder when it acts, andstim gc.target/, and what the pre-build headroom estimate (LLP 0006) is based on. If the estimate can use the measured size of the last build, say so in the pull request; do not change it without a decision.janicduplessis commentedon Nov 15, 2026
Findings from the lane for the build-end measurement (it stopped after tracing, as briefed, and changed nothing). Paths are under
packages/stim-cli/src.Where builds end. iOS and Android compiles go through
engine/acquire-artifact.ts:createBuildHeadroom(line 152), withbegin()andfinish()aroundlifecycle.build.compile(156-162). macOS does the same atmacos/build.ts:282-286.begin()runs only for a local compile, so a cache hit or an offload never reaches it.finish()runs only on success: a failed compile throws past it, so a failed build has no end hook today. Exact uses the same artifact path.There are two size stores, and they do not feed each other.
refreshMeasure(disk-usage.ts). Readers:stim statusand Desktop through it, and the gc worktree sizes. The status watcher fills it only whilestatus --watchruns, and only for the checkout,node_modules, the workspace state folder and device folders. It never measures a Cargotarget/orandroid/*/build. In the daemon the onlyrefreshMeasureuser is the measure job (stimd/main.ts:1844,daemon/dependencies.ts:546), which coversnode_modulesandios/Podsevery 10 minutes.state.sizesin the maintenance state, filled only by the hourly size scan with directmeasuredDirectorySizeAsyncwalks (stimd/passes/measure.ts:175,commands/gc/workspaces.ts:329,stimd/passes/cargo-target.ts:164). The caps (maintenance.workspaceOutputsMaxGb,maintenance.cargoTargetMaxGb) read this one.So a
refreshMeasureafter a build would make status fresher and would not reach the caps.No stale marker exists. A missing cache entry cannot mean "stale": dropping entries makes every never-measured build folder look stale, and a daemon scan would then walk all of them.
How a command reaches the daemon today. Slot release calls
notifyReleased('build')(engine/slot-queue.ts:336), which requests a tick, but it is skipped for an unlimited slot and for a retained claim, so it is not a reliable build-end signal.Headroom (LLP 0006) is the median of the ten newest positive build-ledger records, each the free-space drop between
beginandfinish, with no walk. A post-build folder size is a total, not a per-build delta, so it adds nothing to that estimate.Options the lane listed. A: drop cache entries at build end and extend the daemon's measure job to build folders (needs a tick change; only status benefits). B: a marker file written by the command and consumed by the tick (new state file; still does not reach the caps). C: feed the post-build measurement into
state.sizesso the caps benefit (a larger change to the maintenance scan). D: no build-end hook; shorten the scan or watcher interval.No option was chosen. The two stores are the underlying problem for this item and for the freshness and budget items above, so they are being designed together before any code.
janicduplessis commentedon Nov 15, 2026
Design pass for the second half of this issue (read-only, at
fa6f21cd0; nothing was changed). It corrects the cause once more: the long steps on 2026-11-14 were not size walks.1. Short answer
The two hour-plus steps on 2026-11-14 were not size walks. Nothing in the silent stretches calls a size walk in its decision path; they are serial per-checkout evidence loops (git, activity, process, lsof, settings reads) that run several times per tick under load 14 to 29 per core, and a long tick or lane step blocks everything queued behind it. So the freshness and budget work (requirements 2 and 3) and the scheduling work are two separate fixes, and the scheduling one is what removes the 3 h 26 min gap.
Recommended design:
walkMs) and a sibling "stale" marker.state.sizesstays as the planner's snapshot (it carriesidleDays,blocked, per-workspace grouping that the cache does not) and its shape does not change, becausemaintenance.sizesis in the status JSON.clamp(k x walkMs, floor, ceiling), applied only to when the daemon walks, never to whether a consumer may use the last value.finallyaround the compile; a small independent timer in stimd starts the existing measure thread when markers exist; the measure thread walks the workspace's build folders and a Cargotarget/throughrefreshMeasure.niceabove normal.runMaintenanceand the measure job on their own, and give the dormant dependency step a deadline with a resume cursor. Leave the ladder's rung order and per-rung logic alone; add per-rung timing first because the log cannot say which rung or loop consumed the hours.2.4 Where the 6,699 s and 4,137 s went (2026-11-14, local times)
Evidence is the
daemon_tick,daemon_maintenanceandpressure_actionrecords.daemon_tickanddaemon_maintenancecarryphases, rungs are not timed, andpressure_actionrecords only mark when an action finished.Tick A, 18:46:00 to 20:37:42 (
ladderMs6,699,466;measureMs393,idleMs2,455,dormantMs183):workspace_idleblueprint-murders-issue-1,-35,-16)dormantCandidatesloop over all projectskeptrecords (27 under the 256M rule)idleStageCandidatesplusrankDependenciesBetween 18:49:58 and 20:06:26 the ladder ran rung 5's take loop (nothing to take), rung 6 (
planRung6: a process-table memory report and a per-project Gradle version scan withmeasure: false,sweep.ts:220-290), rung 7 (planWorktrees, a sweep of every discovered worktree,sweep.ts:340-420) and rung 8's source dependency step (ladder.ts:735+, anotheridleStageCandidateswithfull: trueplusrankSourceDependencies). None of these has a size walk in its decision path. I cannot split the 76 minutes among them: no record marks a rung boundary. Between rung 8 actions,planHistory(records.ts:187) and the deletions themselves run; I did not read their cost (marked inferred: slow deletion of archive trees at 0.2 GB free).The 4,339 s ladder of tick B (20:37:42 to 21:51:43) shows the same shape: 2,771 s of silence from the precompute record (20:38:34) to the first rung 3 action (21:24:45) covers the pre-ladder passes, rung 1
stoppedOutputCandidates, rung 2 and the start of rung 3 (at critical rung 3 does not walk the build or Metro cache,ladder.ts:433-438); then rung 4 loops until 21:27:42; rung 5'skeptrecords appear at 21:30:59 and 21:31:39 (197 s and 40 s of evidence loops); then 1,204 s with no record until the tick ended at 21:51:43, which is rungs 6 and 7 at high (rung 8 runs only at critical). So rungs 6 and 7 cost about 20 minutes at high and up to 76 minutes at critical.Lane, 20:38:33 to 21:59:58 (
daemon_maintenance4,885,657 ms:fingerprintMs630,dependenciesMs4,137,365,maintenanceMs747,662):runDormantDependenciesemitted nodormant_dependenciesrecord in those 69 minutes. The next records are 22:01:12 and 22:04:14, from the following lane run (235 s). The function does not walk (it readstrustedMeasure,dependencies.ts:134-135); it runsidleStageCandidatestwice (linked, then source withfull: true,:418-419),rankDependencies(collectArtifactsAsync: per worktree, per candidate pathinUseOfand a process listing,artifacts.ts:409-470) andrankSourceDependencies(per source checkoutsourceCheckoutIdle,dependencies.ts:197). That is the whole 69 minutes, by elimination. Marked inferred at the per-item level.maintenanceMs747,662 is the first size scan and Cargo pass since 18:21: 21:47 to 21:59. It started only after the dependency step, which started only after tick A, which gives the 3 h 26 min between Cargo measurements (18:21 to 21:47). fix: trim Cargo targets at high and critical disk without waiting #5763 now runs the Cargo trim in rung 3 and drops the load gate above normal, so that path is fixed for Cargo; the chain (tick, then lane, then dependency step, then maintenance) is not.Other evidence:
eventLoopDelay.maxMsis 14 to 62 s on these records. The hours are awaits on children, not synchronous work.dependenciesMsof 8 to 32 minutes recur in the log (for example 17:15, 21:11, 22:03, 23:29, 00:33, 01:02 on the 14th and 15th), and ticks of 12 to 14 minutes (23:59, 00:36). Marked as pattern, not traced individually.collectWorktreeSweepand says the dormant candidate loop is sequential and discovery is cached only within one operation. fix: keep the hourly dependency check across worker restarts #5745 fixed the cadence reset. feat: discover tool folders and scan the home folder for checkouts #5764 adds checkouts found through tool folders and a home scan, so the per-tick item count on main is higher than what ran.config.jsonregisters 53 projects today against 146 in the issue (gc --dry-runon 2026-11-14). The registry changed since; the discovered-checkout count at the time is unknown. I cannot compute a per-item cost for 2026-11-14 and do not claim one.2.5 What the brief's premises look like on main
defer.ts:18, anddue.ts:26also stops the 5-minute retry window above normal.ladder.ts:430-470(caches at high) needs a size to decide; it walks immediately. The report-only walks at:161, 370, 479, 544andidle.ts:224are not needed to decide and could be replaced by the free-space delta (a deletion-path change, so not proposed here).4. Pull request sequence
D marks a step that touches code deciding or performing a deletion. The hard constraint: a speed or freshness change must not alter what is deleted, reduced or held. By construction I can guarantee it only for steps that add logging or add data nobody reads yet. Every other step either moves a start time or feeds a fresher number into a decision, so it is marked and carries a docs-only alternative.
idleWorkspaces,collectArtifactsAsync,planRung6,planWorktrees,planHistory): start, end, items visited, indaemon.ndjson. Also recordwalkMsin cache entries written byrefreshMeasure, read through a separate reader.daemon/ladder.ts(runRung,:961-978),daemon/passes.ts(lane laps:218-249),daemon/stages.ts,disk-usage.ts,core/state/status-measures.ts.state.sizes.getExecutorseam): assert record fields and that a rung that throws still logs.ladder.tsmeans the reviewer must see the wrapper does not change control flow. Not D by construction, because the added code is read-only.runMaintenancebecomes its own job on its own thread; the precompute and dormant dependency step stay on the passes thread; the measure job and maintenance start from a timer and not afterrunHomePasses. The dormant dependency step gets a 10-minute deadline and a resume cursor.daemon/passes.ts:201-274,stimd/threads.ts:8, 187-250,stimd/main.ts:1006, 1844, 1970,daemon/dependencies.ts:403-421.runMaintenance, the plans, the claims (maintenanceRunClaimsstill allows one pass), rung order.stim guide lifecycle budgetand LLP 0001.finallyat the two compile sites; timer; the measure job consumes markers and writes the cache. Consumers of the cache for these folders see fresher numbers (status, gc worktree report); the caps do not (they readstate.sizes).engine/acquire-artifact.ts:152-162,macos/build.ts:282-286,core/state/status-measures.ts(marker path and reader),daemon/dependencies.ts:547-552,stimd/main.ts. Also add the marker file name toforgetStatusMeasures(status-measures.ts:43-62) so worktree removal clears it;.failed.jsonis not inisCacheFileName(core/state/status-measures.ts:58-60), so check that markers are pruned like failures.finish()and the ledger, the headroom estimate, all caps,trustedMeasure.refreshMeasureand clock show a marker consumed within one timer period and kept if a newer one appeared; no real build or device.freshwith no entry; drop it); many agents ending builds at once (the walk claim serialises; budget in PR 5). Unverified: whether any decision readsbuildStatebytes from the same cache entries (section 2.2). If one does, this becomes D and the docs-only alternative is to describe the marker as status-only.walkMsandstaleAt. Consumers keep their ownmaxAgeMs.disk-usage.ts(due function),daemon/dependencies.ts, a small new module for the queue.(walkMs, age, staleAt)to due or not; ordering by overdue ratio.walkMsinflated under load stretches TTL; documented.disk-usage.ts:181). Worktree, sweep and idle keep the gate.stimd/passes/defer.ts,run.ts:341-377,measure.ts:76-80,disk-usage.ts:178-182,guide/cleanup.ts:43,guide/settings.ts:799.measureSizes,measureCargoTargets,measureOwnedDevicesand thestim gccopy callrefreshMeasurewith the TTL, and walk only stale folders;state.sizeskeeps its shape.stimd/passes/measure.ts:107-191,cargo-target.ts:136-173,devices.ts,commands/gc/workspaces.ts:486-493,level-run.ts:181-185.staleAtis unset andmeasuredAtis under 60 min old, otherwise walks. A discovered Cargotarget/then keeps its size between scans, which today it loses (run.ts:516-521). That too can make a cap fire that did not. Docs-only alternative: record in LLP 0005 that capped sizes are per scan and that discovered targets rotate.staleAtnewer thanmeasuredAtforces a walk, a failed walk keeps the previous value (not unknown).walkMsmissing on old entries (treat as unknown cost: use the ceiling).Not in this sequence: evidence memo across rungs, reordering rungs, replacing the report-only walks before deletion,
.stalefor dependency trees.5. Product decisions needed
sizedue at build endrefreshMeasureabove normal: removenice(LLP 0001 already says normal priority)stim gcwalks follow the budgetmaintenance.*settings (new settings are product decisions)6. What I could not establish
runDormantDependenciesthe 4,137 s. The log has no rung or loop timing, and the code running that evening differs from main by fix: trim Cargo targets at high and critical disk without waiting #5763 and feat: discover tool folders and scan the home folder for checkouts #5764. PR 1 exists to answer this.buildStatebytes from the cache (section 2.2).isCacheFileNamesuggests files other than<32 hex>.jsonare not listed, which is also what happens to.failed.json.dutiming on a Cargotarget/,niceandtaskpolicyeffects on IO): not exercised; only the issue's numbers.janicduplessis commentedon Nov 15, 2026
Claimed: feat/5759-timing-records (timing-records item, pull request 1 of the design; the issue stays open). Worktree .worktrees/5759-timing-records.
janicduplessis commentedon Nov 16, 2026
Maintainer decision, 2026-11-15: "For rest for with ur recs", in answer to a list of open decisions each carrying a recommendation. The recommendations it accepts for this issue, as they were put to the maintainer:
What follows from it:
disk-usage.ts.daemon_steprecords from a daemon running a build that includes feat: record timing of each ladder rung and evidence loop #5768.janicduplessis commentedon Nov 16, 2026
Claimed: feat/5759-build-end-measure (step 3, build-end marker and independent trigger). Worktree .worktrees/5759-build-end-measure.
janicduplessis commentedon Nov 16, 2026
Step 3 merged: #5782 as 452d880. When a local native build ends the command writes a marker and the daemon measures that workspace build folders within about 30 seconds, so
stim status, Desktop and the gc report show fresh sizes. No deciding code reads these entries; the reader trace is in the pull request. Limits and the one finding left open are in its human-review comment. Not exercised against a real build or a real daemon.State of the six steps:
daemon_steprecords from a daemon running a build that includes feat: record timing of each ladder rung and evidence loop #5768.4, 5, 6: not planned for 2.0, per the decision above.
Related, filed from the review of step 3: #5784, the dormant dependency scan reads project paths without a time limit.
janicduplessis commentedon Nov 16, 2026
First
daemon_steprecords from a real daemon (read-only look at one MacBook, engine 2.0.0-next.20261115170139.g286452f, running 3.3 hours from 2026-11-15 22:42 local, disk at normal, load average between about 5 and 80 during that window). 257 records.dormant_dependencies(whole step)idle_stage_candidatesrank_source_dependenciescollect_artifactsrank_dependenciesdormant_candidatesrung,plan_rung6What it supports: the dormant dependency step is about two minutes at this size, and most of it is
idle_stage_candidates, which runs twice per step at roughly a quarter of a second per item for 151 items. That matches the design pass: serial per-checkout evidence loops, not size walks.What it does not show: the ladder above normal (the disk never left normal, so rungs 3 to 8 did not act), and the same loops at the load of 2026-11-14 (14 to 29 per core). Step 2 still needs records from a period at high or critical before its numbers mean anything; a 10 minute deadline for the dependency step looks generous against 110 s here and is untested against the 69 minutes seen under load.