Skip to content

feat: record timing of each ladder rung and evidence loop - #5768

Merged
janicduplessis merged 3 commits into
mainfrom
feat/5759-timing-records
Nov 15, 2026
Merged

janicduplessis merged 3 commits into
mainfrom
feat/5759-timing-records

Conversation

@janicduplessis

@janicduplessis janicduplessis commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Description

On 2026-11-14 one daemon tick spent 6,699 s in the pressure ladder, and the next maintenance lane spent 4,137 s in the dormant dependency step, while a disk went from 34 GB to 0.2 GB free. The log cannot say where that time went: daemon_tick and daemon_maintenance carry phase totals, rungs are not timed, and pressure_action records only mark when an action finished. Between two of them there are 4,587 s with no record at all. The design on #5759 concludes that these were serial per-checkout evidence loops, not size walks, and that every later step depends on knowing which loop.

This change adds the records and decides nothing.

Solution

One new record in daemon.ndjson, daemon_step, written when a step ends: its name, when it started, how long it took, how many items it counted, and whether it finished, threw or was aborted. It is written for every ladder rung (with the rung and the pressure level, as pressureLevel) and for the evidence loops the design names: idleWorkspaces with its candidate loops, collectArtifactsAsync, the two dependency rankings, planRung6, planWorktrees, planHistory and runDormantDependencies.

The wrapper, timedStep, takes two timestamps around the call it is given, returns its value and rethrows the same error object. The record is written in a finally, a failing logger is ignored, describing an error can never replace it, and with no logger it only calls the function. A caller cannot pass a field named like one the log writers set themselves (src, level, event, msg, ts): an earlier head named the pressure level level, which overwrote the record's severity. A loop is recorded once with a count, never per item. items means the actions recorded for a rung, the checkouts visited for the idle loops, and what the step returned for the others; the guide says which.

refreshMeasure also stores how long each measurement took, as walkMs in the disk-usage cache entry, from just before the install-signature and target reads to the end of the walk. Nothing reads it yet; entries without it stay valid, and DiskMeasure and the status JSON are unchanged.

Volume: at most 1 extra record a tick at normal, 16 at high and 22 at critical, plus about 7 an hour from the dormant dependency step. At critical that is roughly 460 KB an hour against an 8 MiB rotation with one older file kept.

Not wrapped: the activity pass runIdlePass.

What a reviewer should check at each call site in daemon/ladder.ts, daemon/stages.ts and daemon/dependencies.ts: the diff is the wrapper around the same call with the same arguments. Where a function gained a parameter to carry the logger down, the value passed for any parameter before it is the default it had.

Test plan

  • timedStep: a step that finishes, one that throws (the caller receives the same error object, also when describing or classifying it throws) and one that is aborted each leave one record with the right outcome; no logger means no record and the same result; a record built the way the log writers build it keeps severity debug and carries pressureLevel; reserved field names are dropped.
  • A wrapped stage call returns the same result with and without a logger.
  • Ladder: rung records for a finishing, a throwing and an aborted rung, under a fake clock.
  • walkMs is written by a walk and an entry without it still reads.
  • Full CI workflow on the head, all jobs green: https://github.com/appandflow/stim/actions/runs/43728767017.
  • Not exercised: a real daemon under load, which is where the records are needed.

Part of #5759: pull request 1 of the design posted there.

@janicduplessis janicduplessis added area: cli packages/stim-cli, including its README and guide area: core packages/core area: website website/ labels Nov 15, 2026

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: I read the whole diff at 68ac716 and the surrounding functions. No behaviour change found in the wrapped calls. One defect in the new record itself.

Findings

  1. BLOCKING, daemon/ladder.ts:1002 with the three daemon log writers (stimd/main.ts:775, daemon/passes.ts:317-320, :438, :518). The rung record passes fields: { rung, level: context.level }, and every writer builds { src, level, event, msg, ...fields }, so the spread overwrites the record's own level. Checked with the same expression: {"src":"stimd","level":"high","event":"daemon_step","msg":"x","step":"rung","rung":2}. Every rung record in daemon.ndjson therefore has level: "high" or "critical" instead of debug. Neither is in LEVELS (core/state/ndjson.ts:14); levelRank maps an unknown level to 0, so logs --level and the block filter in log-reader.ts:365 treat it as debug by accident, and logs --json shows an invalid level. The guide text ("one debug record ... for a rung: rung and level") describes a record the daemon cannot write. The ladder test misses it because log is a vi.fn that receives level and fields as separate arguments and never goes through a writer. Rename the field (for example pressureLevel) and update the guide, the website text and the test, or assert through createNdjsonWriter.

  2. NON-BLOCKING, daemon/step-timing.ts:42. In the catch, String(error) can throw for a non-Error thrown value (a null-prototype object, a throwing toString). That TypeError would replace the original error, which is the one thing the wrapper promises not to do. Errors in these call sites are Errors, so this is unlikely. Move the message computation into the finally try, or wrap it.

  3. NON-BLOCKING, __tests__/daemon-step-timing.test.ts. The throw test asserts only rejects.toBe(failure) and toHaveBeenCalledTimes(1), not outcome: 'threw' or error. There is no unit test for aborted (only the ladder test covers it, and it does not check error identity). No test covers a changed return value at the dependency or stage call sites. The PR description lists both under "Test plan".

  4. NON-BLOCKING, core/state/status-measures.ts:215. readDiskUsageWalkMs has no production caller, only the test. CLAUDE.md says not to export an internal solely to test it. Either drop it and read the file in the test, or keep it as the reader for the follow-up, and say so.

  5. NON-BLOCKING, disk-usage.ts:~199. walkMs is now() - startedAt, where startedAt is taken before treeSignature and options.targets(), so it includes those. The measure is the same bytes and measuredAt as before. It also adds one now() call on the measured path; fine with a pure clock.

Transparency (checked by reading old and new code)
Every wrapped call is the same function with the same arguments, awaited at the same point, result used the same way. timedStep is async, so the wrapped call adds only microtask ticks. idleWorkspaces was async and is now a non-async wrapper that returns timedStep(...), so a synchronous throw still becomes a rejection. readIdleWorkspaces iterates the same array built in the same order as before (checkouts is the old inline expression), after the same discoverCheckouts await. runDormantDependencies keeps the cadence block outside the wrapper and moves only the actions body into a closure with the same two conditionals. runRung keeps the same catch (claim-unavailable rethrow, throwIfAborted, pressure_rung_failed, failed action push); before is read outside try.

Changed signatures and callers

  • rankDependencies(idle, protectedRoot, operation?, source = false, log?): callers ladder.ts:586 and dependencies.ts:522, both (idle, protectedRoot, operation, false, log). false is the old default. No other callers (tests included).
  • rankSourceDependencies(idle, keeps, protectedRoot, rule, operation?, log?): callers ladder.ts:768 and dependencies.ts:474, both the same first five arguments as before, then log. No other callers.
  • idleWorkspaces(step, config, reading, ...): private; step is prepended and all three callers (idleStageCandidates, stoppedOutputCandidates, dormantCandidates) pass their original arguments unchanged. The exported function signatures are unchanged.
  • DeviceReading gains optional log. It is already present on every RungContext and IdlePassInput spread into it, so the loops now log wherever those objects reach them.
  • collectArtifactsAsync and takeIdleDependencies: signatures unchanged.

Other checks

  • Reads and writes: readDiskUsage, trustedMeasure and freshMeasure are unchanged. readDiskUsage picks named fields, so an older Stim ignores walkMs, and this one reads an entry without it (the test covers that). Nothing reads walkMs or any step timing to decide anything.
  • Logging: log is a synchronous writeSync append per record on the calling thread, inside a finally guarded by try/catch. A throwing logger is swallowed (tested). debug reaches daemon.ndjson by default: the writers apply no level filter, the same path as daemon_maintenance.
  • Types: the daemon log type now allows debug (IdlePassInput['log'] in idle.ts). It is internal; nothing in api.api.txt or the RPC schema mentions it.
  • Volume: from the code I count about 20 at critical, 16 at high, and 7 per hour for the dormant dependency step. This is consistent with the stated bounds.
  • Comments follow the policy; no invented quotes.

Not checked: I ran no tests or builds (read-only). I did not exercise a real daemon.

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: clear at fe90b55. I read the whole diff against main and the surrounding functions, with "no behaviour change" as the claim to check. I found no blocking defect. I ran no tests or builds (read-only).

Signatures and callers (from the code at this head)

Function Change Callers (all of them)
rankDependencies(idle, protectedRoot, operation?, source = false, log?) log appended ladder.ts:586, dependencies.ts:522; both pass (…, operation, false, log), false being the old default. No other caller in src or tests.
rankSourceDependencies(idle, keeps, protectedRoot, rule, operation?, log?) log appended ladder.ts:768, dependencies.ts:474; same first five arguments as before, then log. No other caller.
idleWorkspaces(step, config, reading, thresholdMs, setting, worthIt, selection) (private) step prepended; body moved to readIdleWorkspaces(tally, …) idleStageCandidates, stoppedOutputCandidates, dormantCandidates (stages.ts:337, 349, 497); each passes its old arguments after the step name. The three exported signatures are unchanged.
DeviceReading.log? new optional field Not a parameter; absent means no record. Set only where a RungContext / IdlePassInput is spread into it.
IdlePassInput.log type now DaemonLog (adds 'debug') Internal type only; api.api.txt is untouched by the diff.
refreshMeasure's inner record(…, walkMs) fifth parameter Three call sites inside refreshMeasure (absent path passes null, measured path passes the elapsed time); the early-failure and fresh paths do not call it.

Wrapped calls: each is the same function with the same arguments in the same order, awaited at the same point, its result used as before (collectArtifactsAsync x2, takeIdleDependencies/rankSourceDependencies/rankDependencies, planRung6, planWorktrees, planHistory, RUNG_ACTS[rung]). runDormantDependencies keeps the cadence gate outside the wrapper and moves only the two conditionals into the closure. runRung reads before outside the try, and its catch is unchanged. readIdleWorkspaces iterates the same array as the old inline expression, after the same discoverCheckouts await.

timedStep

  • Returns result; with no log it returns run(tally) directly (a synchronous throw inside the async function is still a rejection).
  • The error object is never replaced: describe has its own try/catch (the earlier non-Error String() hazard is fixed and tested), and everything in finally (clock reads, toISOString, items, the log call) is inside one try/catch. The only unguarded expression in the catch is error.name on an Error, which cannot realistically throw.
  • Reserved names (src, level, event, msg, ts) are filtered out of caller fields. The only caller fields in the diff are rung and pressureLevel. The record's own keys (step, startedAt, durationMs, items, outcome, error) collide with nothing a writer sets (stimd/main.ts:775, passes.ts:317-320, 438, 518 all spread ...fields after src/level/event/msg). The writer-shape test shows level: 'debug' survives and pressureLevel is present.
  • debug is in LEVELS (core/state/ndjson.ts:14).

walkMs: written only by refreshMeasure. git grep finds no reader in production code and no readDiskUsageWalkMs remnant, so knip has nothing to flag. readDiskUsage, trustedMeasure and freshMeasure are unchanged; readDiskUsage picks named fields, so entries with and without walkMs read identically (the test checks both, and that the legacy entry still counts as fresh). The guide's span ("just before the install signature and target reads to the end of the walk") matches startedAt at disk-usage.ts:174.

Status JSON and existing records: no existing record's fields changed; no status payload code is touched.

Tests (against the CLAUDE.md rules): throw asserts the same object rethrown plus outcome: 'threw' and error; aborted asserts the outcome and no error field; the writer-shape test goes through the spread the real writers use; dormantCandidates returns an equal result with and without a logger; performance.now and the walk clock are faked. The ladder test asserts the ladder continues after a throwing rung and that an abort still reaches the caller.

Guide and website: step names, pressureLevel, and what items counts match the code for each step (candidates+skips for collect_artifacts; ranked (+kept) for the rank steps; actions/worktrees for the plans; checkouts visited for the three evidence loops; rung actions for rung). I did not re-derive the "up to 16 at high / 22 at critical" counts by enumerating every path.

NON-BLOCKING nit, no change needed to merge: guide/logs.ts:381 says "each disk-usage cache entry also records walkMs". Entries for an absent dependency path (record(…, null)) carry none, and the "no targets" case records a near-zero value. Say "each entry written after a walk".

Labels: adding review: ai-approved. Not adding review: needs-human: the change adds a debug log record and a cache field, no command, flag, setting or default, and no user-visible cleanup behaviour; the record is observability the issue's design lists as item 1, and diagnosing the hours-long ticks is the stated purpose. The one thing a maintainer might want to know is log volume (up to 22 small records per tick at critical, in a file rotated at 8 MiB), which is documented in the guide.

@janicduplessis janicduplessis added the review: ai-approved A fresh AI review found nothing left to fix label Nov 15, 2026
@janicduplessis
janicduplessis marked this pull request as ready for review November 15, 2026 15:12
@janicduplessis
janicduplessis marked this pull request as draft November 15, 2026 15:14
@janicduplessis janicduplessis removed the review: ai-approved A fresh AI review found nothing left to fix label Nov 15, 2026

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: clear at 02bde7a. No defect found. Read-only: I ran no tests or builds.

timedStep in step-timing.ts

  • Success returns result; the throw path rethrows error, the same object, unconditionally. With no logger it returns run(tally) directly.
  • The catch holds only classify(...) and a destructuring assignment of a plain object, so it cannot throw. classify wraps every read of the thrown value (instanceof, .name, .message, String()) and of signal (.aborted) in one try, and its fallback is { outcome: 'threw' }, so a failed classification records threw with no error field. The finally is a single try/catch around the clock reads, items, and the log call.
  • Reserved names (src, level, event, msg, ts) are still filtered from caller fields.
  • Aborted precedence is unchanged: signal.aborted or an AbortError name gives aborted, otherwise threw.

Scope of 02bde7a: git diff fe90b55cc 02bde7a61 --stat shows only step-timing.ts, its test and guide/logs.ts, so the call sites tabulated in the earlier review are unchanged. I checked ladder.ts (rank_dependencies, plan_rung6, plan_worktrees, rank_source_dependencies, plan_history, rung) against origin/main. Each wraps the same call with the same arguments, plus the trailing context.log where the signature gained it.

Regression test: an Error whose name getter throws. It asserts rejects.toBe(odd) (the same object) and that the record has outcome: 'threw'. The log mock is reset in beforeEach, so calls[0] is this test's record.

Guide: "written after a walk ... from just before the install signature and target reads to the end of the folder walk" matches disk-usage.ts (startedAt before treeSignature, walkMs = now() - startedAt, omitted when null).

Label: review: needs-human does not apply. The change adds a debug log record and a cache field, and no command, flag, setting, default or cleanup behaviour. I am adding review: ai-approved and marking the PR ready.

@janicduplessis janicduplessis added the review: ai-approved A fresh AI review found nothing left to fix label Nov 15, 2026
@janicduplessis
janicduplessis marked this pull request as ready for review November 15, 2026 15:32
@janicduplessis
janicduplessis merged commit ecbc48e into main Nov 15, 2026
66 of 67 checks passed
@janicduplessis
janicduplessis deleted the feat/5759-timing-records branch November 15, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli packages/stim-cli, including its README and guide area: core packages/core area: website website/ review: ai-approved A fresh AI review found nothing left to fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant