docs(claude): journal daemon/dashboard learnings (auto-updater race, UTC drift, flaky tests)

CLAUDE.md

@@ -47,6 +47,18 @@ npm test
 - **Weekly limits are rolling windows:** aaronmsachs-max20 was hard-blocked Tuesday but had usable capacity again by Thursday, two days before its stated "resets Jul 20" — a "dead" account can't be assumed dead for a live verify, hence the shim approach.
 - **PTY banner text has cursor-positioning escapes BETWEEN words** — after ANSI-stripping, text reads `Whatdoyouwanttodo?`. Any matching against agent PTY output must normalize whitespace away first.
 
+## Learnings - 2026-08-04
+
+- **`CLAUDE_CONFIG_DIR` isolation gave each agent a private config but they still SHARE one binary.** Every agent's Claude Code therefore believes it is a standalone install and independently schedules its own auto-update against `~/.local/bin/claude -> versions/<v>`. N private updaters, one shared file, no lock. Two fired 250ms apart (`14:05:30.564Z` ruby, `14:05:30.812Z` pearl), both `install_failed`, and left the symlink dangling at an already-deleted `2.1.220` for ~12 minutes. This is the hidden cost of the 2026-07-14 per-agent-config fix.
+- **A dangling binary reads as an agent crash, and that is the damaging part.** node-pty hands back a pid for a dangling symlink, then the child exits 1 having written ZERO bytes — reproduced exactly: `pid assigned: 69035 / exitCode: 1 signal: 0 / output bytes: 0`. The daemon charged the daily crash budget with exponential backoff; `boss` burned 8 of 10, `analyst` hit the cap and HALTED. Fixed both ways (`DISABLE_AUTOUPDATER=1` pin + a binary-unavailable exemption in `handleExit()`), but the diagnostic tell is worth keeping: **exit_code=1 with NOTHING appended to `stdout.log` means the process never started — check the binary before reading agent logs.** A real agent crash always emits something first.
+- **Only agents that RESTART during such a window die; already-running ones are unaffected.** So the fleet looks half-healthy and the failure masquerades as agent-specific. `boss` and `analyst` crash-looped while ruby/pearl/warden kept beating, purely because the latter hadn't respawned. Corollary: `.last-update-result.json` in each agent's `claude-config/` is the forensic record — it timestamps every updater run and its outcome, and is what proved the race.
+- **The same shape had already fired 13 hours earlier** (analyst updated 00:46Z, crash-looped 01:08–01:28Z, HALTED) and went unnoticed because the hang-detector rescued it. Silent recoveries hide recurring structural bugs — grep `restarts.log` for `CRASH: exit_code=1` bursts when auditing.
+- **"Deliberately duplicated" code drifts silently — and the duplicate is where the bug survives.** `dashboard/src/lib/cron-utils.ts` carries a header saying it mirrors daemon logic and "any changes to the core parsing logic should be reflected here as well." PR #21 fixed cron-expression evaluation (local time → UTC) in the daemon and did not reflect it. Result: the daemon fired `0 9 * * *` at 09:00 UTC while the dashboard displayed 09:00 **local** — a 4-5h lie in the UI that no test caught because the dashboard had *two* inline copies of the evaluator and zero cross-implementation tests. Fix pattern: consolidate to one function per side, then add a test that imports BOTH and asserts equality. That test lives in the ROOT tree — a dashboard test importing root `src/` drags root files into the dashboard's lower-ES-target TS program and breaks `tsc --noEmit`.
+- **Tests that assert superseded behavior fail only on hosts where the old and new semantics diverge.** The FM-8 block asserted local-wall-clock cron behavior and passed for anyone running UTC; it failed 1 test on EDT and 2 under `TZ=Pacific/Auckland`. When changing timezone/locale semantics, grep for tests deriving expectations from ambient `Date` getters (`setHours`, `getDay`) and pin them to absolute `Date.UTC` instants instead.
+- **`Intl.DateTimeFormat.formatToParts()` is ~27x slower than native UTC getters** (measured: 9,382ms vs 340ms over 291K calls). It's the right tool for real timezone work but ruinous inside a per-minute scan. `nextFireFromCron` walks up to a year minute-by-minute, so a sparse expression cost seconds of CPU per computation until UTC got a getter fast path. Reach for Intl at the boundary, not in the loop.
+- **Two flaky-test root causes worth recognizing:** (1) a test that leaves an unawaited infinite loop running — when `afterEach` calls `vi.useRealTimers()`, that loop becomes a REAL polling loop that degrades every later test in the file, so *which* test fails looks random; put teardown in `afterEach`, never at the end of the test body. (2) An endpoint doing O(items) full file reads where O(groups) would do — `GET /crons` re-read each agent's whole execution log once per cron (p50 1980ms → 166ms after one pass per agent). Both presented as "flaky perf tests" and were neither flaky nor about the tests.
+
+
 ## Learnings - 2026-08-14
 
 - **A cancelled Anthropic subscription still AUTHENTICATES — the rotation preflight cannot see it.** `aaronmsachs-max20` was cancelled, yet a clean-room one-word opus `-p` ping returned `alive` exit 0, exactly like the three healthy accounts. It only fails on real workloads: hermes' 90k-token / 381-msg request got `rate_limit_error` (`req_011Ce2ms*`) while the 5-token ping sailed through. **The setup-token liveness ping proves the token authenticates, not that the account has capacity** — so `rotate-oauth` will happily rotate *onto* a cancelled account and report success. Corollary for diagnosis: "all accounts ping alive" is not evidence the credential layer is healthy; check a large-request log instead.