investigate: slow production loading and session switching #40

Activity

1. johoja12 commented on Nov 2, 2026

   

   johoja12

   on Nov 2, 2026

   OwnerAuthor

   More actions

   Read-only production investigation after merging PRs #39 and #37.

   Merge/push: PR #39 -> keyboard candidate da68e4e, then PR #37 -> origin/local b497746. The shared checkout fast-forwarded on local without stashing or switching branches; five unrelated assistant WIP files remain. local...origin/local is 0/0. Merged tree matches the previously tested #39 head. No production build/restart was requested or performed, so the new case-group UI is merged but not yet deployed.

   Primary evidence from native client terminal-load timings:

   • 09:57:04: 1,048,507-character tail: 6,268 ms total, 1,483 ms TTFB, 4,674 ms body/JSON, 86 ms terminal parse; server total 62.9 ms.
   • 11:43:58: 5,459-character remote full capture: 3,271 ms total; server 429 ms; browser TTFB 2,981 ms.
   • 11:56:23: 6,713-character full capture: 2,478 ms total, 1,814 ms TTFB, 567 ms body/JSON, 11 ms terminal parse; server total 43.2 ms.
   • Desktop samples in the trace were 92–313 ms, including two 1 MB tails. Thus phone connection/request/body delivery dominates these measured delays; the trace does not identify the exact Wi-Fi/mobile/proxy hop.
   • Recent phone heartbeat shows EventSource CONNECTING across tens of seconds. Source foreground handling invokes _checkSseStale, whose policy checks streams considered connected; already-connecting/reconnecting attempts follow the 15 s connection deadline and up-to-30 s retry backoff. This is a concrete recovery gap worth fixing, not proof that every delayed load has that cause.

   Fresh server-side checks:

   • 20 localhost /api/sessions samples: median 2.8 ms, max 28.1 ms, no failures; initial normal API probes 1–12 ms; HTML mostly 8–31 ms with one ~397 ms outlier.
   • New SSE connections: first init in 2–30 ms (~80 KB), so the local stream bootstrap itself is fast.
   • Local canonical terminal capture ~42–53 ms. Remote worker capture ~440–868 ms; mac-mini ~313–422 ms. Local wrapper frame/mirror reads ~34–50 ms. No session input, activation or resizing was sent by these measurements.
   • Public unauthenticated redirect probe ~5.08 s, including ~2.63 s connect; this is not an authenticated browser load measurement. Outbound GitHub also hit a TLS handshake timeout during the audit.
   • Host compute mostly 65–72% idle, no sustained memory pressure. Elevated system I/O pressure exists, but Codeman and sampled preview processes were not doing disk I/O in the sampled interval. Do not attribute the measured phone latency solely to disk or CPU.

   Separate capacity finding: root reports 100% with ~1.3–1.9 GB available. Docker reports 46.12 GB reclaimable build cache, 53.21 GB reclaimable images and 34.45 GB reclaimable volumes. These are inventories, not cleanup authorization; active/unidentified caches, images and volumes were not removed. Removed only the private node_modules install I created for completed issue #38 tests (829 MB logical); ~1.8 GB available afterward. Source, screenshots, shared dependencies, live sessions and unrelated WIP preserved.

   Next engineering work: (1) bounded immediate foreground/online recovery for already-connecting streams; (2) reduce unnecessary 1 MB terminal tail replay without sacrificing canonical history/scroll behavior; (3) verify authenticated-client transfer compression/resource timings and the user's actual URL/device/network; (4) separately approved disk-capacity cleanup. No performance source change or service restart in this investigation.

   Private supporting native snapshot: /tmp/codeman-slow-native.json. Additional localhost timing sample: /tmp/codeman-prod-slow-api-sample.json. Device/network clarification was requested; no reply yet.