c2c stalls ~10s via codex during host IO waves (fsync-heavy hook path, idle pi-c2c snapshot churn, stale broker debris)

[XertroV]

Symptom: c2c invocations from codex visibly take ~10s. Reproduced independently: a repo .mcp.json c2c MCP server missed its 30s client startup timeout at ~02:40 AEST while the same handshake later completed in 0.95s.

Root cause: bursty host-wide IO saturation. Measured in one window: PSI io some 18-21% sustained, full 6-9%; nvme0n1p5 at ~1.0s average latency per I/O (only ~3MB/s throughput — latency-bound, not bandwidth-bound); btrfs flush kworkers in D-state; find ~/.c2c/repos 4.9s. Codex runs c2c hook codex on UserPromptSubmit + PostToolUse + SessionStart + SessionEnd with timeout = 10, and the hook path does several atomic (fsync) writes + registry reads + cross-broker alias scan — so a wave multiplies into the full 10s budget per prompt and per tool call.

c2c itself is fast when the disk is calm (all measured this session): whoami ~25ms, hook codex 172-572ms real broker / 97ms hermetic, MCP handshake 0.95s.

Contributors measured during waves

• kitty scrollback writes ~2.2MB/s (TUI output churn from many agent panes — hermes ×3, codex ×2, claude, pi, chromium)
• pi-c2c idle snapshot churn: idle Pi session in ~/src/amaroo (c2c oc-plugin stream-write-statefiles, v0.14.4) writes ~37KB every ~5s ≈ 26MB/h of fsync fan-out, 63MB in its first 16 min, with zero activity
• connector-state sweep: 161/229 ~/.c2c/repos/*/broker/connector-state.json written within 30 min (writer unidentified — long-lived c2c daemons had near-zero cumulative writes)

c2c-side aggravators / housekeeping debt

• stale untracked repo .mcp.json ("tmp-restore-hook", Jul 19) pointed every MCP client at a dead /tmp broker → 30s timeouts + ghost alias; moved aside this session (.mcp.json.stale-tmp-restore-hook.bak)
• 229 dirs in ~/.c2c/repos/ walked by the B188/B191 cross-broker scan on every auto-register
• ~/.c2c/locks/ ≈ 20k+ session-reg-*.lock files (1.1MB dirents) since July, never pruned
• stale kimi notifier daemon (3.5d old, cwd = deleted worktree); prior notifier log hit 52MB

Proposed follow-ups

1. Prune ~/.c2c/repos test/worktree debris + stale session-reg-*.lock files (add a c2c dev housekeeping command or extend c2c dev instances clean-stale scope).
2. pi-c2c: gate/throttle stream-write-statefiles fan-out when the session is idle (skip no-change snapshots); rebuild the pi extension against current c2c (it runs 0.14.4).
3. Hook fast-path: when the session is already registered locally, skip the cross-broker alias scan; consider PSI-aware backoff so a wave does not burn the whole 10s hook budget on registry writes.
4. Identify the connector-state sweep writer; make it incremental (skip unchanged brokers).
5. c2c doctor: add an IO-pressure check (PSI) so "slow c2c" triage starts with the host, not the binary.

Full evidence + timings: .collab/findings/2026-09-02T16-55-00Z-grok-c2c-10s-stalls-host-io-waves.md

[XertroV]

Correction — primary host-side hog identified: plasmashell ("Claude Usage" widget).

After this issue was filed, the previous plasmashell instance turned out to have been SIGKILLed at 02:43:57 (status=9/KILL, no coredump, no oomd) — which is why the per-process IO ranking in this issue (sampled 02:50+) missed it. The journal gives the full picture:

• The old plasmashell ran a "Claude Usage" plasmoid in a credential-refresh spawn loop: startProfileRefresh / readRefreshCredentials shelling out (cu-id=<profile>; cat ~/.kimi-for-coding, ~/.mmx/config.json, ~/.minimax, ~/.claude-w/.credentials.json, …) across 12 uiProfiles (codex-w, grok-1, kimi-default, minimax-default-*), generation counter reaching 9245 over its 3d10h life ≈ one full refresh cycle every ~30s, continuously.
• systemd epitaph: Consumed 1d 19min CPU over 3d 10h wall clock, 2.5G memory peak, 536.2M swap peak — ~23% of a CPU since boot, 24/7, on top of per-cycle exec/config/journal churn.
• The restarted instance resumed the same loop: gen counter back to 34 within ~17 min (same ~30s cadence, 200 journal lines), ~10% CPU. The churn is live again; it has not yet re-degraded to the old instance's state.

So the attribution ranking is: 1. plasmashell Claude-Usage refresh loop (primary, bursty), 2. kitty TUI scrollback churn (~2.2MB/s during waves, secondary), 3. idle pi-c2c statefile snapshots (~26MB/h, secondary). The core diagnosis is unchanged: bursty host IO saturation × c2c's fsync-heavy hook path with timeout = 10 = visible 10s stalls; c2c operations measured fast whenever the disk was calm.

New follow-up for the widget owner (host side): make the plasmoid's refresh event-driven or interval-throttled (30s → minutes), cache credential reads across profiles, and cut per-cycle journal verbosity. Watch for it re-degrading: the old instance needed a SIGKILL after 3.5 days at 23% CPU / 2.5GB peak.

[XertroV]

Follow-up 2 fixed in pi-c2c — commit 559e2d4 ("perf(statefile): gate idle snapshot writes behind change + keepalive").

Idle pi sessions no longer rewrite their statefile on every heartbeat tick:

• Writes are gated on the snapshot's semantic fingerprint (identity + status, excluding the always-changing timestamps). The 10s heartbeat tick is now a cheap in-process check, not a disk write.
• While idle, at most one liveness keepalive write per 60s (configurable idleKeepaliveMs, clamped to at least the tick interval). Status transitions, alias updates, and child reconnects still write immediately.
• heartbeat_interval_ms in the snapshot now reports that honest write bound.
• Also removed a pre-existing double-write of the first snapshot when the multi-key hub spawns its child on attach.

Verified against the real c2c binary (35s probe, tick 1s, keepalive 20s): exactly initial write + one keepalive + one record() write — 3 writes where the old code performed ~32. Steady state: idle churn drops 6× (1 write/60s vs 1/10s). pi-c2c suite: 499 tests, 494 pass, 0 fail; tsc clean; independent review pass found no blockers (its minor findings — keepalive clamping, single clock sample, doc supersession notes — are included in the commit).

Live pickup: pi loads the extension from ~/src/pi-c2c directly (~/.pi/agent/settings.json → ../../src/pi-c2c), so new pi sessions get the fix immediately; already-running sessions (e.g. the idle amaroo one) pick it up on restart or extension reload.