feat(cli): lazy background worker — daemon-less consolidation, metered and detached

[rabbiveesh]

Ships the lazy-background-work proposal: the CLI is its own scheduler — no daemon, no cron. Two features plus two adversarial review rounds each.

The worker tier (crates/memory-cli/src/worker.rs)

• Trigger on writes: remember / promote / onboard --commit count toward a due-check (default: 5 writes + 10-min throttle, 24-h ceiling). When it trips, the CLI re-execs itself as a detached hidden mem __worker (own process group, stdio → .worker.log, rotated past ~1 MB).
• Work: consolidate bounded by the pending backlog (capped at worker.max_targets); dream piggybacks weekly. Scope = the triggering invocation's active scope (the spec's "trigger is the user already being in the right place"), never store-wide.
• Serialization: zero-dep pidfile lock with dead-holder//stale/future-mtime steal; a .worker-state.lock micro-lock serializes state read-modify-writes (µs hold, ~250 ms spin, then proceed-unlocked — the foreground can never hang).
• Honest state: a failed or empty-scope pass keeps the backlog so it retries; a capped pass names its tail (capped at 12; cover the rest: mem consolidate --limit 40); recall/ask surface a one-line 🛠 background … summary exactly once.
• Controls: mem worker (status), mem worker --now (forced pass), MEM_NO_BG=1 / worker.enabled=false (kill switch), knobs under [worker] in config.toml.

Instrumentation + backgrounded mem consolidate

• Metrics ledger .worker-metrics.jsonl: one JSON line per pass — what happened (job, trigger, pending, targets, new edges, review flags, probes/bridges, ok/error) and what it cost (wall ms, chat calls + chat ms, embed texts). Costs come from new atomic counters in memory-embed, the single choke point all LLM traffic flows through; attempts count even on failure. mem worker shows per-job totals + the recent tail.
• mem consolidate detaches by default, running the same locked/metered worker body with the caller's --limit (trigger=cli, no dream piggyback). --fg (or a disabled tier) keeps the synchronous path — also metered, drains the backlog it covers. A detached consolidate that loses the lock to an in-flight worker surfaces consolidate SKIPPED … rerun: mem consolidate --limit N instead of vanishing.
• mem dream stays foreground but is metered, and resets the weekly piggyback cadence.

Review rounds (write→review→fix loop)

Each feature commit was followed by an adversarial review + verification round; the fix commits document what they caught, including: a failed pass permanently draining the retry backlog, a lost-update race on the state file, a clamp panic on max_targets=0, --limit 0 refactor regressions, a clock-skew hole in stale-lock detection, counter-test races, and the silent lock-held drop above. Accepted tradeoffs (store-global counter imprecision, double-steal race, log-rotation race) are documented in the design doc's "As built" section.

Testing

• 31 unit tests in the touched crates (state round-trips, due-check matrix, lock exclusion/steal, drain semantics, metric ledger round-trip + aggregation, counter deltas); full workspace green.
• Live smoke tests: kick-on-threshold, concurrent-kick lock exclusion, kill switch, failed-pass retry, one-time surfacing, detached + --fg consolidate, lock-held skip surfacing.
• Not exercised here: a happy-path pass against a live ollama (this environment has none) — the LLM path reuses the pre-existing consolidate/dream machinery unchanged, but a real mem remember ×5 on a machine with ollama is the last-mile check.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WMTLGECmHZB6HKXADbWzsD

---

Generated by Claude Code