Skip to content

kv-unified: a pre-flight CanonicalForestError takes the agent down but leaves compression/merge production running (and billing) — observed 19 rejections × 5 min on a live resident #96

Description

@Anarchid

If a kv-unified resident's store fails canonicalization (strict flags + one non-contiguous summary), the agent stops answering — every compile throws before the solve — but the autobiographical merge ladder keeps executing merges on every tick, each one a real model call, indefinitely. Nothing in the failure path pauses production, and the only signal is the repeating Canonical summary forest rejected … line in the log. An operator who does not happen to be watching the log pays for a resident that is down. This is a report from a production incident, filed with the evidence we still have; the detailed logs have since rotated, so the "merges kept running" part is as recorded at the time, not re-verified today.

What we saw (2026-09-09, Lynx Knowledge Resident, conhost 0.8.0 / CM 0.8.0 @ bf41338)

  • Recipe switched to foldingStrategy: "kv-unified" with treeifyNonContiguousSummaries: false and preserveGapBearingSummaries: false (the fail-closed default).
  • From 17:40Z: 19 consecutive Canonical summary forest rejected 2 structural issue(s): summary L2-4 owns non-contiguous live leaves, five minutes apart (the heartbeat cadence), no accepted turn.
  • During that window merge executions continued and produced summaries (the archive audit shows L4-85 minted 07:11Z that day and further nodes after the flag flip; the operator note at the time reads "merges still running and still billing").
  • Recovery: preserveGapBearingSummaries: true hot-edited into the recipe + restart; next turn solved (planned=259049 actual=259049 moves=1517 solver=kv-unified).

The two offending summaries were minted on 2026-09-02 by CM 0.6.3's merge grammar (see the sibling issue on enqueueMergeForRange); the point here is not the scar, it is what the system does while the scar is fatal.

Why it matters

  • Strict canonicalization is the default, and every inherited store we have looked at so far (Mythos: 67 → 54 gaps after repair; this one: 2) has some gap. So "kv-unified switch on a real store with default flags" reliably produces exactly this state.
  • The failure is per-compile and permanent (no self-heal), so the cost is unbounded in time.
  • The rejection happens before any solve, so there is no plan, no cache marker, nothing that would justify continuing to deepen the pyramid.

Suggested behaviour

  1. When CanonicalForestError is thrown from the kv-unified pre-flight, set a "compile hard-failed" latch on the strategy that pauses speculative production and merge execution (the existing merge-quarantine klaxon machinery is the natural home — a repeating alarm plus a paused ladder, cleared when a compile succeeds). Compression of the raw tail could arguably continue (it is bounded and protects the budget); merges above L1 should not.
  2. Surface the structural issues once, with the offending ids, as a console.error + trace event, rather than the same one-line throw every five minutes; the issues[] array on the error already has everything needed.
  3. Add a read-only preflight (the migration guide's "improvement A2") so operators can count non-contiguous reachable summaries before flipping a live recipe. The chunk → L1 → parent chain + per-summary leaf-index contiguity check is ~40 lines and is what we used to audit this store after the fact.

Repro without a production store: build any archive where one L2's leaf set has a hole (e.g. the fixture from test/merge-contiguity.test.ts "any hole containing live messages splits merge runs", but persisted as a parented L2), run the kv-unified strategy with both structural flags false, and watch tick() keep dequeuing merges while compile() throws.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions