Skip to content

kv-unified: a stale presentation receipt makes the Pareto label count superlinear (extensionTokens is exact in the state key) — resident hard-down after a kv-stable → kv-unified switch #97

Description

@Anarchid

Switching a resident to kv-unified after it has run another folding strategy for a while fails on the first compile with kv-unified exact label propagation exceeded ceiling 100000 at 125796, and every turn after it, so the agent is down until the recipe is reverted. The forest is not the problem: the same store solves in 133 ms / 363 MB under the same 100k ceiling once the old kvunified:presentation-receipt is removed. The receipt is the last presentation kv-unified accepted (2026-09-09 here); everything folded since by kv-stable counts as "extension" in isExtension(), and extensionTokens is the one token quantity in the Pareto label state key that is not bucketed. Labels that differ only in how many of their rendered tokens are "new" never collapse under dominance (dominates() does not look at extension at all), so the live label set multiplies by the number of distinct extension-token sums over the stale region. Raising labelCeiling is not a fix: on the production store the propagation was still climbing at 1.64M labels and 5.3 GB RSS.

Minimal reproduction (no store needed)

ParetoKvUnifiedPolicySolver on a synthetic chain forest from test/adaptive/harness.ts (buildChronicleWithChain, mergeThreshold 2, varied raw sizes), buckets 100/100/100, maxTokens = 75 % of raw, and a presentation whose leaves map covers only the first covered chunks (repHash: raw:<id>, level 0). Node, ulimit -v 4000000, whole table in 0.47 s / 183 MB:

chunks summaries covered stale leaves labelsCreated maxLabelsPerState ms
20 18 20 (fresh) 0 5,612 635 6
20 18 10 10 14,701 2,047 15
20 18 0 20 12,332 1,579 11
24 22 24 (fresh) 0 14,256 1,388 17
24 22 12 12 60,791 7,313 52
24 22 0 24 41,714 4,384 28
28 25 28 (fresh) 0 23,630 2,355 26
28 25 14 14 178,797 25,785 198
28 25 0 28 78,031 7,296 60

The stale/fresh ratio grows with forest size (2.6× → 4.3× → 7.6×). A half-covered presentation is the worst case because extension is then only partly correlated with rendered tokens, so the two dimensions multiply; fully stale (covered = 0) degrades to "exact tokens instead of bucketed" (3.3×). Script: a ~40-line circle.mjs, happy to attach to the PR.

Production instance (Lynx Knowledge Resident, conhost 0.8.0 / CM 0.8.0 @ bf41338)

  • 2026-09-09: switched to kv-unified, solved fine (planned=259049 … solver=kv-unified), receipt written. 2026-09-10: reverted to kv-stable (unrelated caution). 2026-09-15 16:03Z: switched back. First compile: [inference-failed] … exceeded ceiling 100000 at 125796; nothing in headless.log; every turn thereafter. Reverted 16:23Z.
  • Offline on the 16:11Z snapshot (241 chunks, 318 summaries, 4 levels), same recipe: ceiling 100k → 125796 (identical to prod); 300k → 486795, 3.2 s, 1.7 GB; 1M → 1643080, 10.8 s, 5.3 GB, still climbing. Receipt state set to null → COMPILE OK in 133 ms, 363 MB, 100k ceiling. Two days after the receipt (Sep 11 snapshot, 159 chunks): 1.4 s / 1.6 GB with the receipt vs 0.1 s / 318 MB without — already 5× memory.
  • The receipt's leaves covered roughly the first half of today's live leaves: the worst-case row above.

Why this is a migration bug, not just a scaling one

The receipt chain is persisted under kvunified:presentation-receipt and loaded whenever foldingStrategy === 'kv-unified', regardless of who presented last. kv-stable (and flat-profile) write autobio:resolutions but never touch the receipt, so after any switch-away-and-back the receipt describes a presentation that is not the previous one. Continuity is then measured against the wrong baseline too (the head/leaves from days ago), not only the K term. The same shape arises from a restore of an older kv-unified session, or a large history import.

Suggested fix (PR to follow)

  1. kv-unified-pareto.ts stateKey: bucket extensionTokens by tokenBucketSize like renderedTokens, and omit it entirely when the cache is not relevant (immutable prefix mismatch / no cache) — it feeds only cacheChurn()'s avoidableRecompute, which is zero-weight in that case. Exact mode (tokenBucketSize: 0) keeps it exact.
  2. autobiographical.ts loadPersistedState: when a non-kv-unified folding strategy loads a store that carries a receipt, supersede it (null + one warning line), so a later switch back starts from an empty chain — one cold-cache turn, which the runbook already expects, instead of a hard-down.
  3. Tests: the circle above as a bounded-growth assertion (stale ≤ ~1.5× fresh), and an integration test that a kv-stable restart clears a persisted receipt.

Sibling: #96 (what happens while a compile keeps failing).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions