Skip to content

Fix: cairn prime --budget-bytes bounds only triggers, payload runs 2.5x over (crn-s3749) - #136

Merged
quad341 merged 2 commits into
mainfrom
fix/crn-s3749-bound-total-prime-payload
Aug 28, 2026
Merged

quad341 merged 2 commits into
mainfrom
fix/crn-s3749-bound-total-prime-payload

Conversation

@quad341

@quad341 quad341 commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Fixes crn-s3749.

The defect

--budget-bytes was enforced only against Items. TopicCounts was exempt as part of the "index view", justified by this comment:

...a pure in-memory map reduction over the already-loaded visible set, no new query and no I/O (NFR-2), so this is as cheap as TotalVisible and independent of what the byte budget lets into Items.

That reasoning is about compute cost, and it is still correct — NFR-2 holds. It was being used to exempt the field from a payload byte budget. Cheap to compute, expensive to emit.

Measured on the mayor's live store, 287 entries visible, default flags:

field chars %
topic_counts 18,350 88.2%
triggers 3,466 16.7%
total 20,807 vs an 8,192-byte budget

At that shape topic_counts is not an aggregation — 278 keys, 277 with count == 1. It is the topic-key list wearing an aggregation's shape, gaining a row every time an agent learns anything. Payload was linear in distinct-topic count with nothing in the path to bound it.

This matters because prime is a SessionStart hook: the overrun is paid on every session, forever.

The fix

  • The index reserve (budgetBytes/4) is carved out of the item budget before itemizing, so items cannot starve the index to nothing. Carved out rather than added on top — adding it lets items spend the full budget and pushes the total over, which is the same bug one line later. (I wrote it that way first; the test caught it at 2459/2048.)
  • Whatever items leave unspent flows back to the index.
  • The index is ranked count-descending before the cap, so topics holding more than one entry — the only rows carrying information Items does not already have — survive ahead of singletons. Kept rows still print key-ascending, preserving the ordering callers had.
  • TopicCountsTruncated plus a Warnings line report what was dropped.

A silent cap would be worse than the overrun it replaces, so it says what it dropped. A short index that looks complete is the same false-success shape the store warns about elsewhere.

The aggregate scalars (TotalVisible, fresh/stale/unknown) stay unconditional — they are O(1) bytes, and bounding them would turn an honest overrun into an undercount.

Verified on the store the bug came from

before:  20,813 chars
after:    5,068 chars
         triggers 2,588 · topic_counts 2,333 · 38 listed, 240 dropped
         total_visible still 287, fresh/stale/unknown still 4/15/268
warning: topic index truncated by the byte budget: 240 of 278 topic keys
         not listed (aggregate counts still cover all 287 entries)

Full repo suite green.

One test is amended, not deleted — please review this specifically

TestPrimeIndexBreakdownIndependentOfByteBudget asserted at budget=1 that every topic was still listed. That assertion was the defect — it pinned the exemption. It now pins what FR-1/NFR-2 actually needed (cost independent of entry content size; scalars covering the whole visible set), adds the report-what-you-dropped guarantee, and keeps a generous-budget case asserting the full breakdown still lists everything when it fits.

If crn-3476 intended the enumeration itself to be unconditional, then this is a deliberate contract change and worth an explicit ruling rather than a silent merge.

Not changed

defaultPrimeBudgetBytes stays at 8192. The effective content of that budget changes here, so the default may deserve revisiting — but that is a sizing decision, separate from the accounting bug.

Same measurement reported upstream in the Memory Beads discussion: gastownhall/beads#5877 (issuecomment-5443702739).

cairn-reviewer-scratch added 2 commits August 27, 2026 12:47
…5x over (crn-s3749)

budgetBytes was enforced only against Items. TopicCounts was exempt as part of
the "index view", justified by a comment reasoning that it is "as cheap as
TotalVisible ... independent of what the byte budget lets into Items". That
reasoning is about COMPUTE cost -- NFR-2's no-I/O requirement, which still
holds -- and it was being used to exempt the field from a PAYLOAD BYTE budget.
Cheap to compute, expensive to emit.

Measured on the mayor's live store (287 entries visible), default flags:

    topic_counts   18,350 chars   88.2%    278 keys, 277 of them count == 1
    triggers        3,466 chars   16.7%
    TOTAL          20,807 chars            against an 8,192-byte budget

topic_counts is not an aggregation at that shape -- it is the topic-key list
wearing an aggregation's shape, growing by one row every time an agent learns
anything. Payload size was linear in distinct-topic count with nothing in the
path to bound it.

This makes budgetBytes bound the whole itemized payload:

- A reserve (budgetBytes/4) is carved OUT of the item budget before itemizing,
  so items cannot starve the index to nothing. It is carved out rather than
  added on top -- adding it would let items spend the full budget and push the
  total over, which is the bug in a different place.
- Anything items leave unspent flows back to the index.
- The index is ranked count-descending before the cap, so topics holding more
  than one entry -- the only rows carrying information the per-entry Items list
  does not -- survive ahead of singletons. Kept rows still PRINT key-ascending,
  preserving the ordering callers had.
- TopicCountsTruncated and a Warnings line report what was dropped. A silent
  cap would be worse than the overrun it replaces: a short index would be
  indistinguishable from a complete one.

The aggregate scalars (TotalVisible, fresh/stale/unknown) stay unconditional.
They are O(1) bytes, and bounding them would convert an honest overrun into an
undercount.

Verified against the same live store the bug was measured on:
20,813 -> 5,068 chars, 38 topics listed, 240 dropped and named in warnings,
total_visible still 287.

TestPrimeIndexBreakdownIndependentOfByteBudget is AMENDED, not deleted: its
budget=1 case asserted the old exempt-forever contract, which was the defect.
It now pins what FR-1/NFR-2 actually needed -- cost independent of entry
content size, scalars covering the whole visible set -- plus the new
report-what-you-dropped guarantee, and keeps a generous-budget case asserting
the full breakdown still lists everything when it fits.
CI lint failed on the previous commit with four issues, all introduced by it:
cyclop (Prime at 16 > 15), intrange x2, and unparam on the test helper. Fixing
cyclop by extracting applyIndexBudget then tripped funlen (81 > 80), because
Prime was already at the limit before this branch touched it.

- applyIndexBudget: the index-budget fill plus its report-what-you-dropped
  warning, which belong together anyway.
- orderByPriority: the itemization comparator. Load-bearing and worth naming --
  byte-budget truncation walks this slice in order and stops at the first entry
  that does not fit, so the ordering decides what survives a cap.
- Test helper drops its always-200 parameter for a named const, and the two
  index loops use Go 1.22 range-over-int.

No behaviour change: verified against the same live store, 5,068 chars, 38
topics listed / 240 dropped, total_visible 287. golangci-lint run ./... reports
0 issues and the full suite is green.

Lesson worth recording: golangci-lint was installed locally the whole time and
I pushed without running it, so CI found what a 30-second local command would
have.
@quad341
quad341 merged commit d9c6053 into main Aug 28, 2026
2 checks passed
@quad341
quad341 deleted the fix/crn-s3749-bound-total-prime-payload branch August 28, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant