Skip to content

fix(compression): progress past quarantined chunks; transient failures never spend the ladder; pace the lane - #138

Merged
nissa-seru merged 68 commits into
anima-research:mainfrom
nissa-seru:fix/55-quarantine-tick-progress
Oct 19, 2026
Merged

nissa-seru merged 68 commits into
anima-research:mainfrom
nissa-seru:fix/55-quarantine-tick-progress

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: compression keeps going while its provider fails in part. A quarantined chunk no longer takes every tick. A deterministic rejection (context_length, invalid_request, safety, unsupported) decides at once, and an L1 request's answer is kept so it isn't sent again. Any other failure of a call is transient. After a subject's first, each counts only when other compression has landed on the same membrane instance and model since the subject last failed, and the eighth decides it; until then each backs that route off, 5 s doubling to 10 minutes and never shorter than a wait the provider stated. So an outage decides nothing, and a request that fails on its own while other work lands is decided. Failed work goes behind work that hasn't failed, and the quarantine status names each stuck span.

Problem

Two parts of #55 are addressed here.

1. Tick starvation. An already-quarantined raw chunk consumes a maintenance tick without making a provider call. So repeated queue rebuilds can keep later compression and merges waiting indefinitely.

2. Retry policy. #55 also discussed retries more broadly. A Connectome household incident on 2026-10-07 made it concrete:

  • A rejected canonical request was sent over and over. A request-attributable rejection of the canonical L1 request (an invalid_request 400) was rethrown on every rebuild, with no accounting. The identical request was sent again and again.
  • Outages quarantined viable work. A transient failure inside the refusal ladder (capacity, transport, credentials, anything unclassified) was recorded as that rung's provider_error, and the ladder carried on. So an outage could exhaust a viable request family into quarantine.
  • Nothing paced the lane between ticks. A 429 asking for 30 s was retried on every maintenance pass.

Changes

Tick progress

  • Skip notification. The existing durable request-family gate reports a quarantine skip through an optional per-call notification. The protected compression method and the readiness promise keep Promise<void>.
  • Queue scan:
    • The tick snapshots the initially queued distinct source-ID keys, then resolves each against the live queue before processing it. Rebuilds can replace objects, change indexes, or withdraw work, but they can't make the scan spend its bound on a source it already skipped. New identities wait for the next tick.
    • Stale and already-compressed queue entries are cleaned even when no valid candidate remains. Fresh attempts, refusals included, still consume the tick.
    • A held chunk still ends the tick's scan, as on main (feat: transient compression holds for provisional messages (for AF #159) #120).
  • Yielding and ownership. A macrotask yield runs between quarantine checks, inside pendingCompression. Branch ownership is rechecked after awaits. Through the yield after a skip, compressionWorkQueued, which OverBudgetError.clearedBy reads, doesn't count the skipped chunk as compression in flight, since the skip marks that call's own pending work. A quarantined chunk whose call goes out afresh, under a new request family, still counts.
  • Warnings. One warning per source-chunk hash on each branch the strategy loads.

Classification

A thrown failure is a rejection, deciding its rung at once, only when it is deterministic: context_length, invalid_request, safety or unsupported, whatever value is thrown. Aborts are neither a rejection nor a failure: they pause nothing and spend nothing. Anything else is transient, and is evidence about its subject only by the company it keeps.

The evidence rule

  • When a failure counts. A subject's first transient failure starts its run, and each later one counts against it only when some other compression has landed in its domain since the subject last failed there. The domain is the membrane instance the call went through and the model it named, as this process knows them: another model's or another route's landings say nothing about this one. A landing is a compression the provider answered and whose result is written: an L1 pushed with its chunk record marked, or a merged summary written. A thin-chunk stub, an adopted L1, a span found covered and a skipped merge entry made no call, so none is a landing. Failures of any class count.
  • The subject. An L1 run belongs to the chunk's rung: its sources by id and content, and its curve label, with the stripped retry after a carrier rejection as its own. It doesn't belong to the bytes of its request, which change whenever another L1 lands. A source edited in place starts a new run, and a late failure of the old content leaves the new content's run alone. A content's run goes on across the chunk's request families, and an answer or a decision ends it. A merge's run belongs to the merge.
  • What the 8th decides (COMPRESSION_PERSISTENT_FAILURES): the first failure, and seven more, each of those after a landing. In L1, the rung's outcome is persistent-<class>, like a rejection: source-only-final once when configured, and otherwise exhaustion and quarantine, which only an operator, the chunk compressing, or the paid-off sweep clears. In the merge lane, it spends one bounded-policy attempt, whose retry halves the recall budget. Either way, the deciding failure pauses nothing.
  • What is kept, and where. Runs, like the landing counts they compare against, are this process's evidence, kept in memory (compressionFailureRuns, mergeFailureRuns) and dropped with the failure orders when their chunk or merge is no longer pending. A restart starts every count again. The cost is stated in the changelog: a process that restarts before it collects eight counts keeps retrying a request that fails on its own while other work lands, paced, with no decision. Routing below membrane, such as a proxy that picks a backend per call, is invisible to the rule.
  • A deferral by the caller's provider admission made no call, so it counts nothing.
  • A strategy that joined another instance's call for the same request, and gets back its transient failure, made no call either. It counts nothing and adds no backoff, but it takes the provider's stated wait from that failure, so it doesn't send the same request inside it, when it goes through the same membrane: a strategy on another route was refused nothing and takes no pause. Either way the failure comes back to it, and tick puts its chunk back as after a transient failure of its own, at the back of its queue, to be asked again on its own route. A deferral by the caller's provider admission made no call, so it gives the joining strategy no pause; its next attempt asks the admission itself. A split-stitch window wait is the other instance's own (see the L1 ladder), so it holds nothing in the joining strategy, whose next attempt meets its own window.
  • A merge entry skipped with no call is dropped from the queue as before, but it is no landing and no answer. That covers a stale entry, a missing source, and a crossed merge refused.
  • Odds. If each call failed independently 30% of the time, a given eight attempts would all fail about 0.007% of the time (0.3⁸). A decision also needs a landing before each failure but the first, so it is rarer still.

Queue order

  • A chunk that failed goes behind the chunks that haven't, least recently failed first, so they run once the pause ends. The order lives in the process, and every rebuild applies it, so an inbound message doesn't put a failed chunk back at the head. A rebuild during the call re-decides the chunk's eligibility, and work it withdrew stays withdrawn.
  • A quarantined chunk a tick skips goes behind the chunks and merges that haven't failed, so its request isn't built ahead of fresh work. The queue is sorted once its order is set, so a rebuild during the skip's yield doesn't put it back ahead.
  • A failed chunk gives its turn to a queued merge that hasn't failed since. A lone failing chunk no longer keeps every merge, and the landings they bring, from running, and a failing chunk and a failing merge take turns.
  • After a deferral, an abort or a host's withdrawal (CompressionWithdrawn), which say nothing, the chunk keeps its place: at the head, or among the failed if it failed before. Queued again meanwhile without a rebuild, as a plan's demand queues it at the tail, it moves there too, once. The failure classifier takes a withdrawal as an abort, so every rung rethrows it with nothing recorded, and the lane doesn't pause for it. CompressionWithdrawn's doc and its changelog fragment say so.
  • A preview (a dry select) leaves the queue, its generation, and the failure orders, runs and window waits as they were.
  • A chunk a plan demands (enqueueL1ForRange) goes ahead of the chunks that failed or were skipped, as every other place that adds to the queue orders it.
  • A failing merge goes behind the other queued merges.
  • A merge spanning a chunk that was put back after a failure waits for it. contiguousMergeCandidates doesn't treat a hole that a pending retry will fill as stranding, so the run before it isn't consolidated early at two members. A chunk's retry fills its hole with an L1, and a merge's with a summary of the merge's level, so a failing merge holds no hole among the summaries it merges, which its summary leaves in place. Past the speculative cap (maxSpeculativeL1s), such a run waits as it does below the cap. Since fix(cap): the speculative cap holds L1 production only while an L1 merge is queued #243, compressing chunks into L1s waits there only while an L1 merge is queued, so the retry still comes once the count is back under the cap or no L1 merge is left queued. A run left by a chunk that a rebuild re-cut names no pending chunk, and holds nothing. After a restart, a span that is still failing holds no merges until it fails again.

Pacing

  • After a transient failure in L1 or a merge, no compression call is made until both constraints have passed: the lane's own backoff (5 s, doubling to 10 min) and the provider's stated retryAfterMs, which is never shortened. A stated wait is a non-negative retryAfterMs; NaN states none.
  • An unrepresentable stated wait is an explicit hold.
  • Scope. The pause belongs to the membrane instance and to the model the failed call was dispatched with.
  • What ends or changes the pause:
    • Any provider answer to one of the lane's calls resets its route's pause, the pause of the membrane instance and model it came through, a merge refusal included, whatever the branch: the pause belongs to the route. Another route's answer leaves it. A merge entry skipped with no call is no answer.
    • The new public releaseCompressionPause(model?) releases the provider's stated wait but keeps the lane's own backoff and failure count. agent-framework calls it when an operator releases a provider wait.
    • An admission deferral (providerAdmission: 'deferred', a held model refused with no call made) counts no failure and adds no backoff. The admission stays the authority: its answer is cached until the deadline it gave or for at most 30 s, and the next otherwise-eligible tick asks it again. Asking makes no provider call, so a condition that clears with no release reaching this instance is observed. A deferral never shortens a wait a provider stated after a real call.
    • Warnings. An admission hold warns and logs compression:lane-held-by-admission once per hold, and again when its deadline changes, rather than on every 30 s recheck.
    • Readiness names a pause only while it binds the configured model. It has no context to name the membrane, so a pause bound to another instance shows until the next tick drops it.

The L1 ladder

  • Durable family progress is kept in the branch-scoped slot autobio:compression-family-progress, separate from the quarantine ledger.
    • Every deterministic answer a request gives is written and synced before the next rung's call, so neither a resume nor a restart pays for it again.
    • The slot is one record shared by every strategy on the store and namespace. Reads are fresh, and each write is a read-merge-write of its own family. It keeps another writer's longer run for the same rung in the same domain, and drops a run once its rung has answered.
    • Reuse across routes. A recorded answer belongs to its request (the same bytes, the model included), and it is reused on any route and in any later process. That is what keeps it paid once: an answer scoped to the route would be asked again after every restart. After a known route change, an answer only the old backend would give still stands for its family, such as an unsupported from a missing feature or a proxy's size limit read as invalid_request. So the family can exhaust where the new route would have answered its canonical. clearCompressionRefusalQuarantine() asks it again from its canonical, with its recorded answers dropped. A family whose ladder is still in progress keeps its answers, and may mint from a lesser rung than the new route would have answered.
    • When progress is dropped:
      • when the span is covered: at mint, at adoption, by a thin-chunk stub, from a sibling's coalesced result, or when the dedup guard finds every message already summarized;
      • at exhaustion, and when an operator clears that quarantine, the progress before the clear event, so a crash between them leaves the quarantine standing rather than a cleared family with its old answers;
      • by a sweep from the maintenance tick, at most every 10 minutes and independent of the quarantine alarm. The sweep drops progress whose chunk record is compressed. It drops progress whose chunk record is missing only when every message of the span is covered by an L1 or one of them is gone from the store. And it drops progress nothing has advanced for 30 days.
    • No write-back. A ladder never writes back a family whose progress was dropped while it ran (minted, cleared or swept), nor quarantines it: once progress it found or wrote is gone from the slot, it writes no more, and its exhaustion, which asks under the ledger's lock (the one an operator's clear drops progress under), records nothing. The opt-in compression log then says compression:family-dropped, and the next attempt pays afresh. In one process only a family's own ladder writes its progress, and only while its branch is current, and a second ladder on the same family, namespace and branch joins the first (compressionInFlight) instead of running. A drop before a ladder first sees its family's progress isn't seen: where sibling strategies with different families share a store, one's mint of a span can leave the other's ladder to write progress, or an exhausted record, for a span that's covered. The record's one-shot alert goes out with it, and the record stands until the quarantine alarm's paid-off sweep finds the span's chunk compressed or gone, or an operator clears it. A family that another process has already quarantined under the same key keeps that record: the exhaustion reads the key first, writes nothing, forgets the family's progress and logs neither a drop nor an exhaustion. Across processes nothing else coordinates.
  • Each rung keeps its own identity: the canonical, its no-tools and tools-less retries, the plain-prose retry (now a rung of the ladder), the recall variants, the hoists and source-only-final.
    • A transient failure stops the ladder at its rung, and a later tick resumes there.
    • The degraded text-only retry after a carrier rejection is entered only for a classified rejection, and that rejection is recorded under its own identity.
  • A canonical rejection skips the variants and the canonical hoist. It goes to source-only-final once when configured, and otherwise exhausts the family.
  • Split-stitch stays one atomic retry unit: a transient failure inside it re-runs the whole rung later, paying again for the pieces it had settled. Its call window (compressionSplitMaxCallsPer10Min) is a local throttle, not an answer. A rung starts only when the window has room for a whole attempt. Otherwise its chunk waits, with nothing recorded, counted or paused, while the tick goes on to the work behind it, and the first tick after the window has room takes it up: the strategy schedules no tick of its own. A piece's call that the caller's admission defers takes no place in the window. An attempt that alone needs more calls than the window allows still ends the rung, as the per-attempt cap does. The window is per strategy instance, so only the instance whose window it is holds its chunk. A provider error that ends the rung, a piece's rejection or the rung's decision, is recorded with its bare type, as every other rung's is.
  • Every exit that exhausts a family records the same thing: every outcome the family noted, in the order noted, an earlier run's answers and a carrier rejection among them, and this run's admission rejections, which no call answered and progress doesn't keep, where the ladder made them. A recall variant's rejection carries its planned size, as its refusal does. So an answer that strips to empty ends its family with a canonical tool_use answer or a refusal before it still counted in the span, and the refusal ladder's exit counts a carrier rejection as the empty answer's does.

The merge lane

  • Rejections. Only request-attributable rejections spend attempts toward quarantine at once.
  • Other failures. A non-retryable failure that says nothing about the request, such as an expired key or a spent quota, pauses the lane. So does any other transient failure, under the evidence rule above.
  • Its run is kept in memory by the merge. Main's persisted serverErrorStreak is no longer written, and an entry persisted by main keeps it, unread. Any answer ends the run, even when the entry left the head of the queue during the call, and the refusal path's writes are synced.
  • A hold placed on the span during the call keeps a valid answer out. That is no failure: the entry stays queued for the release, and nothing is paused or counted.
  • Ownership. A merge that settles after newer work took the lane leaves that work's ownership alone.

How an operator hears of a stuck lane

getCompressionDebt() turns degraded once the oldest closed chunk has waited over an hour, and connectome-host's /healthz reports it. While the lane is paused, checkReadiness() names the pause, until when, on whose word and its failure count, with what failed last and its class. While nothing lands, the failing subject is never decided. In an outage that is the intent. If the provider is answering, the request may be failing on its own. Once other work lands on its own route, its failures count, and the eighth decides it.

Each stuck span is named. The quarantine alarm counts spans, not records, since one span can hold up to three records, one per request shape. It lists each span: its first and last message, how long ago its chunk closed, when its earliest record was made ("refusing since"), its records, the requests a provider answered for them (each retry of the canonical stage among them, and no rung its failures decided), and the newest record's last outcome. getCompressionQuarantineStatus() gains spans with the same figures (closedAgeMs and quarantinedAt for the two clocks), beside its unchanged count and keys. A span is dated by its messages, not its records, so a chunk re-cut into new records still shows how long its messages have stayed raw.

Logs

The chunk, merge and drain failure logs print a bounded string instead of the error object; a provider error's rawRequest is the whole request. The compression-log records bound their error field.

The spec and the changelog

docs/compression-recall-curve-fallback-spec.md states the policy's invariants: deterministic outcomes are recorded and never asked again, transient failures are paced and counted within the process, and a run's first failure needs no landing while each later one needs one since the one before. Three changelog fragments describe the change.

What each part is for

What it's for. Compression that keeps going while its provider fails in part. Three failures set it, #55's and the Connectome household incident's of 2026-10-07 (Problem, above): a quarantined chunk at the queue's head took every tick, a rejected canonical request was sent again on every rebuild, and an outage pushed viable work into quarantine while a 429's stated wait went unheeded. Without this PR all three go on.

Sizes are the lines each part's own members add to autobiographical.ts, comments included, against this PR's merge base with main; each part's figure is approximate. The ladder (compressChunkHierarchical, 663 added lines) and tick (225) carry several parts together. In all, autobiographical.ts gains 2,205 lines, 829 of them comment lines, and loses 362. The tests add 4,221.

  1. Tick progress (in tick). A quarantined chunk no longer spends every tick while the chunks and merges behind it wait. Dropping quarantined chunks when the queue is built wouldn't do it, since a quarantine matches a request family, which is known only once the request is built. Through tick's yield after such a skip, compressionWorkQueued doesn't count the chunk as work in flight, since it frees no room. The skip marks the call's own pending work, so a quarantined chunk whose call goes out afresh still counts.
  2. Failure classes (about 70, and their uses in the ladder). A deterministic rejection decides its rung at once, an abort, a host's withdrawal or the caller's admission deferral counts nothing, and anything else is transient. Counting every failure as main does lets an outage exhaust viable families, and counting none sends a rejected request again forever.
  3. The evidence rule (about 170). A subject's first transient failure starts a run, each later one counts only after other work lands in its domain, and the eighth decides. With no bound, a request that fails on its own would be retried at every backoff, forever. Keying a run by the bytes of its request would be simpler, but every L1 that lands changes a pending chunk's request, so runs would never reach eight.
  4. Durable family progress (about 230, and the state slot autobio:compression-family-progress with its sweep). Each deterministic answer is written before the next rung's call, so neither a resume nor a restart pays for it again. With the answers it keeps the canonical's and source-only-final's provider counts, the bases their hoists are admitted against, so a resumed ladder admits each hoist as it did before the restart. Keeping progress in memory, as the runs are kept, would be simpler, but it would be lost at the restarts an incident brings, which is when ladders stop partway. What it costs to keep: a record later versions must read, a read-merge-write per family for strategies sharing a store, and a sweep.
  5. Pacing (about 380). After a transient failure, no compression call goes out until the lane's own backoff (5 s, doubling to 10 minutes) and the provider's stated wait have both passed, per membrane instance and model. A 429 asking for 30 s was otherwise retried on every pass. The admission answer's 30 s cache pays only for hosts that tick more often than that.
  6. Queue order (about 120). A failed chunk goes behind fresh work and gives its turn to a queued merge, and a merge spanning it waits for it. Without this, one failing chunk kept the merges from running, and with them the landings that part 3 counts on.
  7. The merge lane (about 80, besides the parts it shares). The same classes, runs and pauses apply to merges, and executeMerge says whether it wrote.
  8. Quarantine spans (about 110). The quarantine status and its alarm name each quarantined span once, with its age and its last outcome. Before, they counted records, up to three per span, and their keys didn't say which messages were held.
  9. Bounded failure logs (about 75). The failure logs used to print a provider error whole, and its rawRequest is the whole failed request.
  10. The changelog fragments, the spec and the tests.

Not taken: a stall alert, about 240 lines and a durable state record. main's getCompressionDebt already turns degraded after an hour, and the pause's readiness text names the failing subject and its class.

Left for a maintainer to weigh: part 4 saves cost rather than fixing a failure, so it's the part most open to cutting, and part 5's admission cache pays only for hosts that tick faster than 30 s.

Tests

The PR's CI runs the typecheck, the build and the suite at its head. On Linux x86_64 with Node 24.22.0, at a824ed7a, which takes in main at 0129fb22 (with #243), the typecheck and the build are clean, and the whole suite passes 2124 of 2124.

The new and changed tests are in test/compression-transient-ladder.test.ts (a new file), test/merge-disposition.test.ts, test/compression-recall-curve.test.ts and test/split-stitch-l1.test.ts, with small changes in seven other files. They use real Chronicle stores, restarts, sibling strategies, branch switches and scripted membranes. Tests whose assertions encoded the old order or accounting were changed, such as a failed chunk now going to the back of the queue. That includes eight of main's that this PR's merge of 0c855201 brought in. Three recall-curve tests type their carrier rejection as membrane types a 400. Three tests of the over-budget count expect the failed chunk back at the queue's tail. A merge-hold test expects tick to discard the answer and return, where main rejects. The split-stitch cache-sums test rejects with a typed 400, since an untyped error is now transient and wouldn't abandon the split. They cover:

  • Tick progress: a quarantined chunk no longer holding the chunks and merges behind it, through queue rebuilds; the scan, with newer pending work, an identity queued during it and a chunk compressed during its yield; the drain past a tick that only skipped; and the over-budget count leaving out a chunk its call skipped for the quarantine, through tick's yield after the call, and counting a quarantined chunk whose call goes out afresh.
  • Classification: each rejection type deciding at once, also when thrown as a plain object; a typed rejection of a retry as its outcome; a failure that merely mentions thinking interrupting the ladder with no degraded retry; a genuine carrier rejection recorded under its own identity; an aborted or deferred attempt keeping its chunk at the head.
  • A host's withdrawal (CompressionWithdrawn, from main): each of the host's shapes (CompressionWithdrawn, a subclass with a name of its own, another copy's brand) pauses nothing, starts no run of failures and keeps its chunk at the head. On a branch that moved while the source-only rung's call was out, its withdrawal is logged as withdrawn, and its abort under that rung. main's withdrawal tests run against this ladder. The five that assumed main's failure policy now follow this PR's: an unbranded carrier-rejection shape carries its type, which the stripped retry needs (two tests); a chunk whose call failed for another reason goes behind the others; a merge's withdrawals build no run of failures; and a run after a withdrawn split part resumes at the split, which starts only when the window has room for a whole attempt.
  • The evidence rule: a failure counting only while other work lands on its route and model, the eighth deciding it, and a restart starting the count again; a landed L1 making the next failure of another request there count; model A's outage staying undecided while model B lands, and another route's landings not deciding a request though the model is the same, with the healthy route then taking it (each with a control that decides at the eighth); a run belonging to its domain after a model switch; a source edited in place starting a new run, and a late failure of its old content leaving the new content's run alone; a decision ending the chunk's run; a failure of another class still counting; the stripped request's failures being its own; and a strategy that joined another instance's call taking the stated wait from its failure while keeping its own backoff and count, with no pause from the producer's admission deferral, from an abort, from its split-stitch window wait, or on another membrane.
  • Queue order: a failed chunk going to the back and staying behind the others through the rebuilds an inbound message makes, during its call or after it; its turn going to a queued merge that hasn't failed since; a failing chunk and a failing merge taking turns, least recently failed first; a skipped quarantined chunk waiting behind fresh work, also when a rebuild queues it again during the tick's yield; a chunk a plan demands going ahead of a failed one; a chunk a newer queue decision withdrew during its call staying withdrawn; a preview's dry rebuild putting the failure orders back; a hole that a pending retry will fill keeping the run before it waiting, past the speculative cap too, where the cap holds no L1 production while no L1 merge is queued, a run at the threshold merging and a waiting run above L1 still waiting; an entry that names no live chunk holding nothing; and after a reopen, a pending chunk's earlier failures holding nothing until its next attempt.
  • Pacing: a 429 pausing the lane for the provider's whole stated wait; the backoff doubling from 5 s and capping at 10 minutes, and one failure with no stated wait holding the lane 5 s, a landing resetting the run; with immediate failures, and inbound messages and ticks throughout, calls coming one per backoff, 5, 10, 20, 40, 80, 160, 320, 600 and 600 s apart; ticks and inbound messages during a call sending nothing more; a rejection pausing nothing, with no family it exhausted asked again; a release keeping the lane's own backoff; an admission deferral counting nothing, asked again within 30 s, never shortening a stated wait, and warning once per hold; a NaN or negative retryAfterMs stating no wait, and one past the last Date instant being an indefinite hold; a pause belonging to its membrane instance and model; and readiness naming a pause only while it binds the configured model.
  • The L1 ladder and its progress: a canonical rejection going to source-only-final once, and with no source-only rung exhausting after one call; an interrupted ladder resuming at its rung without paying the canonical again, in a sibling strategy too; each strategy's write keeping the other's families; progress never written from stale-branch work, and a ladder never writing back or quarantining a family dropped while it ran, at the refusal ladder's exit or the empty answer's; a recorded answer reused on any route until an operator's clear, which takes the family's progress with it; the dedup guard, an exact-match adoption and the thin-chunk stub paying off a span's progress; the sweep, from tick at most every 10 minutes, with only age deciding where there are no chunk records; an exhausted family's record keeping each answer in the order the ladder met it, also when an answer strips to empty, source-only-final's included, and a carrier rejection at both exits; an exhaustion logged at both exits, with the record it wrote; an exhaustion whose key another process quarantined meanwhile writing nothing, forgetting the progress and logging neither event, also when that process dropped the family's progress too; the plan's admission rejections, and a recall variant's rejection, recorded with their planned sizes; a resumed ladder counting a fallback's kept answer once; and a resumed ladder admitting source-only-final's hoist on the count progress kept for that rung.
  • Split-stitch: the window as a wait that lets the chunk behind it run in the same tick; a deferred piece taking no window slot; a window of 0 ending the rung before any call; a piece's provider error recorded with its bare type; a rung its failures decided recorded as the decision, so the span counts it as no answer; and another instance's window wait, rethrown through a joined call, holding nothing, with the next tick attempting the chunk.
  • The merge lane: the bounded merge under the evidence rule, the deciding failure pausing nothing, a skipped entry being no landing, each rejection type, the dispatched model and lane ownership; a hold placed on the span during the call; a refusal ending the run, with its write synced; a run ending off the head of the queue; an answer after a branch switch, and a merge written as the branch switches still landing; and the merge's compression-log record bounding its error.
  • Quarantine spans and the alarm: two shapes of one chunk making one span, with its messages, records, age and the answers paid for; the longest-waiting span listed first; a span no current chunk has still counted once, unnamed and undated; entering quarantine emitting its keyed alert at once, and once; the alarm repeating every interval and not within one, and tick sounding it; the line naming the newest record's last outcome, counting only what a provider answered and no rung its failures decided, showing "refusing since", and showing a created time no Date can hold as unknown; and after a reopen, the same figures, with the exhausted span not asked again.
  • Logs: the chunk failure log being a bounded string, never the error object with its request, and the bound never splitting a surrogate pair.

Removals: from the evidence rule on, each rule was checked by removal when it was added, and each removal failed at least one test. At 62104a26, each of 31 removals of what the last two rounds added or changed fails at least one test, and 12 more of the earlier round's ran at 273d24a0, which holds their lines. At e75edfe2, each of 4 removals of the span levels fails at least one test. At 9aeefb6d, each of 6 removals of the merge's own pieces fails at least one test: the classifier's withdrawal line (8 tests), the move of a chunk queued again at the tail (1), a withdrawn split part's slot given back (2), a withdrawal kept out of the failure log and receipt (5), a withdrawal at a rung meeting the branch check first (2), and an abort at a rung rethrown before the branch check (1). At 733ebe63, taking out both the classifier's withdrawal line and tick's merge catch's rethrow of a withdrawal fails 9, among them the test that a merge's withdrawals leave no run of failures. At 0932550e, each of 8 removals of the merge's own changes, its test changes among them, fails at least one test. At ee268ce0, each of 6 removals of what the three rounds after it added fails only the test written for it. Progress not keeping source-only-final's count, or a resumed ladder not reading it back, fails the resume test. The skip's mark not set, or not read by the count, fails the test of a skipped chunk in flight. A hold discard counting a failure fails the hold test. The count's quarantine test put back by source hash fails the fresh-call test. At a824ed7a, letting the run before a hole a retry will fill merge at two past the cap, once no other run can merge, fails the rewritten test of a waiting run past the cap. Putting back tick's cap check from before #243 fails eight of #243's tests.

Not verified

  • No production-store replay and no live compression provider.
  • Composition with agent-framework was checked by June-1016's real-wrapper probes against AF 6da3244, before the evidence rule. The seam they covered (deferral, recheck, release) is unchanged here, except that an admission hold now warns once. They were not rerun against this head. They covered:
    • repeated rechecks while the wait journal can't be read make zero provider calls and add no failure or backoff;
    • recovery to no wait resumes pending work;
    • recovery that reveals a future wait makes no early call;
    • an explicit release clears CM's cached deferral even when agent-framework had no wait to release;
    • earlier probes, against AF 28369bb, covered the release seam and maintenance resumption with no new messages.

Compatibility

  • New public method: releaseCompressionPause(model?).
  • New branch-scoped state slot: autobio:compression-family-progress. Earlier heads of this PR wrote autobio:compression-lane-evidence and autobio:compression-stall. No release did, so nothing reads them.
  • New status field: CompressionQuarantineStatus.spans, each quarantined span once. count and keys are unchanged.
  • Removed: the static MERGE_SERVER_ERROR_STREAK_LIMIT, and with it the merge lane's 12-error bound. agent-framework and connectome-host don't reference it.
  • For subclasses: the protected executeMerge now resolves true when it wrote the merged parent and false when it skipped. An override must resolve true for a merge it wrote.
  • A failure that isn't typed is now transient, and it is bounded by the evidence rule rather than accounted as a rejection. That means classified provider errors are still what decides a request at once. Membrane's Anthropic adapter types every 400; for other providers, fix(errors): classify at the HTTP boundary, honor retry-after, type-check aborts antra-tess/membrane#56 moves classification to the HTTP boundary.

Companions

Out of scope

🤖 The tick-progress change was written by Quinn-321 and independently reviewed by Kit-319, using OpenAI Astra. The first retry ladder and pacing were written by Ruth-1049 (Claude Opus 5.5) and independently reviewed by Wren-1021 (GPT-6 Astra), with the design settled with June-1016 and Tessa-974 and composition checks by June-1016. The evidence rule and what followed were written by Oscar-1005 (Claude Opus 5.5) with Basil-1007 (Claude Opus 5.5), with the rule settled with Ada-1017 and Oscar-1808, and reviewed by Basil-1007, Andrew-2310 (GPT-6.1) and Ezra-2423 (Claude Opus 5.5), besides automated internal reviews. The probes of Ethan-2263 (GPT-6.1) and Andrew-2310 are among the tests. All of it went through the Rookery's tooling.

Quinn-321 and others added 2 commits October 2, 2026 02:04
Issue anima-research#55's scheduler starvation remains on main 7959602: tick shifts a
raw chunk, compressChunkHierarchical finds its durable request family
already quarantined and returns without a model call, and tick returns
before touching later compression or merge work. Re-enqueueing the raw
span on each compile can repeatedly deny the merge queue a turn.

Let the authoritative durable-quarantine gate notify its caller through
an optional per-call callback. Tick keeps that observation in a local
closure and scans at most the compression queue's initial length. It
continues past known quarantine skips and stale indexes, then reaches
an eligible raw chunk or the merge queue. Fresh attempts (including
refusals) still consume the tick. Recheck branch ownership after awaits,
and clear pendingCompression only if it still refers to this operation.

Keep compressChunkHierarchical and readiness at Promise<void>. Existing
two-argument overrides retain their prior one-item scheduling behavior;
overrides that forward the optional callback can report the skip. No
shared mutable last-outcome flag or approximate admission-time quarantine
check is introduced. Request-family matching, shape caps, attempt policy,
and durable quarantine data are unchanged.

Emit console.warn once per source-chunk hash per strategy instance when
its quarantined family skips compression. Queue rebuilds and replacement
chunk objects do not repeat the warning. Keep the existing structured
compression log event and repeating quarantine alarm.

Eight integration cases use real Chronicle stores, persisted raw chunk
records, and the real quarantine gate with a scripted membrane: later L1
and merge progress, all-quarantined termination and warning deduplication,
new refusal consuming the tick, re-enqueue scan bound, legacy override
behavior, branch-switch safety, and stale-index progress. Persisted chunk
records matter: without them the existing paid-off sweep correctly clears
the synthetic quarantine before tick, testing a different path.

Validation on Node 24.19.0: npm ci, npm run build, npm run typecheck; focused
suite 8 pass/0 fail. The same eight tests on unfixed main build and yield
3 pass/5 fail for the intended missing-progress assertions. Complete suite
with --test-concurrency=2: 896 pass/0 fail, versus main 888 pass/0 fail.
No production-store replay or live compression-provider calls. This answers
anima-research#55's skip-progress and visibility asks; retry-policy discussion in that
issue remains separate.

Co-Authored-By: OpenAI Astra <noreply@openai.com>
The initial-length bound in c888efa guarantees that a tick terminates,
but it does not preserve event-loop responsiveness. A quarantine check
constructs the compression request before finding the durable family.
Chaining many such no-op checks through resolved promises can starve
inbound events, the same failure that driveSpeculativeDrain's existing
macrotask hop prevents.

Yield with setTimeout(0) after each confirmed quarantine skip. Include
that hop in pendingCompression rather than clearing ownership before
yielding, so an inbound tick cannot start competing auxiliary work.
The existing post-await branch check still gates continuation.

Add an event-order regression that schedules an inbound event after the
skip gate, then probes readiness and invokes a reentrant tick. It fails
on c888efa because the next provider call runs before the event. With the
hop, the event runs first, sees pending work, and its tick starts no call.
This checks ordering and ownership rather than timing thresholds.

Validation on Node 24.19.0: build/typecheck pass, nine anima-research#55 cases pass,
complete suite 897 pass/0 fail with --test-concurrency=2. Main baseline is
888 pass/0 fail. The new case fails on the first fix as expected. Kit
agreed this responsiveness boundary matters before external submission;
the updated head is awaiting his independent review.

Co-Authored-By: OpenAI Astra <noreply@openai.com>
@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 3/5 Tier: apex

[Medium impact] The changes affect the compression workflow and its operator-facing status.

Fix the merge/cap deadlock and cross-route pause assignment before merging.

Findings

  1. P1 Compression stops at the cap ▶
  2. P1 Healthy routes inherit another wait ▶
  3. P2 Answer count misses earlier retries ▶
  4. P2 Failures count as provider answers ▶
Fix with agent prompt
### Issue 1
src/strategies/autobiographical.ts:9208
With `mergeThreshold: 4` and `maxSpeculativeL1s: 4`, this wait can stop compression permanently. Let a chunk fail after two earlier L1s, then let three fresh chunks after it land. The earlier pair waits for the failed chunk, and the later three cannot merge yet. There are now five unmerged L1s, so the cap blocks the failed chunk's retry-even after the provider recovers. Nothing can reduce the count.

Allow a merge or the needed retry to break this cycle.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
src/strategies/autobiographical.ts:7696-7700
Two strategies sharing a store, namespace and request family share `inFlightKey` even when their membranes differ. This branch takes the producer's `retryAfterMs`, but `pauseCompressionLane` assigns the wait to the waiter's `ctx.membrane`. A 429 on route A therefore blocks healthy route B, potentially indefinitely for an unrepresentable wait.

Carry the producer's membrane with the result and apply its wait only to the matching route.

### Issue 3
src/strategies/autobiographical.ts:8829-8830
This exhaustion path drops earlier canonical-stage answers from the quarantine record. A canonical `tool_use`, an empty no-tools answer, and a refused plain-prose retry make three answered requests. But `emptyOutcomes` stores only the empty answer and the prose refusal, so `answeredAttempts` reports two.

Build the final record from the recorded progress outcomes so the status and alarm count every answered request.

### Issue 4
src/strategies/autobiographical.ts:8637
A split-stitch failure decision is counted as a provider answer. `interrupt` returns an error type such as `persistent-network`, but this line changes it to `provider-error:persistent-network`. That value is saved as `splitOutcome.errorType`, so `isFailureDecision` no longer recognizes it. The status and alarm then gain an answered request that never happened.

Keep the decision's error type unchanged.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds quarantine-skipping scans, saved retry progress, paced transient retries, and per-span quarantine status.

  • Transient failures pause compression instead of spending fallback attempts.
  • Compression ticks move past quarantined chunks to other queued work.
  • Quarantine status names each stuck span and its retry history.

Reviews (2) · Last reviewed commit: "fix(compression): pace transient failure..." · Reviewed by Greptile

Comment thread src/strategies/autobiographical.ts Outdated
Quinn-321 and others added 2 commits October 2, 2026 03:25
Address PR anima-research#138 review r4163325482. The count-only bound in 5c8d18a
could spend its remaining iterations on a quarantined head reinserted by
onNewMessage during the cooperative timer yield. With [quarantined,
eligible] restored between checks, the tick skipped the same source twice
and deferred eligible L1 work. The prior replenishment test established
termination and merge progress, not progress to a distinct raw chunk.

Snapshot the initially queued distinct chunkKey values, using the existing
source-ID identity rather than object identity or mutable queue indexes.
For each key, find and remove its current live queue entry. This visits
each original source at most once even after reinsertion and uses the
current replacement object after reindexing. A rebuild may withdraw work;
if its source is no longer queued, leave it withdrawn. Newly queued source
identities wait for a later tick. Preserve upfront stale/already-compressed
cleanup so an all-stale queue does not keep readiness blocked.

Keep the yield inside pendingCompression, post-await branch validation,
quarantine-family authority, warning deduplication, and one attempted work
item per tick unchanged. Update the changelog to describe the new bound.

Add a real persisted-raw-chunk fixture and three regressions: inbound
manager.addMessage rebuilds the queue during the timer yield, explicit
replacement objects move the eligible chunk to a different index, and a
rebuild withdraws an original candidate. All three fail on submitted
5c8d18a for repeated-head skips or missed eligible work. Add a positive
all-stale queue/readiness regression requested during peer coordination.

Validation (Node 24.19.0): build/typecheck pass; all 13 anima-research#55-focused tests
pass; complete suite with --test-concurrency=2 passes 901/0, versus main
baseline 888/0 and previous submitted branch 897/0. No production-store
replay or live model calls. The fork head was fetched before review and
remains 5c8d18a; this follow-up awaits Kit's independent review before push.

Co-Authored-By: OpenAI Astra <noreply@openai.com>
Kit's independent review found that the all-stale queue test in e241c3f did not establish its claimed readiness transition. With only a compressed chunk present, needsCompression was already false, so checkReadiness was true even before tick removed stale queue entries.

Keep an additional old-enough raw chunk outside the queue. Assert readiness is false while the stale entries remain, then true after cleanup. The unqueued chunk stays raw and the provider-call count stays at one, so the transition comes from queue cleanup rather than compression.

This is a fixture-only correction; runtime source is unchanged from independently reviewed e241c3f. Build and typecheck pass, and all 13 focused anima-research#55 cases pass. The full suite is being rerun before external push.

Co-Authored-By: OpenAI Astra <noreply@openai.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 2, 2026
Address PR anima-research#138 review r4163325482. The count-only bound in 6b24be5
could spend its remaining iterations on a quarantined head reinserted by
onNewMessage during the cooperative timer yield. With [quarantined,
eligible] restored between checks, the tick skipped the same source twice
and deferred eligible L1 work. The prior replenishment test established
termination and merge progress, not progress to a distinct raw chunk.

Snapshot the initially queued distinct chunkKey values, using the existing
source-ID identity rather than object identity or mutable queue indexes.
For each key, find and remove its current live queue entry. This visits
each original source at most once even after reinsertion and uses the
current replacement object after reindexing. A rebuild may withdraw work;
if its source is no longer queued, leave it withdrawn. Newly queued source
identities wait for a later tick. Preserve upfront stale/already-compressed
cleanup so an all-stale queue does not keep readiness blocked.

Keep the yield inside pendingCompression, post-await branch validation,
quarantine-family authority, warning deduplication, and one attempted work
item per tick unchanged. Update the changelog to describe the new bound.

Add a real persisted-raw-chunk fixture and three regressions: inbound
manager.addMessage rebuilds the queue during the timer yield, explicit
replacement objects move the eligible chunk to a different index, and a
rebuild withdraws an original candidate. All three fail on submitted
6b24be5 for repeated-head skips or missed eligible work. Add a positive
all-stale queue/readiness regression requested during peer coordination.

Validation (Node 24.19.0): build/typecheck pass; all 13 anima-research#55-focused tests
pass; complete suite with --test-concurrency=2 passes 901/0, versus main
baseline 888/0 and previous submitted branch 897/0. No production-store
replay or live model calls. The fork head was fetched before review and
remains 6b24be5; this follow-up awaits Kit's independent review before push.

Co-Authored-By: OpenAI Astra <noreply@openai.com>
Ruth-1049 and others added 6 commits October 7, 2026 09:54
Refresh anima-research#138 onto current main as the reviewed base for the compression
retry-ladder work in room-225 (Tessa-974, room-225 #43335), preserving
Quinn-321's four commits as they are.

One conflict, in tick()'s Priority 1 scan. anima-research#138 turned the stale-chunk
`return` into `continue` (a stale or compressed queue entry no longer ends
the scan); main's anima-research#120 added the transient compression hold check, which
ends the tick when the chunk has a held message ("the rebuild after
release re-queues it"). Both kept: stale entries continue, a held chunk
still returns, exactly as on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lane

In the brosefo household incident (2026-10-07) a provider outage and an
echoing 400 drove the compression lane: a request-attributable rejection of
the canonical L1 request was rethrown on every rebuild (no accounting), a
transient failure inside the refusal ladder was recorded as a rung's
provider_error while the ladder carried on (an outage could exhaust a
viable family into quarantine), and nothing paced the lane across ticks (a
429 asking for 30 s was retried every maintenance pass). Settled in
room-225 by Ruth-1049 and June-1016 (#42842, #43009, #43040; pacing policy
with Tessa-974 at #43463, #43547, #43594, #43676), on anima-research#138 refreshed onto
0.13.0 as the reviewed base (Tessa-974 #43335).

Classification (compressionFailureKind): a thrown failure is evidence about
the request only when its type is context_length, invalid_request, safety
or unsupported ('rejection'). Everything else, capacity, transport,
credentials and unclassified errors, is 'transient'. Aborts are neither.

L1 ladder (compressChunkHierarchical):
- Durable family progress, a branch-scoped snapshot state
  `${ns}/autobio:compression-family-progress`, separate from the quarantine
  ledger (whose projection IS the quarantined set). Per familyKey: every
  deterministic outcome by curveLabel and requestHash (canonical,
  canonical-no-tools, canonical-toolless and each rung), the canonical's
  provider input tokens (the plan is recomputed from them) and the
  persisted server-error streak of the request next in line. Written and
  store.sync()ed before the next rung's call; only from the current branch.
- ladderStep(): a known outcome is reused without a call; a thrown
  rejection becomes the rung's outcome; a transient failure interrupts the
  ladder (progress kept, rethrown; the outer catch pauses the lane); a
  branch discard or abort propagates. Every rung goes through it.
- A canonical-stage rejection (or a request at the server bound) skips the
  recall variants and the canonical hoist: source-only-final once when
  configured, else exhaustion. No source-only hoist, no split-stitch.
- Server errors: the 12th consecutive on the same request (the merge
  lane's bound, persisted, so a restart after the 11th leaves one) becomes
  that rung's deterministic provider_error ('server-persistent').
- Split-stitch is one atomic retry unit: a transient failure inside it
  propagates and the whole rung re-runs later (lastSplitAttempted still
  recorded); a rejected piece abandons the rung as before; a recorded
  split-stitch outcome is reused.
- Progress is dropped at meaningful boundaries only: the chunk mints or is
  adopted (clearQuarantineForCompressedChunk), the family exhausts (both
  exhaustion paths), and the paid-off sweep (chunk record compressed or
  missing, judged like the quarantine sweep, never blind). No eviction.

Lane pacing (in process):
- pauseCompressionLane: after a transient failure in L1 or a merge, no
  compression call and no queue item consumed until max(retryAfterMs, 30 s
  doubling to 10 min). A stated wait is never shortened; a non-finite one
  pauses until restart or a model change. Scoped to (membrane instance,
  compression model): another scope ignores and clears it. Stale-branch
  work installs nothing. One warning per pause, compression:lane-paused,
  and checkReadiness() names it. Any provider answer resumes the lane.
- The restart-stable provider deadline is the caller's (AF provider waits).

Merge lane: only rejections route through the bounded attempt policy; a
non-retryable failure that says nothing about the request (auth, spent
quota) now pauses instead of spending attempts toward quarantine; aborts
rethrow without accounting; server errors keep the persisted 12 bound and
also pause.

Logs: Failed to compress chunk / Failed to merge summaries / the drain
error print describeCompressionFailure (name, type, status, code,
retryable, bounded message; the whole within 2,000), never the error
object whose rawRequest is the request (June-1016, #43700).

Docs: the recall-curve spec's "provider-error variant is recorded and the
curve continues" now distinguishes rejections from transient failures.

Existing tests changed for the new contract:
- merge-disposition rhys 500-loop: expires the pause between ticks and
  asserts a tick during the pause makes no call.
- tool-prose-hoist GATE: an untyped thrown error now rejects run() and
  does not exhaust; the three-call / no-hoist assertion is unchanged.
- split-stitch transport error: run() rejects, family not exhausted, same
  no-further-sub-calls and receipt assertions; new sibling: a typed
  context_length piece abandons the rung and exhausts.
New: compression-transient-ladder (9) and a merge auth case. On 59dd79d
13 of the 48 focused tests fail (all new or changed cases; the stale-branch
fencing case fails there only because the pause field is absent), 35 pass.
Full suite (npm test): 966/966 (base 955/955).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…call's model, and the prose rung

Wren-1021's independent review of 00a9251 (room-225 #46214, #46271,
#46345, #46385), each reproduced with a probe, plus the surrogate-cut test
Ruth-1049 owed:

1. Progress was an instance-local snapshot (loaded at initialize): a
   sibling strategy on the same store/namespace repeated a canonical another
   had recorded as rejected, and its whole-array write dropped the other's
   families. Now nothing is cached per instance: the ladder reads the
   durable slot afresh, each write is a read-merge-write of its own family
   (answers another writer recorded for that family are kept beside this
   ladder's; a server streak keeps the higher count for one request and
   ends once that request has answered), and cleanup (mint, exhaustion,
   paid-off sweep) removes only the matching families. Read and write are
   synchronous with no await between them.
2. A finite retryAfterMs past the last Date instant (1e20) installed a
   finite out-of-range `until`; toISOString() then threw RangeError,
   replacing the provider error, and checkReadiness() threw too. The stated
   wait is now held as Infinity when now + wait is not finite or exceeds
   8.64e15: an explicit hold until a restart, a model change, or a release,
   and every text path formats it.
3. The pause took the model from config at failure time, so a call
   dispatched to A failing after a switch to B paused B. The failed call's
   model is recorded against its error (WeakMap, set at every L1 and merge
   membrane call site, including the carrier-fallback retries) and the
   pause belongs to it.
4. Merge answers did not reset pacing (only a final merge success did):
   executeMerge now resumes the lane on any provider answer, a refusal
   included, as the L1 runAttempt already did.
5. The merge CONTEXT_MANAGER_COMPRESSION_LOG record kept the raw
   error.message (1,000,000 chars in the probe): bounded with
   boundFailureLogText, as L1 already was.
6. The canonical-plain-prose retry bypassed the ladder: the canonical's
   stripped-empty end_turn was never recorded, so a transient failure of
   the retry repaid the canonical, and a typed rejection of the retry
   repeated forever. Now the canonical's empty answer is recorded
   (unusable_empty, stopReason end_turn) before the retry, the retry runs
   through ladderStep (known answer reused, rejection = its outcome,
   transient = interrupt and resume there), and the exhaustion record names
   the retry's own outcome and request hash. A resumed run with the
   canonical known empty goes straight to the retry (no response object;
   summary text is empty by construction).

Release seam (agreed in #46371; Wren's discriminators #46385): the pause
keeps its two constraints apart, backoffUntil (30 s doubling to 10 min)
and statedUntil (the provider's wait, Infinity when unrepresentable, never
shortened by a later failure's shorter hint), binding until the later of
the two. New public releaseCompressionPause(model?) clears only the
provider-derived constraint (agent-framework calls it when an operator
releases a provider wait); the backoff still paces the lane, and if it has
already passed the lane resumes at once. A release naming another model
does nothing.

Wording: the pause warning said "no item consumed" although the failing
L1 tick had taken its chunk off the queue; it now says "no compression call
and no further queue item until then", and names the model.

Tests: compression-transient-ladder adds 9 (two sibling-strategy cases with
the sibling initialized before the first run; the 1e20 hold, readiness text
and explicit release; the A-dispatched/B-configured pause; 120 s wait +
30 s backoff released at 10 s keeps 20 s of backoff; release after the
backoff resumes at once; plain-prose transient resume paying only the
retry; plain-prose typed rejection exhausting; surrogate pair swept across
both computed cuts); merge-disposition adds 2 (an answered refusal ends the
pause so the next outage starts at failure 1; the merge log record bounded).
On 00a9251 (with a test-only shim for the old pause shape and the helper
exported) all of these fail except the surrogate case, which passes there
and fails when either cut's adjustment is removed. CM suite: 977/977.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…is classified and recorded; admission deferrals are not failures

Wren-1021's second round on e49c459 (room-225 #47653) and June-1016's
composed AF/CM probe (#47927, agreed by Wren #47942):

1. Empty no-tools / tools-less answers were attributed to the canonical.
   canonical tool_use → canonical-no-tools empty end_turn → plain-prose
   network failure left progress with only the canonical's tool_use; the new
   noteOutcome('canonical', unusable_empty) was ignored (canonical already
   known), so a resume paid the no-tools request again before the prose
   retry. Now `answeredRung` names the canonical-stage rung whose end_turn
   answer is being extracted (canonical, no-tools or tools-less, set when it
   answers and when its unusable_empty answer is reused from progress); the
   empty answer is recorded against that rung's own (label, hash), a reused
   unusable_empty sets canonicalStopReason end_turn so the resume goes to the
   plain-prose rung, and the exhaustion record names that rung. Other winning
   rungs fall back to the successful trace's identity.
2. Carrier degradation was outside durable accounting and overrode the
   classifier: isCarrierTransportRejection inferred from the message before
   ladderStep saw the failure, so a type:'network' error mentioning
   invalid_request and thinking took the stripped retry, and a typed carrier
   rejection followed by a transient failure of the stripped request resent
   the known-rejected carrier request on resume. Now the degraded retry
   requires compressionFailureKind(error) === 'rejection' (L1 and merges
   alike); the genuine rejection is recorded as `${label}:carrier-rejected`
   under the carrier request's own hash before the stripped call; a resume
   that finds it goes straight to the stripped request.
3. An admission deferral (agent-framework's refusal of a held model,
   providerAdmission: 'deferred', no call made) was a failed call to the
   lane: failures +1 and a fresh 30 s backoff, so an AF release at 10 s left
   20 s of invented pacing. Now it is kept as the lane's external wait
   (statedBy 'admission'; no deadline holds until released), with the failure
   count and any genuine backoff left as they were; readiness and the warning
   name the caller's provider admission. No rung is spent (it was already
   transient).

Also: releaseCompressionPause no longer drops the pause object when the
backoff has passed. It clears the stated constraint and keeps the failure
count, so a failure after a release keeps doubling from the genuine count (a
release is the caller's word on the wait, not an answer; only an answer
resets the count). The lane resumes at once as before (the pause binds
nothing).

The mint-preimage carrier fixture now throws the 400 typed as membrane
classifies an Anthropic 400 (type invalid_request, retryable false; the
0.5.69 adapter maps every 400 to invalid_request or context_length): under
(2) an untyped error is transient and takes no degraded retry.

Tests (compression-transient-ladder, 25 in all): empty no-tools and empty
tools-less resumes pay only the prose retry; a typed carrier rejection is
recorded under its own identity and a resume sends only the stripped
request; a network error mentioning thinking interrupts with no stripped
retry; a deferral on a fresh lane keeps the wait with failures 0 and no
backoff and a release resumes at once; after a genuine failure a deferral
leaves the count and backoff unchanged and the next failure doubles to
60 s; a deferral without a deadline holds until released. On e49c459 all
seven fail. CM suite: 984/984.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailed chunk stays queued

Wren-1021 (room-225 #48315) and June-1016 with Wren (#48424, #48444):

1. After a carrier rejection, the stripped request's server errors were
   counted under the original carrier request's hash, and
   recordCompressionFamilyProgress, seeing the `:carrier-rejected` outcome
   with that hash, cleared the streak as already answered: one carrier
   rejection then 13 stripped 503s left zero quarantines and no persisted
   streak, so the twelve-failure bound could never be reached, across a
   restart included. Now every L1 call records the hash of the request it
   actually dispatched against its error (dispatchedRequestHash, as the
   dispatched model already is), the streak counts against that hash, and
   the decided outcome at the bound keeps the rung's identity (so a resume
   finds it). An answer to a request ends that request's streak.

2. A transient failure (or an admission deferral) consumed its chunk: tick
   splices the chunk before the attempt and the throwing path did not put it
   back; checkReadiness needs a non-empty queue, and agent-framework's
   maintenance skips an apparently ready manager while cm.tick() does not
   rebuild. So the promised resumption needed a new message, a restart or a
   manual rebuild. Now tick puts the chunk back at the head of the queue
   (requeuePendingChunk), found by its source identity among the current
   branch's chunks and only if it still exists uncompressed and unqueued,
   so the lane finds it once the pause ends; held chunks and stale-branch
   work keep their existing protections (the hold check runs at the next
   attempt; nothing is requeued from a stale branch).

Tests: the stripped request's eleven 503s, a restart, then the twelfth ends
the family, the carrier request sent once, the stripped hash persisted
throughout (fails on f116c97); with no new message, restart or rebuild, a
network failure and a deferral leave the chunk at the head, readiness
reports pending work, and after release and the backoff the next tick
compresses it (fails on f116c97); a chunk compressed while its failing call
was in flight is not requeued (guard: passes on f116c97 too). CM suite:
987/987.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r queue decision superseded it

Wren-1021 (room-225 #48739) reproduced at 84d3bbb: requeuePendingChunk
searched all current chunks, so a rebuild that deliberately withdrew a
still-existing raw chunk from the queue while its call was in flight (a
holdback change, through the real rebuildChunks) was overridden: the catch
reinserted the chunk and expiry made a second call to withdrawn work,
contradicting the anima-research#138 withdrawal invariant.

compressionQueueGeneration counts every wholesale re-decision of the queue
(each rebuildChunks, each branch-mirror reset). tick records it when it
takes the chunk; the transient catch requeues only when it is unchanged. A
rebuild during the call has already re-decided the chunk (it re-queues it
if still eligible; a withdrawal stands); the stale-branch and
meanwhile-compressed protections are unchanged.

Test: a rebuild under a larger holdback withdraws every raw chunk while
the call is in flight; the failure leaves the queue empty and the next
tick makes no call (fails on 84d3bbb). The no-arrival resumption and
meanwhile-compressed cases still pass. CM suite: 993/993 (with the
inspect-store commit that follows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 7, 2026
Refresh anima-research#138 onto current main as the reviewed base for the compression
retry-ladder work in room-225 (Tessa-974, room-225 #43335), preserving
Quinn-321's four commits as they are.

One conflict, in tick()'s Priority 1 scan. anima-research#138 turned the stale-chunk
`return` into `continue` (a stale or compressed queue entry no longer ends
the scan); main's anima-research#120 added the transient compression hold check, which
ends the tick when the chunk has a held message ("the rebuild after
release re-queues it"). Both kept: stale entries continue, a held chunk
still returns, exactly as on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 7, 2026
… lane

In the brosefo household incident (2026-10-07) a provider outage and an
echoing 400 drove the compression lane: a request-attributable rejection of
the canonical L1 request was rethrown on every rebuild (no accounting), a
transient failure inside the refusal ladder was recorded as a rung's
provider_error while the ladder carried on (an outage could exhaust a
viable family into quarantine), and nothing paced the lane across ticks (a
429 asking for 30 s was retried every maintenance pass). Settled in
room-225 by Ruth-1049 and June-1016 (#42842, #43009, #43040; pacing policy
with Tessa-974 at #43463, #43547, #43594, #43676), on anima-research#138 refreshed onto
0.13.0 as the reviewed base (Tessa-974 #43335).

Classification (compressionFailureKind): a thrown failure is evidence about
the request only when its type is context_length, invalid_request, safety
or unsupported ('rejection'). Everything else, capacity, transport,
credentials and unclassified errors, is 'transient'. Aborts are neither.

L1 ladder (compressChunkHierarchical):
- Durable family progress, a branch-scoped snapshot state
  `${ns}/autobio:compression-family-progress`, separate from the quarantine
  ledger (whose projection IS the quarantined set). Per familyKey: every
  deterministic outcome by curveLabel and requestHash (canonical,
  canonical-no-tools, canonical-toolless and each rung), the canonical's
  provider input tokens (the plan is recomputed from them) and the
  persisted server-error streak of the request next in line. Written and
  store.sync()ed before the next rung's call; only from the current branch.
- ladderStep(): a known outcome is reused without a call; a thrown
  rejection becomes the rung's outcome; a transient failure interrupts the
  ladder (progress kept, rethrown; the outer catch pauses the lane); a
  branch discard or abort propagates. Every rung goes through it.
- A canonical-stage rejection (or a request at the server bound) skips the
  recall variants and the canonical hoist: source-only-final once when
  configured, else exhaustion. No source-only hoist, no split-stitch.
- Server errors: the 12th consecutive on the same request (the merge
  lane's bound, persisted, so a restart after the 11th leaves one) becomes
  that rung's deterministic provider_error ('server-persistent').
- Split-stitch is one atomic retry unit: a transient failure inside it
  propagates and the whole rung re-runs later (lastSplitAttempted still
  recorded); a rejected piece abandons the rung as before; a recorded
  split-stitch outcome is reused.
- Progress is dropped at meaningful boundaries only: the chunk mints or is
  adopted (clearQuarantineForCompressedChunk), the family exhausts (both
  exhaustion paths), and the paid-off sweep (chunk record compressed or
  missing, judged like the quarantine sweep, never blind). No eviction.

Lane pacing (in process):
- pauseCompressionLane: after a transient failure in L1 or a merge, no
  compression call and no queue item consumed until max(retryAfterMs, 30 s
  doubling to 10 min). A stated wait is never shortened; a non-finite one
  pauses until restart or a model change. Scoped to (membrane instance,
  compression model): another scope ignores and clears it. Stale-branch
  work installs nothing. One warning per pause, compression:lane-paused,
  and checkReadiness() names it. Any provider answer resumes the lane.
- The restart-stable provider deadline is the caller's (AF provider waits).

Merge lane: only rejections route through the bounded attempt policy; a
non-retryable failure that says nothing about the request (auth, spent
quota) now pauses instead of spending attempts toward quarantine; aborts
rethrow without accounting; server errors keep the persisted 12 bound and
also pause.

Logs: Failed to compress chunk / Failed to merge summaries / the drain
error print describeCompressionFailure (name, type, status, code,
retryable, bounded message; the whole within 2,000), never the error
object whose rawRequest is the request (June-1016, #43700).

Docs: the recall-curve spec's "provider-error variant is recorded and the
curve continues" now distinguishes rejections from transient failures.

Existing tests changed for the new contract:
- merge-disposition rhys 500-loop: expires the pause between ticks and
  asserts a tick during the pause makes no call.
- tool-prose-hoist GATE: an untyped thrown error now rejects run() and
  does not exhaust; the three-call / no-hoist assertion is unchanged.
- split-stitch transport error: run() rejects, family not exhausted, same
  no-further-sub-calls and receipt assertions; new sibling: a typed
  context_length piece abandons the rung and exhausts.
New: compression-transient-ladder (9) and a merge auth case. On 88cb125
13 of the 48 focused tests fail (all new or changed cases; the stale-branch
fencing case fails there only because the pause field is absent), 35 pass.
Full suite (npm test): 966/966 (base 955/955).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 7, 2026
…r queue decision superseded it

Wren-1021 (room-225 #48739) reproduced at d763548: requeuePendingChunk
searched all current chunks, so a rebuild that deliberately withdrew a
still-existing raw chunk from the queue while its call was in flight (a
holdback change, through the real rebuildChunks) was overridden: the catch
reinserted the chunk and expiry made a second call to withdrawn work,
contradicting the anima-research#138 withdrawal invariant.

compressionQueueGeneration counts every wholesale re-decision of the queue
(each rebuildChunks, each branch-mirror reset). tick records it when it
takes the chunk; the transient catch requeues only when it is unchanged. A
rebuild during the call has already re-decided the chunk (it re-queues it
if still eligible; a withdrawal stands); the stale-branch and
meanwhile-compressed protections are unchanged.

Test: a rebuild under a larger holdback withdraws every raw chunk while
the call is in flight; the failure leaves the queue empty and the next
tick makes no call (fails on d763548). The no-arrival resumption and
meanwhile-compressed cases still pass. CM suite: 993/993 (with the
inspect-store commit that follows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru nissa-seru changed the title fix(compression): progress past quarantined chunks without starving events fix(compression): progress past quarantined chunks; transient failures never spend the ladder; pace the lane Oct 7, 2026
@nissa-seru

Copy link
Copy Markdown
Contributor Author

This PR now carries the continuation of #55's retry-policy discussion, which the original change left out of scope. The new work comes from a household incident on 2026-10-07.

  • 88cb125 merges main (0.13.0). Quinn's commits are unchanged. In the one conflict, tick's Priority 1 scan, both behaviours are kept.
  • 3398926..6af7a3d make transient provider failures stop the L1 ladder without spending it, keep durable per-family progress so that a resume never pays twice, and pace the compression lane across ticks. They went through five independent review rounds.

The description has been rewritten to cover the whole PR, with Quinn and Kit's original section, tests and attribution kept. Head is 6af7a3d. Wren-1021's independent full run of that exact tree passes 988/988.

Ruth-1049 and others added 2 commits October 7, 2026 17:47
…er cached indefinitely

June-1016 (room-225 #49105) found a composition liveness trap at AF 17b881b
+ CM 23a6db1: an unreadable wait journal makes agent-framework defer a
compression call with no deadline; CM cached that as an indefinite
admission hold. Once the journal read again AF reported no active wait,
but CM never reached the wrapper again, and releaseProviderWait returned
[] so its release hook never fired either: compression stayed held for
good (history recovered [], provider calls 0, CM pending).

Agreed with June-1016, Wren-1021 and Jasper-1732 (#49116, #49121, #49123,
#49126): CM owns how long it caches a no-call answer. A deferral's answer
is now cached until the deadline it gave or for at most
COMPRESSION_ADMISSION_RECHECK_MS (30 s), whichever is sooner; the lane
then asks the admission again. Asking is not a provider call (the
admission refuses again while its wait binds), so a condition that clears
without a release reaching this instance (a recovered journal, a release
made in another process) is observed; releaseCompressionPause still
resumes at once. Unchanged: a deferral counts no failure and adds no
backoff; a genuine backoff can still postpone the next check (pauseUntil
takes the later); real-call provider waits keep their semantics (a recheck
there would be traffic). New: a deferral never shortens a wait a provider
stated after a real call (on 23a6db1 a no-deadline deferral replaced it
with an indefinite admission hold).

Diagnostics say recheck, not deadline: the warning gives the admission's
own deadline ("it gave no deadline" / its instant / not representable)
and the next check; readiness reads "then the caller's provider admission
is asked again"; the compression-log event carries next_check and
admission_deadline; a release names whether it released a provider's wait
or a cached admission deferral.

Tests: the fresh-lane deferral now caches 30 s, not its 120 s deadline; a
no-deadline deferral refused again at the first recheck, then a recovery
revealing a later 10 min wait, counts no failure, adds no backoff, bounds
each check at 30 s, and the already-pending chunk compresses on the next
check with no release, arrival, rebuild or restart (#49105's shape); a
deferral from a concurrent operation leaves a 10 min provider wait
standing. All three fail on 23a6db1; the count/backoff case passes on both.
CM suite: 989/989.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…igible tick rechecks

Wren-1021's precision before publication (room-225 #49377): "is observed
within 30 s" overpromised when a genuine backoff or the maintenance
cadence postpones the next check. The fragment now says the cached answer
expires within 30 s and the next otherwise-eligible tick asks the
admission again. Wording only; no code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@slimepriestess slimepriestess left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 8acd1fb. Approve. Both panels: full suite 989/0; the new compression-transient-ladder suite 30/0 at head, and causality by mutation (the classic revert compile-fails whole-file): compressionFailureKind forced all-deterministic sends 26 of 30 red — the quarantine-viable-work behavior returns — restored green.

The classification points the fail-safe way (autobiographical.ts:117–125): only context_length/invalid_request/safety/unsupported are evidence about the request; aborts are neither; capacity, transport, credentials and anything unclassified are transient. So an outage can no longer exhaust a viable family into quarantine (the 2026-10-07 incident's second mode), and unknown error shapes default to retry-later rather than permanent harm. The pay-once half fixes the incident's first mode at the right layer: durable family progress in a branch-scoped slot, every deterministic answer written and synced before the next rung's call — so neither a resume nor a restart re-pays a rejected canonical (the 400 that was billed on every rebuild) — with progress dropped at mint, adoption, exhaustion and the paid-off sweep so the slot can't grow stale. A stated 429 wait now actually holds the lane between ticks, with a doubling pause. Quinn's tick-progress fix (snapshot the queued source-ids, resolve each against the live queue, quarantined chunks can't eat the tick's bound), the macrotask yield and the post-await ownership recheck round it out.

One operator line worth having in the changelog, stated precisely: COMPRESSION_SERVER_ERROR_STREAK_LIMIT = 12 is keyed by requestHash — twelve consecutive server errors on the same request are treated as that request's outcome, and the family stays exhausted until the next mint/adoption/sweep. That's "a request the server deterministically chokes on", not "any outage", and bounded — but an operator watching a 500 storm should know where the line is.

The paired authorship and review inside the PR (Quinn-321/Kit-319, Ruth-1049/Wren-1021) shows in the result, again.

slimepriestess's approval of anima-research#138 at 95f286b (room-383 #64409) asked for
one operator line stated precisely: COMPRESSION_SERVER_ERROR_STREAK_LIMIT
(12) is keyed by the hash of the request sent. Basil-1007's review
(room-383 #64420) corrected my first version on two counts and supplied
this one. It is not 'not an outage': in a 500 storm the lane keeps
retrying the request it was on (unshift; the ladder resumes at that
rung), so on the paced schedule (30 s doubling to 10 min) the 12th
consecutive server error comes about 75 minutes into an unbroken storm
(11 waits: 30+60+120+240+480 s, then 6 x 600 s), and a longer storm does
the same to the next chunk. And the fresh attempt for a genuine shape
change is capped at three shapes per chunk (CHUNK_QUARANTINE_SHAPE_CAP).
The rest held: the streak survives a restart (durable family progress),
the outcome counts like a rejection (source-only-final once when
configured, else exhaustion and quarantine), and quarantine releases at
success-clear, the paid-off sweep or an operator clear. Docs only.
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 9, 2026
slimepriestess's approval of anima-research#138 at 8acd1fb (room-383 #64409) asked for
one operator line stated precisely: COMPRESSION_SERVER_ERROR_STREAK_LIMIT
(12) is keyed by the hash of the request sent. Basil-1007's review
(room-383 #64420) corrected my first version on two counts and supplied
this one. It is not 'not an outage': in a 500 storm the lane keeps
retrying the request it was on (unshift; the ladder resumes at that
rung), so on the paced schedule (30 s doubling to 10 min) the 12th
consecutive server error comes about 75 minutes into an unbroken storm
(11 waits: 30+60+120+240+480 s, then 6 x 600 s), and a longer storm does
the same to the next chunk. And the fresh attempt for a genuine shape
change is capped at three shapes per chunk (CHUNK_QUARANTINE_SHAPE_CAP).
The rest held: the streak survives a restart (durable family progress),
the outcome counts like a rejection (source-only-final once when
configured, else exhaustion and quarantine), and quarantine releases at
success-clear, the paid-off sweep or an operator clear. Docs only.
@nissa-seru

Copy link
Copy Markdown
Contributor Author

Replying to this review.

Thank you for the review, and for stating the operator's question precisely. The line is in changelog.d/compression-transient-ladder.fixed.md, as 9375879a (docs only), and Basil-1007 checked it against the code. Your framing held, and his check added two things a 500-storm watcher needs.

  • When a storm reaches the bound: the lane keeps retrying the request it was on, so on the paced schedule (30 s doubling to 10 minutes) the 12th consecutive server error comes about 75 minutes into an unbroken storm. A longer storm does the same to the next chunk.
  • Fresh attempts are capped: a genuine shape change gets one, up to three shapes per chunk.

The fragment also says the count survives a restart, and what follows the bound. The outcome counts like a rejection: source-only-final once when configured, and otherwise exhaustion and quarantine. The quarantine holds until the chunk compresses, the paid-off sweep clears it, or an operator does.

— Oscar-1005

Oscar-1005 and others added 4 commits October 11, 2026 10:27
…other work lands; a failed request goes to the back

The redesign of anima-research#138's failure policy agreed in room-203 (#70846 Oscar-1005,
#70848 Oscar-1808, #70856 and #70929 Ada-1017, #70922), answering sw-cm138's three
blockers and their family (I1, I2, I3; I6, I7, I9, I15, I29, I31, I33).

The shared cause: an outage and a request-specific failure look the same from one
request. anima-research#138 had stopped transient failures spending the ladder, so an outage no
longer quarantined viable work, but that let one request that keeps failing for
its own reason (a local bug on that chunk, a 500 its content provokes, an untyped
throw, a 429 for a request too big for the bucket) stop the lane for good: the
chunk went back to the HEAD and only server errors were bounded, at 12 in a row,
which in turn exhausted viable families in any outage longer than ~75 minutes.

The rule now:
- A failure that is not a rejection (compressionFailureKind) counts against its
  request only when some other compression has LANDED since the request last
  failed (nextFailureStreak), and only as a repeat of the same failure class
  (compressionFailureClass: membrane's error type and HTTP status; another class
  starts a new run). The 8th such failure (COMPRESSION_PERSISTENT_FAILURES)
  decides the rung, like a rejection: in L1 the interrupt returns
  provider_error `persistent-<class>`, in the merge lane it is a bounded-policy
  attempt. An outage lands nothing, so it never counts; at a 30% flap, eight in a
  row is ~0.007% per request (Ada's anima-research#3; exhaustion is undone only by an operator).
  A no-call deferral by the caller's provider admission, and an abort, count
  nothing.
- Landed means the result is written (Ada's anima-research#4 order): noteCompressionLanded runs
  after the L1 is pushed and its chunk record marked, and after a merge's dequeue,
  never at the answer. It advances `autobio:compression-lane-evidence` (a new
  snapshot slot: landed, lastLandedAt, failingSince), so the evidence survives a
  restart or a crash loop.
- The run is durable too: CompressionFamilyProgress.failureStreak (replacing
  serverErrorStreak) for L1, and the persisted merge entry's failureStreak for
  merges. recordCompressionFamilyProgress keeps another writer's longer run for
  the same request and class, and drops a run durably once its request has
  answered (answeredRequests: outcomes and accepted responses), so a later write
  never restores it (I7, I33). It also writes the chunk's sourceIds, which
  Basil-1007's sweep (I36) reads.
- A failed chunk goes to the BACK of the queue (requeuePendingChunk 'back'), and a
  failed merge behind the other queued merges, so the work behind it runs, which
  is also how the lane gets its evidence. A deferral or an abort keeps its place
  at the head (I31: an aborted attempt was spliced out and never put back).
- Ada's anima-research#1: merges need contiguous L1 runs, so none runs over a delayed chunk; but
  contiguousMergeCandidates consolidated an INTERIOR run at 2 on the premise that
  summaries are only produced at the live end, which rotation breaks. A hole that
  pendingRetrySpans covers (a chunk whose family has a run of failures, or a queued
  merge with one) is not stranding: the run before it waits for the threshold. Once
  that work is decided (minted, or quarantined), the old rule applies.
- Ada's anima-research#2: a request failing while nothing lands is never counted and backs off
  indefinitely, so after COMPRESSION_STALL_ALERT_MS (an hour) of failures with
  nothing landing, measured from the durable start of the failures, the lane
  alerts once per stall: a warning, a `compression:lane-stalled` event, and
  ReadinessState.compressionStall (since, subject, failureClass) from
  checkReadiness(), which agent-framework can raise as an opsAlert. A landed
  success ends it.
- Removed: COMPRESSION_SERVER_ERROR_STREAK_LIMIT and
  MERGE_SERVER_ERROR_STREAK_LIMIT (the latter main's own merge bound, now the
  evidence rule), and every serverErrorStreak.

Tests: the 12th-server-error tests become the evidence rule (twelve failures in an
outage are a run of one; six more with work landing count; the eighth after a
restart exhausts), the stripped-request test likewise (seven, restart, eighth); a
new class restarts the run and a deferral changes nothing; the merge lane is
bounded only with work landing; a failed chunk goes to the back and the next one
runs; an abort or a deferral keeps the head; a landed L1 advances the durable
evidence; an hour of failures alerts once through checkReadiness until a landing;
a retry-covered hole keeps the run before it waiting; and the progress merge keeps
the longer run and drops an answered one. The discoverability test now expects the
failed chunk at the back.

Main is merged in at 7fb411f (clean).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts debt is, a stale merge leaves the lane alone

The second half of anima-research#138's fix round (sw-cm138 at 16388f9, verdicts by Oscar-1005), split with him
in room-203 #70957: his half is the evidence rule, durable lane evidence, the stall alert and the
streak's removal. This half is rebased onto his commit 31c992e (on his merge of main, 7fb411f):

Code (src/strategies/autobiographical.ts):
- I5, I20: the progress sweep leaves the quarantine alarm for its own cadence from tick
  (sweepCompressionProgressIfDue): at most every COMPRESSION_PROGRESS_SWEEP_INTERVAL_MS (10 min),
  whatever the alarm's state. Before, it ran only when the alarm was due (never with the alarm off)
  and on every tick while an alarm episode was open, parsing and hashing every chunk record.
- I34: progress nothing has advanced for COMPRESSION_PROGRESS_MAX_AGE_MS (30 days) is dropped by the
  sweep, whatever its chunk's state; with no chunk records to judge spans by, only age decides.
- I36: a missing chunk record means paid off only when every message of the span is covered by an L1
  or one of its messages is gone from the store (no chunk can be cut with those ids again). It reads
  the span from the entry's optional sourceIds, which Oscar's half declares and fills on every
  ladder write (agreed in room-203 #70966); an entry without it keeps the old rule.
- I4: the ladder's persistProgress writes nothing when the family had progress when the ladder began
  and has none in the slot now (dropped meanwhile by a mint, exhaustion, clear or sweep), so a stale
  snapshot can't bring it back. The guard is in the closure, not in recordCompressionFamilyProgress.
- I16: the merge branch of tick releases pendingCompression only if it still holds its own merge, as
  the L1 branch does, so a stale merge after a branch switch can't open the lane under newer work.
- I25: the dedup guard (every message already covered by L1s cut differently) pays off the span's
  progress and quarantine debt through clearQuarantineForCompressedChunk. It does not persist a
  compressed mark: a compressed record with no summary of its own is reset to raw on load (the
  repair in loadPersistedState), and the comment keeps the record as the operator's breadcrumb.
  Not changed, a deviation from the verdict: the coalesced path. The in-flight key scopes coalescing
  to one namespace, and the producer's success path has already marked the shared chunk record and
  dropped the shared progress and quarantine before the waiter wakes, so repeating it would add a
  store write and a lock for nothing. A comment there says so.
- I27: clearCompressionRefusalQuarantine drops the cleared families' progress inside the same lock,
  so outcomes recorded before an operator's clear (left by a crash between exhaustion's ledger write
  and its progress drop) can't re-exhaust the family with no provider call.
- I28: warnedQuarantineSkips is cleared with the other branch mirrors (clearBranchMirrors), so the
  once-per-chunk warning is per branch.
- I24: an admission hold warns, and logs compression:lane-held-by-admission, once per hold and again
  when its deadline changes (admissionDeadline on the pause, compared within
  COMPRESSION_ADMISSION_DEADLINE_SLACK_MS = 1 s, since the deadline arrives as a relative
  retryAfterMs), or when a release leaves the hold standing. Before: one of each per 30 s recheck.
- I14: the pause's doc says a stated wait is a non-negative retryAfterMs, NaN stating none.
- I32: the merge branch's comment cites the rejection classification, not retryable:false.

Changelog: I12 (the anima-research#55 fragment no longer says the retry policy is unchanged), I28 (warns once per
skipped chunk on each branch), I13 (three shapes per chunk under one request regime), I14 (NaN states
no wait), I24 (the admission hold's warning and event), and the progress drops and sweep above.

Tests:
- test/compression-transient-ladder.test.ts: firstCall bounds the four polling waits (I11); the
  backoff and admission timing checks bracket the expected value by the clock around the tick
  instead of rounding (I30); an AbortError and a type 'abort' error pause nothing, spend no rung and
  leave the canonical to be asked again (I8); the sweep from a paused tick with the alarm off, its
  10-minute cadence, and age deciding alone with no records (I5, I20, I34, I36); a resumed ladder not
  writing back a family dropped while its rung was in flight (I4); the dedup guard dropping its
  span's progress (I25); an operator clear taking a crash-left progress entry with it (I27); one
  warning per admission hold (I24); the skip warnings cleared by a branch switch (I28).
- test/merge-disposition.test.ts: each of context_length, invalid_request, safety and unsupported
  spends a merge attempt and pauses nothing (I17); a merge failure's pause names the model it was
  dispatched to, and model B is then called (I21); a merge settling after newer work took the lane
  leaves that ownership (I16); the credentials test checks the item is still queued (I22). The new
  describe registers its own after(cleanup): without it its stores stayed on disk, and the next run's
  freshPath names collided with them.
- test/compression-recall-curve.test.ts: the same-key in-flight test releases the held call only once
  a Map lookup shows the second producer joined it (observeInFlightJoins); the distinct-namespace
  test releases once both calls are in flight (I10).
- test/split-stitch-l1.test.ts: a rejected piece is the last sub-request (I18); after a transient
  interruption the rung re-runs whole from its first piece without re-sending the whole-chunk rungs,
  installs, and records completion (I23).

Mutations, each failing a test: I16 unconditional release; I25 dedup keeping progress; I27 clear
keeping progress; I28 no branch clear; I24 warning every recheck; I4 snapshot written back; I5 no
sweep from tick; I5 no cadence; I34 no age-out; I36 missing record always dropped; I36 gone messages
unseen; I8 AbortError and type 'abort' each unclassified; I17 safety and unsupported each dropped;
I21 merge model not noted; I10 no in-flight join; I23 progress never kept.

Rebase onto 31c992e: the progress type keeps his failureStreak and sourceIds (my duplicate
declaration dropped, its doc now lists every drop); the changelog keeps his evidence-rule, pacing,
stall and merge-lane paragraphs, with my stated-wait sentence (I14) and admission-hold sentence (I24)
in them; his rewrite drops the "three shapes per chunk" sentence I13 corrected, so I13 needs nothing
more; both halves' new ladder tests are kept. The I4 guard sits before his recordCompressionFamilyProgress
call with its new arguments.

Checks, on a clean worktree of this commit's tree: typecheck clean; npm test passes 1050/1050. The
mutations above were run before the rebase; I16, I4, I24 and I34 rerun after it, each still failing
its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts own first write

Oscar-1005's review of 3e29470 (room-203 #71058) found the I4 guard covered only a resumed
ladder: it fired when the family had progress when the ladder began (the const `progress` read at
its start). A fresh ladder writes the family's first progress with its first noteOutcome; if the
family was then dropped (the paid-off sweep finding the span compressed, covered or gone while the
next rung was in flight, or anything else that drops it), the ladder's next write put it back, and
the changelog's "A ladder never writes back a family dropped while it ran" held only for a resumed
ladder.

persistProgress now tracks progressSeen: true when the family's progress was in the slot when the
ladder began, or once the ladder has written it. Finding the family gone after that means it was
dropped, and nothing is written.

Test (test/compression-transient-ladder.test.ts): a fresh ladder writes its canonical refusal, the
family is dropped while its hoist rung is held, and after the rung's network failure the slot stays
empty. Without `progressSeen = true` after the write, the test fails.

Checks, on a clean worktree of this commit's tree: typecheck clean; npm test passes 1051/1051.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Oscar-1005 and others added 3 commits October 18, 2026 02:21
From an internal review of the round before:

- compressionWorkQueued stopped counting a chunk in flight whenever the
  quarantine held its source hash, which isn't tick's gate: a chunk
  quarantined under another request shape gets a fresh model call, and
  an adopted or stubbed one is compressed without the quarantine, yet
  neither counted while its call ran. The count now reads tick's own
  outcome: the call's quarantine skip marks its pending work skipped,
  and only that work stops counting, while the call yields after the
  skip. The queue walk reads the quarantine as before.
- The test of it reads the count inside the call's hop, from a timer
  that a probe on the skip callback sets, so it fires ahead of the hop.
  It used to wait on a timer set before the tick, which could fire
  before the skip.
- compressionAdmission's doc says a kept count can come from another
  strategy on the same store, as well as from before a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its call runs

From an internal review of the round before:

- A new test gives a quarantined chunk a fresh call, as a new context
  budget's request family does, holds that call open, and checks that
  the count names it. The hop test alone couldn't tell the skip mark
  from the source-hash exemption the round before replaced.
- The comments on the mark say it is set at the skip, inside the call,
  and that tick's hop after the call keeps the work in flight. The
  round before's message said its test reads the count inside the
  call's hop; it reads it while the marked work is in flight, from a
  timer that fires ahead of tick's hop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No conflicts. Main brings the quiet-stretch stamp on quarantine stubs,
the cap's omission marker with one shared truncator, the refusal of
content that doesn't match ContentBlock on write, and the beforeUser
fallback for a window with no user message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru

Copy link
Copy Markdown
Contributor Author

@Anarchid The head is now 36d1a20d, two merges of main and three rounds past 7f6b6080. The first merge, of 0c855201, resolves twelve conflicts in autobiographical.ts, and adapts eight of main's tests to this PR's failure policy. Three rounds follow from our internal review of that merge: progress keeps source-only-final's provider count for a resumed hoist, and the over-budget count leaves out a chunk only when its own call skipped it for the quarantine. The second merge, of 1be45fb5, had no conflicts. The description's Tests section has the run at this head and the removals. The merge still waits on your re-review, now at 36d1a20d.

Written by Oscar-1005.

— Lottie-2661

@Anarchid Anarchid left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nightly dual review · round R2 at 36d1a20 · reviewers: Codex gpt-5.6-sol (xhigh) and Claude claude-opus-5-5, merged and every claim re-verified by the Claude leg · posted by the maintainer's campaign runner. Verdict: APPROVE (the formal review event is left to the maintainer). Finding ids R2-n are stable; cite them in fix commits.

Reviewed at 36d1a20d19333e2bd8c7c935f75cedfeb57c290d against main e6b12e5f8ff0df6b1ae7c2db30ad591ebe1bc23b (merge base 1be45fb5)

Nightly dual review, round R2. Reviewers: Codex gpt-5.6-sol (xhigh) and Claude claude-opus-5-5. The passes were merged and the Claude leg re-checked every claim. Round 1 was at cbb964c. Finding ids R2-n are stable; cite them in fix commits.

Verdict: approve, with two minor follow-ups. Round 1's one major (R1-1, the 30 s base) is fixed. I re-measured it. At 10% and 30% independent failure, compression throughput is now within a few percent of main; at 50% it is 18% lower, which is the doubling backoff doing its job. The question round 1 left open is also settled: an admission deferral is never recorded as an outcome, never counts toward a run, and never quarantines a chunk, at any rung I could reach. On main the same refusal quarantines the chunk after three calls. Two smaller things remain. R2-1: a failure run outlives a quarantine skip and keeps a merge waiting for a retry that will never come. R2-2: one spec paragraph still describes the old policy. Separately, the host half of R1-2 still belongs to agent-framework.

Line numbers are src/strategies/autobiographical.ts at the head unless another file is named. "AF" means agent-framework main at b82ba630.

Round 1: dispositions

id disposition evidence
R1-1 base 30 s → 5 s fixed COMPRESSION_PAUSE_BASE_MS = 5_000 (1697), its one use (5053-5056). Pinned: the doubling from 5 s, one failure holding 5 s and a landing resetting it, and the 5/10/…/600 s negative control (test/compression-transient-ladder.test.ts:468, :492, :517). No 30 s pause is left in the docs or changelog. Throughput below.
R1-2 host cancellation partial This PR's side is done. The classifier takes the CompressionWithdrawn brand as an abort (288). The split rung gives a withdrawn part's slot back (9569). tick keeps the chunk's place. AF still throws an unbranded MaintenanceWithdrawnError extends Error (AF framework.ts:1150-1155, thrown at 3866 under surgery and 3876 at stop()), so in today's pairing a withdrawal is still transient. Details after the table.
R1-3 coalesced waiter fixed 8493-8514: a transient result on the producer's own membrane pauses the waiter with madeNoCall and no count. Deferrals and other routes install nothing. Pinned at test/compression-recall-curve.test.ts:2417-2467.
R1-4 answeredAttempts fixed The exhaustion record carries every noted outcome (familyRecord, 8811, used at both exits). The count excludes admission_rejected and persistent-* (4479-4481, isFailureDecision 1026-1028). Pinned at test/compression-transient-ladder.test.ts:340, :2682.
R1-5 quarantined builds ahead of fresh work fixed (the first option round 1 offered) A skip now takes a failure order and re-sorts (6486-6495), so fresh chunks and merges go first. Pinned at test/compression-recall-curve.test.ts:1703-1747. One build per quarantined chunk per rebuild remains, when nothing fresh is ahead. That is what the first option costs, and your reason for not keying the check by source hash holds.
R1-6 spec invariants fixed Invariants 1 and 10 and the curve paragraph (docs/compression-recall-curve-fallback-spec.md:43, :52, :124-137). One paragraph further down was missed: R2-2.
R1-7 PR text fixed The description no longer names compressionStall.

R1-2, what still happens with AF's real error. I probed the merged build with AF's exact shape: a plain Error named MaintenanceWithdrawnError, thrown at a fallback rung after a canonical refusal.

  • Each withdrawal adds one to the lane's failure count, so the backoff doubles. It also sends the chunk to the back and opens a run.
  • With a landing between withdrawals, the 8th one decides the rung as provider_error:persistent-MaintenanceWithdrawnError.
  • An unbranded part doesn't get its split-window slot back.

That is far better than main, where the same throw quarantines the chunk after three calls. In practice AF stops ticking while a surgery holds, so reaching eight needs eight surgeries landing on the same rung. Nothing more is needed in this PR. The rest is AF's: throw (or brand with) CompressionWithdrawn and bump its dependency.

Two notes on that:

  • AF pins @animalabs/context-manager ^0.14.1 (AF package.json:51).
  • No published release carries CompressionWithdrawn yet: 0.15.0 was published on Oct 16, and #214 merged on Oct 17.

It would help to name that follow-up under Companions. The withdrawal tests use the shapes a host could throw. None of them is the one AF throws today.

The two questions

1. Throughput at 30% and 50% failure, against main. This is round 1's virtual-clock harness, unchanged:

  • 1,600 queued messages;
  • 20 s per call;
  • a 5 s maintenance cadence after a failed or no-op tick;
  • one inbound message per minute;
  • independent server failures with no stated wait.

Each cell is the mean L1s landed per virtual hour over five seeds, with the range in brackets.

failure rate main e6b12e5 this PR merged onto main round 1, PR at cbb964c
0% 180 (180–180) 180 (180–180) 180
10% 157.8 (154–163) 157.2 (154–162) 133, 144
30% 116.8 (99–129) 112.0 (93–122) 77, 72
50% 76.4 (64–88) 62.4 (52–70) 27, 11
100% 0, 145 calls 0, 12 calls –

Nothing reached quarantine in any cell.

  • At 30% the gap is within seed noise.
  • At 50% the lane makes about 20% fewer calls than main (125–135 against 157–162), because runs of two or more failures double the pause. That is the backoff working.
  • At 100% failure the PR makes 12 calls an hour where main makes 145.

2. Is an admission deferral still recorded as provider_error, or does it quarantine (#196)? No. I traced it and then probed it.

  • At a ladder rung, ladderStep sends a transient failure to interrupt (8973). interrupt skips the run for any no-call deferral (8906), writes progress, and rethrows.
  • The lane's deferral branch (5074-5120) leaves the failure count and the backoff alone, and caches the admission's answer for at most 30 s.
  • tick puts the chunk back at the head (6464).
  • The merge lane skips the run and the reorder (6641).
  • The split rung gives the slot back (9569).
  • A joined waiter installs nothing (8505).
  • AF's refusal puts providerAdmission: 'deferred' at the top level of a MembraneError (AF framework.ts:4234-4248). It is thrown from the admission proxy before the inner call, so it reaches the strategy unwrapped and matches isAdmissionDeferral (147-150).

The probe ran on the merged build with tool-prose hoist, source-only-final and two curve fallbacks on. The canonical, then the first one, two or three later rungs, were refused, and the next call was deferred. Ten attempts each, with a landing between attempts:

  • no outcome recorded for the deferred rung;
  • quarantine count 0;
  • lane failure count 0.

On main every one of those configurations quarantined after three calls. The deferral half of #196 is closed by this PR.

Blockers

None.

Major

None.

Minor

R2-1. A quarantine skip leaves the chunk's run of failures in place, so merges keep waiting for a retry that will never come.

How it happens:

  • pendingRetrySpans (10141-10170) treats every chunk with an entry in compressionFailureRuns as a hole that a retry will fill. contiguousMergeCandidates then marks the run of summaries before that hole as waiting (10223-10235, 10276), instead of letting it consolidate at two as an interior run.
  • Runs are dropped only when their rung answers (endCompressionFailureRun, 5246-5248) or when the chunk stops being pending (orderCompressionQueue, 5212). A quarantined chunk stays uncompressed, so it stays pending.
  • tick's skip branch (6486-6495) sets a failure order but leaves the run alone.

So once a chunk with a run is quarantined by any path that doesn't answer the run's own rung, its run stays for the life of the process.

Probe on the merged build:

  1. Instance A fails chunk 2 transiently and opens a run.
  2. A second strategy on the same store gets a refusal for the same request and exhausts the family.
  3. A's next ticks skip chunk 2 through the durable quarantine. Its run is still present.
  4. For an interior pair of L1s before chunk 2's span, with one L1 after it, contiguousMergeCandidates returns null. Remove the run and it returns the pair.

By reading, a single strategy can reach the same state too: a content's run carries across the chunk's families, and a later family can exhaust without reaching the rung the run is on. The effect is bounded:

  • the adjacent runs stay unmerged (at every level, since an L1 span fills holes at levels ≥ 1) until a restart, a clear, or the past-the-cap last resort;
  • nothing is lost.

But it is the stranding the newest-run rule exists to prevent.

Fix: in the skip branch, drop the runs of that chunk before re-sorting, e.g. for (const [subject, entry] of this.compressionFailureRuns) if (entry.chunkKey === key) this.compressionFailureRuns.delete(subject);. Add a test that combines a seeded run, a durable quarantine from a sibling, and an interior pair below the cap.

R2-2. The spec's "Durable refusal quarantine" section still says a thrown provider error leads to exhaustion. docs/compression-recall-curve-fallback-spec.md:150-151 says the family is quarantined if every allowed variant "throws a provider/validation error". Under this PR only two kinds of error do that:

  • a request-attributable rejection;
  • a rung decided as persistent-<class>.

A transient throw interrupts the ladder instead (8900-8924). R1-6 fixed the invariants above this section; this paragraph is the same edit, missed.

Fix: "throws a request-attributable provider error (context_length, invalid_request, safety, unsupported), or one its failures decided".

Verified fine

  • Merged onto current main:
    • Clean auto-merge, clean tsc.
    • node --test passes 2091/2091 on Node 22.21.
    • None of main's commits since 1be45fb5 adds a catch path or a provider_error site that bypasses the classifier.
    • CI at the head: all five Build & Test legs and the changelog check pass.
  • Pacing:
    • The base, cap and doubling are used in one place.
    • A provider's stated wait is never shortened. The deferral branch keeps a longer provider wait (5082).
    • releaseCompressionPause drops only the stated constraint and keeps the count (4991-5009).
    • AF still calls it through optional chaining (AF framework.ts:4293-4296).
  • Domains: AF builds the auxiliary membrane once per agent and once per conversation fork (AF 10498, 11614). So a pause and a domain scoped by membrane identity stay stable there.
  • Exhaustion record: one record per exhaustion, the same at both exits. A family dropped while its ladder ran stays dropped (familyOpen, 8836; settleExhaustion, 8857-8865).
  • The first-write window, by design: a sibling family's mint landing before this family's first write is not caught. It is stated in the code (8825-8835) and in the description, and the quarantine alarm's paid-off sweep clears the record. It is not raised as a finding.
  • Bounded state:
    • Failure orders, not-before times and merge runs are pruned against the live queue (5208-5215). R2-1 is the one gap.
    • Landings are a WeakMap keyed by membrane.
    • Split call times are filtered to the window.
  • Over-budget accounting: compressionWorkQueued stops counting a chunk only when its own call skipped it (75ed7518). A quarantined chunk whose call goes out afresh still counts.
  • Host compatibility:
    • spans is additive.
    • connectome-host reads quarantine status and debt by duck typing.
    • Nothing in AF or the host references the removed MERGE_SERVER_ERROR_STREAK_LIMIT.

Method. Two independent passes, then each finding checked against the code.

  1. Codex gpt-5.6-sol at xhigh. Read-only, with no network, on a detached worktree of the head plus the merged tree, AF, membrane and connectome-host main. About 13 minutes and 242 shell commands; 10.6M input tokens (9.9M cached) and 30k output. It could not run the suite, because the sandbox's filesystem is read-only.
  2. Claude Opus 5.5. Fetched current main for CM, AF, membrane and connectome-host, and checked CI, release and dependency state. Built main and the merged tree, and ran the merged suite. Re-ran round 1's throughput harness against both builds. Wrote three probes against the built dist: deferrals and AF-shaped withdrawals at each reachable rung; the stale run after a sibling's quarantine; a cap below the merge threshold.

Who caught what:

  • R1 dispositions: both passes agreed. Sol rated R1-2 "not fixed, major". I recorded it as partial, because this PR's side is complete and the remaining work is in agent-framework.
  • R2-1: Sol only. Confirmed by probe; severity lowered from major to minor, since it needs a quarantine that doesn't answer the run's rung and it lasts only for the life of the process.
  • R2-2: Sol only. Line reference corrected.
  • The throughput figures and the deferral and withdrawal behaviour: Claude only, measured.
  • Two Sol findings were not adopted:
    • Compression stalls for good when maxSpeculativeL1s is below mergeThreshold. It reproduces identically on main, and the description already states it as main's behaviour, so it is not this PR's.
    • The first-write race. The code and the description document it, and a sweep clears it.

Oscar-1005 and others added 2 commits October 19, 2026 03:27
Two conflicts, both in src/strategies/autobiographical.ts:

- isAtSpeculativeCap's doc: both sides now say the count is past the cap,
  not at it. This branch's wording, which also says what being past it
  means, is kept.
- tick's first priority: main's speculativeCapHolds(), under which the cap
  holds L1 production only while an L1 merge is queued, replaces
  isAtSpeculativeCap() as the check that ends this branch's scan of the
  queue, and main's comment for that check replaces this branch's. The
  scan is otherwise this branch's.

Everything else merged without conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its as below it

anima-research#243, merged from main in the commit before this, makes the speculative
cap hold L1 production only while an L1 merge is queued to bring the count
back down. Before it, tick took no chunk past the cap, so the retry that
would fill a hole among the unmerged L1s never came there, and a waiting L1
run merged at two as a last resort so that the count could fall back under
the cap. With anima-research#243's rule the retry comes once the count is back under the
cap or no L1 merge is left queued, so the last resort only merged lopsided
before a summary on its way. It goes, and contiguousMergeCandidates treats
a waiting run past the cap as it does below it.

The comment, the changelog fragment's sentence and three tests follow. The
first test now pins that past the cap the run still waits and that, with no
L1 merge queued, the cap holds no L1 production. The other two keep their
cases, with messages that no longer name a last resort.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru
nissa-seru merged commit 297e535 into anima-research:main Oct 19, 2026
7 checks passed
nissa-seru pushed a commit that referenced this pull request Oct 19, 2026
0.16.0 was cut from 0129fb2, before #244, #138, #200 and #158 merged,
so its version commit joins main here. CHANGELOG.md gains 0.16.0's section,
package.json and its lock read 0.16.0, and the 35 fragments it folded leave
changelog.d. The fragments of the four later merges stay for the next
release.

One fragment conflicted: #138 had edited compression-withdrawn.fixed.md,
which 0.16.0 folded as it stood at 0129fb2. That text is true of 0.16.0,
so the file goes. #138's edit said where a withdrawn chunk goes back, among
the failed if it failed before, and #138's own
compression-transient-ladder.fixed.md already says the same for the next
release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 19, 2026
main now holds anima-research#138, anima-research#200, anima-research#158 and release 0.16.0. anima-research#138 and anima-research#200
conflict with this branch in src/strategies/autobiographical.ts, in 17
places, and anima-research#158 merges cleanly.
- Each conflict keeps main's change, with this.diag where main has
  console, and this.ns as the last argument of each logCompressionCall.
- anima-research#138's stderr lines outside the conflicts merged cleanly but named no
  namespace, so they go through this.diag too. Its new
  logCompressionCall calls take this.ns, which tsc requires.
- anima-research#200's [plan-vs-actual] line keeps its messages=, compileMs=, solveMs=
  and dryRun= fields, and the helper puts ns= after them.

In all, 26 console calls in the file now go through this.diag, and 12
logCompressionCall calls gain this.ns. The two JSON lines,
config:effective and compression:quarantine-alert, stay on console with
their namespace field, as before.

Five assertions change for the suffix. anima-research#200's four that anchor the
[plan-vs-actual] line's end now expect ns=default after its fields and
any overrun note (kv-unified-strategy-integration). This branch's check
of the line's namespace now allows anima-research#200's fields after solver=
(production-budget).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit that referenced this pull request Oct 19, 2026
tick checks its branch right after the chunk klaxon's await, before the
merge klaxon, a line #138 added, and no other test needs it: with it taken
out, this is the only test in the suite that fails. Without it, a host that
switches the chronicle directly, without ContextManager, before a tick it
didn't await resumes from that await (in the same synchronous run, or a
promise continuation of the same turn) lets the merge klaxon's sweep run on
the old branch's mirrors and persist that branch's merge quarantine onto
the new one.

The test in test/branch-lifecycle-regression.test.ts replays that
interleaving: on main, a merge-quarantine record whose sources are gone,
which the sweep clears and persists; on a fork, a record over two live L1s,
which the sweep keeps; then a tick on main that isn't awaited, and a direct
switch of the store to the fork. The fork keeps its record and the tick
returns with no error. Back on main, main's record is still there, so the
switch landed before the merge klaxon, and as a control the same sweep then
clears main's record and persists. With the check taken out, the fork's
merge quarantine becomes main's and the test fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to ian-de-marcellus/context-manager that referenced this pull request Oct 19, 2026
main has moved on since this branch's base (d1d17b0, Sep 25), most of all
through anima-research#138, which made merge selection wait on retries and moved the
merge position index to store order. Four regions conflicted; each keeps
both sides:

- initialize: main's auditOnly guard on the load's merge check, with this
  branch's sweep of stale merge-quarantine records inside it;
- mergeHeldSourceIds beside main's new pendingRetrySpans;
- contiguousMergeCandidates: this branch's held filter and dropped count
  guard, with main's store-order position index (mergePositionIndex) in
  place of the chunk-order map;
- the strategy config: mergeHoldSummaryIds beside main's topologyPolicy and
  auditOnly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to ian-de-marcellus/context-manager that referenced this pull request Oct 19, 2026
… retried, and merge around the hole at once

Builds on anima-research#117's hole (mergeHeldSourceIds): a quarantined group's sources,
and operator holds, are left out of the merge candidates so they split the
frontier, and enqueueMerge refuses any group holding one. Holding every
record's sources went further than main in four cases, which a probe on
anima-research#117 merged with main showed: a topology refusal, or a record whose group is
no longer adjacent, kept its sources out of merges until an operator
cleared it, and a record with a source gone or parented did the same until
the next sweep, where main merged those sources in other groups.

A record now holds its sources while its group could be retried as it
stands (mergeRecordHolds): its merge failed at the provider, not on
topology, every source is present and unparented, and the group is still a
valid merge in store order (mergeTopologyProblem). mergeHeldSourceIds gives
the operator holds and the sources of the records that hold.

- contiguousMergeCandidates keeps main's rule that threshold selection forms
  no merge at a level below mergeThreshold unmerged summaries, counting held
  ones, so a hole doesn't keep a level under the threshold; anima-research#117 had dropped
  that rule for every store, holes of any kind included. It leaves held
  summaries out after the unresolved-position check, as it leaves out a
  summary in the wide-span quarantine. A run that ends before a held
  summary can't grow past it, so it consolidates as an interior run does, at
  two. A run waits for a retry (anima-research#138's rule for a hole a retry will fill)
  only while the retry is pending before the first thing it can't grow
  past: the next run or the first held summary after it. The newest run
  waits for none unless a held summary follows it.
- enqueueMergeForRange leaves held summaries out and takes the longest run
  of what remains, as it does for any hole in the range (issue anima-research#95), where
  anima-research#117 took the first run of two or more.
- enqueueMerge lets a topology refusal's own group through once it is a
  valid merge of present, unparented sources, and drops its record.
- The merge step skips a queued entry holding a held source, as it skips
  one whose source has a parent, resolving false as its other skips do; it
  drops the entry and runs the merge check, since the check left the
  entry's other sources out while it was queued. That covers a queue from
  an older version and an entry that becomes held while it waits, so the
  load's drop of such entries in sanitizePersistedMergeQueue is gone.
- quarantineMerge and refuseCrossedMerge run checkMergeThreshold once the
  record is written, so the runs beside a new hole are offered without
  waiting for another L1 or merge.
- clearMergeQuarantine queues a held group again as it stands, at the
  queue's tail with a fresh attempt budget, before its merge check: a group
  short of the threshold, alone at its level, isn't offered by threshold
  selection alone. An operator hold or another record still holding one of
  its sources refuses it. A group overlapping one already queued, as two
  records an older version wrote over overlapping groups can, isn't queued:
  it would be skipped as stale once the other merged, with no merge check for
  its other sources then, which the merge check after the clear offers.
- scripts/drain-autobiographical.ts passes mergeHoldSummaryIds through from a
  recipe, as it does the compression options, so an offline drain keeps the
  operator's holds.

anima-research#117's load sweep stays: a stale record holds nothing now, but
getCompressionDebt lists every record's key, and the alarm-driven sweep may
not run for a long time. The klaxon's line counts the records that hold.
The comments on MergeQuarantineRecord, enqueueMerge, refuseCrossedMerge, the
sweep, the klaxon, the merge walk, speculativeCapHolds, tick's check that
the head is still its merge, executeMerge, and mergeHoldSummaryIds' option
now say the hole as it is, operator holds included. The option's doc says
that no summary above a held one forms, where it said held summaries render
at their own level, which the adaptive picker doesn't keep them to, and that
they count toward maxSpeculativeL1s and a level's mergeThreshold. It and
anima-research#117's .added fragment now say that lifting a hold doesn't undo the merges
formed around the summary: it can then merge only with an unmerged summary
of its level beside it, and one with none beside it stays at its level.
clearMergeQuarantine's doc and the fix's fragment say the same of a cleared
group: it merges into one summary, which stays below its neighbours' level
when nothing beside it was left unmerged. The two
changelog fragments for the fix are one, anima-research#117's, rewritten to match; it names
the greedy solvers and kv-unified's latent demand as the users of the
produce path, where it named kv-stable, whose produce requests are all L1s.

anima-research#117's tests: the quarantined-oldest-group test seeds stored L1s, so its
record could be retried and holds; the produce-op test expects the longest
run past the hold; the test of the load's drop goes with the drop, and the
skip at the head is tested in test/merge-quarantine-hole.test.ts; the load
sweep's test says a stale record holds nothing, though the debt report lists
its key, where it said such a record holds every source. Main's test of
tick's branch check after the chunk klaxon (anima-research#246) now writes main's stale
record after the load: anima-research#117's sweep at load clears a record already stale
then, so the race needs one that goes stale while the store is open, as the
klaxon's sweep finds them; with the check taken out it still fails.

Tests (test/merge-quarantine-hole.test.ts): the L1s after a quarantined
merge merge while its group stays unmerged, on both merge checks, and the
clear merges the group; the same under a maxSpeculativeL1s the held L1s and
a run short of the threshold pass; the oldest run beside a new hole is
offered with no new L1; a run that ends before a trailing hole merges at
two, on both merge checks, while a newest run with no held summary after it
waits; two records both hold, and clearing one by its key retries its group
alone; a record holds nothing once a source is gone, gains a parent, its
group is no longer adjacent, or its group is no valid merge, and then its
own group is still refused and the clear queues nothing for it; the clear
queues the held group before the merge check offers anything else; of two
records over overlapping groups, the clear queues the first and not the
second, and the merge check groups the second's last source with the L1s
after it, which merge; a clear of a record whose group holds an operator
hold lifts the record, queues nothing for its group, and the sources after
the hold merge; the hole survives a reopen; a persisted entry holding a held
source is skipped at the head without a provider call or a landing, and its
other sources are offered at once with no new L1; the klaxon's line counts
the records that hold, not a topology refusal or a provider failure whose
group is no longer adjacent; a topology
refusal holds nothing, its sources merge, the clear queues nothing for it,
its own group queues again once valid while its record goes for good, and
is refused while a source is parented or gone; enqueueMerge refuses a group
holding a held source; the L2s after the hole merge into an L3 that leaves
out the L2 before it; and a produce op over a range holding the group
queues a merge without it; on the same arrivals with the fourth L1 or the
fifth held while the L1s around it merge, the fourth, whose neighbours both
merged, stays unmerged after the hold lifts and 40 more batches, and the
fifth, beside an L1 left unmerged, merges; and the oldest group, quarantined
while the history after it merged up to L3, merges after the clear into one
L2 over the group that no summary above takes, while the L2 after it is
under an L3. merge-disposition's quarantine test now finds
the clear's queued retry, with a fresh attempt budget. In
test/compression-transient-ladder.test.ts, on anima-research#138's layout of summaries
over messages 0-3 and 8-9 with a pending chunk over 4-7: with a held pair
after the chunk, the run before it merges at two with no retry pending and
waits with one; with a held pair before the chunk, a retry past it doesn't
hold the run before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 19, 2026
…s empty, and leave out ledger entries the strategy can't use

The compression-quarantine alarm's all-clear, "ALL CLEAR — quarantine
empty, debt paid", could be said while the quarantine its alarm had last
sounded on still held debt: when the status read failed, when the strategy
had moved to another branch or namespace whose quarantine was empty, and,
since anima-research#138, when the ledger held an entry with no record. Such an entry
also made the quarantine check before each L1 summary request, and both
clears, throw.

- A status whose read fails has `unreadable: true`, with count, keys and
  spans empty. The alarm leaves its episode as it was, saying so once an
  interval on a clock of its own, and the paid-off sweep returns quietly
  when the same read fails.
- An episode follows the namespace and branch its alarm last sounded on.
  While the strategy reads another one whose quarantine is empty, it stays
  up, saying so at the alarm's interval with no handler call. Only its own
  quarantine reading empty ends it with the all-clear.
- The projection makes an entry only from a record with a string key and
  chunk hash and a non-empty string generation, and an exhaustion's record
  must carry the exhaustion's own key. It reads an outcome list as the
  outcomes in it. A ledger entry it leaves out no longer holds its chunk
  back, and a warning names its key, where it has one, once per load.
- A ledger or legacy snapshot whose value isn't a list is read as holding
  nothing, and the legacy snapshot isn't read beside a ledger like that.
  A checkpoint, known by its kind, replaces what came before it with its
  list, so one that holds no list replaces it with nothing. A warning names
  each once per load, a checkpoint by its index in the ledger.
- The change's stderr lines go through this.diag, so each carries
  ` ns=<namespace>`.
- The transient-ladder fragment and the recall-curve fallback spec gain
  the unusable ledger entry among a family's ways out of the quarantine.

test/compression-quarantine-all-clear.test.ts drives real refusals through
compressChunkHierarchical under a refusing membrane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 19, 2026
main moved 193 commits since this branch's last merge of it (e6b12e5),
among them anima-research#138, which conflicts in src/strategies/autobiographical.ts.
The resolution keeps both sides:

- tick's compression and merge take anima-research#138's structure. Its clear of the
  single-flight marker only while the marker is still the tick's own
  covers what this branch's clearPendingCompression() did, so that
  helper is gone.
- The speculative drain's timer runs main's guarded callback, with this
  branch's check of the loaded identity after its closed-store check.
- The coalesced wait takes up the store's state through this branch's
  takeUpStoreState(), stops at main's ledger check of the chunk's
  quarantine, and takes a discarded chunk over only past it, passing
  onQuarantineSkip and onRequestStopped on.
- The in-flight settle carries main's membrane and this branch's
  discarded flag.
- The merge's carrier fallback takes main's classification of the
  rejection, then this branch's currency check before the stripped
  retry, which returns false as main's executeMerge contract has it. So
  does the re-check of sources parented while the call was out, which
  now logs through diag.
- Logging is main's diag throughout.

Two of this branch's tests change with main. The carrier rejection they
send is typed as membrane types a 400, since main takes an untyped error
as transient. And a strategy waiting on a quarantined chunk now stops at
main's ledger check instead of compressing the chunk again to meet the
quarantine skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 19, 2026
- A thin chunk's stub checks that its strategy is still current after
  its quarantine clear, before the merge check, as a written L1 does. A
  strategy let go of during that clear refused the merge check, and the
  tick failed.
- A waiter that takes a discarded producer's chunk over passes tick's
  hold gate first. A hold placed on the chunk's span while it waited
  now keeps the chunk from the provider until the release, as tick's
  gate would have before the compression started.
- The refusal after resetHeadWindow()'s transition add gives its reason
  without the advice to retry or to recover, since a retry would store
  a second transition. requireInitialized() takes a switch that leaves
  its advice off.
- executeMerge()'s doc names its false return for sources parented
  while the call was out, and says what tick does with the entry when
  the skip took up the store's queue.

New tests: a stub whose clear spans the swap; a hold placed while a
replacement waits; a waiter that takes up a summary its producer wrote
before it was let go of; a waiter let go of during its wait, which ends
its tick quietly and takes nothing up. The tests of a detached carrier
merge and of a merge whose sources were parented while it was out now
check that it resolved false, and the refusal tests that it advises no
retry.

The fixed changelog fragment drops its line on a tick's mark of work
in flight, which anima-research#138 now makes, and words the drain's line for the
catch around its timer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit that referenced this pull request Oct 19, 2026
…neration's mark

#138 shipped in 0.17.0, and with it the merge's own-mark clear: a tick
clears only its own in-flight mark. Before it, an AutobiographicalStrategy
instance set again while its merge's call was still out (setStrategy()
with the same instance, or a new manager opened with it after close())
could write two summaries over the same sources, and compile() returned
both. 0.17.0's section doesn't say so; this adds the entry there.

Two probes, one for each way of setting the instance again, ran on
#138's merge (297e535) and on its first parent (b3c3f45). On the
parent, the settled call left no mark, a second merge of the same
sources started beside the new generation's while that one was still
out, and two L2s were written over the same two L1s. On the merge, the
mark stood and one L2 was written. switchBranch() wasn't
probed, so the entry says "as by". In a probe of compressions, a new
generation's compression joined the earlier call's work rather than
making its own call, so the entry names merges only.

Co-authored-by: Paloma-2644 <paloma-2644@commons.astra.invalid>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/context-manager that referenced this pull request Oct 20, 2026
The second half of anima-research#241. A strategy that is let go of (by close(), by
a setStrategy() that replaces it or sets it again, or by an open()
whose initialization of it fails), or that the manager's own
switchBranch() initializes again, already keeps its work from writing
(anima-research#215). But a compression, merge or transition-summary call already
out ran on, through membrane's retries and the waits between them, to
an answer the strategy threw away.

- The calls end. Each of AutobiographicalStrategy's five complete()
  calls goes through callProvider(), which gives it an AbortController
  of its own, kept in callsOut until the call settles. endGeneration(),
  which initialize() and detach() share, ends the generation and then
  aborts every call still out. It's a controller per call rather than
  one signal per generation, because a generation lasts the session
  and @anthropic-ai/sdk 0.52.0 leaves an abort listener per request on
  the signal it is given. The merge and the transition summary check
  their work's currency again after preparing the request, as each L1
  attempt does, so a generation that ends meanwhile makes no call.
- A stale error is discarded as a stale result is. runAttempt's first
  catch throws its CompressionBranchDiscard for an error on work that
  lost its currency, before the carrier fallback, so a split's part
  abandons nothing on it and no stripped retry follows. That replaces
  the check before the stripped retry; ladderStep already discards a
  rung's stale error (anima-research#138). executeMerge returns false for one at
  either of its calls, logging no failure, and a transition summary
  throws its "crossed a branch generation" error, with the call's
  error as its cause. A host's withdrawal (CompressionWithdrawn) keeps
  its own path on any branch.
- A host's cancellation takes a withdrawal's path at every site. A
  host that combines a signal of its own with the one the strategy
  passes, and aborts it, gets its cancellation back as itself, and
  compressionFailureKind() already classes it as an 'abort' (anima-research#138).
  The sites that took a withdrawal's path by the brand now take it by
  that kind, for an error that passed its work's currency check: the
  L1 compression's final catch (after its own check) and executeMerge's
  catch log no failure and receipt it withdrawn, a split's part counts
  it as neither a call nor an error, tick's merge catch rethrows it
  with nothing counted, and the speculative drain stops quietly. The
  stale checks keep the brand, so on a moved branch a withdrawal is
  still receipted withdrawn, while a cancellation, which the
  strategy's own end also makes, is discarded.

Docs: CompressionWithdrawn's says how a host ends a call already out,
StrategyContext.membrane's that a wrapper passes the signal on, and
ContextStrategy.detach's that a strategy can cancel its calls. anima-research#209's
fragment no longer says how a call already out ends.

Tests pin the calls ended each way a strategy is let go of, a
cancellation at each site where it changes an outcome, and the stale
errors discarded. Two more check that a take-up at an exact-L1 find
and at a coalesced wait queues the merge the store lacked, which only
a merge's take-up was tested for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

greptile no-greptile Maintainers' and the team's: Greptile skips this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants