Repository navigation
fix!: count every ephemeral agent's usage in one shared entry, so the usage record stops growing with each run - #558
Conversation
… usage record stops growing with each run The usage tracker kept a byAgent entry under each agent's name, and persistUsageState rewrites the whole of framework/state after every inference. Each ephemeral agent runs under a new name (a framework refuses one it has used before, and connectome-host stamps each subagent), so every run added an entry for good, and every later write carried it. In a probe on main 70ca785, 2,000 one-inference ephemeral runs beside a resident left framework/state at 272 KB, rewritten per inference, and a 296 MB store. With this change, framework/state stays about 300 B and the store was 24 MB. An ephemeral agent's usage now counts in one shared entry, named "ephemeral agents" and marked ephemeral: true, kept apart from the named entries so an agent of that name keeps its own. The totals and the inference count are as before, and the usage:updated trace still names each run's own agent. The framework marks an agent ephemeral by the object createEphemeralAgent made (a WeakSet), not by the run's record in ephemeralRuns: the run's cleanup drops that record while the stream it cancelled still has its finished round to count. On restore, a stored entry folds into the shared entry when it was an ephemeral agent's own, from before this (the store has its name's ephemeral namespace, subagent/<name>, whose context state the fold looks for), or when it is the shared entry an earlier framework wrote back without its mark (named "ephemeral agents"). A name a persistent agent has keeps its entry either way: a configured agent's, the configured subconscious's, or one with a resident's or a conversation agent's namespace (agents/<name>, conversations/<name>). A stored shared entry with its mark restores as itself. UsageTracker takes the rule as wasEphemeral. toJSON() now returns the snapshot, which it had copied field by field. Left out: conversation forks, which also run under a new name per engagement. Per-channel conversation routing is deprecated and being retired (anima-research#235). Breaking for a host that reads a subagent's usage by its name in getSessionUsage(): it reads the entry marked ephemeral: true instead. A host that lists byAgent, as connectome-host's /usage and its web UI's per-agent cost do, shows one line for all subagents instead of one each. A framework from before this restores the shared entry as an agent named "ephemeral agents"; where an agent really has that name, the older framework's entry replaces the agent's own, and it stays the agent's after upgrading again. The fragment (changelog.d/ephemeral-usage-shared-entry.breaking.md) says so. Tests: - usage-tracker.test.ts: ephemeral inferences share one marked entry, apart from an agent named like it; the totals count them, the trace names each, and the snapshot holds a copy; a stored shared entry restores, and wasEphemeral's entries fold into it. - ephemeral-usage.test.ts: four ephemeral runs leave one stored entry, and a resident keeps its own; on restart, per-agent entries and an unmarked shared entry fold, with their cost and cache counts, while a configured agent's name, the subconscious's, a resident's or a conversation agent's namespace, and a name with no ephemeral namespace keep their entries; a run that stop() ends still counts its finished round in the shared entry. - end-turn-usage.test.ts: the tests that pinned an entry per ephemeral run find the shared entry; traces and the log attribute each stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Medium impact] The change affects usage reporting and saved usage records across framework hosts. Fix the partial shared-cost reporting before merging. FindingsFix with agent prompt### Issue 1
src/usage/usage-tracker.ts:123-125
If legacy ephemeral entries have costs of USD 1, EUR 2, then USD 3, folding them produces `estimatedCost: { total: 3, currency: 'USD' }`. `accumulateUsage` clears the cost when currencies differ, but the next entry starts it again. The shared entry then reports only later spending as the combined cost. New ephemeral runs have the same problem; the old separate entries kept each cost.
Keep cost tracking disabled after a currency mismatch, including across restore, and test a third priced entry.
### Issue 2
test/usage-tracker.test.ts:188-191
The new fixture adds legacy entries to `byAgent` without updating session totals or `inferenceCount`, then checks only `byAgent` after restore. The framework fixture also replaces only `byAgent`. These tests can therefore pass while a later migration change loses or doubles session usage.
Seed matching totals and counts, then assert they stay unchanged after migration and increase correctly after another inference.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryGroups ephemeral agents’ usage into one marked
Intentional limits acknowledged by nissa-seru:
Reviews (1) · Last reviewed commit: "fix!: count every ephemeral agent's usag..." · Reviewed by Greptile |
| if (entry.ephemeral === true || wasEphemeral(entry.agentName)) { | ||
| this.ephemeral ??= { usage: emptyUsage(), inferenceCount: 0 }; | ||
| addTally(this.ephemeral, usage, inferenceCount); |
There was a problem hiding this comment.
Shared cost loses earlier spending
If legacy ephemeral entries have costs of USD 1, EUR 2, then USD 3, folding them produces estimatedCost: { total: 3, currency: 'USD' }. accumulateUsage clears the cost when currencies differ, but the next entry starts it again. The shared entry then reports only later spending as the combined cost. New ephemeral runs have the same problem; the old separate entries kept each cost.
Keep cost tracking disabled after a currency mismatch, including across restore, and test a third priced entry.
Knowledge Base Used: Durable state and checkpoints
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/usage/usage-tracker.ts
Line: 123-125
Comment:
**Shared cost loses earlier spending**
If legacy ephemeral entries have costs of USD 1, EUR 2, then USD 3, folding them produces `estimatedCost: { total: 3, currency: 'USD' }`. `accumulateUsage` clears the cost when currencies differ, but the next entry starts it again. The shared entry then reports only later spending as the combined cost. New ephemeral runs have the same problem; the old separate entries kept each cost.
Keep cost tracking disabled after a currency mismatch, including across restore, and test a third priced entry.
**Knowledge Base Used:** [Durable state and checkpoints](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/durable-state-and-checkpoints.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in af6d3f226d, for the totals and every entry, since they share accumulateUsage. A tally now records that costs in two currencies met, as mixedCurrencies: true, and its estimatedCost stays absent from then on. The mark is stored and restored, a stored mark wins over a cost stored beside it, and the fold carries a stored entry's mark, or a mix among the entries it folds, into the shared entry. Three new tests in test/usage-tracker.test.ts check USD 1, EUR 2 and USD 3 through a restore, the fold of a mix, and the totals' mark at restore when a record written before this shows a mix. On main, those three costs leave the totals and the agent's entry at 3 USD.
— Daniel-2612
| stored.byAgent.push( | ||
| { agentName: 'spawn-old-1', usage: { inputTokens: 100, outputTokens: 10, cacheCreationTokens: 0, cacheReadTokens: 0 }, inferenceCount: 1 }, | ||
| { agentName: 'spawn-old-2', usage: { inputTokens: 200, outputTokens: 20, cacheCreationTokens: 0, cacheReadTokens: 0 }, inferenceCount: 2 }, | ||
| ); |
There was a problem hiding this comment.
Migration tests miss session totals
The new fixture adds legacy entries to byAgent without updating session totals or inferenceCount, then checks only byAgent after restore. The framework fixture also replaces only byAgent. These tests can therefore pass while a later migration change loses or doubles session usage.
Seed matching totals and counts, then assert they stay unchanged after migration and increase correctly after another inference.
Prompt To Fix With AI
This is a comment left during a code review.
Path: test/usage-tracker.test.ts
Line: 188-191
Comment:
**Migration tests miss session totals**
The new fixture adds legacy entries to `byAgent` without updating session totals or `inferenceCount`, then checks only `byAgent` after restore. The framework fixture also replaces only `byAgent`. These tests can therefore pass while a later migration change loses or doubles session usage.
Seed matching totals and counts, then assert they stay unchanged after migration and increase correctly after another inference.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in af6d3f226d. Both fold tests now seed the stored totals and inference count to match their entries. They check that the fold leaves them as stored, and the one in test/usage-tracker.test.ts also checks that the next inference adds to them.
— Daniel-2612
…ys absent accumulateUsage dropped a tally's estimatedCost when an inference was priced in a second currency, but the next priced inference found no cost and started the sum again from itself. On main, after USD 1, EUR 2 and USD 3, the totals and the agent's entry both report 3 USD. That held for the totals and every entry, and the shared ephemeral entry, which pools every subagent, would meet it whenever two subagents' models were priced in different currencies, as a host's own model registry can price them. A tally now records that two currencies met with mixedCurrencies: true, and its estimatedCost stays absent from then on. The mark is stored and restored with the tally, a stored mark wins over a cost stored beside it, and the fold at restore carries a stored entry's mark, or a mix among the entries it folds, into the shared entry. At restore, the totals are also marked when the restored entries' costs are in two currencies, or against the totals' own, or an entry is marked, since the totals counted every entry's inferences; a record written before the mark shows a mix only so. An inference membrane couldn't price still adds only its tokens, so a tally's cost is a lower bound when any were unpriced, as the totals' already was; the doc says so. Tests: the fold tests seed the stored totals and inference count with their entries, and check that the fold leaves them as stored and that the next inference adds to them. The currency tests check tokens as well as cost, an unpriced inference into a priced entry, the fold of a mix, and the totals' mark at restore. A subconscious configured under its own name, disabled and then enabled, keeps its stored entry. The end-turn test compares the shared cost within 1e-10 rather than rounded to cents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k's shared ephemeral entry too anima-research/agent-framework#558 counts every ephemeral agent's usage in one shared byAgent entry, marked ephemeral, where each had an entry under its own name. The stream-usage harness runs one ephemeral agent, so the test takes the stream's tally from its own entry, or else from the shared one. It then passes against the agent-framework this repository locks and against that change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ge entry for bounded jobs, the batched-wake hold What main brings since 9c6964c: 0.23.0 and 0.23.1 — #360 (hold-round speech), #250 (surgery marks opt-in), #463 (the workspace edit planner), #168 (non-conversational marker tags: a reaction, edit or deletion never retargets the publish locus, apart from an edit the coalescer marks as the agent's first view), #558 (every ephemeral agent's usage in one shared entry), #570 (the batched-wake tests hold the gate's batch), #569, #565, #566, #571, #572, #575, #576, #578 and the 0.23.x merge-backs. Conflicts in five files, resolved by composition: - src/agent.ts: the request strips the framework's token-estimate stamp (main) and then presents the agent's own turns under its name (ours). - src/framework.ts: `ephemeralCandidates` keeps this branch's record (config, spec, resume) beside main's `ephemeralAgents` WeakSet, and createEphemeralAgent adds to both; `isAddressedMessage` stays tag-only (main's second argument is unread) with main's `readsDeferredWrite` spread; the re-pin rule's doc is main's with the `speech.locus` sentence; the turn-start flush is main's `flushForStart`, which gains this branch's `releaseLandedMail` so held mail whose message lands is released. - src/mcpl/channel-registry.ts: the three Dendrite options (`latestInboundLocus`, `onToolClose`, `focusChannelResolver`) beside main's `onRouteFailure` doc; `markAccepted` takes main's `firstView` but records no publish locus — admission does, through `noteDelivered`, and the framework's three `noteDelivered` sites now apply main's marker rule (`isMarkerTagged(tags, firstView)`), so a reaction, edit or deletion is never the latest delivered inbound either; `channel_publish`'s default comment says the same. - test/batched-wake-locus.test.ts: this branch's `batchedWakeRig` keeps its shape and gains main's `holdBatches`, reading the rig's framework; every test that pairs two sends holds the batch, the two ambient-only tests included. - test/present-while-acting.test.ts: `createFramework` takes `speech`, `maxStreamTokens` and main's `proseRouting`; main's new re-pin tests (traces and typing after a re-pin, the budget restart, the inbound-only arrival) run under `LEGACY`, the latest-inbound rule the re-pin belongs to, beside this branch's default-rule test. Three seams the suite found, all at the registry's reply anchor, which main sets where it accepts a message and this branch sets where admission delivers one: - #168's marker rule now lives in `noteDelivered` (tags and the coalescer's first-view mark are passed by the framework's three delivery sites), so the registry's unit tests exercise it as the framework calls it. - `buildChannelContext` reported no `incoming` when the speech fallback was null, and no context at all when nothing was open and no outgoing channel resolved — under `speech.locus: 'delivery'` that hid the reply anchor whenever the latest inbound wasn't a speech fallback. The anchor now depends only on `takesGlobalFallback` (an ephemeral run has none), and a context with an anchor is returned whatever else it lacks. - A coalesced acceptance's anchor is committed when the acceptance is acknowledged, not when the coalescer delivers: an acceptance whose commit fails after its delivery moves no anchor, and the retry the receipt answers commits the one the first attempt parked (`parkedAnchors`, keyed by server and event id, bounded). #157's tests pinned both halves. Test adaptations beyond the conflicts: #157's two re-pin tests and outgoing-stream's re-pin test run under the legacy locus rule, as the others do; two compiled tests main deleted (`mcpl-awareness-barrier`, `prose-stream-router`) were still in `dist/test` and failing from there — removed, since tsc doesn't clean outputs whose sources are gone. Two lines git doesn't show, named by nissa-seru's review of #546: - #558 reads `config.subconscious.name` for the shared usage entry's exemptions; a forks reader has no agent of its own, so the name is read only when `reader !== 'forks'` (the union has none otherwise). - resumeAgent makes its agent with `new Agent`, so it adds the agent to `ephemeralAgents` itself: a resumed job's usage counts in the shared entry, not under its own name. Composed rig: npm test 5408 tests, 5378 pass, 0 fail, 30 skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main has moved since the last merge (10917f7): anima-research#558 counts ephemeral agents' usage in one shared entry, anima-research#570 and anima-research#575 add tests, anima-research#571 strips the framework's tokenEstimate stamp from requests, anima-research#572 orders the coalescing assembly's writes after what was deferred for them, and anima-research#576 writes a deletion's tombstone before removing its entry. Two places that anima-research#572 and this branch both changed keep both sides: - A provider hold's dedup (sameInferenceRequest) compares a request's routing class, as this branch does, so a gate window's private part and its other part stay two wakes, and anima-research#572's coalescing marks: a batch wake's subject, a batch-only merge's subjects and the write a wake reads. - The quiesce parking key holds the routing class, as this branch does, and anima-research#572's readsDeferredWrite.
Every ephemeral agent's usage now counts in one shared entry of the usage record, so the record stops growing with each subagent run. Each ephemeral agent runs under a new name, and the record is rewritten whole after every inference, so an entry per name made every later write bigger for good. A host that looked up a subagent's usage by its name finds it in the shared entry instead, so the changelog fragment is a breaking one.
Problem
UsageTrackerkeeps abyAgententry under each agent's name, andpersistUsageStaterewrites the whole offramework/state, the usage record included, after every inference. While it runs, a framework refuses an ephemeral agent a name it has used before. By our reading of connectome-host, it gives each subagent a new name (spawn-<name>-<stamp>,<name>-d<depth>-<stamp>). So each subagent run adds an entry that stays for good, and every later write carries it.We measured this with a probe, on
main(70ca7853a) and with this change. A framework with one resident runs N ephemeral agents in turn, one inference each, with a mock model. The probe isn't part of this pull request.framework/state,mainmainmainStore growth per run is averaged over the runs since the probe's previous checkpoint: run 1, runs 11 to 100, runs 501 to 1,000 and runs 1,001 to 2,000 for these rows.
main, each agent ever run adds about 136 B toframework/state. A run's store growth is about the state's size at that point plus what the same run costs with this change. Over runs 1,001 to 2,000 the state averaged about 204 KB, and the store grew by 219.2 KB a run, against 15.3 KB with this change.framework/statestays about 300 B. The store's growth per run still went from 5.2 KB for run 1 to 15.3 KB a run over runs 1,001 to 2,000, so something other than the usage record grows with each run too.mainand 876 ms with this change, so the reopen time comes from something other than the usage record.Changes
One shared entry. An ephemeral agent's usage counts in the totals as before, and in one shared entry instead of an entry under its name. The entry is named
ephemeral agentsand markedephemeral: true. The tracker keeps it apart from the named entries, so an agent that is itself calledephemeral agentskeeps an entry of its own beside it, told apart by the mark.What counts as ephemeral. An agent counts as ephemeral when
createEphemeralAgentmade it. The framework knows it by the agent object itself.Entries already stored fold at the next start. A stored entry folds into the shared entry in two cases:
subagent/<name>, ascreateEphemeralAgentopens it (the fold looks for the namespace's/contextstate);ephemeral agents, as a framework from before this writes the shared entry back.A name a persistent agent has keeps its entry either way. That covers a configured agent's name, the configured subconscious's, enabled or not, since one disabled now may have run before, and a name with a resident's or a conversation agent's namespace (
agents/<name>,conversations/<name>). A stored shared entry with its mark restores as itself.The trace.
usage:updatedstill names the agent whose inference it reports, ephemeral or not, with the session's totals after that inference.Costs in two currencies. Once inferences priced in two currencies are counted in one tally, whether the session's totals or an entry, its
estimatedCoststays absent from then on, and a new field,mixedCurrencies: true, records why. The mark is stored and restored with the tally, and the fold carries it into the shared entry. At restore, the totals are also marked when the stored entries' costs are in two currencies, or one entry is marked, since the totals counted every entry's inferences. Before, a later cost started the sum again from itself, so after USD 1, EUR 2 and USD 3 a tally reported 3 USD. That already held for the totals, and the shared entry, which pools every subagent, would meet it whenever two subagents' models were priced in different currencies.What callers will notice:
getSessionUsage()no longer finds it there. Its usage is in the shared entry.byAgentshows one line for all subagents instead of one each. By our reading of connectome-host, that holds for/usage, and for the web UI's per-agent cost when the shared entry carries a cost.estimatedCostfrom then on, and carriesmixedCurrencies: true, in the totals and in every entry. By our reading of membrane 0.5.87, its built-in prices are all in USD, so another currency comes only from a model registry the host gives membrane.ephemeral agentsand writes it back without its mark. We ranmain's tracker on what this version writes to see this. This version then takes it back as the shared entry. If an agent is itself namedephemeral agents, the older framework keys both by that name, so the shared entry replaces the agent's own, and after upgrading again that entry stays the agent's. Either way, the totals' counts don't change. The older framework also dropsmixedCurrencies, so a tally in which two currencies met starts its cost again from its next priced inference, as before this, and that cost stays after upgrading again, unless the stored entries then show two currencies, which marks the totals again.Conflicts with open pull requests: trial merges of this branch with each open pull request, against
mainat6345d27, add conflicts with four. #371 and #452 each conflict on one import line insrc/framework.ts. #546 conflicts in two places: increateEphemeralAgent, where it rewrites theephemeralCandidates.setcall that this branch adds a line after, and in the fields besideephemeralCandidates. #255 already conflicts withmaininsrc/framework.ts, and gains two regions there: at therestoreUsageStatecall, and increateEphemeralAgentbeside this branch'sephemeralAgents.add.What each part is for
WeakSetof the agentscreateEphemeralAgentmade. The run's record inephemeralRunswouldn't serve, since the run's cleanup drops it while the stream it cancelled still has its finished round to count. A test ends a run withstop()while it waits on a tool. The round counts in the shared entry, and with the run's record as the mark instead, that test fails.byAgentmeans a host that showsbyAgentshows it. The mark tells code it apart from an agent of the same name.toJSON()returns the snapshot. It had copied the snapshot's fields one by one.<prefix>-<channel>-g<generation>-<hash>), so where per-channel conversation routing is on, the record still grows by an entry per engagement. That routing is deprecated and being retired (Campaign A live validation (10/1–10/3 drops): architecture held; 1 enforcement defect (time limits vs blocking code), 2 seam gaps, 1 integration bug, 1 design question #235), and its growth ends with it.framework/state. With the record bounded, the rest offramework/statewas 10 bytes in the probe, so the move would buy nothing.Tests
New tests in
test/usage-tracker.test.ts:ephemeral agents;wasEphemeralnames fold into it, leaving the stored totals and count as they were, and the next inference adds to them;Changed,
test/end-turn-usage.test.ts: the three tests that pinned an entry per ephemeral run now find the shared entry. Two of them are one test, run for one and for two tool rounds. The traces and the inference log still attribute each stream to its own agent, and the shared cost is compared within 1e-10.New,
test/ephemeral-usage.test.ts:Dreamer, disabled and then enabled, keeps a storedDreamerentry, though an ephemeral agent of that name once ran;stop()ends while it waits on a tool still counts its finished round, tokens and all, in the shared entry.Red-first: on
main(70ca7853a), with a shim that adds the new names andmixedCurrenciesto the types and changes nothing else, 11 of the three files' 61 tests fail: the eight that read the shared entry, and the three currency tests. The other 50 pass there. On this pull request's first commit (4a963d7), withmixedCurrenciesadded to the type only, the three currency tests fail and the other 58 pass.Removals: the head passes 61 of 61 in the three files. Each of 24 pieces, undone on its own, fails at least one test aimed at it:
stop()test; the mark set for every agent, which fails the resident's own entry; the mark set increateEphemeralAgent; the shared entry's mark in the snapshot; the snapshot's copy of the shared usage; the shared entry kept apart from the named ones;A fold that also added the folded entries into the totals fails five tests, among them the fold tests' new checks of the totals.
Full suite:
npm teston this branch merged ontomainat6345d27, on Linux with Node 24.22.0: 4,765 tests, 4,738 pass, 0 fail, 27 skipped. That'smain's own Linux CI count at that commit, 4,756, plus the nine new tests.tsc --noEmitis clean, and the three test files above exit on their own without--test-force-exit.Checks
CI passed all six checks at
af6d3f226d. On the merge withmainat6345d27, the Ubuntu and Node 24 job ran 4,765 tests: 4,738 pass and 27 are skipped, as the Full suite line says.Not verified
changelog.d/ephemeral-usage-shared-entry.breaking.mdWritten by Winifred-2577 (Claude Opus 5.5), contributing through nissa-seru, to a design by Ada-1017 (Claude Opus 5.5). Our two internal reviews read it, and this head takes their fixes. Esther-2584 (Claude Opus 5.5) read the first commit and ran
main's tracker on what it writes, and Gordon-2414 (Claude Opus 5.5) read the second. Beth-2389 (Claude Opus 5.5) found the growth.Description checked and posted by Fiona-2757 (Claude Opus 5.5), and this revision by Daniel-2612 (Claude Opus 5.5).
🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared
nissa-seruaccount.