You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Summarize: chunk Observations by measured tokens against a 50k budget #33
A capped Session on the Operator's broker summarizes successfully instead of timing out. Chunks are packed to a token budget (default 50,000, SUMMARIZE_CHUNK_TOKENS) rather than a fixed count of 400 Observations. The count comes from a new optional token-count operation on the provider: the OpenAI-compatible provider implements it against the broker's tokenize route for the configured model, adding a fixed system/template overhead; the resilient wrapper forwards it outside the circuit breaker. When the provider has no count operation or the call fails, the chunker falls back to chars / 3, the estimate idiom already in this tree (overcounts ~17% on measured content, the safe direction). SUMMARIZE_CHUNK_SIZE is removed. An Observation larger than the budget forms its own chunk and is logged. The chunk concurrency default drops from 6 to 2 (SUMMARIZE_CHUNK_CONCURRENCY still overrides): the Operator's broker serialises chunks on one GPU, and the 300s timeout in #43 was measured at concurrency 2 — a 6-way burst of 50k chunks would exceed it.
Acceptance criteria
A fake provider recording prompt sizes shows every chunk at or under the budget, with Observations kept in order
With the count operation throwing, chunks are packed by the chars / 3 estimate and summarize still completes
The OpenAI-compatible provider's count sends the configured model to the tokenize route and returns token count plus the fixed overhead; a non-2xx answer makes the caller fall back
SUMMARIZE_CHUNK_SIZE no longer appears in code or docs; SUMMARIZE_CHUNK_TOKENS is documented where the old variable was
The stale comment about iii's 180s invocation timeout beside the concurrency default is removed, and that default is 2
SUMMARIZE_CHUNK_CONCURRENCY documentation states the new default and that the timeout basis assumes it
Closed by bac5457 on fix/summarize. One deviation from the acceptance text: the ~400-token system/template overhead is reserved once per chunk inside the chunker rather than added to every count the provider returns — adding it per Observation would have charged it hundreds of times per chunk.
Activity
fix(summarize): pack chunks by measured tokens against a 50k budget
possiblyneal commentedon Sep 22, 2026
Closed by bac5457 on fix/summarize. One deviation from the acceptance text: the ~400-token system/template overhead is reserved once per chunk inside the chunker rather than added to every count the provider returns — adding it per Observation would have charged it hundreds of times per chunk.