Skip to content

Summarize: chunk Observations by measured tokens against a 50k budget #33

Closed

Description

@possiblyneal

Parent

Part of #31.

Branch: fix/summarize

What to build

A capped Session on the Operator's broker summarizes successfully instead of timing out. Chunks are packed to a token budget (default 50,000, SUMMARIZE_CHUNK_TOKENS) rather than a fixed count of 400 Observations. The count comes from a new optional token-count operation on the provider: the OpenAI-compatible provider implements it against the broker's tokenize route for the configured model, adding a fixed system/template overhead; the resilient wrapper forwards it outside the circuit breaker. When the provider has no count operation or the call fails, the chunker falls back to chars / 3, the estimate idiom already in this tree (overcounts ~17% on measured content, the safe direction). SUMMARIZE_CHUNK_SIZE is removed. An Observation larger than the budget forms its own chunk and is logged. The chunk concurrency default drops from 6 to 2 (SUMMARIZE_CHUNK_CONCURRENCY still overrides): the Operator's broker serialises chunks on one GPU, and the 300s timeout in #43 was measured at concurrency 2 — a 6-way burst of 50k chunks would exceed it.

Acceptance criteria

  • A fake provider recording prompt sizes shows every chunk at or under the budget, with Observations kept in order
  • With the count operation throwing, chunks are packed by the chars / 3 estimate and summarize still completes
  • The OpenAI-compatible provider's count sends the configured model to the tokenize route and returns token count plus the fixed overhead; a non-2xx answer makes the caller fall back
  • SUMMARIZE_CHUNK_SIZE no longer appears in code or docs; SUMMARIZE_CHUNK_TOKENS is documented where the old variable was
  • The stale comment about iii's 180s invocation timeout beside the concurrency default is removed, and that default is 2
  • SUMMARIZE_CHUNK_CONCURRENCY documentation states the new default and that the timeout basis assumes it
  • npm test green

Blocked by

Activity

  1. added a commit that references this issue on Sep 22, 2026
    bac5457
  2. possiblyneal commented on Sep 22, 2026

    @possiblyneal
    OwnerAuthor

    Closed by bac5457 on fix/summarize. One deviation from the acceptance text: the ~400-token system/template overhead is reserved once per chunk inside the chunker rather than added to every count the provider returns — adding it per Observation would have charged it hundreds of times per chunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions