Skip to content

Design question: why not use model metadata for contextWindow + built-in compaction? #246

Description

@Lynskylate

Context

I noticed that Bub currently:

  1. Has no contextWindow metadata — It doesn't track each model's max context length (unlike pi, where models.json contains contextWindow per model)
  2. Has no proactive compaction — It relies entirely on the API returning a context-length error, then does a reactive auto-handoff (anchor-based truncation, retrying once)
  3. Uses handoff for truncation, not summarization — When context overflows, older tape entries before the anchor are simply dropped, not summarized

Current behavior

AgentSettings:
  - max_tokens: 16384  (output limit only, not context window)
  
On context overflow:
  - is_context_length_error() regex-matches the API error
  - tape.handoff() creates a new anchor → older messages are silently dropped
  - MAX_AUTO_HANDOFF_RETRIES = 1

Question

Is this by design? Specifically:

  1. Model metadata: any-llm-sdk doesn't appear to expose contextWindow for providers/models. Is there a plan to add this? Or is the philosophy that Bub should be model-agnostic and not rely on hardcoded limits?

  2. Built-in compaction: Would the project consider built-in summarization-based compaction (like pi's /compact), or is the anchor-based handoff considered sufficient? The current approach drops history rather than preserving it in compressed form.

  3. Prevention vs recovery: Is the preference for "try first, recover on error" over "track context usage proactively"? The latter would allow Bub to trim/compact before hitting the API limit.

I'm happy to contribute if there's alignment on a direction. Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions