fix: populate global context-limit cache from named provider model config (#262)#269
Open
mmacedoeu wants to merge 3 commits into
Open
fix: populate global context-limit cache from named provider model config (#262)#269mmacedoeu wants to merge 3 commits into
mmacedoeu wants to merge 3 commits into
Conversation
mmacedoeu
force-pushed
the
feat/262-per-model-context-limit
branch
from
06b651a to
df55912
Compare
…nfig (1jehuang#262) Named OpenAI-compatible providers that don't expose GET /v1/models fall back to DEFAULT_CONTEXT_LIMIT (200k) even when the user has configured explicit context_window values in their provider config. Populate CONTEXT_LIMIT_CACHE from NamedProviderModelConfig.context_window values during provider init, so all resolution paths (TUI info widget, compaction budget, remote model switching) see the user-configured limits instead of falling back to 200k.
1jehuang#262) NamedProviderConfig now accepts a context_window field at the provider level. When a model entry doesn't specify its own context_window, the provider-level value is used as fallback. This matches the common config pattern where all models served by a gateway share the same context limit. [providers.opengateway] context_window = 1000000 default_model = "mimo-v2.5-pro" [[providers.opengateway.models]] id = "mimo-v2.5-pro" # inherits context_window from provider level
…ng#262) The TUI and server run in separate processes. The CONTEXT_LIMIT_CACHE was only populated in the server process during provider init. The TUI process had an empty cache, causing context_limit_for_model_with_provider to return None and fall back to DEFAULT_CONTEXT_LIMIT (200k). Add populate_context_limits_from_config() which reads all named provider model configs and populates the cache. Call it during config initialization (both initial load and reload) so every process has the cache populated from the start. Use a direct config-ref variant during LazyLock init to avoid a deadlock.
mmacedoeu
force-pushed
the
feat/262-per-model-context-limit
branch
from
df55912 to
7f18942
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #262
When using an OpenAI-compatible provider that doesn't expose , user-configured values in were only used by the provider instance's own method. All other resolution paths (TUI info widget, compaction budget, model switching) fell through to .
Changes
**** - In , populate the global from values after building . This ensures all codepaths that call or resolve the correct limit.
**** - Added test verifying that:
**** - Fix pre-existing compile error: make module .
Usage
Users can now set per-model context limits in their provider config:
And the configured limit will be respected globally, even without a working
/v1/modelsendpoint.