Repository navigation
Conversation
A prompt plus an answer cap over the context window now gets the cap shortened to the room the context leaves instead of a 400, matching how Strata serves the same request. --no-fit-max-tokens keeps the old hard reject. Docs row flipped off -> on. Rationale: clients that top out max_tokens at the full window (the habit a Strata deployment trains) hit a wall of 400s after switching backends; issue jesdga95#460. Validation (interpreter D:/Strata/.venv/Scripts/python.exe, cwd D:/Strata/.scratch/basalt-src, branch strata-context-fit): - three suites test_config + test_context_overflow + test_server at the full final tree: Ran 257 tests, OK (skipped=1) - full serve discover: Ran 1146, failures=10 errors=22 skipped=41; the clean origin/main worktree run: Ran 1144, same 10F+22E names, same order (env-dependent Chinese-Windows strerror/TLS/process-group failures, pre-existing; the +2 test delta is this change's new tests) Seat: engineer
/v1/messages 400s for an over-the-context request now lead with Anthropic's own sentences - "prompt is too long: N tokens > M maximum" and "input length and `max_tokens` exceed context limit: N + M > C" - the keyword contract harnesses (Claude Code and friends) react to, then the basalt sentence. OpenAI and Responses paths keep their words. Also fixes the reject text pointing at the wrong config place: fit-max-tokens hangs off a model entry of the --config file (serve/config.py MODEL_OPTIONS), not the "server" section; and it now says the switch is on by default. Rationale: issue jesdga95#460, second half of the Strata-parity gap. Validation: covered by test_context_overflow (Anthropic wording assertions on both over-room and prompt-only cases, stream both ways) inside the suite run recorded on commit 1 (Ran 257 OK (skipped=1)). Seat: engineer
The old hard-reject cases now ask for it explicitly (fit_max_tokens=False, restored in finally) instead of inheriting it from the default, and assert the corrected "model entry of the --config file" pointer. New positive cases pin the new default: an over-the-context budget gets clamped and answers (openai/anthropic). test_config's EveryOption samples the no- form for switches that are on by default, so the CLI-vs-default assertion keeps its bite (same family as the existing grammar handling); the sweep skips --no-* twins whose semantics ride with their positive form. Rationale: issue jesdga95#460; contracts follow the flipped default without silently deleting the old expectations. Validation: three suites at the step-0 tree read Ran 256 OK (skipped=1) (architecture and QA seats each reproduced this independently); final-tree run Ran 257 OK (skipped=1). Seat: engineer
The real engine reports text_prompt and the server takes text, so the overflow is raised after the engine's count, through PromptText.settle re-raised by primed() before any SSE header - outside the try that only wrapped prepare. The keyword contract therefore broke on the production path even after the wording landed. _anthropic's try now also wraps primed, so that path answers 400 with Anthropic's sentences as well. New test drives the takes_text path (engine.takes_text with generate wrapping settle, the existing technique from test_answer_room_uses_the_engine_clipped_cap): stream both ways, asserts "input length and `max_tokens` exceed context limit", plus a prompt-only case asserting "prompt is too long". Rationale: issue jesdga95#460; the MockEngine stream=True case could not see this gap (id path only). Validation: the new test alone OK (Ran 1, 0.78s); three suites final tree Ran 257 OK (skipped=1); single run of serve.tests.test_server.MaxTokens. test_the_text_prompt_path_says_the_overflow_like_anthropic OK. Seat: engineer
Tests only, no production change: the Responses path already converts the overflow to code context_length_exceeded with param input (three catches around prepare, the count, and collect). Asserts that shape in test_overflow_is_a_recognisable_400_before_streaming, and folds /v1/responses (max_output_tokens at the full window) into the default-clamp 200 case so the flipped default is pinned across all three API shapes. Rationale: issue jesdga95#460; the same contract should hold for the Responses-facing clients too. Validation: inside the suite run recorded on commit 1 (three suites Ran 257 OK (skipped=1) at the final tree). Seat: engineer
When the cap gets shortened instead of a 400, the server now prints one [server] line saying what was asked, what it became, and why (shortening_note in serve/prompt.py, printed from PromptText.settle and from Service._fit). Without it an operator sees an answer stop early with no trace of why. Strata logs the same kind of line; this keeps the parity. The line is stdout-side, not the engine's stderr channel. Self-contained on purpose: drop this one commit and the rest of the series stands. Validation: the default-clamp tests capture stdout and assert the note appears (both paths exercised: ids through _fit, text through settle); three suites final tree Ran 257 OK (skipped=1). Seat: engineer
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #460.
Scope first: nothing in the engine (
src/) is touched. All six commits live in the Pythonserve/layer — a startup default, 400 wording, and tests. Decoding, scheduling, KV, and every performance path are untouched; deployments running today see zero throughput change. Only requests that currently eat a 400 change behavior.Problem and result
Strata and basalt pick opposite defaults for a prompt whose
max_tokensplus prompt overflows the context: Strata shortensmax_tokensto the room left and answers (fit_max_tokens=trueby default); basalt hard-rejects with a 400. A deployment that switched backends behind a reverse proxy watched a stream of instant 400s from clients that top outmax_tokensat the full window — a habit the previous backend trained. After this change basalt answers those requests like Strata does, and its/v1/messagesrejections speak Anthropic's own keyword sentences so harnesses (Claude Code and friends, the contract from Strata's #4838) can react to them.Approach
Six commits, each independently meaningful:
fit-max-tokensdefaults on (BooleanOptionalAction default=True, service kwarg, docs row off→on).--no-fit-max-tokenskeeps the old hard reject.ContextOverflow.anthropic_text()—/v1/messagesoverflows lead with "prompt is too long: N tokens > M maximum" / "input length andmax_tokensexceed context limit: N + M > C", then the basalt sentence. OpenAI and Responses paths keep their words. The reject text also stops pointing at the wrong config place: the key hangs off a model entry of the--configfile (serve/config.pyMODEL_OPTIONS), not the "server" section, and says the switch is on by default.fit_max_tokens=Falseinstead of inheriting it — and pin the new default positively. The config option sweep samples the--no-form for on-by-default switches (same family as the existinggrammarhandling).text_prompt, so the overflow raises after the engine's count, throughPromptText.settlere-raised byprimed()before any SSE header, outside thetrythat only wrappedprepare. Thattrynow also wrapsprimed, and atakes_texttest pins the keyword contract on that path (stream both ways).context_length_exceeded/param=input) — the conversion already existed, it just wasn't asserted — and fold/v1/responsesinto the default-clamp 200 case.[server]log line when a cap is shortened (asked, became, why). Strata logs the same kind of line; without it an operator sees an answer stop early with no trace. Deliberately self-contained: drop this one commit and the series stands.Validation (all run on this machine, Chinese Windows; interpreter = a project venv with jinja2)
test_config+test_context_overflow+test_server) at the final tree:Ran 257 tests, OK (skipped=1)— reproduced independently twice (architecture seat, QA seat). The G1 test alone:Ran 1, OK.test_text_prompt:Ran 6, OK.servediscover on the changed treeRan 1146vs cleanorigin/main(63cb68a) worktreeRan 1144— identical failure set, 32 names, same order, verified by name-list diff (32=32, empty diff). The +2 test delta is this change's new tests. All 32 failures are environment-dependent (localizedstrerrortext, CRLF byte offsets, TLS, process-group primitives, a missing optional pypdf in the test venv) and pre-exist on the baseline; none is in a file this PR touches.src/or native files are touched; nothing to build.Limits and notes
--no-fit-max-tokens(or set it false in a model entry). Help text and the docs row say so._fitpath; the settle path is exercised but not stdout-asserted), and commit 1's failure attribution omits the missing-pypdf family (the largest part of the environment set). Reading counts quoted in messages are all real.