Skip to content

serve: answer an over-the-context request the way Strata does - #465

Open
petercham wants to merge 6 commits into
jesdga95:mainfrom
petercham:strata-context-fit
Open

petercham wants to merge 6 commits into
jesdga95:mainfrom
petercham:strata-context-fit

Conversation

@petercham

Copy link
Copy Markdown

Closes #460.

Scope first: nothing in the engine (src/) is touched. All six commits live in the Python serve/ layer — a startup default, 400 wording, and tests. Decoding, scheduling, KV, and every performance path are untouched; deployments running today see zero throughput change. Only requests that currently eat a 400 change behavior.

Problem and result

Strata and basalt pick opposite defaults for a prompt whose max_tokens plus prompt overflows the context: Strata shortens max_tokens to the room left and answers (fit_max_tokens=true by default); basalt hard-rejects with a 400. A deployment that switched backends behind a reverse proxy watched a stream of instant 400s from clients that top out max_tokens at the full window — a habit the previous backend trained. After this change basalt answers those requests like Strata does, and its /v1/messages rejections speak Anthropic's own keyword sentences so harnesses (Claude Code and friends, the contract from Strata's #4838) can react to them.

Approach

Six commits, each independently meaningful:

  1. fit-max-tokens defaults on (BooleanOptionalAction default=True, service kwarg, docs row off→on). --no-fit-max-tokens keeps the old hard reject.
  2. ContextOverflow.anthropic_text() — /v1/messages overflows lead with "prompt is too long: N tokens > M maximum" / "input length and max_tokens exceed context limit: N + M > C", then the basalt sentence. OpenAI and Responses paths keep their words. The reject text also stops pointing at the wrong config place: the key hangs off a model entry of the --config file (serve/config.py MODEL_OPTIONS), not the "server" section, and says the switch is on by default.
  3. Tests keep the old hard-reject contract — explicitly via fit_max_tokens=False instead of inheriting it — and pin the new default positively. The config option sweep samples the --no- form for on-by-default switches (same family as the existing grammar handling).
  4. G1: the real production path was still broken — the engine reports text_prompt, so the overflow raises after the engine's count, through PromptText.settle re-raised by primed() before any SSE header, outside the try that only wrapped prepare. That try now also wraps primed, and a takes_text test pins the keyword contract on that path (stream both ways).
  5. Tests pin the Responses shape (context_length_exceeded / param=input) — the conversion already existed, it just wasn't asserted — and fold /v1/responses into the default-clamp 200 case.
  6. A [server] log line when a cap is shortened (asked, became, why). Strata logs the same kind of line; without it an operator sees an answer stop early with no trace. Deliberately self-contained: drop this one commit and the series stands.

Validation (all run on this machine, Chinese Windows; interpreter = a project venv with jinja2)

  • Passed: three suites (test_config + test_context_overflow + test_server) at the final tree: Ran 257 tests, OK (skipped=1) — reproduced independently twice (architecture seat, QA seat). The G1 test alone: Ran 1, OK. test_text_prompt: Ran 6, OK.
  • Passed: full serve discover on the changed tree Ran 1146 vs clean origin/main (63cb68a) worktree Ran 1144 — identical failure set, 32 names, same order, verified by name-list diff (32=32, empty diff). The +2 test delta is this change's new tests. All 32 failures are environment-dependent (localized strerror text, CRLF byte offsets, TLS, process-group primitives, a missing optional pypdf in the test venv) and pre-exist on the baseline; none is in a file this PR touches.
  • Not run: CUDA/HIP/SYCL builds — no src/ or native files are touched; nothing to build.

Limits and notes

  • Default change: users who relied on the hard 400 must now pass --no-fit-max-tokens (or set it false in a model entry). Help text and the docs row say so.
  • Bisect heads-up: commits 1–2 land production changes before commit 3 rewrites the old contract tests, so the tree at commits 1–2 fails the pre-existing overflow contract tests (verified by checkout). Squash-on-merge or bisect-around; reordering was open to the maintainer's preference.
  • Two commit-message phrasings are slightly stronger than the evidence: the log-line commit's claim that both shortening paths are stdout-asserted (the assertion covers the _fit path; the settle path is exercised but not stdout-asserted), and commit 1's failure attribution omits the missing-pypdf family (the largest part of the environment set). Reading counts quoted in messages are all real.
  • Measured on a local deployment switched from Strata to basalt behind a proxy; the issue report (serve: clamp an over-context max_tokens by default, like Strata does (and speak Anthropic's wording on /v1/messages) #460) carries the reproduction.

A prompt plus an answer cap over the context window now gets the cap
shortened to the room the context leaves instead of a 400, matching how
Strata serves the same request. --no-fit-max-tokens keeps the old hard
reject. Docs row flipped off -> on.

Rationale: clients that top out max_tokens at the full window (the
habit a Strata deployment trains) hit a wall of 400s after switching
backends; issue jesdga95#460.

Validation (interpreter D:/Strata/.venv/Scripts/python.exe, cwd
D:/Strata/.scratch/basalt-src, branch strata-context-fit):
- three suites test_config + test_context_overflow + test_server at the
  full final tree: Ran 257 tests, OK (skipped=1)
- full serve discover: Ran 1146, failures=10 errors=22 skipped=41; the
  clean origin/main worktree run: Ran 1144, same 10F+22E names, same
  order (env-dependent Chinese-Windows strerror/TLS/process-group
  failures, pre-existing; the +2 test delta is this change's new tests)

Seat: engineer
/v1/messages 400s for an over-the-context request now lead with
Anthropic's own sentences - "prompt is too long: N tokens > M maximum"
and "input length and `max_tokens` exceed context limit: N + M > C" -
the keyword contract harnesses (Claude Code and friends) react to,
then the basalt sentence. OpenAI and Responses paths keep their words.

Also fixes the reject text pointing at the wrong config place:
fit-max-tokens hangs off a model entry of the --config file
(serve/config.py MODEL_OPTIONS), not the "server" section; and it now
says the switch is on by default.

Rationale: issue jesdga95#460, second half of the Strata-parity gap.

Validation: covered by test_context_overflow (Anthropic wording
assertions on both over-room and prompt-only cases, stream both ways)
inside the suite run recorded on commit 1 (Ran 257 OK (skipped=1)).

Seat: engineer
The old hard-reject cases now ask for it explicitly
(fit_max_tokens=False, restored in finally) instead of inheriting it
from the default, and assert the corrected "model entry of the
--config file" pointer. New positive cases pin the new default: an
over-the-context budget gets clamped and answers (openai/anthropic).
test_config's EveryOption samples the no- form for switches that are
on by default, so the CLI-vs-default assertion keeps its bite
(same family as the existing grammar handling); the sweep skips
--no-* twins whose semantics ride with their positive form.

Rationale: issue jesdga95#460; contracts follow the flipped default without
silently deleting the old expectations.

Validation: three suites at the step-0 tree read Ran 256 OK
(skipped=1) (architecture and QA seats each reproduced this
independently); final-tree run Ran 257 OK (skipped=1).

Seat: engineer
The real engine reports text_prompt and the server takes text, so the
overflow is raised after the engine's count, through PromptText.settle
re-raised by primed() before any SSE header - outside the try that
only wrapped prepare. The keyword contract therefore broke on the
production path even after the wording landed. _anthropic's try now
also wraps primed, so that path answers 400 with Anthropic's
sentences as well.

New test drives the takes_text path (engine.takes_text with
generate wrapping settle, the existing technique from
test_answer_room_uses_the_engine_clipped_cap): stream both ways,
asserts "input length and `max_tokens` exceed context limit", plus a
prompt-only case asserting "prompt is too long".

Rationale: issue jesdga95#460; the MockEngine stream=True case could not see
this gap (id path only).

Validation: the new test alone OK (Ran 1, 0.78s); three suites final
tree Ran 257 OK (skipped=1); single run of
serve.tests.test_server.MaxTokens.
test_the_text_prompt_path_says_the_overflow_like_anthropic OK.

Seat: engineer
Tests only, no production change: the Responses path already converts
the overflow to code context_length_exceeded with param input (three
catches around prepare, the count, and collect). Asserts that shape in
test_overflow_is_a_recognisable_400_before_streaming, and folds
/v1/responses (max_output_tokens at the full window) into the
default-clamp 200 case so the flipped default is pinned across all
three API shapes.

Rationale: issue jesdga95#460; the same contract should hold for the
Responses-facing clients too.

Validation: inside the suite run recorded on commit 1 (three suites
Ran 257 OK (skipped=1) at the final tree).

Seat: engineer
When the cap gets shortened instead of a 400, the server now prints
one [server] line saying what was asked, what it became, and why
(shortening_note in serve/prompt.py, printed from PromptText.settle
and from Service._fit). Without it an operator sees an answer stop
early with no trace of why. Strata logs the same kind of line; this
keeps the parity. The line is stdout-side, not the engine's stderr
channel.

Self-contained on purpose: drop this one commit and the rest of the
series stands.

Validation: the default-clamp tests capture stdout and assert the
note appears (both paths exercised: ids through _fit, text through
settle); three suites final tree Ran 257 OK (skipped=1).

Seat: engineer

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

serve: clamp an over-context max_tokens by default, like Strata does (and speak Anthropic's wording on /v1/messages)

1 participant