Truncated model output is misreported as a schema failure and retried at the same cap

[seonghobae]

Gap

The model clients never read finish_reason. A response truncated because it hit the output cap is indistinguishable from a complete one, and the recovery path makes the situation worse rather than better.

Verified on main (@8c6a2fa): grep -rn "finish_reason\|incomplete" src/four_pillars/ returns nothing. NimClient._content accepts any non-empty choices[0].message.content string, so an HTTP 200 carrying a half-written JSON object is treated as a successful generation.

The generation caps have no recorded basis

location	value
nim.py:156	max_tokens: int = 4096
generation.py:35	max_tokens: int = 4096
analysis.py:159	max_tokens=8192 (synthesis)
analysis.py:189	max_tokens=8192 (editorial repair)

Nothing in docs/, CLAUDE.md, or AGENTS.md states where 4096 or 8192 came from — not a provider constraint, not a model catalog entry, not an administrator policy. There is no catalog of per-model output limits anywhere in the repository, so the caps cannot follow the selected model or deployment.

Why truncation gets worse on retry, not better

NimClient.generate treats a parse or schema failure as a repairable schema error (nim.py:185-208). On each round it appends the previous truncated output as an assistant message plus a repair instruction, then re-posts with the same max_tokens:

messages.extend([
    {"role": "assistant", "content": raw_content},     # the truncated text
    {"role": "user", "content": "Return the complete answer again ..."},
])
continue                                                # same max_tokens

So the input grows every round while the output ceiling stays fixed. If the first attempt was cut off for lack of output budget, the retry has strictly less room to succeed. With nim_max_schema_repairs defaulting to 1 there is exactly one such retry, after which the client raises

{provider} output failed schema validation after N repair attempts: ...

which attributes an exhausted output budget to a schema defect. The operator sees a schema error, the job records a quality failure, and the real cause — the cap — is never named.

A second, quieter outcome: if a truncation happens to leave syntactically valid JSON that still satisfies the model schema, the report is accepted and delivered to the customer with content silently missing. Nothing in the pipeline can detect it, because the quality gate in quality.py checks section presence and copy, not whether generation completed.

Required behavior

1. Read the provider's completion status (finish_reason, and status/incomplete_details where the API provides them) and classify a truncated response as its own outcome, separate from a schema violation and from success.
2. Do not retry a truncation at the same cap. Either raise the limit for the retry on evidence, or fail explicitly as incomplete; the current same-cap retry is guaranteed to have less room than the attempt that already failed.
3. Derive the generation limit from the selected model, provider, and deployment rather than from a literal, and record the source, the computed value, and the value actually sent alongside the existing trace fields.
4. Surface the distinction in GenerationTrace and in the stored job failure reason, so "the model ran out of output budget" is never reported as "the model returned invalid JSON".

Regression cases

• a 200 response whose finish_reason is length and whose body is valid JSON satisfying the schema must not be accepted as a complete report
• a truncated response must not be retried at the same cap, and the raised error must name the cap, not the schema
• a genuine schema violation must still take the existing repair path unchanged
• the limit must change when the selected model or deployment changes, and tests must not pin it to a literal

Scope note

src/four_pillars/nim.py and generation.py are not claimed by any open PR, but tests/test_nim.py and tests/test_nim_errors.py are owned by #39, and the routing change in #39 decides which catalog the limit should be read from. Filed rather than patched so the limit source and the routing land together.

Internal decision target: 2026-09-21. Next review: 2026-09-18.

[seonghobae]

PR #53 repairs the finish_reason half of this issue rather than leaving it filed.

What the repair covers: finish_reason did not appear anywhere in the repository, so a reply the gateway cut short at the token ceiling was validated like any complete reply. Because every report section model in analysis.py has optional later fields, a truncated prefix validates, and a half-written interpretation reached the customer's PDF as a finished section. _content now raises NimTruncationError for length and the max_tokens variant. An absent or unrecognised value still means unknown, so gateways that omit the field are unaffected.

What stays open here: the token-ceiling half. max_tokens: int = 4096 is a signature default with no catalog behind it, and there is no nim_max_tokens setting. Deriving the limit from the model or deployment catalog has to change settings.py, which PRs #31 and #39 both currently occupy, so it is not forced into #53.

Evidence on #53's branch: 257 passed, 1 deselected, 100% statement and branch coverage, Ruff, compileall, check_docs (19 documents), and product_gap_audit all pass. The new test file failed with an ImportError before the source change.

[seonghobae]

A third site, so whoever takes the remaining half does not miss it

The token-ceiling work is not confined to nim.py. max_tokens: int = 4096 also appears on the StructuredGenerationClient protocol in src/four_pillars/generation.py, which is the provider-neutral contract every backend implements.

Every occurrence of the constant today:

file	site
src/four_pillars/generation.py	the protocol's generate signature
src/four_pillars/nim.py	the implementation's generate signature
src/four_pillars/analysis.py	two call sites that override it with 8192
scripts/nim_eval.py	the evaluation harness overrides it with 1024

The two overrides in analysis.py are for the synthesis and the editorial repair, and they are equally unsourced: 8192 is asserted rather than derived. The evaluation harness asserts a third number, 1024, for the same operation. So a catalog-driven limit has to reach the protocol, the implementation, and all three call sites together, or the contract and the callers will disagree about where the ceiling comes from.

generation.py is claimed by no open PR. analysis.py is likewise unclaimed. settings.py, where a nim_max_tokens setting would live, is still occupied by both #31 and #39, which remains the reason I am not landing this.

Related: the finish_reason half is already repaired in #53, and #58 measures a separate unsourced constant in the same family, the equation-of-time approximation in calendar.py.