Preserve scored terminal model failures - #386
Conversation
|
Added commit 22996e9 for corrected-v3 provider availability evidence. Import configs may now declare provider_unavailable models with reviewed receipt and incomplete coverage. They render in a non-scoring lane with canonical_score=null and are excluded from aggregate rows, pass rates, costs, leaderboard, and Pareto. Import fails if an annotated model also appears in canonical scored traces. Nemotron 3 Super remains unscored; no corrected-v3 data was ingested. Full npm run check is green. |
|
Added the exact Poolside Laguna XS 2.1 context-window message shape in commit HEAD: ProviderError 400 containing Input length N exceeds the maximum allowed input length of M tokens. Both supplied real automation traces classify as context_window_exceeded with aggregate score 0 while raw errors remain untouched. Generic and nearby 400s remain rejected. Focused tests and full npm run check are green. |
|
Corrected-v3 integration hardening added in this commit: diagnostic audit configs can render visibly non-authoritative private scorecards but import-prime rejects aggregate publication; authoritative parity retains strict 100% incumbent calibration. Recognized transient 429/5xx call attempts are accepted only inside otherwise-scored terminal traces when a later usage-bearing call proves recovery; terminal/unknown transport failures remain rejected. Exact OpenAI OverlongPromptError context-limit rows remain score-zero model outcomes. All 10 Cedar scorecards built, all five parity imports calibrated 100%, full npm run check green. |
What changed
context_lengthandmax_turnsterminal rows when they are error-free and carry final verifier reward/partial creditWhy
The previous
agent_completed-only gate censored genuine model capability and context-fit failures. Corrected-v3 produced both scored Prime context-limit stops and deterministic Anthropic prompt-too-long failures. Those are benchmark outcomes, not missing transport attempts.Evidence
npm run check: 1,077 tests, build, typecheck, 37 public skills, package smokecontext_length, normalized context-window ProviderError 400, and rejection of 429/503/invalid-JSON/network casesNo corrected-v3 data was imported or committed.