Skip to content

Preserve scored terminal model failures - #386

Draft
lluisinthedesert wants to merge 5 commits into
mainfrom
agent/prime-terminal-model-failures
Draft

Preserve scored terminal model failures#386
lluisinthedesert wants to merge 5 commits into
mainfrom
agent/prime-terminal-model-failures

Conversation

@lluisinthedesert

Copy link
Copy Markdown
Contributor

What changed

  • accept Prime context_length and max_turns terminal rows when they are error-free and carry final verifier reward/partial credit
  • narrowly normalize deterministic ProviderError 400 context-window overflows to aggregate score 0 while preserving the raw native trace and error
  • keep generic 400, invalid JSON, 429, 5xx, and network/provider failures rejected
  • surface the native/display stop reason and model-failure outcome in imported rows and the private scorecard

Why

The previous agent_completed-only gate censored genuine model capability and context-fit failures. Corrected-v3 produced both scored Prime context-limit stops and deterministic Anthropic prompt-too-long failures. Those are benchmark outcomes, not missing transport attempts.

Evidence

  • validated the classifier against the three actual Haiku 4.5 automation context-window traces
  • npm run check: 1,077 tests, build, typecheck, 37 public skills, package smoke
  • focused regressions cover scored context_length, normalized context-window ProviderError 400, and rejection of 429/503/invalid-JSON/network cases

No corrected-v3 data was imported or committed.

@lluisinthedesert

Copy link
Copy Markdown
Contributor Author

Added commit 22996e9 for corrected-v3 provider availability evidence. Import configs may now declare provider_unavailable models with reviewed receipt and incomplete coverage. They render in a non-scoring lane with canonical_score=null and are excluded from aggregate rows, pass rates, costs, leaderboard, and Pareto. Import fails if an annotated model also appears in canonical scored traces. Nemotron 3 Super remains unscored; no corrected-v3 data was ingested. Full npm run check is green.

@lluisinthedesert

Copy link
Copy Markdown
Contributor Author

Added the exact Poolside Laguna XS 2.1 context-window message shape in commit HEAD: ProviderError 400 containing Input length N exceeds the maximum allowed input length of M tokens. Both supplied real automation traces classify as context_window_exceeded with aggregate score 0 while raw errors remain untouched. Generic and nearby 400s remain rejected. Focused tests and full npm run check are green.

@lluisinthedesert

Copy link
Copy Markdown
Contributor Author

Corrected-v3 integration hardening added in this commit: diagnostic audit configs can render visibly non-authoritative private scorecards but import-prime rejects aggregate publication; authoritative parity retains strict 100% incumbent calibration. Recognized transient 429/5xx call attempts are accepted only inside otherwise-scored terminal traces when a later usage-bearing call proves recovery; terminal/unknown transport failures remain rejected. Exact OpenAI OverlongPromptError context-limit rows remain score-zero model outcomes. All 10 Cedar scorecards built, all five parity imports calibrated 100%, full npm run check green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant