Skip to content

Add runtime model fallback to the agent loop (WI-041) - #122

Merged
nateshpp merged 1 commit into
mainfrom
feature/agent-model-fallback-runtime
Jul 7, 2026
Merged

nateshpp merged 1 commit into
mainfrom
feature/agent-model-fallback-runtime

Conversation

@nateshpp

@nateshpp nateshpp commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

A role with a prioritized models list (ADR-039) already picked the best model a budget
affords, statically, before any call was made. This closes the gap that left open: when the
resolved primary model turns out unreachable at invocation time (a network-layer failure, or
a timeout), scripts/agent/run-cycle.mjs's openai-http transport (invokeRoleOpenAI) now
falls back to the next model in the role's chain instead of failing the whole cycle. A
candidate the current daily budget cannot afford is skipped without an attempt, so a fallback
can never silently spend past the budget. A real answer from a reachable endpoint, an auth
failure, a bad status, a malformed response, is never retried, only genuine unreachability is,
so a broken diff or a rejected request is never silently re-tried against a different model.
The metric recorded for a fallback run reflects the model that actually answered, plus a
model_fallback_from field listing every unreachable candidate tried first, so the audit
trail never misattributes a diff to the originally planned model.

The change is fully backward compatible: a plan built without a runtime chain (any hand-built
plan, or a role with a single configured model) takes the exact original one-shot code path,
with no new checks and no behavior change. Closes WI-041.

Governance checklist

  • npm run verify passes locally (drift guard + style check + tests + AgentProof)
  • No em dashes in any tracked file (the style check rejects them)
  • Prose states conclusions directly (style check rejects hedging phrases)
  • No AI co-author lines in commit messages
  • If ratchet logic changed: updated docs/specs/ratchet-spec.md and added/updated a fixture (N/A, no ratchet logic touched)
  • If a new AgentProof scenario: fixture is zero-false-positive (N/A, no new AgentProof scenario)
  • If config schema changed: schema_version bumped and CHANGELOG entry added (N/A, no schema change; CHANGELOG entry added regardless)
  • If prompt changed: npm run build:prompt regenerated the bundle (N/A, no prompt change)

Test evidence

# tests 832
# suites 17
# pass 832
# fail 0
# cancelled 0
# skipped 0
# todo 0

AgentProof: Score: 25/25 normative | 10/10 extended (35/35 total)
Level: HARDENED

New coverage: tests/agent-model-fallback.test.mjs (10 tests) covers the unreachable-error
classifier, the fallback chain builder (including CLI-override ordering and registry pruning),
planCycle attaching a resolved chain to maker/checker, and invokeRoleOpenAI integration:
falls back on a genuinely unreachable primary, never falls back on a real non-network error,
skips an unaffordable paid candidate without attempting it, and is behavior-identical for a
plan with no chain.

A role with a prioritized models list (ADR-039) already picked the best
model a budget affords, statically, before any call. This closes the
gap that left open: when the resolved primary model turns out
unreachable at invocation time, the openai-http transport now falls
back to the next model in the role's chain instead of failing the
whole cycle, skipping any candidate the current budget cannot afford
and never retrying a real (non-network) error from a reachable
endpoint. The metric for a fallback run records the model that
actually answered and which candidates were tried first, so the audit
trail stays accurate.
@nateshpp
nateshpp requested a review from techseek4vr as a code owner July 7, 2026 02:41
@nateshpp
nateshpp merged commit 4743baa into main Jul 7, 2026
9 of 10 checks passed
@nateshpp
nateshpp deleted the feature/agent-model-fallback-runtime branch July 7, 2026 05:22
nateshpp added a commit that referenced this pull request Jul 7, 2026
Resolves conflicts against WI-041 (runtime model fallback, #122),
which merged first: both branches touched run-cycle.mjs and
CHANGELOG.md's Unreleased section for complementary, non-overlapping
features. Regenerates the snapshot from scratch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant