Skip to content

Add a live probe for the Decisions API contract - #32

Merged
Deep-Sixed merged 1 commit into
mainfrom
claude/zealous-edison-ik9ez1
Oct 6, 2026
Merged

Deep-Sixed merged 1 commit into
mainfrom
claude/zealous-edison-ik9ez1

Conversation

@Deep-Sixed

Copy link
Copy Markdown
Owner

Summary

The jev decision adapter (Deep-Sixed/Cerberus#29, Deep-Sixed/Cerberus#30) was built against a contract that is only partly confirmed. The field names inside a single question and a single answer were inferred. scripts/probe-jev-decisions.py checks them against the live endpoint:

OPENROUTER_API_KEY=... uv run --frozen python scripts/probe-jev-decisions.py

What it does

  • Sends one decision through OpenRouterJevDecider itself, so the request body, headers, deadline and answer reader are the ones production uses. A pass-through transport records the raw response so it can be printed next to what Cerberus read from it.

  • Uses synthetic input: a made-up three-model pool described by the real state builder, plus a fixed user message (or --prompt). Nothing from a configuration or a real request is sent, and the key is never printed. One decision call is billed.

  • Asks both the route and effort questions by default; --no-effort asks the route question only. --model, --endpoint, --key-env and --timeout override the defaults, which are those of the jev policy.

  • Names the outcome:

    Outcome What it tells you
    OK Request accepted, and every answer is one of the options offered
    REQUEST REJECTED (HTTP n) The question shape is wrong. Compare the error body with the printed request; the fix goes in build_body
    ANSWER NOT READ for … Request accepted, but the answer sits under a key Cerberus doesn't read. The fix goes in _answer
    ANSWERED OUTSIDE THE OPTIONS An answer was read, but it isn't an option Cerberus offered
    NO ANSWER / UNREADABLE RESPONSE Network, endpoint or key problem; or a non-JSON body
  • Exits 0 only when the request is accepted and every answer is an offered option. It exits 1 otherwise, and 2 when it can't run (no key).

Test plan

  • tests/test_probe_jev_decisions.py runs the probe offline against a mocked endpoint. It covers:
    • the production request shape
    • a passing run
    • route-only mode
    • each failure verdict
    • the key never appearing in output
    • refusal to run without a key
  • Ran over the real network path with a fake key. openrouter.ai is blocked here, so this checked the transport and proxy setup: it reported NO ANSWER (unreachable) and exit 1, and didn't print the key.
  • ./scripts/ci.sh: ruff clean, 440 passed.
  • python scripts/check-public-history.py: 0 findings.
  • Run it once with a real key. This is the run that settles Add jev: Jev decides which model, Cerberus executes it #29's open question.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P4Nm4MVSMmG6JyZi8WumtK


Generated by Claude Code

The jev decision adapter was written against a partly confirmed
contract: the fields inside one question and one answer were inferred.
scripts/probe-jev-decisions.py settles it against a live endpoint.

- Sends one decision through OpenRouterJevDecider itself, so the request
  body, headers, deadline and answer reader are production's, recorded
  at the transport so the raw response can be shown beside what
  Cerberus read from it.
- The decision state is synthetic: a made-up three-model pool described
  by the real state builder, and a fixed or --prompt user message.
  Nothing from a configuration or a real request is sent; the key is
  never printed.
- Asks the route and effort questions by default (--no-effort for the
  route alone), then names the outcome: accepted and read, request
  rejected (with the error body), unreadable response, answer not read
  (pointing at _answer), or answered outside the offered options.
- Exit 0 only when the request is accepted and every answer is an
  offered option; 1 otherwise; 2 when it cannot run (no key).
- tests/test_probe_jev_decisions.py drives it offline against a mocked
  endpoint for each outcome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4Nm4MVSMmG6JyZi8WumtK
@Deep-Sixed
Deep-Sixed marked this pull request as ready for review October 6, 2026 00:41
@Deep-Sixed
Deep-Sixed merged commit 299a8ac into main Oct 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants