Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,9 @@ that model first through its own failover loop, the rest in configured order.
silently change routing.
- If no usable decision comes back, the request runs in configured order; a
decision outage never takes the alias down.
- Optionally (`jev.reasoning_efforts`), the same call also asks how much
reasoning effort the request needs; the answer replaces the effort of pool
candidates that already set one, and a caller's own effort still wins.
- Streams, fails over and cools down like any dispatch alias.

## Dispatch
Expand Down
4 changes: 3 additions & 1 deletion config/cerberus.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -87,14 +87,16 @@ aliases:
# first through its own failover loop. No usable answer -> configured order.
candidates: # configured order is the fallback order
- { provider: google, credential: free-1, model: gemini-flash }
- { provider: openrouter, credential: primary, model: anthropic/claude-sonnet }
# a candidate that sets reasoning_effort is one Jev's effort answer may replace
- { provider: openrouter, credential: primary, model: anthropic/claude-sonnet, reasoning_effort: medium }
jev:
decider: { provider: openrouter, credential: primary }
model: typesafe/jev-1.13 # pinned; bump deliberately after testing
input: last_user_message # or full_conversation, or metadata (no text)
max_input_chars: 4000
timeout_seconds: 5 # decision deadline; then configured order
allow_paid_pool: true
reasoning_efforts: [low, medium, high] # also ask Jev for an effort, among these only

identities:
example-reader:
Expand Down
16 changes: 15 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,23 @@ error, an unreadable response, an absent or unknown choice — runs the request
configured order instead, which every pool member already satisfies; the alias
stays routable while no decision can be had.

With `jev.reasoning_efforts` set, the same call asks a second question: how much
reasoning effort the request needs, answered from that list only. The answer
replaces the configured `reasoning_effort` of every planned candidate that sets
one, and is never added to a candidate that sets none, because a route without
one may not accept the parameter at all; the list is how an operator keeps Jev
to values every such route accepts. The egress rule is unchanged: a caller's own
stated effort wins unless the candidate sets `reasoning_effort_override`. The
effort is vetted on its own — an answer outside the list leaves the configured
effort standing, whatever the model answer was — and with one model left the
call is still made for the effort alone. A plan may change the effort of a
target that sets one and nothing else about it; the dispatch loop refuses any
other difference from a configured target.

Each routing event carries a `jev` record: options offered, whether the decision
was `chosen`, `skipped` or a `fallback`, the reason, the choice and its
probability, decision latency, status and usage. It never carries request text.
probability, the effort applied with its probability or why none was, decision
latency, status and usage. It never carries request text.
The confirmed Decisions API contract covers the endpoint, the model, the
`model`/`state`/`questions` request and `answers` keyed by question id; the
fields inside one question and one answer are not yet confirmed, so the answer
Expand Down
1 change: 1 addition & 0 deletions src/cerberus/control/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,7 @@ def _jev(
"credential_ref": policy.decider.credential,
"model": policy.model,
"input": policy.input,
"reasoning_efforts": policy.reasoning_efforts,
"max_input_chars": policy.max_input_chars,
"timeout_seconds": policy.timeout_seconds,
"readiness": decider_readiness(
Expand Down
12 changes: 8 additions & 4 deletions src/cerberus/jev/__init__.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,31 @@
"""jev mode: Jev as Cerberus's model-selection intelligence, Cerberus as executor.

Cerberus filters a pool that may span every provider it knows — local hosts,
Cloudflare, OpenRouter — and asks Jev, through OpenRouter's Decisions API, one
question: which surviving model is the cheapest one strong enough for this
request. Cerberus then runs that model through its own failover loop. Jev
decides; Cerberus governs and executes.
Cloudflare, OpenRouter — and asks Jev, through OpenRouter's Decisions API,
which surviving model is the cheapest one strong enough for this request and,
by policy, how much reasoning effort it needs. Cerberus then runs that model
through its own failover loop. Jev decides; Cerberus governs and executes.
"""

from cerberus.jev.backend import (
Answer,
Decision,
DecisionBackend,
DecisionError,
DecisionRequest,
OpenRouterJevDecider,
Question,
)
from cerberus.jev.dispatch import decider_readiness, jev_dispatch

__all__ = [
"Answer",
"Decision",
"DecisionBackend",
"DecisionError",
"DecisionRequest",
"OpenRouterJevDecider",
"Question",
"decider_readiness",
"jev_dispatch",
]
56 changes: 33 additions & 23 deletions src/cerberus/jev/backend.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
"""Decision-service seam plus the OpenRouter Decisions API implementation.

A jev alias asks one bounded question — which pool model should serve this
request — and gets back one of the option ids it offered, or nothing usable.
Nothing outside this module knows the wire shape of a decision service.
A jev alias asks bounded questions in one call — which pool model should serve
this request and, when its policy says so, how much reasoning effort the request
needs — and gets back, per question, one of the options it offered or nothing
usable. Nothing outside this module knows the wire shape of a decision service.

Confirmed contract for OpenRouter's Decisions API: ``POST /api/alpha/decisions``
with a bearer key; the body carries ``model``, ``state`` and ``questions``;
Jev answers ``choice``, ``noul`` and ``score`` questions; the response carries
``answers`` keyed by question id, with typed results and probabilities, and
``usage``. The field names inside one question and one answer are not yet
confirmed, so the request uses the plainest form and the answer is read
tolerantly. A misread can only ever produce "no usable answer", because the
choice must equal an option id Cerberus sent; the request then runs in
configured order, which every pool member already satisfies.
tolerantly. A misread can only ever produce "no usable answer", because each
choice must equal an option Cerberus sent; the request then runs in configured
order and configured effort, which policy already allows.
"""

from __future__ import annotations
Expand All @@ -26,31 +27,45 @@

DecisionFailure = Literal["timeout", "unreachable", "http_error", "invalid_response"]

# the one question a jev alias asks, by id
# the questions a jev alias asks, by id
ROUTE_QUESTION_ID = "route"
ROUTE_QUESTION = (
"Which candidate model is the least costly one that is still strong enough "
"to handle this request well?"
)
EFFORT_QUESTION_ID = "effort"
EFFORT_QUESTION = "How much reasoning effort should the chosen model spend on this request?"
# keys an answer's chosen option may sit under, most specific first
_CHOICE_KEYS = ("choice", "result", "value", "answer", "selected")
_USAGE_KEYS = ("prompt_tokens", "completion_tokens", "total_tokens", "cost")


@dataclass(frozen=True, slots=True)
class Question:
id: str
question: str
options: list[str] # opaque model ids or effort levels; never a provider name


@dataclass(frozen=True, slots=True)
class DecisionRequest:
endpoint: str
api_key: str
model: str
state: str # everything the decision service may read, already scoped by policy
options: list[str] # opaque option ids; never a provider name
questions: list[Question]
timeout_seconds: float


@dataclass(frozen=True, slots=True)
class Decision:
choice: str | None # an option id exactly as answered, or None when absent
class Answer:
choice: str | None # exactly as answered, or None when absent; not yet vetted
confidence: float | None


@dataclass(frozen=True, slots=True)
class Decision:
answers: dict[str, Answer] # one per question asked, keyed by question id
usage: dict[str, float] | None
http_status: int
latency_ms: float
Expand Down Expand Up @@ -86,12 +101,8 @@ def build_body(request: DecisionRequest) -> dict[str, Any]:
"model": request.model,
"state": request.state,
"questions": [
{
"id": ROUTE_QUESTION_ID,
"type": "choice",
"question": ROUTE_QUESTION,
"options": list(request.options),
}
{"id": q.id, "type": "choice", "question": q.question, "options": list(q.options)}
for q in request.questions
],
}

Expand Down Expand Up @@ -125,10 +136,9 @@ def elapsed() -> float:
if not isinstance(payload, dict):
raise DecisionError("invalid_response", latency_ms=elapsed(), http_status=response.status_code)

choice, confidence = _route_answer(payload.get("answers"))
answers = payload.get("answers")
return Decision(
choice=choice,
confidence=confidence,
answers={q.id: Answer(*_answer(answers, q.id)) for q in request.questions},
usage=_usage(payload.get("usage")),
http_status=response.status_code,
latency_ms=elapsed(),
Expand All @@ -138,13 +148,13 @@ async def aclose(self) -> None:
await self._client.aclose()


def _route_answer(answers: Any) -> tuple[str | None, float | None]:
"""The chosen option and its probability from the route answer, if present."""
def _answer(answers: Any, question_id: str) -> tuple[str | None, float | None]:
"""The chosen option and its probability for one question, if present."""

if isinstance(answers, list):
answer = next((a for a in answers if isinstance(a, dict) and a.get("id") == ROUTE_QUESTION_ID), None)
answer = next((a for a in answers if isinstance(a, dict) and a.get("id") == question_id), None)
elif isinstance(answers, dict):
answer = answers.get(ROUTE_QUESTION_ID)
answer = answers.get(question_id)
else:
return None, None
if isinstance(answer, str):
Expand Down
Loading
Loading