Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,15 +253,16 @@ The generated shell hooks expect Git Bash; PowerShell-only setups are not covere

Use `SMART_ROUTER_CONFIG_VERSION` at launch to select a smart-routing configuration.
Each version defines separate Claude Code and Codex flag mappings; the existing versions
use identical mappings for both agents:
use identical mappings for both agents. The Claude-only version disables routing for Codex:

| Version | Subagent routing | First-prompt routing | Orchestrator |
| --- | --- | --- | --- |
| `first_prompt_and_subagent_no_orch_v0` | On | On | Off |
| `subagent_only_v0` | On | Off | Off |
| `subagent_only_v1` | On | Off | Off |
| `subagent_orch_v0` | On | Off | On |
| `subagent_orch_v1` | On | Off | On |
| Version | Routed agents | Subagent routing | First-prompt routing | Orchestrator |
| --- | --- | --- | --- | --- |
| `first_prompt_and_subagent_no_orch_v0` | Claude, Codex | On | On | Off |
| `subagent_only_v0` | Claude, Codex | On | Off | Off |
| `subagent_only_v1` | Claude, Codex | On | Off | Off |
| `subagent_orch_v0` | Claude, Codex | On | Off | On |
| `subagent_orch_v1` | Claude, Codex | On | Off | On |
| `subagent_orch_v0_claude_only` | Claude | On | Off | On |

`first_prompt_and_subagent_no_orch_v0` is the customer configuration for first-prompt
and subagent routing without orchestration: `ENABLE_SMART_ROUTING_V2=1`,
Expand All @@ -274,6 +275,11 @@ so first-prompt routing remains off; orchestration is also off.
`subagent_orch_v1` enables all three legacy flags. Like `subagent_only_v1`, it routes
subagents rather than the first prompt, and it additionally enables orchestration.

`subagent_orch_v0_claude_only` keeps Claude's `subagent_orch_v0` flags (`0`, `1`, `1`)
and sets all three flags to `"0"` for Codex. Codex uses its ordinary launch and
managed default model, without routing or orchestration. Explicit launch/session
on/off controls retain their existing precedence over presets.

`SMART_ROUTER_NAME` still selects the router independently of the preset.

Smart-routed Claude and Codex sessions install `smart-router`. The `subagent_orch_v0`
Expand Down
33 changes: 32 additions & 1 deletion src/ucode/smart_routing/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
# Route only subagents with V2, subagent-only, and orchestration all enabled.
SUBAGENT_ORCH_V1 = "subagent_orch_v1"

# Preserve subagent_orch_v0 for Claude, with all Codex routing flags disabled.
SUBAGENT_ORCH_V0_CLAUDE_ONLY = "subagent_orch_v0_claude_only"

_BASE_VERSIONS = {
FIRST_PROMPT_AND_SUBAGENT_NO_ORCH_V0: {
ENABLE_SMART_ROUTING_ENV_VAR: "1",
Expand Down Expand Up @@ -60,7 +63,35 @@

# Agents share the base preset unless explicitly changed; comment any intentional difference.
_VERSIONS = {
version: {AGENT_CLAUDE: flags, AGENT_CODEX: flags} for version, flags in _BASE_VERSIONS.items()
FIRST_PROMPT_AND_SUBAGENT_NO_ORCH_V0: {
AGENT_CLAUDE: _BASE_VERSIONS[FIRST_PROMPT_AND_SUBAGENT_NO_ORCH_V0],
AGENT_CODEX: _BASE_VERSIONS[FIRST_PROMPT_AND_SUBAGENT_NO_ORCH_V0],
},
SUBAGENT_ONLY_V0: {
AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ONLY_V0],
AGENT_CODEX: _BASE_VERSIONS[SUBAGENT_ONLY_V0],
},
SUBAGENT_ONLY_V1: {
AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ONLY_V1],
AGENT_CODEX: _BASE_VERSIONS[SUBAGENT_ONLY_V1],
},
SUBAGENT_ORCH_V0: {
AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ORCH_V0],
AGENT_CODEX: _BASE_VERSIONS[SUBAGENT_ORCH_V0],
},
SUBAGENT_ORCH_V1: {
AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ORCH_V1],
AGENT_CODEX: _BASE_VERSIONS[SUBAGENT_ORCH_V1],
},
# Codex opts out of routing and orchestration; Claude keeps subagent_orch_v0.
SUBAGENT_ORCH_V0_CLAUDE_ONLY: {
AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ORCH_V0],
AGENT_CODEX: {
ENABLE_SMART_ROUTING_ENV_VAR: "0",
ENABLE_SUBAGENT_ROUTING_ENV_VAR: "0",
ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0",
},
},
Comment thread
lilly-luo marked this conversation as resolved.
}


Expand Down
5 changes: 5 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ CUJ4's Claude preset sessions also use it after verifying delegated file tasks:
answer does not prove that a resumed child has stopped running.
They cover Claude/Codex helper dispatch and rejection of routing decisions without
the agent-specific prompt-submission evidence.
The shared preset CUJ runs both agents for every version and derives routing and
orchestration assertions from each agent's resolved flags. Both must complete a native
child task. `subagent_orch_v0_claude_only` routes Claude's child; Codex uses the managed
default without routing requests, new router decisions, recipe headers, or orchestrator context.

Agent-specific smart-routing preset component checks cover shared base presets, explicit agent mappings,
complete per-agent validation, session precedence, and preserving the preset selector
until managed-default agent selection.
Expand Down
11 changes: 7 additions & 4 deletions tests/e2e_cuj/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,11 @@ uv run pytest -c tests/e2e_cuj/pytest.ini --confcutdir=tests/e2e_cuj \
## Smart-routing CUJ

`test_cuj4_smart_routing.py` uses its own read-only workspace with managed smart routing enabled.
It runs the original routed/explicit-model journeys and five
It runs the original routed/explicit-model journeys and six
`SMART_ROUTER_CONFIG_VERSION` cases using the shared version constants; each launches Claude
and Codex once.
The independent expectation table is:
The CUJ derives routing and orchestration assertions from each agent's resolved preset flags.
Component tests independently verify the preset values. The resulting expectations are:

| Selector | First prompt routed | Orchestrator context | Child on first prompt | Explicitly requested child |
| --- | --- | --- | --- | --- |
Expand All @@ -77,6 +78,7 @@ The independent expectation table is:
| `subagent_only_v1` | No | No | No | Yes, routed |
| `subagent_orch_v0` | No | Yes | No | Yes, routed |
| `subagent_orch_v1` | No | Yes | No | Yes, routed |
| `subagent_orch_v0_claude_only` | No | Claude only | No | Yes for both; only Claude routed |

First prompts explicitly forbid delegation to isolate first-prompt routing. Assertions require
the expected presence/absence of a prompt-correlated router request, successful inference on
Expand All @@ -91,8 +93,9 @@ only known display/safeguards rejections followed by native removal of the rejec
and a final non-empty 200 are accepted. Model, prompt, budget, and effort must remain unchanged.
Each preset session then explicitly requests one child for a separate hidden-value
file task. Assertions require a new native child transcript containing the value,
a correlated spawn-routing decision, and successful child inference on the
router's selected model. This tests requested delegation, not automatic orchestrator delegation.
a correlated spawn-routing decision and successful child inference on the
router's selected model when routing is enabled. Disabled routing requires no router
requests, new routing decisions, recipe headers, or orchestrator context. This tests requested delegation, not automatic orchestrator delegation.
The wait uses the child's answer directly; parent-answer reconstruction is reserved for first-prompt tasks.
Offline checks reject parent-only answers and accept child answers before the parent replies.
Offline evidence regressions do not establish a live CUJ pass.
Expand Down
97 changes: 60 additions & 37 deletions tests/e2e_cuj/test_cuj4_smart_routing.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
SUBAGENT_ONLY_V0,
SUBAGENT_ONLY_V1,
SUBAGENT_ORCH_V0,
SUBAGENT_ORCH_V0_CLAUDE_ONLY,
SUBAGENT_ORCH_V1,
resolve_environment,
)

from .base import BaseCujTest
Expand Down Expand Up @@ -230,22 +232,22 @@ class TestCujSmartRouting(BaseCujTest):
WORKSPACE_URL = "https://dbc-1a9622fc-2e91.cloud.databricks.com/"

@pytest.mark.parametrize(
"SMART_ROUTER_CONFIG_VERSION, first_prompt_routed, orchestrator_enabled",
"SMART_ROUTER_CONFIG_VERSION",
[
(FIRST_PROMPT_AND_SUBAGENT_NO_ORCH_V0, True, False),
(SUBAGENT_ONLY_V0, False, False),
(SUBAGENT_ONLY_V1, False, False),
(SUBAGENT_ORCH_V0, False, True),
(SUBAGENT_ORCH_V1, False, True),
FIRST_PROMPT_AND_SUBAGENT_NO_ORCH_V0,
SUBAGENT_ONLY_V0,
SUBAGENT_ONLY_V1,
SUBAGENT_ORCH_V0,
SUBAGENT_ORCH_V1,
SUBAGENT_ORCH_V0_CLAUDE_ONLY,
],
)
def test_smart_router_config_version(
self, cuj, SMART_ROUTER_CONFIG_VERSION, first_prompt_routed, orchestrator_enabled
):
def test_smart_router_config_version(self, cuj, SMART_ROUTER_CONFIG_VERSION):
"""Scenario: launch both agents with a preset, then explicitly request a subagent.

Expected: first-prompt routing and orchestrator context match the preset;
one routed native child completes the delegated task.
Expected: each agent's resolved flags determine first-prompt routing,
orchestration, and child routing. One native child completes the delegated
task even when routing is disabled.
"""
session, workspace, recorder = cuj
previous = session.env.get("SMART_ROUTER_CONFIG_VERSION")
Expand All @@ -254,9 +256,21 @@ def test_smart_router_config_version(
configs = _assert_published_config_matches_expectations(workspace.config())
recorder.configure_session(session, ["configure", "--disable-databricks-ai-tools"])
for agent in AGENTS:
expected_env = resolve_environment(
{"SMART_ROUTER_CONFIG_VERSION": SMART_ROUTER_CONFIG_VERSION}, agent=agent
)
v2_enabled = expected_env["ENABLE_SMART_ROUTING_V2"] == "1"
subagent_only = expected_env["ENABLE_SMART_ROUTING_SUBAGENT_ONLY"] == "1"
subagent_routed = v2_enabled or subagent_only
first_prompt_routed = v2_enabled and not subagent_only
orchestrator_enabled = (
expected_env["ENABLE_SMART_ROUTER_ORCHESTRATOR"] == "1" and subagent_routed
)
supported = workspace.model_ids(agent)
evidence = SessionEvidence(session.home, agent)
existing_sessions = set(agent_sessions(session, agent))
decisions_path = session.home / ".ucode" / f"{agent}-smart-routing-decisions.jsonl"
original_decisions = read_jsonl(decisions_path)
task = FileTask(session)
task.prompt += " Do not delegate."
checkpoint = recorder.checkpoint()
Expand All @@ -282,6 +296,9 @@ def test_smart_router_config_version(
expected_model
)
evidence.assert_applied(task, supported, expected=expected_model)
assert (
"databricks-smart-router-recipe" in inference.headers
) == subagent_routed, agent
assert (
ORCHESTRATOR_CONTEXT.encode() in inference.body
) == orchestrator_enabled, agent
Expand All @@ -293,9 +310,6 @@ def test_smart_router_config_version(

child_task = FileTask(session)
child_task.prompt = child_task.delegate_prompt
decisions_path = (
session.home / ".ucode" / f"{agent}-smart-routing-decisions.jsonl"
)
decision_count = len(read_jsonl(decisions_path))
checkpoint = recorder.checkpoint()
existing_sessions = set(agent_sessions(session, agent))
Expand All @@ -318,31 +332,40 @@ def test_smart_router_config_version(
for answer in assistant_answers(agent, records)
), agent
decisions = read_jsonl(decisions_path)[decision_count:]
assert_subagent_routed(
session,
agent,
child_task,
decision_ids={decision["decision_id"] for decision in decisions},
)
requests = recorder.requests_after(checkpoint)
routes = [request for request in requests if request.path == ROUTING_PATH]
assert len(routes) == 1, (agent, routes)
route_prompt = routes[0].payload["task"]["prompt"]
assert child_task.filename in route_prompt
response = recorder.response_for(routes[0])
assert response.status_code == 200
selections = response.payload["route_selection"]
assert len(selections) == 1
inference = _task_inference_request(
requests,
agent,
route_prompt,
after=routes[0].sequence,
)
inference = served_inference_request(recorder, requests, inference, agent)
assert canonical_model(inference.payload["model"]) == canonical_model(
selections[0]["route_option"]["model"]
)
assert len(routes) == int(subagent_routed), (agent, routes)
if subagent_routed:
assert_subagent_routed(
session,
agent,
child_task,
decision_ids={decision["decision_id"] for decision in decisions},
)
route_prompt = routes[0].payload["task"]["prompt"]
assert child_task.filename in route_prompt
response = recorder.response_for(routes[0])
assert response.status_code == 200
selections = response.payload["route_selection"]
assert len(selections) == 1
inference = _task_inference_request(
requests,
agent,
route_prompt,
after=routes[0].sequence,
)
inference = served_inference_request(recorder, requests, inference, agent)
assert canonical_model(inference.payload["model"]) == canonical_model(
selections[0]["route_option"]["model"]
)
assert "databricks-smart-router-recipe" in inference.headers
else:
assert read_jsonl(decisions_path) == original_decisions
assert not any(
"databricks-smart-router-recipe" in request.headers
or ORCHESTRATOR_CONTEXT.encode() in request.body
for request in requests
), agent
if agent == CLAUDE:
tui.wait_for_background_tasks()
tui.exit_normally()
Expand Down
9 changes: 7 additions & 2 deletions tests/integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,15 @@ inference, and Claude inference. The config equality also accounts for the works
skill name as data; skill download and invocation are not covered.

The dedicated smart-routing CUJ in `../e2e_cuj/test_cuj4_smart_routing.py` leaves the original
managed-default and explicit-model cases unchanged. One additional test runs the five supported
managed-default and explicit-model cases unchanged. One parameterized test runs all six
`SMART_ROUTER_CONFIG_VERSION` values, checking both agents' first-prompt routing, orchestrator
context in inference input, and completed explicitly requested routed subagents.
context in inference input, and completed explicitly requested native subagents.
This is not automatic orchestrator-delegation coverage. No workspace configuration is modified.
The shared preset CUJ runs both agents for every version and derives routing and
orchestration assertions from each agent's resolved flags. Both must complete a native
child task. `subagent_orch_v0_claude_only` routes Claude's child; Codex uses the managed
default without routing requests, new router decisions, recipe headers, or orchestrator context.

Agent-specific smart-routing preset component checks cover shared base presets, explicit agent mappings,
complete per-agent validation, session precedence, and preserving the preset selector
until managed-default agent selection.
Expand Down
Loading