diff --git a/tests/README.md b/tests/README.md index dc5cf381e..b05b22ca1 100644 --- a/tests/README.md +++ b/tests/README.md @@ -138,12 +138,18 @@ tool-result confirmation after each toggle, and explicitly request their childre including while routing is off. `test_integration_evidence.py` checks native tool-result extraction for both agents, including collapsed-output records, and excludes user echoes and assistant claims. - -Dedicated regression coverage is missing for root-only Smart Router Orchestrator activation, -compaction, retained skills in ineligible sessions, -role-contract preservation, and isolation from legacy preference files. Codex's -native hook merging and project trust, automatic delegation, and execution of -pre-existing hooks remain outside the integration suite. +Terminal error checks distinguish client failures from error names quoted in reviews. + +`test_ug_orchestrator.py` requests a review of three source modules without mentioning +subagents. It requires a completed native child response and a root report containing +traceability values read from all three files. Each harness then performs native +`/compact`, receives the full workflow again for the next review, and completes it with +new child work. Codex delivers its compact hook before the next model request. +The journeys check orchestration and continuation, not review accuracy. +Retained skills in ineligible sessions, role-contract preservation, and isolation +from legacy preference files still lack dedicated regression coverage. Codex's +native hook merging and project trust, and execution of pre-existing hooks remain +outside the integration suite. The portable Windows routing test checks native executable forwarding, generated hooks/plugins, caller arguments, and cleanup without Unix imports. It does not @@ -195,6 +201,7 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi | `test_ug_codex_app_server_client_initializes` | Connect a stdio client, direct/`--` separator, routing off/on | Actual JSON-RPC initialize response; no non-JSON stdout; no routing | | `test_smart_routing_claude_route_subagent_hook`, `test_smart_routing_codex_route_subagent_hook` | Pipe a real PreToolUse spawn payload to the installed route-subagent hook with subagent-only routing enabled | Allow decision against the live router; requested model replaced by a routed agent definition (Claude) or bundled catalog slug (Codex) from the offered models; one audited decision matching the session and task | | `test_smart_router_skill_toggles_claude_subagent_routing`, `test_smart_router_skill_toggles_codex_subagent_routing` | Configure, launch a real subagent-only TUI with orchestration unset or opted in, then spawn tagged children while invoking the installed Smart Router skill to switch routing on -> off -> on in the same session | Only `smart-router` is installed by default; opt-in also installs `smart-router-orchestrator`; all three native children complete; only routing-enabled phases show the subagent banner and produce a live routing decision correlated with the child; no first-prompt routing wrapper; normal exit | +| `test_orchestrator_claude_delegates_and_recovers_after_compaction`, `test_orchestrator_codex_delegates_and_recovers_after_compaction` | Opt into orchestration, request a multi-file review without asking for subagents, compact natively, and request a different review | Both reviews have new completed child work and a root answer with the files' withheld review IDs; the full workflow is delivered before the first review and after native compaction | | `test_ug_configure_claude_repeat_and_revert`, `test_ug_configure_codex_repeat_and_revert` | Configure twice over user settings; complete a task; revert twice | Settings preserved; no bearer in ug state; generated config removed; status unconfigured | | `test_ug_configure_claude_cleans_stale_skills_mcp_on_workspace_switch` | Configure the first workspace, register its skills MCP, switch to a second real workspace, and use Claude | Old registration removed from Claude and the new workspace state; old workspace bucket preserved; repeat configure stays clean; real file task completes on the second workspace | | `test_ug_configure_claude_rejects_invalid_credentials`, `test_ug_configure_codex_rejects_invalid_credentials` | Configure with a rejected bearer against the real workspace | Authentication failure; no successful saved setup | diff --git a/tests/integration/README.md b/tests/integration/README.md index 9bf85f70b..4e74dd2e9 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -226,6 +226,7 @@ test_ug_claude_commands.py # command help forwarding test_ug_codex_commands.py # command help and parser error forwarding test_ug_codex_app_server.py # actual client/server initialize exchange test_ug_smart_routing_hooks.py # live hook contract plus skill-driven subagent toggles +test_ug_orchestrator.py # automatic delegation and native compaction continuation test_ug_configure_claude_lifecycle.py # repeat setup, revert, rejected credentials test_ug_configure_claude_workspace_switch.py # real skills MCP cleanup across two workspaces test_ug_configure_codex_lifecycle.py # repeat setup, revert, rejected credentials @@ -289,11 +290,22 @@ assistant answer after each skill invocation. Collapsed terminal output is allowed; the answer need not repeat the CLI's exact wording. Each following child still verifies whether a routing decision occurred. Their off-phase child is an explicit user-requested delegation; -these journeys do not establish automatic orchestration behavior. Root-only -activation, compaction, retained skills in ineligible sessions, -role-contract preservation, and isolation from legacy preference files -lack dedicated regression coverage. Codex's native hook merging, project trust, -and execution of pre-existing hooks are not exercised by this integration suite. +these toggle journeys do not establish automatic orchestration behavior. + +`test_ug_orchestrator.py` covers automatic delegation separately in the managed-fixture +lane. Each real harness reviews three source modules without a request to use subagents, +then performs native `/compact` and reviews three different modules. Each review needs +new completed native child work and a root report containing the exact review IDs from +all source headers; the IDs are never included in the prompt. The test also requires +the full workflow in fresh native hook context for both reviews. Codex runs its compact +hook before the next model request, so the second review checks that delivery. +It checks orchestration and continuation, not the accuracy of review findings. +Run these two journeys with `-- -m managed_fixture -k test_orchestrator_`. + +Retained skills in ineligible sessions, role-contract preservation, and isolation +from legacy preference files still lack dedicated regression coverage. Codex's native +hook merging, project trust, and execution of pre-existing hooks are not exercised by +this integration suite. The portable `../test_claude_windows_smart_routing.py` checks the Windows subagent-only fallback without Unix imports. Native Windows TUI and hook execution @@ -755,7 +767,8 @@ onboarding state or relax the prompt/task assertions. Test homes are deleted aft each case; redacted diagnostics remain. For manual interaction, configure a fresh home with the same installed binaries and recorded public CLI arguments. Definitive API errors and the client's exhausted retry limit fail the TUI wait -immediately with the actual screen. A transient 429/503 while the client is still +immediately with the actual screen. Only displayed client error lines count; +error names quoted in review prose do not. A transient 429/503 while the client is still retrying is not treated as terminal; the suite adds no task retries of its own. ## Colima / Docker diff --git a/tests/integration/test_ug_orchestrator.py b/tests/integration/test_ug_orchestrator.py new file mode 100644 index 000000000..fea760779 --- /dev/null +++ b/tests/integration/test_ug_orchestrator.py @@ -0,0 +1,195 @@ +"""Real automatic delegation and continuation after native context compaction.""" + +import uuid +from pathlib import Path + +import pytest +from utils.evidence import ( + agent_sessions, + assert_no_terminal_api_error, + completed_answers, + completed_child_answers, + is_child_session, + orchestrator_contexts, +) +from utils.managed import use_managed_config_fixture +from utils.terminal import AgentTerminal + + +class ReviewTask: + """Substantive source review whose traceability values are absent from the prompt.""" + + def __init__(self, session, phase, sources): + directory = session.cwd / phase + directory.mkdir() + self.phase = phase + self.values = [] + source_root = Path(__file__).resolve().parents[2] / "src" / "ucode" + for source in sources: + value = uuid.uuid4().hex + self.values.append(value) + text = f"# REVIEW_ID: {value}\n" + (source_root / source).read_text() + (directory / Path(source).name).write_text(text) + self.prompt = ( + f"Review all three Python modules in {phase}/ for correctness, error handling, " + "and preservation of user data. Read each module and report concrete findings " + "with file references, or explain why you found none. These are standalone " + "review copies; dependencies outside this directory are not part of the review. " + "Do not modify files or execute the source. Include each module's exact " + "REVIEW_ID from its header in your final report so the review is traceable." + ) + + def completed(self, session, agent): + return any( + all(value in answer for value in self.values) + for answer in completed_answers(agent, _root_records(session, agent)) + ) + + +def _root_records(session, agent): + return [ + row + for path, records in agent_sessions(session, agent).items() + if not is_child_session(agent, path, records) + for row in records + ] + + +def _review(tui, session, agent, task): + before_children = completed_child_answers(session, agent) + before_records = len(_root_records(session, agent)) + tui.submit(task.prompt) + + def finished(screen): + assert_no_terminal_api_error(screen) + return task.completed(session, agent) + + tui.wait_for(finished, "a completed review of all three modules", timeout=360) + children = completed_child_answers(session, agent) + new_answers = { + path: answers[len(before_children.get(path, [])) :] for path, answers in children.items() + } + session.record( + f"{task.phase}-completion.json", + {"review_ids": task.values, "new_completed_child_answers": new_answers}, + ) + assert any(answer.strip() for answers in new_answers.values() for answer in answers), ( + "The root completed the review without a completed native child review" + ) + assert orchestrator_contexts(agent, _root_records(session, agent)[before_records:]), ( + "The native root transcript did not receive a fresh orchestrator workflow" + ) + + +def _compact(tui, session, agent): + before = len(_root_records(session, agent)) + tui.submit("/compact") + + def reloaded(screen): + assert_no_terminal_api_error(screen) + records = _root_records(session, agent)[before:] + if agent == "claude": + compacted = any(row.get("subtype") == "compact_boundary" for row in records) + contexts = [ + row + for row in records + if row.get("attachment", {}).get("hookEvent") == "SessionStart" + ] + else: + compacted = any( + row.get("type") == "compacted" + or ( + row.get("type") == "event_msg" + and row.get("payload", {}).get("type") == "context_compacted" + ) + for row in records + ) + # Codex runs the compact hook before the next model request. + # The follow-up review requires its newly delivered workflow. + return compacted + return compacted and bool(orchestrator_contexts(agent, contexts)) + + tui.wait_for(reloaded, "native compaction", timeout=180) + session.record("compaction-records.json", _root_records(session, agent)[before:]) + + +@pytest.mark.claude +@pytest.mark.managed_fixture +def test_orchestrator_claude_delegates_and_recovers_after_compaction( + live_session, workspace, tmp_path +): + """Scenario: opt into orchestration and ask Claude to review three source modules, + without requesting subagents; compact natively, then review three different modules. + + Expected: the prompt and compaction hooks deliver the complete workflow; both reviews + have a completed native child and a root report containing values read from all files. + This checks delegation and continuation, not the accuracy of the review's findings. + """ + session = live_session + session.env.update( + ENABLE_SMART_ROUTING_V2="1", + ENABLE_SMART_ROUTING_SUBAGENT_ONLY="1", + ENABLE_SMART_ROUTER_ORCHESTRATOR="1", + TMPDIR=str(tmp_path), + ) + use_managed_config_fixture(session, "claude_smart_routing") + session.run( + "configure", "--workspace", workspace, "--skip-upgrade", "--disable-databricks-ai-tools" + ) + first = ReviewTask( + session, "initial", ["config_io.py", "gateway_proxy.py", "managed_config.py"] + ) + with AgentTerminal( + session, "claude", [str(session.binary), "claude"], "automatic-orchestration" + ) as tui: + tui.boot() + _review(tui, session, "claude", first) + _compact(tui, session, "claude") + second = ReviewTask( + session, + "followup", + ["skills_download.py", "codex_config.py", "smart_routing/session_env.py"], + ) + _review(tui, session, "claude", second) + tui.exit_normally() + + +@pytest.mark.codex +@pytest.mark.managed_fixture +def test_orchestrator_codex_delegates_and_recovers_after_compaction( + live_session, workspace, tmp_path +): + """Scenario: opt into orchestration and ask Codex to review three source modules, + without requesting subagents; compact natively, then review three different modules. + + Expected: the prompt and compaction hooks deliver the complete workflow; both reviews + have a completed native child and a root report containing values read from all files. + Parent history copied into a child is excluded from child-completion evidence. + """ + session = live_session + session.env.update( + ENABLE_SMART_ROUTING_V2="1", + ENABLE_SMART_ROUTING_SUBAGENT_ONLY="1", + ENABLE_SMART_ROUTER_ORCHESTRATOR="1", + TMPDIR=str(tmp_path), + ) + use_managed_config_fixture(session, "codex_smart_routing") + session.run( + "configure", "--workspace", workspace, "--skip-upgrade", "--disable-databricks-ai-tools" + ) + first = ReviewTask( + session, "initial", ["config_io.py", "gateway_proxy.py", "managed_config.py"] + ) + with AgentTerminal( + session, "codex", [str(session.binary), "codex"], "automatic-orchestration" + ) as tui: + tui.boot() + _review(tui, session, "codex", first) + _compact(tui, session, "codex") + second = ReviewTask( + session, + "followup", + ["skills_download.py", "codex_config.py", "smart_routing/session_env.py"], + ) + _review(tui, session, "codex", second) + tui.exit_normally() diff --git a/tests/integration/utils/evidence.py b/tests/integration/utils/evidence.py index d3f77a893..15e628215 100644 --- a/tests/integration/utils/evidence.py +++ b/tests/integration/utils/evidence.py @@ -11,12 +11,13 @@ def assert_no_terminal_api_error(screen: str) -> None: - """Fail on definitive client errors, not an in-progress transient retry.""" + """Recognize client error lines without mistaking review prose for an API failure.""" error = re.search( - r"unexpected status (?:400|401|403|404|405|409|422)\b|PERMISSION_DENIED" - r"|exceeded retry limit", + r"^[ \t]*(?:[■⎿][ \t]*)?" + r"(?:(?:unexpected status|API Error:)[ \t]*(?:400|401|403|404|405|409|422)\b" + r"|exceeded retry limit\b)", screen, - re.IGNORECASE, + re.IGNORECASE | re.MULTILINE, ) assert error is None, "Agent returned a terminal API error:\n" + screen @@ -52,6 +53,58 @@ def assistant_answers(agent: str, records: list[dict]) -> list[str]: return helper.assistant_answers(records) if helper is not None else [] +def completed_answers(agent: str, records: list[dict]) -> list[str]: + """Exclude Claude's intermediate commentary from task-completion evidence.""" + if agent == "claude": + records = [ + row for row in records if row.get("message", {}).get("stop_reason") == "end_turn" + ] + return assistant_answers(agent, records) + + +def completed_child_answers(session, agent: str) -> dict[str, list[str]]: + """Do not count parent turns copied into a Codex child's rollout as child work.""" + sessions = agent_sessions(session, agent) + parent_turns = { + row["payload"]["turn_id"] + for path, records in sessions.items() + if not is_child_session(agent, path, records) + for row in records + if row.get("type") == "turn_context" and row.get("payload", {}).get("turn_id") + } + return { + path: completed_answers( + agent, + [row for row in records if row.get("payload", {}).get("turn_id") not in parent_turns], + ) + for path, records in sessions.items() + if is_child_session(agent, path, records) + } + + +def orchestrator_contexts(agent: str, records: list[dict]) -> list[str]: + """Read delivered hook context, excluding user text and assistant claims.""" + contexts = [] + for row in records: + if agent == "claude": + attachment = row.get("attachment", {}) + if attachment.get("type") == "hook_additional_context": + contexts.extend(attachment.get("content", [])) + elif row.get("type") == "response_item": + payload = row.get("payload", {}) + if payload.get("type") == "message" and payload.get("role") == "developer": + contexts.extend(part.get("text", "") for part in payload.get("content", [])) + return [ + text + for text in contexts + if isinstance(text, str) + and text.startswith("Smart Router Orchestrator is on for this session.") + and "## Workflow" in text + and "### Claude Code adapter" in text + and "### Codex adapter" in text + ] + + def tool_outputs(agent: str, records: list[dict]) -> list[str]: """Read native tool results even when their terminal output is collapsed.""" contents = [] diff --git a/tests/test_integration_evidence.py b/tests/test_integration_evidence.py index cf04efc76..6d67b55c1 100644 --- a/tests/test_integration_evidence.py +++ b/tests/test_integration_evidence.py @@ -10,6 +10,9 @@ SubagentCalculation, assert_no_terminal_api_error, assistant_answer_contains, + completed_answers, + completed_child_answers, + orchestrator_contexts, tool_outputs, ) @@ -65,6 +68,8 @@ def _write_answer(home, agent, *, child, value): "■ exceeded retry limit, last status: 429 Too Many Requests", "■ unexpected status 403 Forbidden: PERMISSION_DENIED", "■ unexpected status 401 Unauthorized", + ' ⎿ API Error: 403 {"error_code": "PERMISSION_DENIED"}', + 'Review complete.\n\n ⎿ API Error: 403\n{"error_code": "PERMISSION_DENIED"}', ], ) def test_terminal_api_failure_reports_the_actual_error(screen): @@ -85,6 +90,18 @@ def test_transient_retries_and_running_tasks_are_not_terminal_errors(screen): assert_no_terminal_api_error(screen) +@pytest.mark.parametrize( + "screen", + [ + " 3. PERMISSION_DENIED silently swallowed when there's no cached fallback.\n❯", + "The code raises 'unexpected status 403' when access is denied.", + "The handler returns 'exceeded retry limit' after repeated errors.", + ], +) +def test_error_mentions_in_review_text_are_not_terminal_errors(screen): + assert_no_terminal_api_error(screen) + + @pytest.mark.parametrize("agent", ["claude", "codex"]) def test_tagged_calculation_requires_the_native_child_answer(tmp_path, agent): session = _Session(tmp_path) @@ -103,6 +120,88 @@ def test_tagged_calculation_requires_the_native_child_answer(tmp_path, agent): assert "1+1" in task.prompt +def test_completed_answers_excludes_claude_intermediate_commentary(): + records = [ + { + "type": "assistant", + "message": { + "role": "assistant", + "stop_reason": reason, + "content": [{"type": "text", "text": answer}], + }, + } + for reason, answer in [("tool_use", "I will inspect the module"), ("end_turn", "Reviewed")] + ] + assert completed_answers("claude", records) == ["Reviewed"] + + +def test_completed_child_answers_excludes_inherited_codex_parent_work(tmp_path): + parent = [ + {"type": "turn_context", "payload": {"turn_id": "parent-turn"}}, + { + "type": "event_msg", + "payload": { + "type": "task_complete", + "turn_id": "parent-turn", + "last_agent_message": "Parent answer", + }, + }, + ] + child = [ + {"type": "session_meta", "payload": {"source": {"subagent": "spawn"}}}, + *parent, + { + "type": "event_msg", + "payload": { + "type": "task_complete", + "turn_id": "child-turn", + "last_agent_message": "Child answer", + }, + }, + ] + session = _transcript_session(tmp_path, "codex", {"parent.jsonl": parent, "child.jsonl": child}) + assert completed_child_answers(session, "codex") == {"child.jsonl": ["Child answer"]} + + +@pytest.mark.parametrize("agent", ["claude", "codex"]) +def test_orchestrator_context_requires_native_delivery_not_echoed_text(agent): + context = ( + "Smart Router Orchestrator is on for this session.\n" + "## Workflow\n### Claude Code adapter\n### Codex adapter" + ) + records = [ + {"type": "user", "message": {"content": context}}, + { + "type": "response_item", + "payload": { + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": context}], + }, + }, + ] + assert orchestrator_contexts(agent, records) == [] + if agent == "claude": + records.append( + { + "type": "attachment", + "attachment": {"type": "hook_additional_context", "content": [context]}, + } + ) + else: + records.append( + { + "type": "response_item", + "payload": { + "type": "message", + "role": "developer", + "content": [{"type": "input_text", "text": context}], + }, + } + ) + assert orchestrator_contexts(agent, records) == [context] + + def test_codex_model_identity_uses_only_the_completed_answer_turn(): records = [ {