Verify automatic orchestration and compaction in CI

tests/README.md

@@ -138,12 +138,18 @@ tool-result confirmation after each toggle, and explicitly request their childre
 including while routing is off.
 `test_integration_evidence.py` checks native tool-result extraction for both agents,
 including collapsed-output records, and excludes user echoes and assistant claims.
-
-Dedicated regression coverage is missing for root-only Smart Router Orchestrator activation,
-compaction, retained skills in ineligible sessions,
-role-contract preservation, and isolation from legacy preference files. Codex's
-native hook merging and project trust, automatic delegation, and execution of
-pre-existing hooks remain outside the integration suite.
+Terminal error checks distinguish client failures from error names quoted in reviews.
+
+`test_ug_orchestrator.py` requests a review of three source modules without mentioning
+subagents. It requires a completed native child response and a root report containing
+traceability values read from all three files. Each harness then performs native
+`/compact`, receives the full workflow again for the next review, and completes it with
+new child work. Codex delivers its compact hook before the next model request.
+The journeys check orchestration and continuation, not review accuracy.
+Retained skills in ineligible sessions, role-contract preservation, and isolation
+from legacy preference files still lack dedicated regression coverage. Codex's
+native hook merging and project trust, and execution of pre-existing hooks remain
+outside the integration suite.
 
 The portable Windows routing test checks native executable forwarding, generated
 hooks/plugins, caller arguments, and cleanup without Unix imports. It does not
@@ -195,6 +201,7 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi
 | `test_ug_codex_app_server_client_initializes` | Connect a stdio client, direct/`--` separator, routing off/on | Actual JSON-RPC initialize response; no non-JSON stdout; no routing |
 | `test_smart_routing_claude_route_subagent_hook`, `test_smart_routing_codex_route_subagent_hook` | Pipe a real PreToolUse spawn payload to the installed route-subagent hook with subagent-only routing enabled | Allow decision against the live router; requested model replaced by a routed agent definition (Claude) or bundled catalog slug (Codex) from the offered models; one audited decision matching the session and task |
 | `test_smart_router_skill_toggles_claude_subagent_routing`, `test_smart_router_skill_toggles_codex_subagent_routing` | Configure, launch a real subagent-only TUI with orchestration unset or opted in, then spawn tagged children while invoking the installed Smart Router skill to switch routing on -> off -> on in the same session | Only `smart-router` is installed by default; opt-in also installs `smart-router-orchestrator`; all three native children complete; only routing-enabled phases show the subagent banner and produce a live routing decision correlated with the child; no first-prompt routing wrapper; normal exit |
+| `test_orchestrator_claude_delegates_and_recovers_after_compaction`, `test_orchestrator_codex_delegates_and_recovers_after_compaction` | Opt into orchestration, request a multi-file review without asking for subagents, compact natively, and request a different review | Both reviews have new completed child work and a root answer with the files' withheld review IDs; the full workflow is delivered before the first review and after native compaction |
 | `test_ug_configure_claude_repeat_and_revert`, `test_ug_configure_codex_repeat_and_revert` | Configure twice over user settings; complete a task; revert twice | Settings preserved; no bearer in ug state; generated config removed; status unconfigured |
 | `test_ug_configure_claude_cleans_stale_skills_mcp_on_workspace_switch` | Configure the first workspace, register its skills MCP, switch to a second real workspace, and use Claude | Old registration removed from Claude and the new workspace state; old workspace bucket preserved; repeat configure stays clean; real file task completes on the second workspace |
 | `test_ug_configure_claude_rejects_invalid_credentials`, `test_ug_configure_codex_rejects_invalid_credentials` | Configure with a rejected bearer against the real workspace | Authentication failure; no successful saved setup |

tests/integration/README.md

@@ -226,6 +226,7 @@ test_ug_claude_commands.py              # command help forwarding
 test_ug_codex_commands.py               # command help and parser error forwarding
 test_ug_codex_app_server.py             # actual client/server initialize exchange
 test_ug_smart_routing_hooks.py           # live hook contract plus skill-driven subagent toggles
+test_ug_orchestrator.py                 # automatic delegation and native compaction continuation
 test_ug_configure_claude_lifecycle.py   # repeat setup, revert, rejected credentials
 test_ug_configure_claude_workspace_switch.py # real skills MCP cleanup across two workspaces
 test_ug_configure_codex_lifecycle.py    # repeat setup, revert, rejected credentials
@@ -289,11 +290,22 @@ assistant answer after each skill invocation.
 Collapsed terminal output is allowed; the answer need not repeat the CLI's exact wording.
 Each following child still verifies whether a routing decision occurred.
 Their off-phase child is an explicit user-requested delegation;
-these journeys do not establish automatic orchestration behavior. Root-only
-activation, compaction, retained skills in ineligible sessions,
-role-contract preservation, and isolation from legacy preference files
-lack dedicated regression coverage. Codex's native hook merging, project trust,
-and execution of pre-existing hooks are not exercised by this integration suite.
+these toggle journeys do not establish automatic orchestration behavior.
+
+`test_ug_orchestrator.py` covers automatic delegation separately in the managed-fixture
+lane. Each real harness reviews three source modules without a request to use subagents,
+then performs native `/compact` and reviews three different modules. Each review needs
+new completed native child work and a root report containing the exact review IDs from
+all source headers; the IDs are never included in the prompt. The test also requires
+the full workflow in fresh native hook context for both reviews. Codex runs its compact
+hook before the next model request, so the second review checks that delivery.
+It checks orchestration and continuation, not the accuracy of review findings.
+Run these two journeys with `-- -m managed_fixture -k test_orchestrator_`.
+
+Retained skills in ineligible sessions, role-contract preservation, and isolation
+from legacy preference files still lack dedicated regression coverage. Codex's native
+hook merging, project trust, and execution of pre-existing hooks are not exercised by
+this integration suite.
 
 The portable `../test_claude_windows_smart_routing.py` checks the Windows
 subagent-only fallback without Unix imports. Native Windows TUI and hook execution
@@ -755,7 +767,8 @@ onboarding state or relax the prompt/task assertions. Test homes are deleted aft
 each case; redacted diagnostics remain. For manual interaction, configure a fresh
 home with the same installed binaries and recorded public CLI arguments.
 Definitive API errors and the client's exhausted retry limit fail the TUI wait
-immediately with the actual screen. A transient 429/503 while the client is still
+immediately with the actual screen. Only displayed client error lines count;
+error names quoted in review prose do not. A transient 429/503 while the client is still
 retrying is not treated as terminal; the suite adds no task retries of its own.
 
 ## Colima / Docker

tests/integration/test_ug_orchestrator.py

@@ -0,0 +1,195 @@
+"""Real automatic delegation and continuation after native context compaction."""
+
+import uuid
+from pathlib import Path
+
+import pytest
+from utils.evidence import (
+    agent_sessions,
+    assert_no_terminal_api_error,
+    completed_answers,
+    completed_child_answers,
+    is_child_session,
+    orchestrator_contexts,
+)
+from utils.managed import use_managed_config_fixture
+from utils.terminal import AgentTerminal
+
+
+class ReviewTask:
+    """Substantive source review whose traceability values are absent from the prompt."""
+
+    def __init__(self, session, phase, sources):
+        directory = session.cwd / phase
+        directory.mkdir()
+        self.phase = phase
+        self.values = []
+        source_root = Path(__file__).resolve().parents[2] / "src" / "ucode"
+        for source in sources:
+            value = uuid.uuid4().hex
+            self.values.append(value)
+            text = f"# REVIEW_ID: {value}\n" + (source_root / source).read_text()
+            (directory / Path(source).name).write_text(text)
+        self.prompt = (
+            f"Review all three Python modules in {phase}/ for correctness, error handling, "
+            "and preservation of user data. Read each module and report concrete findings "
+            "with file references, or explain why you found none. These are standalone "
+            "review copies; dependencies outside this directory are not part of the review. "
+            "Do not modify files or execute the source. Include each module's exact "
+            "REVIEW_ID from its header in your final report so the review is traceable."
+        )
+
+    def completed(self, session, agent):
+        return any(
+            all(value in answer for value in self.values)
+            for answer in completed_answers(agent, _root_records(session, agent))
+        )
+
+
+def _root_records(session, agent):
+    return [
+        row
+        for path, records in agent_sessions(session, agent).items()
+        if not is_child_session(agent, path, records)
+        for row in records
+    ]
+
+
+def _review(tui, session, agent, task):
+    before_children = completed_child_answers(session, agent)
+    before_records = len(_root_records(session, agent))
+    tui.submit(task.prompt)
+
+    def finished(screen):
+        assert_no_terminal_api_error(screen)
+        return task.completed(session, agent)
+
+    tui.wait_for(finished, "a completed review of all three modules", timeout=360)
+    children = completed_child_answers(session, agent)
+    new_answers = {
+        path: answers[len(before_children.get(path, [])) :] for path, answers in children.items()
+    }
+    session.record(
+        f"{task.phase}-completion.json",
+        {"review_ids": task.values, "new_completed_child_answers": new_answers},
+    )
+    assert any(answer.strip() for answers in new_answers.values() for answer in answers), (
+        "The root completed the review without a completed native child review"
+    )
+    assert orchestrator_contexts(agent, _root_records(session, agent)[before_records:]), (
+        "The native root transcript did not receive a fresh orchestrator workflow"
+    )
+
+
+def _compact(tui, session, agent):
+    before = len(_root_records(session, agent))
+    tui.submit("/compact")
+
+    def reloaded(screen):
+        assert_no_terminal_api_error(screen)
+        records = _root_records(session, agent)[before:]
+        if agent == "claude":
+            compacted = any(row.get("subtype") == "compact_boundary" for row in records)
+            contexts = [
+                row
+                for row in records
+                if row.get("attachment", {}).get("hookEvent") == "SessionStart"
+            ]
+        else:
+            compacted = any(
+                row.get("type") == "compacted"
+                or (
+                    row.get("type") == "event_msg"
+                    and row.get("payload", {}).get("type") == "context_compacted"
+                )
+                for row in records
+            )
+            # Codex runs the compact hook before the next model request.
+            # The follow-up review requires its newly delivered workflow.
+            return compacted
+        return compacted and bool(orchestrator_contexts(agent, contexts))
+
+    tui.wait_for(reloaded, "native compaction", timeout=180)
+    session.record("compaction-records.json", _root_records(session, agent)[before:])
+
+
+@pytest.mark.claude
+@pytest.mark.managed_fixture
+def test_orchestrator_claude_delegates_and_recovers_after_compaction(
+    live_session, workspace, tmp_path
+):
+    """Scenario: opt into orchestration and ask Claude to review three source modules,
+    without requesting subagents; compact natively, then review three different modules.
+
+    Expected: the prompt and compaction hooks deliver the complete workflow; both reviews
+    have a completed native child and a root report containing values read from all files.
+    This checks delegation and continuation, not the accuracy of the review's findings.
+    """
+    session = live_session
+    session.env.update(
+        ENABLE_SMART_ROUTING_V2="1",
+        ENABLE_SMART_ROUTING_SUBAGENT_ONLY="1",
+        ENABLE_SMART_ROUTER_ORCHESTRATOR="1",
+        TMPDIR=str(tmp_path),
+    )
+    use_managed_config_fixture(session, "claude_smart_routing")
+    session.run(
+        "configure", "--workspace", workspace, "--skip-upgrade", "--disable-databricks-ai-tools"
+    )
+    first = ReviewTask(
+        session, "initial", ["config_io.py", "gateway_proxy.py", "managed_config.py"]
+    )
+    with AgentTerminal(
+        session, "claude", [str(session.binary), "claude"], "automatic-orchestration"
+    ) as tui:
+        tui.boot()
+        _review(tui, session, "claude", first)
+        _compact(tui, session, "claude")
+        second = ReviewTask(
+            session,
+            "followup",
+            ["skills_download.py", "codex_config.py", "smart_routing/session_env.py"],
+        )
+        _review(tui, session, "claude", second)
+        tui.exit_normally()
+
+
+@pytest.mark.codex
+@pytest.mark.managed_fixture
+def test_orchestrator_codex_delegates_and_recovers_after_compaction(
+    live_session, workspace, tmp_path
+):
+    """Scenario: opt into orchestration and ask Codex to review three source modules,
+    without requesting subagents; compact natively, then review three different modules.
+
+    Expected: the prompt and compaction hooks deliver the complete workflow; both reviews
+    have a completed native child and a root report containing values read from all files.
+    Parent history copied into a child is excluded from child-completion evidence.
+    """
+    session = live_session
+    session.env.update(
+        ENABLE_SMART_ROUTING_V2="1",
+        ENABLE_SMART_ROUTING_SUBAGENT_ONLY="1",
+        ENABLE_SMART_ROUTER_ORCHESTRATOR="1",
+        TMPDIR=str(tmp_path),
+    )
+    use_managed_config_fixture(session, "codex_smart_routing")
+    session.run(
+        "configure", "--workspace", workspace, "--skip-upgrade", "--disable-databricks-ai-tools"
+    )
+    first = ReviewTask(
+        session, "initial", ["config_io.py", "gateway_proxy.py", "managed_config.py"]
+    )
+    with AgentTerminal(
+        session, "codex", [str(session.binary), "codex"], "automatic-orchestration"
+    ) as tui:
+        tui.boot()
+        _review(tui, session, "codex", first)
+        _compact(tui, session, "codex")
+        second = ReviewTask(
+            session,
+            "followup",
+            ["skills_download.py", "codex_config.py", "smart_routing/session_env.py"],
+        )
+        _review(tui, session, "codex", second)
+        tui.exit_normally()

tests/integration/utils/evidence.py

@@ -11,12 +11,13 @@
 
 
 def assert_no_terminal_api_error(screen: str) -> None:
-    """Fail on definitive client errors, not an in-progress transient retry."""
+    """Recognize client error lines without mistaking review prose for an API failure."""
     error = re.search(
-        r"unexpected status (?:400|401|403|404|405|409|422)\b|PERMISSION_DENIED"
-        r"|exceeded retry limit",
+        r"^[ \t]*(?:[■⎿][ \t]*)?"
+        r"(?:(?:unexpected status|API Error:)[ \t]*(?:400|401|403|404|405|409|422)\b"
+        r"|exceeded retry limit\b)",
         screen,
-        re.IGNORECASE,
+        re.IGNORECASE | re.MULTILINE,
     )
     assert error is None, "Agent returned a terminal API error:\n" + screen
 
@@ -52,6 +53,58 @@ def assistant_answers(agent: str, records: list[dict]) -> list[str]:
     return helper.assistant_answers(records) if helper is not None else []
 
 
+def completed_answers(agent: str, records: list[dict]) -> list[str]:
+    """Exclude Claude's intermediate commentary from task-completion evidence."""
+    if agent == "claude":
+        records = [
+            row for row in records if row.get("message", {}).get("stop_reason") == "end_turn"
+        ]
+    return assistant_answers(agent, records)
+
+
+def completed_child_answers(session, agent: str) -> dict[str, list[str]]:
+    """Do not count parent turns copied into a Codex child's rollout as child work."""
+    sessions = agent_sessions(session, agent)
+    parent_turns = {
+        row["payload"]["turn_id"]
+        for path, records in sessions.items()
+        if not is_child_session(agent, path, records)
+        for row in records
+        if row.get("type") == "turn_context" and row.get("payload", {}).get("turn_id")
+    }
+    return {
+        path: completed_answers(
+            agent,
+            [row for row in records if row.get("payload", {}).get("turn_id") not in parent_turns],
+        )
+        for path, records in sessions.items()
+        if is_child_session(agent, path, records)
+    }
+
+
+def orchestrator_contexts(agent: str, records: list[dict]) -> list[str]:
+    """Read delivered hook context, excluding user text and assistant claims."""
+    contexts = []
+    for row in records:
+        if agent == "claude":
+            attachment = row.get("attachment", {})
+            if attachment.get("type") == "hook_additional_context":
+                contexts.extend(attachment.get("content", []))
+        elif row.get("type") == "response_item":
+            payload = row.get("payload", {})
+            if payload.get("type") == "message" and payload.get("role") == "developer":
+                contexts.extend(part.get("text", "") for part in payload.get("content", []))
+    return [
+        text
+        for text in contexts
+        if isinstance(text, str)
+        and text.startswith("Smart Router Orchestrator is on for this session.")
+        and "## Workflow" in text
+        and "### Claude Code adapter" in text
+        and "### Codex adapter" in text
+    ]
+
+
 def tool_outputs(agent: str, records: list[dict]) -> list[str]:
     """Read native tool results even when their terminal output is collapsed."""
     contents = []