diff --git a/tests/README.md b/tests/README.md index c4450b54d..6ccf14346 100644 --- a/tests/README.md +++ b/tests/README.md @@ -151,7 +151,7 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi | Test | User action | Expected evidence | | --- | --- | --- | -| `test_cuj_configuration`, `test_cuj_codex_inference`, `test_cuj_claude_inference` in `test_cuj2_mps_mcp.py` | Read the permanently preconfigured two-agent MPS/MCP config; run separate configure, Codex TUI, and Claude TUI cases | Configuration verifies both native MPS APIs, generated settings, and sandbox inclusion with web_search excluded; each TUI case records its exact MPS target header and model, paired HTTP 200 response, and final marker. Live sandbox execution is deferred because MAS cannot downscope the CI service principal. No workspace config CRUD is performed | +| `test_cuj_configuration`, `test_cuj_codex_inference`, `test_cuj_claude_inference` in `test_cuj2_mps_mcp.py` | Read the permanently preconfigured two-agent MPS/MCP config; run separate configure, Codex TUI, and Claude TUI cases | Configuration verifies both native MPS APIs, generated settings, and the `ug_e2e.tools` MCP selector with `read_fixture`; each TUI case records its connected native MCP inventory, calls `read_fixture` with a fresh nonce, checks the SHA-256 receipt in the parent transcript, and records the exact MPS target header/model with a paired HTTP 200 response. No workspace config CRUD is performed | | `test_ug_configure_claude_databricks` | Configure Databricks Hosted; execute the generated auth helper; launch plain `ug claude`, read a file, and open `/model` | Generated helper invokes `ug` with clean token stdout; assistant returns an unpredictable file value; native discovery caches `system.ai` models and the picker shows a discovered model without an opt-in flag; normal exit; reopen with working keyboard input | | `test_ug_configure_claude_anthropic_mps` | Select Anthropic MPS in the real configure picker; launch Claude | Saved provider in status; completed TUI file task; normal exit | | `test_ug_configure_codex_databricks` | Configure Databricks Hosted; execute the generated auth helper; open Codex TUI and read a file | Generated helper invokes `ug` with clean token stdout; completed assistant answer contains the file value; normal exit and reopen | @@ -227,13 +227,15 @@ Managed discovery expectations come from separate read-only, provider-scoped model-list requests; they do not rely solely on ug's generated catalog. The dedicated-workspace CUJ suite adds three independently runnable CUJ2 MPS+MCP cases: -one configuration case and separate Codex and Claude TUI task cases. The workspace is +one configuration case and separate Codex and Claude TUI MCP-task cases. The workspace is preconfigured for CUJ2 and is read-only; every case validates it, and the fixture compares it again during teardown. The TUI cases forward agent requests through the shared per-test loopback recorder to the real workspace and assert the provider-service header, model, -paired HTTP 200 response, and final answer. The configuration case verifies sandbox MCP -selection and generated client listings; live sandbox execution remains deferred because -MAS cannot downscope the CI service principal. Failed TUI runs +paired HTTP 200 response, and final answer. The configuration case verifies the +`ug_e2e.tools` MCP selector and generated `fixture_reader` listings. Each TUI case opens +the native MCP inventory, requires the connected `read_fixture` tool, calls it with a fresh +nonce, and checks the nonce-derived receipt and actual tool-call record in the parent transcript. +Failed TUI runs retain terminal output/actions/screen and best-effort agent transcript snapshots under the runner artifact directory before the temporary homes are deleted. The suite requires `UG_CUJ_SP_CLIENT_ID` and `UG_CUJ_SP_CLIENT_SECRET`, with `ug`, `claude`, `codex`, and @@ -322,7 +324,7 @@ These are unit/component checks; they do not establish live sudo password-prompt | Scenario | Status / requirement | | --- | --- | -| Broad live MCP functionality | CUJ2 covers sandbox MCP configuration and generated client listings. Live sandbox execution is deferred because MAS cannot downscope the CI service principal. Installation tests cover the local web-search MCP handshake and tool listing, not upstream proxying or a real search request. Other MCP services, live parent/child search, and permission decisions remain deferred | +| Broad live MCP functionality | CUJ2 covers the preconfigured `ug_e2e.tools` MCP schema, native inventory connection/tool listing, and real `read_fixture` calls with nonce-derived parent-transcript receipts. Installation tests cover the local web-search MCP handshake and tool listing, not upstream proxying or a real search request. Other MCP services, live parent/child search, and permission decisions remain deferred | | Dedicated CUJ Skill coverage | Deferred because of a backend storage-path issue; CUJ2 has no Skill configuration, download, invocation, or assertion path | | Broad configure flags, multiple workspaces, and PAT flows | Deferred while focusing on basic CUJs | | Workspace-switch MCP cleanup | The `workspace_switch` CUJ covers real registration, cleanup, repeat configure, and a completed Claude task. Unit/component tests cover duplicate attempts and injected removal failures; the CUJ does not force an agent timeout. It runs in the required managed CI lane for full/live runs. | diff --git a/tests/e2e_cuj/AGENTS.md b/tests/e2e_cuj/AGENTS.md index 6961d9093..097ae065b 100644 --- a/tests/e2e_cuj/AGENTS.md +++ b/tests/e2e_cuj/AGENTS.md @@ -47,13 +47,14 @@ Model evidence combines observed gateway requests/responses with native complete records; neither routing banners nor native records alone prove an applied decision. `test_cuj2_mps_mcp.py` runs three independently collected cases against a permanently -preconfigured workspace: configuration, Codex inference, and Claude inference. Each case -reads and validates the exact two-agent MPS/MCP shape. No case creates, updates, deletes, +preconfigured workspace: configuration, Codex MCP tool use, and Claude MCP tool use. Each +case reads and validates the exact two-agent MPS/MCP shape. No case creates, updates, deletes, or restores CodingAgentConfig. The fixture's final read-only equality check is the only workspace mutation guard. The test uses the shared `UserSession`, `Workspace`, and -`TuiRequestRecorder`, `AgentTerminal`, and integration transcript readers directly. The -configuration case verifies the selected sandbox service and generated MCP listings; live -`run_code` execution remains deferred because MAS cannot downscope the CI service principal. +`TuiRequestRecorder`, `AgentTerminal`, MCP inventory helper, and integration transcript readers +directly. The configuration case verifies the `ug_e2e.tools` selector and generated +`fixture_reader` listings; each TUI case verifies the connected native inventory, calls +`read_fixture` with a fresh nonce, and checks its SHA-256 receipt in the parent transcript. Use a clean disposable POSIX runner without existing machine-wide agent settings. Install the intended versions of `ug`, `claude`, `codex`, and `databricks` on PATH. diff --git a/tests/e2e_cuj/helpers/constants.py b/tests/e2e_cuj/helpers/constants.py index 3a934d550..d226c3765 100644 --- a/tests/e2e_cuj/helpers/constants.py +++ b/tests/e2e_cuj/helpers/constants.py @@ -34,3 +34,6 @@ class CodingAgent(StrEnum): SANDBOX_MCP_SERVICE_NAME = "system.ai.sandbox" WEB_SEARCH_MCP_SERVICE_NAME = "system.ai.web_search" +MCP_FIXTURE_LOCATION = "ug_e2e.tools" +MCP_FIXTURE_SERVICE_NAME = "ug_e2e.tools.fixture_reader" +MCP_FIXTURE_TOOL_NAME = "read_fixture" diff --git a/tests/e2e_cuj/test_cuj2_mps_mcp.py b/tests/e2e_cuj/test_cuj2_mps_mcp.py index 0af6311df..b9099cd50 100644 --- a/tests/e2e_cuj/test_cuj2_mps_mcp.py +++ b/tests/e2e_cuj/test_cuj2_mps_mcp.py @@ -2,6 +2,7 @@ from __future__ import annotations +import hashlib import json import shutil import tomllib @@ -15,6 +16,7 @@ assistant_answers, is_child_session, ) +from tests.integration.utils.mcp import open_claude_mcp_inventory, open_codex_mcp_inventory from tests.integration.utils.terminal import AgentTerminal from .base import BaseCujTest @@ -23,8 +25,10 @@ CODEX, CODING_AGENT_BY_CLI_NAME, INFERENCE_PATHS, + MCP_FIXTURE_LOCATION, + MCP_FIXTURE_SERVICE_NAME, + MCP_FIXTURE_TOOL_NAME, MODEL_PROVIDER_SERVICE_FIXTURES, - SANDBOX_MCP_SERVICE_NAME, WEB_SEARCH_MCP_SERVICE_NAME, ) from .helpers.workspace import Workspace @@ -52,7 +56,7 @@ def _assert_cuj2_config(config: dict) -> None: "spec_version": 1, "default_agent": CODING_AGENT_BY_CLI_NAME[CODEX], "enabled_agents": [expected_agent_config(CLAUDE), expected_agent_config(CODEX)], - "mcp_servers": {"names": [SANDBOX_MCP_SERVICE_NAME]}, + "mcp_servers": {"unity_catalog_location": MCP_FIXTURE_LOCATION}, } assert actual == expected, config @@ -99,9 +103,10 @@ def _assert_mps_catalog(workspace) -> None: def _assert_mcp_services(workspace) -> None: - for name in (SANDBOX_MCP_SERVICE_NAME, WEB_SEARCH_MCP_SERVICE_NAME): - service = workspace.client.ai_gateway.get_mcp_service(f"mcp-services/{name}") - assert service.name and service.name.endswith(name), service + service = workspace.client.ai_gateway.get_mcp_service( + f"mcp-services/{MCP_FIXTURE_SERVICE_NAME}" + ) + assert service.name and service.name.endswith(MCP_FIXTURE_SERVICE_NAME), service def _assert_configured_files(session, workspace_url: str) -> None: @@ -149,7 +154,7 @@ def _assert_generated_mcp_listings(session) -> None: assert binary, f"Required agent is not on the isolated PATH: {agent}" listing = session.run("mcp", "list", binary=binary, timeout=120) output = f"{listing.stdout}\n{listing.stderr}" - assert _mcp_name_listed(output, SANDBOX_MCP_SERVICE_NAME), output + assert _mcp_name_listed(output, MCP_FIXTURE_SERVICE_NAME), output assert not _mcp_name_listed(output, WEB_SEARCH_MCP_SERVICE_NAME), output @@ -161,29 +166,81 @@ def _parent_transcripts(session, agent: str) -> dict[str, list[dict]]: } -def _task_complete(session, agent: str, marker: str) -> bool: +def _task_complete(session, agent: str, task) -> bool: transcripts = _parent_transcripts(session, agent) - return any( - marker in answer - for records in transcripts.values() - for answer in assistant_answers(agent, records) - ) + return any(task.completed(records, agent) for records in transcripts.values()) -def _assert_task_evidence(session, agent: str, marker: str) -> None: +def _assert_task_evidence(session, agent: str, task) -> None: transcripts = _parent_transcripts(session, agent) answers = [ answer for records in transcripts.values() for answer in assistant_answers(agent, records) ] - assert any(marker in answer for answer in answers), answers + assert task.completed_in_answers(answers), answers + assert any(task.tool_call_observed(records) for records in transcripts.values()), transcripts session.record( f"cuj2-{agent}-evidence", - {"marker": marker, "answers": answers}, + { + "expected": task.expected, + "run_id": task.run_id, + "answers": answers, + "agent_sessions": transcripts, + }, ) -def _agent_prompt(marker: str) -> str: - return f"Reply with the exact marker `{marker}` and no other text. Do not use tools." +class _McpFixtureTask: + """A nonce-bound MCP call whose receipt is checked in the native parent transcript.""" + + def __init__(self, marker: str): + self.marker = marker + self.run_id = uuid.uuid4().hex + self.receipt = hashlib.sha256( + b"unity-gateway-cuj-fixture\0" + + MCP_FIXTURE_TOOL_NAME.encode("ascii") + + b"\0" + + self.run_id.encode("ascii") + ).hexdigest() + self.expected = {"marker": marker, MCP_FIXTURE_TOOL_NAME: self.receipt} + self.prompt = ( + f"Call the read-only MCP tool {MCP_FIXTURE_TOOL_NAME} on " + f"{MCP_FIXTURE_SERVICE_NAME.replace('.', '-')}, with run_id={self.run_id}. " + f"Reply with only this JSON object: include marker {marker!r} and the exact text " + f"returned by the tool under the {MCP_FIXTURE_TOOL_NAME!r} key. Do not use shell " + "commands, files, or subagents." + ) + + def completed_in_answers(self, answers: list[str]) -> bool: + for answer in answers: + try: + if json.loads(answer) == self.expected: + return True + except json.JSONDecodeError: + continue + return False + + def completed(self, records: list[dict], agent: str) -> bool: + return self.completed_in_answers(assistant_answers(agent, records)) + + def tool_call_observed(self, records: list[dict]) -> bool: + def visit(value): + if isinstance(value, dict): + kind = value.get("type", "") + name = value.get("name", "") + if kind in { + "tool_use", + "tool_call", + "function_call", + "custom_tool_call", + } and MCP_FIXTURE_TOOL_NAME in str(name): + arguments = value.get("input", value.get("arguments", {})) + return self.run_id in json.dumps(arguments, sort_keys=True) + return any(visit(child) for child in value.values()) + if isinstance(value, list): + return any(visit(child) for child in value) + return False + + return visit(records) def _assert_inference_request(recorder, request, *, provider: str, model: str, marker: str) -> None: @@ -203,9 +260,10 @@ class TestCuj2Configuration(_Cuj2Base): def test_cuj_configuration(self, cuj): """Scenario: configure ug from the preconfigured two-agent MPS/MCP workspace. - Expected: the read-only CodingAgentConfig selects the two exact MPS resources and sandbox - MCP; both native MPS APIs advertise their selected model, generated agent settings use - the exact provider/model values, and agent MCP listings exclude web_search. + Expected: the read-only CodingAgentConfig selects the two exact MPS resources and the + fixture MCP schema; both native MPS APIs advertise their selected model, generated agent + settings use the exact provider/model values, and agent MCP listings expose the fixture + reader while excluding web_search. """ session, workspace, _ = cuj published = workspace.config() @@ -228,11 +286,12 @@ def test_cuj_configuration(self, cuj): class TestCuj2CodexInference(_Cuj2Base): def test_cuj_codex_inference(self, cuj): - """Scenario: configure through the recorder and complete a real Codex inference. + """Scenario: configure through the recorder and complete a real Codex MCP task. Expected: Codex sends a Responses request for gpt-5-nano with the ug_e2e.providers.openai target header and receives its paired HTTP 200 response; its - parent transcript contains the final marker. + native inventory connects the fixture MCP and lists read_fixture; its parent transcript + records the nonce-bound receipt returned by the tool. """ session, workspace, recorder = cuj published = workspace.config() @@ -252,16 +311,20 @@ def test_cuj_codex_inference(self, cuj): _assert_generated_mcp_listings(session) recorder.prepare_launch() - marker = f"CUJ2-CODEX-{uuid.uuid4().hex}" + task = _McpFixtureTask(f"CUJ2-CODEX-{uuid.uuid4().hex}") checkpoint = recorder.checkpoint() with AgentTerminal(session, CODEX, [str(session.binary), CODEX], "cuj2-codex") as tui: tui.boot() - tui.submit(_agent_prompt(marker)) + inventory = open_codex_mcp_inventory( + tui, {MCP_FIXTURE_SERVICE_NAME: MCP_FIXTURE_TOOL_NAME} + ) + assert _mcp_name_listed(inventory, MCP_FIXTURE_SERVICE_NAME), inventory + tui.submit(task.prompt) def completed(screen): assert_no_terminal_api_error(screen) assert "Do you want to proceed?" not in screen, screen - return _task_complete(session, CODEX, marker) + return _task_complete(session, CODEX, task) tui.wait_for(completed, "completed Codex task", timeout=240) tui.exit_normally() @@ -277,18 +340,19 @@ def completed(screen): request, provider=MODEL_PROVIDER_SERVICE_FIXTURES[CODEX][0], model=MODEL_PROVIDER_SERVICE_FIXTURES[CODEX][1], - marker=marker, + marker=task.marker, ) - _assert_task_evidence(session, CODEX, marker) + _assert_task_evidence(session, CODEX, task) class TestCuj2ClaudeInference(_Cuj2Base): def test_cuj_claude_inference(self, cuj): - """Scenario: configure through the recorder and complete a real Claude inference. + """Scenario: configure through the recorder and complete a real Claude MCP task. Expected: Claude sends a Messages request for claude-haiku-4-5-20251001 with the ug_e2e.providers.anthropic target header and receives its paired HTTP 200 response; its - parent transcript contains the final marker. + native inventory connects the fixture MCP and lists read_fixture; its parent transcript + records the nonce-bound receipt returned by the tool. """ session, workspace, recorder = cuj published = workspace.config() @@ -308,15 +372,20 @@ def test_cuj_claude_inference(self, cuj): _assert_generated_mcp_listings(session) recorder.prepare_launch() - marker = f"CUJ2-CLAUDE-{uuid.uuid4().hex}" + task = _McpFixtureTask(f"CUJ2-CLAUDE-{uuid.uuid4().hex}") checkpoint = recorder.checkpoint() with AgentTerminal(session, CLAUDE, [str(session.binary), CLAUDE], "cuj2-claude") as tui: tui.boot() - tui.submit(_agent_prompt(marker)) + inventory = open_claude_mcp_inventory( + tui, {MCP_FIXTURE_SERVICE_NAME: MCP_FIXTURE_TOOL_NAME} + ) + assert _mcp_name_listed(inventory, MCP_FIXTURE_SERVICE_NAME), inventory + tui.submit(task.prompt) def completed(screen): assert_no_terminal_api_error(screen) - return _task_complete(session, CLAUDE, marker) + assert "Do you want to proceed?" not in screen, screen + return _task_complete(session, CLAUDE, task) tui.wait_for(completed, "completed Claude task", timeout=240) tui.exit_normally() @@ -332,6 +401,6 @@ def completed(screen): request, provider=MODEL_PROVIDER_SERVICE_FIXTURES[CLAUDE][0], model=MODEL_PROVIDER_SERVICE_FIXTURES[CLAUDE][1], - marker=marker, + marker=task.marker, ) - _assert_task_evidence(session, CLAUDE, marker) + _assert_task_evidence(session, CLAUDE, task) diff --git a/tests/integration/README.md b/tests/integration/README.md index 542309acc..0ecf4c42b 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -6,8 +6,8 @@ It reuses the session, terminal, file-task, and transcript helpers in `utils/`. Its Claude/Codex evidence helpers keep scenario-specific assertions separate from shared mechanics. Workspace config/catalog reads use its base class's Databricks SDK client. Configuration is read-only and checked for changes at teardown; concurrent readers need no reservation. -CUJ2 adds three separately collected cases for exact MPS/MCP configuration, Codex inference, -and Claude inference. +CUJ2 adds three separately collected cases for exact MPS/MCP configuration and real +`read_fixture` tool use through Codex and Claude. This suite runs the **installed product** through subprocesses, against the same `UCODE_TEST_WORKSPACE` used by the existing e2e tests. It does not import `ucode`, @@ -370,8 +370,9 @@ The old focused checks are now descriptive CUJs with setup and outcomes visible in each test. Duplicate boot-only checks are incorporated into the Databricks configuration TUI journeys. Real failures, including generated config left after revert and banners on app-server stdout, remain assertions. -Live MCP/skills functionality, the broad configure-option matrix, and other -agents are outside this focused revision. +Live skills functionality, the broad configure-option matrix, and other agents are outside +this focused revision. The CUJ2 cases exercise the preconfigured `ug_e2e.tools` MCP schema; +MCP status/list commands remain separate follow-ups. Custom OAuth search dispatch and refresh are covered by component tests in `../test_mcp_web_search.py`; no live search request, delegated search, or classifier permission decision is asserted by those tests or this integration suite. diff --git a/tests/integration/utils/mcp.py b/tests/integration/utils/mcp.py new file mode 100644 index 000000000..3550d881f --- /dev/null +++ b/tests/integration/utils/mcp.py @@ -0,0 +1,104 @@ +"""Inspect MCP inventories through the real agent terminals.""" + +import re + +import pyte + + +def open_claude_mcp_inventory(tui, services: dict[str, str]) -> str: + """Open Claude's native MCP menu and verify each requested tool is connected and listed.""" + tui.submit("/mcp") + tui.wait_for( + lambda screen: "Manage MCP servers" in screen and tui.selected_line(), + "Claude's MCP menu", + timeout=120, + ) + tui.wait_for( + lambda screen: "connecting" not in screen.casefold(), + "Claude's MCP startup checks to finish before opening server details", + timeout=120, + ) + first = tui.selected_line() + screens = [] + for _ in range(100): + screens.append(tui.visible) + current = tui.selected_line() + tui.send("\x1b[B", "inspect the next MCP inventory row") + tui.wait_for( + lambda screen, current=current: tui.selected_line() != current, + "the next MCP inventory row", + ) + if tui.selected_line() == first: + break + else: + raise AssertionError("Claude's MCP inventory did not wrap within 100 rows") + inventory = "\n".join(screens) + tui.actions.append({"reason": "claude-mcp-inventory", "screen": inventory}) + for service, tool in services.items(): + name = service.replace(".", "-") + tui.choose("Manage MCP servers", name) + tui.wait_for( + lambda screen, name=name: ( + name in screen.casefold() + and re.search(r"(?m)^\s*Status:\s*(?:[✓✔]\s*)?connected\s*$", screen) + and "View tools" in screen + and "tools fetch failed" not in screen + ), + f"{name} connected with tools loaded", + timeout=120, + ) + tui.actions.append({"reason": f"mcp-connected-{name}", "screen": tui.visible}) + tui.choose("MCP Server", "View tools") + tui.wait_for( + lambda screen, tool=tool: re.search(rf"\b{re.escape(tool)}\b", screen), + f"{tool} in {name}'s tools menu", + ) + tui.actions.append({"reason": f"mcp-tools-{name}", "screen": tui.visible}) + tui.send("\x1b", "return from the MCP tools view") + tui.wait_for( + lambda screen: "MCP Server" in screen or "Manage MCP servers" in screen, + "the MCP server detail or inventory", + ) + if "Manage MCP servers" not in tui.visible: + tui.send("\x1b", "return to the MCP inventory") + tui.wait_for( + lambda screen: "Manage MCP servers" in screen, + "Claude's MCP inventory", + ) + tui.send("\x1b", "close Claude's MCP menu") + tui.wait_for( + lambda screen: ( + "Manage MCP servers" not in screen and re.search(r"(?m)^\s*[❯›>]\s*(?!\d+[.)])", screen) + ), + "Claude's prompt after closing the MCP menu", + ) + return inventory + + +def open_codex_mcp_inventory(tui, services: dict[str, str]) -> str: + """Show Codex's complete MCP inventory and verify each requested tool is connected.""" + output_start = len(tui.output) + tui.submit("/mcp verbose") + for service, tool in services.items(): + name = re.escape(service.replace(".", "-")) + tui.wait_for( + lambda screen, name=name, tool=tool: re.search( + rf"(?m)^\s*• {name}: connected \(1 tool\)\s*\n" + rf"\s*• Auth: [^\n]*\n" + rf"\s*• Tools: {re.escape(tool)}\s*$", + screen, + ), + f"{service} connected with {tool} loaded in Codex's MCP inventory", + timeout=120, + ) + screen = pyte.HistoryScreen(140, 60, history=1000) + pyte.Stream(screen).feed("".join(tui.output[output_start:])) + assert len(screen.history.top) < screen.history.size, "Codex's MCP inventory was truncated" + history = [ + "".join(line[column].data for column in range(screen.columns)) + for line in screen.history.top + ] + inventory = "\n".join([*history, *screen.display]) + assert "MCP Tools" in inventory, inventory + tui.actions.append({"reason": "codex-mcp-inventory", "screen": inventory}) + return inventory