diff --git a/tests/README.md b/tests/README.md index d479c8110..01a411650 100644 --- a/tests/README.md +++ b/tests/README.md @@ -232,8 +232,8 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi | `test_ug_configure_codex_databricks` | Configure Databricks Hosted; execute the generated auth helper; open Codex TUI and read a file | Generated helper invokes `ug` with clean token stdout; completed assistant answer contains the file value; normal exit and reopen | | `test_ug_configure_codex_openai_mps` | Select OpenAI MPS in the real configure picker; launch Codex | Saved provider in status; completed TUI file task; normal exit | | `test_ug_claude_custom_oauth_cli_boots`, `test_ug_codex_custom_oauth_cli_boots` | Launch with `ENABLE_CUSTOM_OAUTH_FROM_CLI=1`, `--workspace`, and `--client-id databricks-cli` | Real TUI reaches a usable prompt, accepts keyboard input, exits normally, and saves `client_id = databricks-cli` in its generated CLI profile; Claude also reads the OS-managed settings and requires a profile-only `apiKeyHelper` | -| `test_case_07_configured_claude_discovers_system_models`, `test_case_09_fresh_claude_discovers_system_models` | Launch configured/fresh Claude with no discovery flag or source override | Claude caches `system.ai` models (including recognized Anthropic gateway aliases), includes ug's discovered family defaults, and shows a discovered picker entry | -| `test_case_08_configured_codex_uses_default_models`, `test_case_10_fresh_codex_uses_default_models` | Launch configured/fresh Codex with no source override | ug discovers `system.ai` models but leaves model/reasoning preferences unset; app-server exposes native GPT entries without a generated scoped catalog | +| `test_case_07_configured_claude_discovers_system_models` (CUJ7), `test_case_09_fresh_claude_discovers_system_models` (integration) | Launch configured/fresh Claude with no discovery flag or source override | Claude caches `system.ai` models (including recognized Anthropic gateway aliases), includes ug's discovered family defaults, and shows a discovered picker entry; case 07 lives in `e2e_cuj/test_cuj7_model_discovery.py` | +| `test_case_08_configured_codex_uses_default_models` (CUJ7), `test_case_10_fresh_codex_uses_default_models` (integration) | Launch configured/fresh Codex with no source override | ug discovers `system.ai` models but leaves model/reasoning preferences unset; app-server exposes native GPT entries without a generated scoped catalog; case 08 lives in `e2e_cuj/test_cuj7_model_discovery.py` | | `test_case_11_*` | Launch configured and fresh Claude with an explicit provider and no managed config | The provider catalog replaces built-in picker rows; the cache contains exactly the provider model, and the picker shows both its row and Default resolving to it | | `test_case_12_*` | Launch configured and fresh Codex with a provider | The provider supplies exactly its model catalog | | `test_case_13_*` | Launch configured and fresh Claude with a model location and no managed config | The explicit parent's catalog replaces built-in picker rows and appears in the real picker; Default resolves to the model in the scoped fixture catalog | @@ -243,9 +243,17 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi | `test_ug_opencode_headless_prompt_argument` | Run OpenCode from a script (`run --format json --auto`) with an argument prompt | Completed Read tool call; final text answer contains the file value; exit zero (non-blocking CI lane) | | `test_ug_agents_self_managed_opencode_journey` | Under the injected `managed_workspace_default` config (enables Claude/Codex, not OpenCode): bare configure, `ug agents list`, refused OpenCode launch, `ug agents add opencode`, real headless task, `ug agents remove opencode` (marker `managed_fixture and opencode`, non-blocking CI lane) | OpenCode absent from the list and launch fails with "doesn't enable OpenCode" / `ug agents add opencode`; after add it is listed self-managed and the headless Read task returns the fixture value; after remove it is hidden and refused again | | `test_ug_agents_admin_managed_guardrails` | `ug agents add` / `remove` on an agent the admin config enables | Add is a no-op noting the admin manages it; remove is rejected with nonzero exit | +| `test_ug_claude_headless_fresh_workspace`, `test_ug_codex_headless_fresh_workspace` | From fresh state, launch an agent with `--workspace` against a workspace with no managed config | Claude Haiku 4.5 and Codex GPT-5.4 Nano read an unpredictable file value through the gateway and return it in a structured completed answer; exit zero; no routing | +| `test_ug_claude_fresh_provider_launch`, `test_ug_codex_fresh_provider_launch` | From fresh state, launch each real agent CLI with `--workspace` and an existing provider | Both CLIs report the selected provider and exit successfully without routing or inference; Claude also checks its provider header. Cases 11/12 cover provider discovery | +| `test_ug_claude_headless_fresh_model_location` | From fresh state, launch Claude in the CUJ7 workspace with `--workspace`, `--model-location ug_e2e.models`, and `ug_e2e.models.claude_haiku` | Haiku reads an unpredictable file value through the selected model location and returns it in a structured completed answer; exit zero; no routing | +| `test_ug_codex_headless_fresh_model_location` | From fresh state, launch Codex in the CUJ7 workspace with `--workspace`, `--model-location ug_e2e.models`, and `ug_e2e.models.gpt_luna` | Luna reads an unpredictable file value through the selected model location and returns it in a structured completed answer; exit zero; no routing | +| `test_ug_claude_preserves_preexisting_managed_family_defaults` | Launch fresh Claude with pre-existing OS-managed family defaults and select the Sonnet family | Every default survives unchanged; the selected family completes a file task through the preconfigured Sonnet service | +| `test_ug_claude_tui_rewrites_managed_settings_preserving_family_defaults` | Launch the Claude TUI with the same pre-existing OS-managed family defaults and `--model sonnet` | ug rewrites the OS-managed file (headless launches leave it untouched) while keeping every default; the TUI file task reaches the Sonnet service; `ug revert` restores the seeded file | +| `test_ug_codex_preserves_preexisting_managed_config` | Launch fresh Codex over pre-existing OS-managed admin settings (a sibling model provider and a display key) and select scoped GPT Luna before ug's separator or in `exec` | Every admin setting survives unchanged; the file task completes through the scoped Luna service with the parent-schema header | +| `test_ug_codex_tui_rewrites_managed_config_preserving_admin_settings` | Launch the Codex TUI with the same OS-managed admin settings, `--model-location ug_e2e.models`, and scoped GPT Luna | ug rewrites the OS-managed file (headless launches leave it untouched) while keeping every admin setting; the TUI file task reaches the Luna service with the parent-schema header; `ug revert` restores the seeded file | | `test_ug_claude_exports_trace_to_configured_table`, `test_ug_codex_exports_trace_to_configured_table` | Configure tracing, complete a headless task carrying a unique trace marker, then wait for ingestion | The configured trace table contains an agent span with the same trace-safe marker and requested model | | `test_ug_claude_headless_explicit_model_bypasses_routing` | Pass `--model VALUE` / `--model=VALUE` before and after ug's separator with smart routing enabled, without workspace policy | Real file task completes; JSON `modelUsage` reports the requested model with output tokens; no routing wrapper | -| `test_ug_codex_headless_explicit_model_bypasses_routing` | Pass `--model VALUE` / `--model=VALUE` / `-m VALUE` with smart routing enabled | Real file task completes; no routing wrapper | +| `test_ug_codex_headless_explicit_model_bypasses_routing` | Pass `--model VALUE` / `--model=VALUE` / `-m VALUE` before ug's separator and within Codex `exec`, with smart routing enabled | Real file task completes; completed-turn model matches the selection; no routing wrapper | | `test_ug_claude_preserves_caller_settings_and_hook` | Pass a settings path containing spaces | Real SessionStart hook executes; caller file unchanged; file task completes | | `test_ug_claude_reports_unsupported_short_model_option` | Pass Claude's unsupported `-m` | Actual agent error and exit status preserved | | `test_ug_claude_auth_help`, `test_ug_claude_mcp_help` | Request subcommand help with legacy smart routing off/on | Real agent help; no routing wrapper | @@ -274,7 +282,7 @@ integration utilities; only CUJ-specific evidence correlation stays in a test fi | `test_ug_and_ucode_auth_helpers_emit_only_the_supplied_bearer` | Run both auth helper commands with the public bearer override, with and without forced refresh | Exact token-only stdout, no warnings or ANSI escapes; no workspace authentication or saved state | | `test_ug_and_ucode_web_search_helpers_preserve_mcp_stdio` | Initialize and list tools through both web-search helper commands | Exactly the MCP JSON-RPC responses; no text/ANSI contamination; existing server/tool identities preserved; no model request | -With Claude and Codex selected there are **80 live cases** (14 marked TUI cases), +With Claude and Codex selected there are **85 live cases** (11 marked TUI cases), **1 two-workspace case** (marker `workspace_switch`), **43 managed-fixture cases** (marker `managed_fixture`, with only the CodingAgentConfig input injected from a JSON file in `fixtures/managed_config/`), and **7 installation checks**. The 14 retained numbered scenarios @@ -338,10 +346,21 @@ dependency graph to reproduce a user's combination. Every relevant same-reposito PR and push to `main` runs both smoke and the full CUJ suite. Smoke covers the Databricks Hosted configure/TUI, custom OAuth CLI TUI, and headless argument journeys for both agents, in two parallel jobs. After smoke finishes, the full -suite runs all 80 live cases across two parallel agent jobs: one Claude VM and one +suite runs all 85 live cases across two parallel agent jobs: one Claude VM and one Codex VM, each running its configure, headless, and commands/lifecycle cases serially. Each agent is installed once for the full suite, and no two full jobs for the same agent overlap within a run. +CUJ7's eight journeys collect twelve cases, six per agent, in the required +`E2E CUJs · CUJ 7 · Unmanaged model discovery` shard: configured Claude picker and +Codex app-server discovery, fresh Claude/Codex model-location file tasks, and +preservation of pre-existing OS-managed input (Claude family defaults, Codex admin +settings) on headless and interactive launch. Task journeys cover model selection before +and after ug's separator. Each case reverts its local setup. The class +pins `WORKSPACE_URL` to `https://dbc-14e376e8-6541.cloud.databricks.com` rather than using +a workspace secret. The workspace must publish no `CodingAgentConfig`, expose discoverable +`system.ai` models, and retain the existing models `ug_e2e.models.claude_haiku`, +`ug_e2e.models.claude_sonnet`, and `ug_e2e.models.gpt_luna`; the shared service principal +needs read/use privileges on all three. CI starts integration alongside unit tests and the existing e2e shards. Integration does not wait for agent e2e or get skipped when an agent shard fails. These suites share workspace capacity; overlapping their requests can still encounter rate limits. diff --git a/tests/e2e_cuj/conftest.py b/tests/e2e_cuj/conftest.py index 04ec5877b..a9aab171a 100644 --- a/tests/e2e_cuj/conftest.py +++ b/tests/e2e_cuj/conftest.py @@ -12,9 +12,9 @@ from .helpers.constants import CLAUDE, CODEX, MANAGED_PATHS from .helpers.session import ( MACHINE_WIDE_LEAK, - MachineWideLeak, UserSession, dirty_runner_message, + record_machine_wide_leak, ) from .helpers.tui_request_recorder import TuiRequestRecorder from .helpers.workspace import Workspace @@ -31,13 +31,6 @@ def _scrubbed(error: DatabricksError) -> RuntimeError: return RuntimeError(f"Workspace API failed: {type(error).__name__}") -def _record_machine_wide_leak(request): - """Name the class that leaked machine-wide settings to the later classes it blocks.""" - leaked = tuple(str(path) for path in MANAGED_PATHS if path.exists()) - if leaked: - request.config.stash[MACHINE_WIDE_LEAK] = MachineWideLeak(request.node.nodeid, leaked) - - @pytest.fixture(scope="class") def cuj(request, setup_workspace, tmp_path_factory): assert os.name == "posix", "Full TUI CUJs require a disposable POSIX runner" @@ -70,7 +63,7 @@ def cuj(request, setup_workspace, tmp_path_factory): "cleanup-revert", "CUJ teardown left machine-wide agent settings" ) finally: - _record_machine_wide_leak(request) + record_machine_wide_leak(request) except DatabricksError as error: raise _scrubbed(error) from None diff --git a/tests/e2e_cuj/helpers/constants.py b/tests/e2e_cuj/helpers/constants.py index a46848f61..132ee0e48 100644 --- a/tests/e2e_cuj/helpers/constants.py +++ b/tests/e2e_cuj/helpers/constants.py @@ -53,6 +53,10 @@ class CodingAgent(StrEnum): "us.anthropic.claude-haiku-4-5-20251001-v1:0", ) UC_MODEL_LOCATION_FIXTURE = ("ug_e2e.models", "ug_e2e.models.codex_primary") +MODEL_SERVICE_SCHEMA = "ug_e2e.models" +CLAUDE_HAIKU_MODEL_SERVICE = f"{MODEL_SERVICE_SCHEMA}.claude_haiku" +CLAUDE_SONNET_MODEL_SERVICE = f"{MODEL_SERVICE_SCHEMA}.claude_sonnet" +CODEX_LUNA_MODEL_SERVICE = f"{MODEL_SERVICE_SCHEMA}.gpt_luna" FIXTURE_READER_MCP_SERVICE_NAME = "ug_e2e.tools.fixture_reader" # UC skill names reject underscores, so the plan's `fixture_summary` is hyphenated. FIXTURE_SUMMARY_SKILL_NAME = "ug_e2e.skills.fixture-summary" diff --git a/tests/e2e_cuj/helpers/evidence.py b/tests/e2e_cuj/helpers/evidence.py index 73a401474..043e7e634 100644 --- a/tests/e2e_cuj/helpers/evidence.py +++ b/tests/e2e_cuj/helpers/evidence.py @@ -10,8 +10,10 @@ import httpx from tests.integration.utils.evidence import FileTask, read_jsonl +from tests.integration.utils.model_discovery import claude_discovery_model_id +from tests.integration.utils.provider_catalog import MODEL_SERVICE_PARENT_SCHEMA_HEADER -from .constants import CLAUDE, CODEX, NATIVE_MODEL_ALIASES +from .constants import CLAUDE, CODEX, INFERENCE_PATHS, NATIVE_MODEL_ALIASES def canonical_model(value): @@ -114,6 +116,67 @@ def served_inference_request(recorder, requests, request, agent): raise AssertionError("Claude compatibility retries did not reach a successful response") +def assert_claude_headless_model(result, expected): + final = None + for line in result.stdout.splitlines(): + try: + payload = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(payload, dict) and payload.get("type") == "result": + final = payload + assert final is not None and not final.get("is_error"), result.stdout + usage = final["modelUsage"] + assert set(usage) == {expected}, {"expected": expected, "observed": sorted(usage)} + assert usage[expected]["outputTokens"] > 0, usage + + +def _request_contains_task(request, agent, task): + field = "messages" if agent == CLAUDE else "input" + entries = request.payload.get(field) + if not isinstance(entries, (list, str)): + return False + if isinstance(entries, str): + return entries == task.prompt + for entry in entries: + if not isinstance(entry, dict) or entry.get("role") != "user": + continue + content = entry.get("content", "") + if isinstance(content, str) and content == task.prompt: + return True + if isinstance(content, list) and any( + message_text([part]) == task.prompt for part in content if isinstance(part, dict) + ): + return True + return False + + +def assert_inference_evidence(recorder, checkpoint, agent, task, expected, *, parent_schema=None): + expected_wire_model = claude_discovery_model_id(expected) if agent == CLAUDE else expected + requests = recorder.requests_after(checkpoint) + inference_requests = [ + request + for request in requests + if request.method == "POST" and request.path == INFERENCE_PATHS[agent] + ] + assert inference_requests, { + "agent": agent, + "path": INFERENCE_PATHS[agent], + "requests": [(request.method, request.path) for request in requests], + } + task_requests = [ + request for request in inference_requests if _request_contains_task(request, agent, task) + ] + assert task_requests, "No inference request contained the submitted task prompt" + for request in task_requests: + if parent_schema is not None: + assert ( + request.headers.get(MODEL_SERVICE_PARENT_SCHEMA_HEADER.lower()) == parent_schema + ), request.headers + served = served_inference_request(recorder, task_requests, request, agent) + assert_served(recorder, served, expected_wire_model) + + def claude_file_task(session): """A FileTask naming its absolute path, so Claude reads it without a `find` permission prompt.""" task = FileTask(session) diff --git a/tests/e2e_cuj/helpers/session.py b/tests/e2e_cuj/helpers/session.py index 2f1a57d2d..163957600 100644 --- a/tests/e2e_cuj/helpers/session.py +++ b/tests/e2e_cuj/helpers/session.py @@ -22,6 +22,13 @@ class MachineWideLeak: MACHINE_WIDE_LEAK = pytest.StashKey[MachineWideLeak]() +def record_machine_wide_leak(request): + """Name the scenario that left machine-wide settings in subsequent setup failures.""" + leaked = tuple(str(path) for path in constants.MANAGED_PATHS if path.exists()) + if leaked: + request.config.stash[MACHINE_WIDE_LEAK] = MachineWideLeak(request.node.nodeid, leaked) + + def dirty_runner_message(leak: MachineWideLeak | None) -> str: if leak is None: return ( diff --git a/tests/e2e_cuj/pytest.ini b/tests/e2e_cuj/pytest.ini index ae59c16cd..69509fda4 100644 --- a/tests/e2e_cuj/pytest.ini +++ b/tests/e2e_cuj/pytest.ini @@ -8,5 +8,6 @@ markers = mcp_registration: managed MCP registration, connection, and tool use workspace_isolated: reads preconfigured CUJ workspace state with isolated local sessions tui: real interactive terminal and completed task + live: requires the CUJ's real workspace claude: only runs when Claude Code is explicitly selected codex: only runs when Codex is explicitly selected diff --git a/tests/e2e_cuj/test_cuj3_models.py b/tests/e2e_cuj/test_cuj3_models.py index d427cbf5f..2129018b9 100644 --- a/tests/e2e_cuj/test_cuj3_models.py +++ b/tests/e2e_cuj/test_cuj3_models.py @@ -1,7 +1,5 @@ """Managed catalog discovery, default launches, and explicit model selection.""" -import json - import pytest from tests.integration.utils.agents import claude, codex @@ -25,13 +23,12 @@ MODEL_SCHEMA, OTHER_MODEL_SCHEMA, ) -from .helpers.constants import CLAUDE, CODEX, INFERENCE_PATHS +from .helpers.constants import CLAUDE, CODEX from .helpers.evidence import ( SessionEvidence, - assert_served, + assert_claude_headless_model, + assert_inference_evidence, claude_file_task, - message_text, - served_inference_request, ) from .helpers.terminal import Terminal @@ -75,63 +72,6 @@ def _catalog_display_names(workspace, agent, schema): raise AssertionError("Anthropic catalog exceeded 20 pages") -def _assert_claude_headless_model(result, expected): - final = None - for line in result.stdout.splitlines(): - try: - payload = json.loads(line) - except json.JSONDecodeError: - continue - if isinstance(payload, dict) and payload.get("type") == "result": - final = payload - assert final is not None and not final.get("is_error"), result.stdout - usage = final["modelUsage"] - assert set(usage) == {expected}, {"expected": expected, "observed": sorted(usage)} - assert usage[expected]["outputTokens"] > 0, usage - - -def _request_contains_task(request, agent, task): - field = "messages" if agent == CLAUDE else "input" - entries = request.payload.get(field) - if not isinstance(entries, (list, str)): - return False - if isinstance(entries, str): - return entries == task.prompt - for entry in entries: - if not isinstance(entry, dict) or entry.get("role") != "user": - continue - content = entry.get("content", "") - if isinstance(content, str) and content == task.prompt: - return True - if isinstance(content, list) and any( - message_text([part]) == task.prompt for part in content if isinstance(part, dict) - ): - return True - return False - - -def _assert_inference_evidence(recorder, checkpoint, agent, task, expected): - expected_wire_model = claude.discovery_model_id(expected) if agent == CLAUDE else expected - requests = recorder.requests_after(checkpoint) - inference_requests = [ - request - for request in requests - if request.method == "POST" and request.path == INFERENCE_PATHS[agent] - ] - assert inference_requests, { - "agent": agent, - "path": INFERENCE_PATHS[agent], - "requests": [(request.method, request.path) for request in requests], - } - task_requests = [ - request for request in inference_requests if _request_contains_task(request, agent, task) - ] - assert task_requests, "No inference request contained the submitted task prompt" - for request in task_requests: - served = served_inference_request(recorder, task_requests, request, agent) - assert_served(recorder, served, expected_wire_model) - - @pytest.fixture(autouse=True) def _revert_catalog_test_state(cuj): yield @@ -184,7 +124,7 @@ def test_catalog_discovery_claude_picker_preserves_default(self, cuj): tui.task(evidence, task) tui.exit_normally() task.assert_completed(session, CLAUDE) - _assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) session.assert_not_routed() @pytest.mark.codex @@ -228,7 +168,7 @@ def test_catalog_discovery_codex_picker_preserves_default(self, cuj): tui.task(evidence, task) tui.exit_normally() assert_completed_task_model(session, CODEX, task.value, CODEX_DEFAULT) - _assert_inference_evidence(recorder, checkpoint, CODEX, task, CODEX_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CODEX, task, CODEX_DEFAULT) session.assert_not_routed() @pytest.mark.claude @@ -254,7 +194,7 @@ def test_catalog_discovery_bare_ug_uses_claude_default(self, cuj): tui.task(evidence, task) tui.exit_normally() task.assert_completed(session, CLAUDE) - _assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) session.assert_not_routed() @pytest.mark.claude @@ -280,7 +220,7 @@ def test_catalog_discovery_claude_tui_uses_default(self, cuj): tui.task(evidence, task) tui.exit_normally() task.assert_completed(session, CLAUDE) - _assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) session.assert_not_routed() @pytest.mark.codex @@ -302,7 +242,7 @@ def test_catalog_discovery_codex_tui_uses_default(self, cuj): tui.task(evidence, task) tui.exit_normally() assert_completed_task_model(session, CODEX, task.value, CODEX_DEFAULT) - _assert_inference_evidence(recorder, checkpoint, CODEX, task, CODEX_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CODEX, task, CODEX_DEFAULT) session.assert_not_routed() @pytest.mark.claude @@ -327,8 +267,8 @@ def test_catalog_discovery_claude_headless_uses_default(self, cuj): timeout=240, ) task.assert_headless_answer(CLAUDE, result) - _assert_claude_headless_model(result, CLAUDE_DEFAULT) - _assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) + assert_claude_headless_model(result, CLAUDE_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CLAUDE, task, CLAUDE_DEFAULT) session.assert_not_routed() @pytest.mark.codex @@ -347,7 +287,7 @@ def test_catalog_discovery_codex_headless_uses_default(self, cuj): ) task.assert_headless_answer(CODEX, result) assert_completed_task_model(session, CODEX, task.value, CODEX_DEFAULT) - _assert_inference_evidence(recorder, checkpoint, CODEX, task, CODEX_DEFAULT) + assert_inference_evidence(recorder, checkpoint, CODEX, task, CODEX_DEFAULT) session.assert_not_routed() @pytest.mark.claude @@ -376,8 +316,8 @@ def test_catalog_discovery_claude_explicit_model_completes_task(self, cuj, model timeout=240, ) task.assert_headless_answer(CLAUDE, result) - _assert_claude_headless_model(result, claude.discovery_model_id(model)) - _assert_inference_evidence(recorder, checkpoint, CLAUDE, task, model) + assert_claude_headless_model(result, claude.discovery_model_id(model)) + assert_inference_evidence(recorder, checkpoint, CLAUDE, task, model) session.assert_not_routed() @pytest.mark.codex @@ -405,5 +345,5 @@ def test_catalog_discovery_codex_explicit_model_completes_task(self, cuj, model) ) task.assert_headless_answer(CODEX, result) assert_completed_task_model(session, CODEX, task.value, model) - _assert_inference_evidence(recorder, checkpoint, CODEX, task, model) + assert_inference_evidence(recorder, checkpoint, CODEX, task, model) session.assert_not_routed() diff --git a/tests/e2e_cuj/test_cuj7_model_discovery.py b/tests/e2e_cuj/test_cuj7_model_discovery.py new file mode 100644 index 000000000..230850164 --- /dev/null +++ b/tests/e2e_cuj/test_cuj7_model_discovery.py @@ -0,0 +1,480 @@ +"""CUJ7: model discovery and scoped inference in a dedicated unmanaged workspace.""" + +import json +import os +import shutil +import tomllib +from pathlib import Path + +import pytest +from databricks.sdk.errors import DatabricksError, NotFound + +from tests.integration.utils.evidence import FileTask +from tests.integration.utils.managed import MANAGED_CONFIGS_PATH, assert_no_managed_config +from tests.integration.utils.model_discovery import ( + assert_claude_system_models_in_picker, + assert_codex_default_models, +) +from tests.integration.utils.terminal import TerminalProcess + +from .base import BaseCujTest, bearer +from .helpers.constants import ( + CLAUDE, + CLAUDE_HAIKU_MODEL_SERVICE, + CLAUDE_SONNET_MODEL_SERVICE, + CODEX, + CODEX_LUNA_MODEL_SERVICE, + MANAGED_PATHS, + MODEL_SERVICE_SCHEMA, +) +from .helpers.evidence import ( + SessionEvidence, + assert_claude_headless_model, + assert_inference_evidence, + claude_file_task, +) +from .helpers.session import ( + MACHINE_WIDE_LEAK, + MANAGED_STATE_FILES, + UserSession, + dirty_runner_message, + record_machine_wide_leak, +) +from .helpers.terminal import Terminal +from .helpers.tui_request_recorder import TuiRequestRecorder + +CUJ_NAME = "CUJ 7 · Unmanaged model discovery" + +pytestmark = [pytest.mark.live, pytest.mark.workspace_isolated] + +CLAUDE_FAMILY_DEFAULTS = { + f"ANTHROPIC_DEFAULT_{family}_MODEL": ( + CLAUDE_SONNET_MODEL_SERVICE if family == "SONNET" else CLAUDE_HAIKU_MODEL_SERVICE + ) + for family in ("FABLE", "OPUS", "SONNET", "HAIKU") +} +# Inert admin-owned Codex settings outside ug's owned keys: a sibling provider and a display key. +CODEX_ADMIN_CONFIG = """file_opener = "none" + +[model_providers.AdminProvider] +name = "Admin provider" +base_url = "https://admin-provider.invalid/v1" +wire_api = "responses" +""" +MANAGED_INPUT = { + CLAUDE: (MANAGED_PATHS[0], json.dumps({"env": CLAUDE_FAMILY_DEFAULTS})), + CODEX: (MANAGED_PATHS[1], CODEX_ADMIN_CONFIG), +} +# Only an interactive launch rewrites the OS-managed file; ug's gateway base URL proves it did. +MANAGED_BASE_URL = { + CLAUDE: (("env", "ANTHROPIC_BASE_URL"), "/ai-gateway/anthropic"), + CODEX: (("model_providers", "Databricks", "base_url"), "/ai-gateway/codex/v1"), +} + + +def _selected_agent(request): + agents = [agent for agent in (CLAUDE, CODEX) if request.node.get_closest_marker(agent)] + assert len(agents) == 1, "CUJ7 cases must select exactly one agent" + return agents[0] + + +def _parse_managed(agent, text): + return json.loads(text) if agent == CLAUDE else tomllib.loads(text) + + +def _read_managed(session, agent): + path = MANAGED_INPUT[agent][0] + return _parse_managed(agent, session.run(str(path), binary="cat", timeout=30).stdout) + + +def _assert_contains(document, seeded, path=()): + """Every seeded leaf survives unchanged at its original path.""" + for key, value in seeded.items(): + location = ".".join((*path, key)) + assert isinstance(document, dict) and key in document, f"Missing seeded {location}" + if isinstance(value, dict): + _assert_contains(document[key], value, (*path, key)) + else: + assert document[key] == value, { + "path": location, + "seeded": value, + "observed": document[key], + } + + +def _assert_rewritten(document, agent, recorder): + keys, suffix = MANAGED_BASE_URL[agent] + value = document + for key in keys: + value = value.get(key) if isinstance(value, dict) else None + assert value == f"{recorder.url}{suffix}", value + + +def _assert_revert_restores(session, agent, seeded, name): + with TerminalProcess(session, "ug", [str(session.binary), "revert"], name) as terminal: + terminal.finish() + assert _read_managed(session, agent) == seeded + + +class TestUnmanagedModelDiscovery(BaseCujTest): + WORKSPACE_URL = "https://dbc-14e376e8-6541.cloud.databricks.com" + + @pytest.fixture(autouse=True) + def unmanaged_workspace(self, setup_workspace): + self._assert_unmanaged() + yield + self._assert_unmanaged() + + def _assert_unmanaged(self): + try: + payload = self.workspace.api_client.do("GET", MANAGED_CONFIGS_PATH) + except NotFound: + return + except DatabricksError as error: + raise RuntimeError(f"Workspace API failed: {type(error).__name__}") from None + assert_no_managed_config(payload) + assert not payload.get("next_page_token"), "Incomplete CodingAgentConfig listing" + + @pytest.fixture + def live_session(self, unmanaged_workspace, tmp_path, request): + """Keep each launch unconfigured; the class-scoped cuj fixture requires a published config.""" + assert os.name == "posix", "CUJ7 requires a disposable POSIX runner" + assert not any(path.exists() for path in MANAGED_PATHS), dirty_runner_message( + request.config.stash.get(MACHINE_WIDE_LEAK, None) + ) + binary = shutil.which("ug") + assert binary, "Install ug before running CUJ7" + for tool in (_selected_agent(request), "databricks"): + assert shutil.which(tool), f"Install the required CLI before running CUJ7: {tool}" + session = UserSession( + tmp_path, + Path(binary), + tmp_path / "artifacts", + bearer(self.workspace), + ) + try: + yield session + finally: + try: + session.revert_machine_wide( + "cleanup-revert", "CUJ7 teardown left machine-wide agent settings" + ) + finally: + record_machine_wide_leak(request) + + @pytest.fixture + def request_recorder(self, live_session): + with TuiRequestRecorder(self.workspace.config.host) as recorder: + yield recorder + + @pytest.fixture + def managed_input(self, live_session, request): + """Seed the selected agent's OS-managed input; stop recording before guarded revert.""" + session = live_session + agent = _selected_agent(request) + path, text = MANAGED_INPUT[agent] + managed_path = str(path) + managed_directory = path.parent + directory_existed = managed_directory.exists() + try: + if not directory_existed: + session.run("install", "-d", "-m", "0755", str(managed_directory), binary="sudo") + session.run("tee", managed_path, binary="sudo", input_text=text) + with TuiRequestRecorder(self.workspace.config.host) as recorder: + yield _parse_managed(agent, text), recorder + finally: + try: + # Revert restores our seeded input; revert_machine_wide would reject that file. + if any((session.home / ".ucode" / name).is_file() for name in MANAGED_STATE_FILES): + with TerminalProcess( + session, "ug", [str(session.binary), "revert"], "family-defaults-revert" + ) as terminal: + terminal.finish() + finally: + session.run("rm", "-f", managed_path, binary="sudo") + if not directory_existed and managed_directory.exists(): + session.run("rmdir", str(managed_directory), binary="sudo") + + @pytest.mark.claude + @pytest.mark.tui + def test_case_07_configured_claude_discovers_system_models(self, live_session): + """Scenario: configure Claude, then launch without source overrides or discovery flags. + + Expected: native discovery caches system.ai models as raw IDs or recognized Claude + gateway aliases and shows a discovered picker entry. + """ + session = live_session + session.configure( + [ + "configure", + "--agents", + CLAUDE, + "--workspace", + self.workspace.config.host, + "--skip-upgrade", + "--disable-databricks-ai-tools", + ] + ) + with Terminal(session, "case-07-system-models", [CLAUDE]) as tui: + tui.boot() + screen = tui.open_model_picker( + model_visible=lambda _text: session.claude_gateway_cache_ready() + ) + tui.exit_normally() + assert_claude_system_models_in_picker(session, screen) + + @pytest.mark.codex + def test_case_08_configured_codex_uses_default_models(self, live_session): + """Scenario: configure Codex, then launch without source overrides. + + Expected: unmanaged configuration leaves model selection to Codex's native default; + ug records system.ai discovery while model and reasoning preferences remain unset; + app-server exposes native GPT entries without a generated provider/parent-scoped catalog. + """ + session = live_session + session.configure( + [ + "configure", + "--agents", + CODEX, + "--workspace", + self.workspace.config.host, + "--skip-upgrade", + "--disable-databricks-ai-tools", + ] + ) + models = session.codex_model_ids(["app-server", "--listen", "stdio://"]) + assert_codex_default_models(session, models) + + @pytest.mark.claude + @pytest.mark.parametrize("model_owner", ["ug", "claude"]) + def test_ug_claude_headless_fresh_model_location( + self, live_session, request_recorder, model_owner + ): + """Scenario: launch fresh scoped Claude with --model before/after ug's separator. + + Expected: Haiku completes a file task without prior configuration or routing. + """ + session = live_session + task = claude_file_task(session) + model = CLAUDE_HAIKU_MODEL_SERVICE + model_args = ["--model", model] + evidence = SessionEvidence(session.home, CLAUDE) + + result = session.run( + CLAUDE, + "--workspace", + request_recorder.url, + "--model-location", + MODEL_SERVICE_SCHEMA, + *(model_args if model_owner == "ug" else []), + "--", + *(model_args if model_owner == "claude" else []), + "-p", + task.prompt, + "--output-format", + "json", + "--allowedTools", + "Read", + timeout=240, + ) + task.assert_headless_answer(CLAUDE, result) + assert evidence.completed(task), "No completed native turn matched the file task" + assert_claude_headless_model(result, model) + assert_inference_evidence( + request_recorder, 0, CLAUDE, task, model, parent_schema=MODEL_SERVICE_SCHEMA + ) + session.assert_not_routed() + + @pytest.mark.claude + @pytest.mark.parametrize("model_owner", ["ug", "claude"]) + def test_ug_claude_preserves_preexisting_managed_family_defaults( + self, live_session, managed_input, model_owner + ): + """Scenario: select Sonnet before/after ug's separator over OS-managed defaults. + + Expected: ug preserves every family default, and the selected Sonnet family + completes a file task through the preconfigured Sonnet service, not a discovered default. + """ + session = live_session + task = claude_file_task(session) + model = CLAUDE_SONNET_MODEL_SERVICE + model_args = ["--model", "sonnet"] + seeded, recorder = managed_input + evidence = SessionEvidence(session.home, CLAUDE) + result = session.run( + CLAUDE, + "--workspace", + recorder.url, + *(model_args if model_owner == "ug" else []), + "--", + *(model_args if model_owner == "claude" else []), + "-p", + task.prompt, + "--output-format", + "json", + "--allowedTools", + "Read", + timeout=240, + ) + task.assert_headless_answer(CLAUDE, result) + assert evidence.completed(task), "No completed native turn matched the file task" + assert_claude_headless_model(result, model) + assert_inference_evidence(recorder, 0, CLAUDE, task, model) + _assert_contains(_read_managed(session, CLAUDE), seeded) + session.assert_not_routed() + + @pytest.mark.claude + @pytest.mark.tui + def test_ug_claude_tui_rewrites_managed_settings_preserving_family_defaults( + self, live_session, managed_input + ): + """Scenario: select Sonnet in an interactive launch over OS-managed defaults. + + Expected: unlike headless launches, the TTY lets ug rewrite the OS-managed file; the + rewrite keeps every family default, the TUI task reaches the preconfigured Sonnet service, + and revert restores the seeded file. + """ + session = live_session + task = claude_file_task(session) + model = CLAUDE_SONNET_MODEL_SERVICE + seeded, recorder = managed_input + evidence = SessionEvidence(session.home, CLAUDE) + with Terminal( + session, + "family-defaults-tui", + [CLAUDE, "--workspace", recorder.url, "--model", "sonnet"], + ) as tui: + tui.boot() + tui.submit(task.prompt) + tui.task(evidence, task) + tui.exit_normally() + task.assert_completed(session, CLAUDE) + assert_inference_evidence(recorder, 0, CLAUDE, task, model) + settings = _read_managed(session, CLAUDE) + _assert_rewritten(settings, CLAUDE, recorder) + _assert_contains(settings, seeded) + session.assert_not_routed() + _assert_revert_restores(session, CLAUDE, seeded, "family-defaults-tui-revert") + + @pytest.mark.codex + @pytest.mark.parametrize("model_position", ["before_separator", "exec"]) + def test_ug_codex_headless_fresh_model_location( + self, live_session, request_recorder, model_position + ): + """Scenario: launch fresh scoped Codex with --model before ug's separator or in exec. + + Expected: GPT Luna completes a file task without prior configuration or routing. + """ + session = live_session + task = FileTask(session) + model = CODEX_LUNA_MODEL_SERVICE + model_args = ["--model", model] + evidence = SessionEvidence(session.home, CODEX) + + result = session.run( + CODEX, + "--workspace", + request_recorder.url, + "--model-location", + MODEL_SERVICE_SCHEMA, + *(model_args if model_position == "before_separator" else []), + "--", + "exec", + "--skip-git-repo-check", + "--json", + *(model_args if model_position == "exec" else []), + task.prompt, + timeout=240, + ) + task.assert_headless_answer(CODEX, result) + turn = evidence.completed(task) + assert turn and set(turn.models) == {model}, turn + assert_inference_evidence( + request_recorder, 0, CODEX, task, model, parent_schema=MODEL_SERVICE_SCHEMA + ) + session.assert_not_routed() + + @pytest.mark.codex + @pytest.mark.parametrize("model_position", ["before_separator", "exec"]) + def test_ug_codex_preserves_preexisting_managed_config( + self, live_session, managed_input, model_position + ): + """Scenario: select GPT Luna before ug's separator or in exec over OS-managed admin config. + + Expected: ug preserves every admin setting, and the scoped Luna service completes a + file task without routing. + """ + session = live_session + task = FileTask(session) + model = CODEX_LUNA_MODEL_SERVICE + model_args = ["--model", model] + seeded, recorder = managed_input + evidence = SessionEvidence(session.home, CODEX) + result = session.run( + CODEX, + "--workspace", + recorder.url, + "--model-location", + MODEL_SERVICE_SCHEMA, + *(model_args if model_position == "before_separator" else []), + "--", + "exec", + "--skip-git-repo-check", + "--json", + *(model_args if model_position == "exec" else []), + task.prompt, + timeout=240, + ) + task.assert_headless_answer(CODEX, result) + turn = evidence.completed(task) + assert turn and set(turn.models) == {model}, turn + assert_inference_evidence( + recorder, 0, CODEX, task, model, parent_schema=MODEL_SERVICE_SCHEMA + ) + _assert_contains(_read_managed(session, CODEX), seeded) + session.assert_not_routed() + + @pytest.mark.codex + @pytest.mark.tui + def test_ug_codex_tui_rewrites_managed_config_preserving_admin_settings( + self, live_session, managed_input + ): + """Scenario: select scoped GPT Luna in an interactive launch over OS-managed admin config. + + Expected: unlike headless launches, the TTY lets ug rewrite the OS-managed file; the + rewrite keeps every admin setting, the TUI task reaches the scoped Luna service, and + revert restores the seeded file. + """ + session = live_session + task = FileTask(session) + model = CODEX_LUNA_MODEL_SERVICE + seeded, recorder = managed_input + evidence = SessionEvidence(session.home, CODEX) + with Terminal( + session, + "managed-config-tui", + [ + CODEX, + "--workspace", + recorder.url, + "--model-location", + MODEL_SERVICE_SCHEMA, + "--model", + model, + ], + ) as tui: + tui.boot() + tui.submit(task.prompt) + tui.task(evidence, task) + tui.exit_normally() + turn = evidence.completed(task) + assert turn and set(turn.models) == {model}, turn + assert_inference_evidence( + recorder, 0, CODEX, task, model, parent_schema=MODEL_SERVICE_SCHEMA + ) + config = _read_managed(session, CODEX) + _assert_rewritten(config, CODEX, recorder) + _assert_contains(config, seeded) + session.assert_not_routed() + _assert_revert_restores(session, CODEX, seeded, "managed-config-tui-revert") diff --git a/tests/integration/README.md b/tests/integration/README.md index cd4809d9c..100395a0f 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -80,6 +80,10 @@ before and after ug's separator, with no workspace policy. It requires a complet file task, the requested model with output tokens in JSON `modelUsage`, and no routing wrapper. Use the runner's `--claude-model` option to reproduce a specific model. +Codex covers `--model VALUE`, `--model=VALUE`, and `-m VALUE` before ug's separator +and within native `exec`, asserting the completed turn's selected model. Codex has +no UG-owned `--model` option: all three spellings are forwarded to Codex in either position. + The `smart_defaults` wire schema, legacy `spend_tiers` cache reads, and recommendation request gating are covered by unit/component tests listed in `../README.md`. This suite does not yet assert live `recommendModel` request counts for configs with and without tiers. @@ -279,8 +283,8 @@ test_ug_configure_claude_workspace_switch.py # real skills MCP cleanup across tw test_ug_configure_codex_lifecycle.py # repeat setup, revert, rejected credentials test_ug_claude_managed_model_discovery.py # Claude MPS policy cases (claude_mps.json) test_ug_codex_managed_model_discovery.py # Codex MPS policy cases (codex_mps.json) -test_ug_claude_model_discovery.py # unmanaged scenarios 7, 9, 11, 13 -test_ug_codex_model_discovery.py # unmanaged scenarios 8, 10, 12, 14 +test_ug_claude_model_discovery.py # unmanaged scenarios 9, 11, 13 +test_ug_codex_model_discovery.py # unmanaged scenarios 10, 12, 14 test_ug_configure_managed.py # stubbed static model list/catalog pointer, no agent selector, cache TTL test_ug_configure_managed_models.py # stubbed model sources, smart-routing banner, Codex fallback metadata test_ug_configure_managed_mcp.py # stubbed managed MCP list @@ -388,6 +392,9 @@ reproduce another existing service. Use `--claude-provider-model` / No service is created or modified. A missing service, permission, or OAuth token fails the selected CUJ, rather than skipping it. +The fresh `--provider` journeys use these preprovisioned services to check provider selection +and real agent CLI launch, without inference. Codex also checks its app-server model catalog. + The tracing journeys are part of their respective Full agent lanes and use the existing e2e workspace and bearer. Because that workspace deliberately has no published managed configuration, each journey injects only a tracing-enabled CodingAgentConfig input (the @@ -441,11 +448,13 @@ provider or parent replacement picker can render the raw gateway ID/display name the assertion accepts either numbered-row form. Custom Model Services must still appear by their gateway IDs or display names in a numbered picker row; startup banners and footer text cannot satisfy discovery assertions. Cases 7–14 send no inference prompts; -they only configure, list models, and open/close the picker. Other live CUJs perform -real model tasks. +they only configure, list models, and open/close the picker. The two dedicated CUJ7 fresh-state +cases instead complete real file tasks through `--workspace` and `--model-location`, using +`ug_e2e.models.claude_haiku` and `ug_e2e.models.gpt_luna`. The two fresh `--provider` +journeys check setup and launch with existing services, without inference. -There are **80 live cases** (including 14 marked TUI journeys) and **7 installation -checks** with Claude and Codex; selecting OpenCode adds one live headless case. One +There are **85 live cases** (including 11 marked TUI journeys) and **7 installation +checks** with Claude and Codex; selecting OpenCode adds one live headless case and two `managed_fixture` cases. One **`workspace_switch` case** uses two real workspaces and checks skills MCP cleanup and a completed Claude task. A further **43 `managed_fixture` cases** (ten of them also `live`) run on the managed workspace with a checked-in JSON CodingAgentConfig from `tests/fixtures/managed_config/` @@ -463,8 +472,11 @@ Direct renderer tests cover default/catalog composition, while focused CLI regre catalog discovery with overall defaults, family defaults, or both, along with explicit model selection and preservation of static model lists. Both cases run on the managed workspace's own bearer; no second workspace or extra secret is involved. -The 14 retained numbered scenarios comprise 24 explicit journeys: 12 managed and 12 unmanaged -executions; the complete integration suite collects 121 executions. See the named coverage and gaps matrix in +Across integration and dedicated CUJ7, the 14 numbered scenarios comprise 24 journeys: +12 managed and 12 unmanaged. Integration contains 22 of them (12 managed, 10 unmanaged); +configured unmanaged cases 7 and 8 run in `e2e_cuj/test_cuj7_model_discovery.py`. +The complete integration suite collects 129 executions (126 with Claude and Codex selected). +See the named coverage and gaps matrix in [../README.md](../README.md). ```bash @@ -597,6 +609,40 @@ Claude lane also passes `CLAUDE_CODE_OAUTH_TOKEN` (the same secret the e2e workf uses) for the relayed hybrid CUJ. Fork PRs run installation checks only because they cannot receive those secrets. +CUJ7's eight journeys collect twelve cases, six per agent, in `tests/e2e_cuj/test_cuj7_model_discovery.py`: +configured Claude picker and Codex app-server discovery, fresh Claude/Codex model-location file +tasks, and preservation of pre-existing OS-managed input (Claude family defaults, Codex admin +settings) on headless and interactive launch. The class pins `WORKSPACE_URL` to +`https://dbc-14e376e8-6541.cloud.databricks.com`; the required +`E2E CUJs · CUJ 7 · Unmanaged model discovery` shard uses +the shared `UG_CUJ_SP_CLIENT_ID` / `UG_CUJ_SP_CLIENT_SECRET` credentials, not a workspace +secret. The workspace must publish no `CodingAgentConfig`, expose discoverable `system.ai` +models, and retain the existing models `ug_e2e.models.claude_haiku`, +`ug_e2e.models.claude_sonnet`, and `ug_e2e.models.gpt_luna`; the shared service principal +needs read/use privileges on all three. +The task journeys exercise model selection before and after ug's separator, including +Claude's UG-owned `--model` and its native option. All twelve cases reuse the CUJ +session/terminal helpers and check that the workspace remains unmanaged; they never +change remote configuration. Function-scoped sessions keep every launch fresh, mint +a bearer per case, and use shared `revert_machine_wide` cleanup. +Both agents correlate the exact user task to requests for the expected UC service and +nonempty HTTP 200 responses through the shared recorder. Scoped launches also require +the `databricks-model-service-parent-schema` header to equal `ug_e2e.models`. +Claude checks structured `modelUsage`; Codex checks its completed native turn's model. +The preservation case seeds distinct Sonnet and other-family defaults only after the +clean-runner preflight, verifies Sonnet inference and unchanged defaults, then reverts +and removes that input. Recording stops before guarded revert; fixture teardown keeps +cleanup failures separate from task failures. An accidental Opus fallback cannot satisfy +the Sonnet assertion. +Headless launches have no TTY, so ug leaves the OS-managed file untouched unless it conflicts; +only an interactive launch rewrites it. Each agent has a matching headless and interactive +preservation pair over a shared seeded-file fixture. Claude seeds distinct family defaults; +Codex seeds a sibling `AdminProvider` model provider and `file_opener`, both outside ug's owned +keys. Every case requires each seeded leaf unchanged plus task-correlated inference through the +recorder (Codex also with the parent-schema header). The interactive cases additionally require +ug's gateway base URL in the rewritten file, proving the write happened, and require `ug revert` +to restore the seeded document. + The workspace check requires the secret to match `https://eng-ml-inference-team-us-east-1.cloud.databricks.com` (a trailing slash is accepted). It never changes the secret or switches workspaces. There is no @@ -607,13 +653,13 @@ each test; only explicit-model scenarios choose and record a discovered Every same-repository PR and push to `main` runs **Smoke journeys**, followed by **Full journeys** even if smoke fails. Smoke runs the Hosted configure/TUI, headless argument, and custom OAuth CLI TUI journeys for each agent (six cases, -two agent jobs). Full runs all 80 live cases, including those smoke cases, in two +two agent jobs). Full runs all 85 live cases, including those smoke cases, in two disjoint agent lanes: | Agent lane | Marker | Cases | | --- | --- | --- | -| Claude | `live and claude` | 38 | -| Codex | `live and codex` | 42 | +| Claude | `live and claude` | 39 | +| Codex | `live and codex` | 46 | A non-blocking **OpenCode** job (`live and opencode`, one case) runs alongside them with `continue-on-error` and is not part of the required `cujs` gate until it is stable. @@ -922,7 +968,7 @@ uv run --no-project --python 3.12 python scripts/run_integration.py \ unset DATABRICKS_BEARER ``` -This runs all 80 live cases. For the seven installation checks, run the same +This runs all 85 live cases. For the seven installation checks, run the same runner/version/index arguments with `--installation-only` and omit `-- -m live`; no bearer or workspace is needed. Results remain under `.integration-runs/`. Each invocation needs a new output directory; an existing one is rejected. diff --git a/tests/integration/test_ug_claude_headless.py b/tests/integration/test_ug_claude_headless.py index f94a4a991..66ea2d5c7 100644 --- a/tests/integration/test_ug_claude_headless.py +++ b/tests/integration/test_ug_claude_headless.py @@ -3,6 +3,7 @@ import json import pytest +from utils.constants import CLAUDE_TEST_MODEL from utils.evidence import FileTask pytestmark = [pytest.mark.live, pytest.mark.claude] @@ -114,6 +115,65 @@ def test_ug_claude_headless_prompt_after_separator(live_session, workspace): session.assert_not_routed() +@pytest.mark.usefixtures("unmanaged_workspace") +def test_ug_claude_headless_fresh_workspace(live_session, workspace): + """Scenario: launch Claude headlessly against an unmanaged workspace from fresh state. + + Expected: ``ug claude --workspace`` starts the real installed Claude CLI without a + configure step, and its structured result contains the unpredictable fixture value after + using the Read tool with the inexpensive Haiku model; the command exits without routing. + """ + session = live_session + task = FileTask(session) + + result = session.run( + "claude", + "--workspace", + workspace, + "--", + "--model", + CLAUDE_TEST_MODEL, + "-p", + task.prompt, + "--output-format", + "json", + "--allowedTools", + "Read", + timeout=180, + ) + task.assert_headless_answer("claude", result) + usage = json.loads(result.stdout)["modelUsage"] + assert set(usage) == {CLAUDE_TEST_MODEL}, usage + assert usage[CLAUDE_TEST_MODEL]["outputTokens"] > 0, usage + session.assert_not_routed() + + +@pytest.mark.usefixtures("unmanaged_workspace") +def test_ug_claude_fresh_provider_launch(live_session, workspace, claude_provider): + """Scenario: launch Claude from fresh state with an explicit provider service. + + Expected: ``ug claude --workspace`` with ``--provider`` starts the real installed Claude + CLI without a configure step, reports the selected provider, and writes the provider header + to its generated settings; ``--version`` exits successfully without inference or routing. + """ + session = live_session + result = session.run( + "claude", + "--workspace", + workspace, + "--provider", + claude_provider, + "--", + "--version", + timeout=180, + ) + assert claude_provider in result.stdout + settings = json.loads((session.home / ".claude/ucode-settings.json").read_text()) + headers = (settings.get("env") or {}).get("ANTHROPIC_CUSTOM_HEADERS", "").splitlines() + assert f"Databricks-Model-Provider-Service: {claude_provider}" in headers + session.assert_not_routed() + + @pytest.mark.parametrize("model_form", ["separate", "equals"]) @pytest.mark.parametrize("model_owner", ["ug", "claude"]) def test_ug_claude_headless_explicit_model_bypasses_routing( diff --git a/tests/integration/test_ug_claude_model_discovery.py b/tests/integration/test_ug_claude_model_discovery.py index 2289da231..5c513a27b 100644 --- a/tests/integration/test_ug_claude_model_discovery.py +++ b/tests/integration/test_ug_claude_model_discovery.py @@ -1,10 +1,13 @@ -"""Claude model-discovery CUJs for repository scenarios 7, 9, 11, and 13.""" +"""Claude model-discovery CUJs for repository scenarios 9, 11, and 13.""" import json import re import pytest -from utils.model_discovery import claude_model_in_picker, claude_system_model_ids +from utils.model_discovery import ( + assert_claude_system_models_in_picker as _assert_system_models_in_picker, +) +from utils.model_discovery import claude_model_in_picker from utils.terminal import AgentTerminal pytestmark = [pytest.mark.claude, pytest.mark.usefixtures("unmanaged_workspace")] @@ -53,25 +56,6 @@ def _system_models_visible(session): return lambda _text: session.claude_gateway_cache_ready() -def _assert_system_models_in_picker(session, screen): - models = session.claude_gateway_models() - ids = claude_system_model_ids(models) - discovered = session.workspace_state()["claude_models"] - assert discovered, "ug configure found no Claude system.ai models" - # ug caches the newest system.ai model per family from UC model-services, which can list a - # just-released model (e.g. a new opus tier) before Claude Code's own gateway-model discovery - # surfaces it in the picker cache. Require the cached defaults to be well-formed system.ai - # Claude ids that overlap Claude Code's catalog, rather than a strict subset of a catalog that - # can legitimately lag behind UC model-services. - assert all(model_id.startswith("system.ai.claude-") for model_id in discovered.values()), ( - discovered - ) - assert set(discovered.values()) & set(ids), (discovered, models) - assert any( - claude_model_in_picker(screen, model["id"], model.get("display_name")) for model in models - ), screen - - def _assert_replacement_picker(session, expected_ids): settings = json.loads((session.home / ".claude" / "ucode-settings.json").read_text()) assert not {"availableModels", "enforceAvailableModels"} & settings.keys(), settings @@ -80,35 +64,6 @@ def _assert_replacement_picker(session, expected_ids): assert [option["model"] for option in picker["options"]] == expected_ids, picker -@pytest.mark.live -@pytest.mark.tui -def test_case_07_configured_claude_discovers_system_models(live_session, workspace): - """Scenario: configure Claude, then launch without source overrides or discovery flags. - - Expected: native discovery caches system.ai models as raw IDs or recognized Claude - gateway aliases and shows a discovered picker entry. - """ - session = live_session - session.run( - "configure", - "--agents", - "claude", - "--workspace", - workspace, - "--skip-upgrade", - "--disable-databricks-ai-tools", - timeout=240, - ) - - command = [str(session.binary), "claude"] - with AgentTerminal(session, "claude", command, "case-07-system-models") as tui: - tui.boot() - screen = tui.open_model_picker(model_visible=_system_models_visible(session)) - tui.exit_normally() - - _assert_system_models_in_picker(session, screen) - - @pytest.mark.live @pytest.mark.tui def test_case_09_fresh_claude_discovers_system_models(live_session, workspace): diff --git a/tests/integration/test_ug_codex_headless.py b/tests/integration/test_ug_codex_headless.py index 289c138eb..5bb5221df 100644 --- a/tests/integration/test_ug_codex_headless.py +++ b/tests/integration/test_ug_codex_headless.py @@ -4,7 +4,7 @@ import pytest from utils.constants import CODEX_TEST_MODEL -from utils.evidence import FileTask +from utils.evidence import FileTask, assert_completed_task_model pytestmark = [pytest.mark.live, pytest.mark.codex] @@ -122,11 +122,14 @@ def test_ug_codex_headless_prompt_after_separator(live_session, workspace): @pytest.mark.parametrize("model_form", ["separate", "equals", "short"]) -def test_ug_codex_headless_explicit_model_bypasses_routing(live_session, workspace, model_form): - """Scenario: choose an explicit model while global smart routing is enabled. - - Expected: the model option is accepted, the real file task completes, and - no routing wrapper overrides the caller's choice. +@pytest.mark.parametrize("model_position", ["before_separator", "exec"]) +def test_ug_codex_headless_explicit_model_bypasses_routing( + live_session, workspace, model_form, model_position +): + """Scenario: choose a model before ug's separator or within Codex exec. + + Expected: the completed file-task turn selects the requested model, and + no routing wrapper overrides it. """ session = live_session session.choose_codex_windows_sandbox() @@ -148,13 +151,68 @@ def test_ug_codex_headless_explicit_model_bypasses_routing(live_session, workspa session.env["ENABLE_SMART_ROUTING_V2"] = "1" result = session.run( "codex", + *(model_args if model_position == "before_separator" else []), "--", "exec", "--skip-git-repo-check", "--json", task.prompt, - *model_args, + *(model_args if model_position == "exec" else []), timeout=180, ) task.assert_headless_answer("codex", result) + assert_completed_task_model(session, "codex", task.value, model) + session.assert_not_routed() + + +@pytest.mark.usefixtures("unmanaged_workspace") +def test_ug_codex_headless_fresh_workspace(live_session, workspace): + """Scenario: launch Codex headlessly against an unmanaged workspace from fresh state. + + Expected: ``ug codex --workspace`` starts the real installed Codex CLI without a + configure step, and its structured completed answer contains the unpredictable fixture + value after using the workspace model; the command exits successfully without routing. + """ + session = live_session + session.choose_codex_windows_sandbox() + task = FileTask(session) + + result = session.run( + "codex", + "--workspace", + workspace, + "--", + "exec", + "--skip-git-repo-check", + "--json", + "--model", + CODEX_TEST_MODEL, + task.prompt, + timeout=180, + ) + task.assert_headless_answer("codex", result) + assert_completed_task_model(session, "codex", task.value, CODEX_TEST_MODEL) + session.assert_not_routed() + + +@pytest.mark.usefixtures("unmanaged_workspace") +def test_ug_codex_fresh_provider_launch(live_session, workspace, codex_provider): + """Scenario: launch Codex from fresh state with an explicit provider service. + + Expected: ``ug codex --workspace`` with ``--provider`` starts the real installed Codex + CLI without a configure step and reports the selected provider; ``--version`` exits + without inference or routing. Case 12 covers the provider's app-server model catalog. + """ + session = live_session + result = session.run( + "codex", + "--workspace", + workspace, + "--provider", + codex_provider, + "--", + "--version", + timeout=180, + ) + assert codex_provider in result.stdout session.assert_not_routed() diff --git a/tests/integration/test_ug_codex_model_discovery.py b/tests/integration/test_ug_codex_model_discovery.py index 178e10657..8f27bdf9e 100644 --- a/tests/integration/test_ug_codex_model_discovery.py +++ b/tests/integration/test_ug_codex_model_discovery.py @@ -1,9 +1,9 @@ -"""Codex model-discovery CUJs for repository scenarios 8, 10, 12, and 14.""" +"""Codex model-discovery CUJs for repository scenarios 10, 12, and 14.""" import os -import tomllib import pytest +from utils.model_discovery import assert_codex_default_models as _assert_default_models from utils.provider_catalog import fetch_codex_parent_catalog pytestmark = [pytest.mark.codex, pytest.mark.usefixtures("unmanaged_workspace")] @@ -19,43 +19,6 @@ def _codex_parent_catalog(unmanaged_workspace, parent_schema, codex_parent_model return catalog -def _assert_default_models(session, models): - discovered = session.workspace_state()["codex_models"] - assert discovered and all(model.startswith("system.ai.") for model in discovered) - config = tomllib.loads((session.home / ".codex/ucode.config.toml").read_text()) - assert "model" not in config, config - assert "model_reasoning_effort" not in config, config - assert "model_catalog_json" not in config, config - assert models and len(models) == len(set(models)), models - assert any(model.startswith("gpt-") for model in models), models - assert not list((session.home / ".ucode").glob("codex-model-catalog-*.json")) - - -@pytest.mark.live -def test_case_08_configured_codex_uses_default_models(live_session, workspace): - """Scenario: configure Codex, then launch without source overrides. - - Expected: unmanaged configuration leaves model selection to Codex's native default; - ug records system.ai discovery while model and reasoning preferences remain unset; - app-server exposes native GPT entries without a generated provider/parent-scoped catalog. - """ - session = live_session - session.run( - "configure", - "--agents", - "codex", - "--workspace", - workspace, - "--skip-upgrade", - "--disable-databricks-ai-tools", - timeout=240, - ) - - models = session.codex_model_ids(["app-server", "--listen", "stdio://"]) - - _assert_default_models(session, models) - - @pytest.mark.live def test_case_10_fresh_codex_uses_default_models(live_session, workspace): """Scenario: launch fresh Codex with --workspace and no source overrides. diff --git a/tests/integration/utils/evidence.py b/tests/integration/utils/evidence.py index 5ea9d189a..79ec8dd57 100644 --- a/tests/integration/utils/evidence.py +++ b/tests/integration/utils/evidence.py @@ -24,7 +24,7 @@ def assert_no_terminal_api_error(screen: str) -> None: def read_jsonl(path: Path) -> list[dict]: if not path.is_file(): return [] - text = path.read_text() + text = path.read_text(encoding="utf-8") lines = text.splitlines(keepends=True) records = [] for index, line in enumerate(lines): diff --git a/tests/integration/utils/model_discovery.py b/tests/integration/utils/model_discovery.py index 926dc063b..ae805cbba 100644 --- a/tests/integration/utils/model_discovery.py +++ b/tests/integration/utils/model_discovery.py @@ -2,6 +2,7 @@ import hashlib import re +import tomllib _CLAUDE_GATEWAY_ALIAS = re.compile(r"^anthropic-aigw-[0-9a-fA-F]{8}-(?P.*)$") _CLAUDE_PICKER_ROW = re.compile(r"(?m)^[ \t]*(?:[❯›>][ \t]*)?\d+\.[ \t]+(?P