AIGTWY-4878 [2/3] Test catalog discovery and configured defaults

[tt-le]

Summary

Catalog discovery consumer, stacked on the behavior-preserving helper extraction #968. Uses the preconfigured CUJ3 workspace; provisioning #964 remains separate.

• Require Claude's Sonnet/Haiku/Kimi set and Codex's GPT Luna/Kimi set under ug_e2e.models; exclude Gemini and ug_e2e.other_models decoys from scoped discovery and native pickers.
• Inspect gateway catalogs and native pickers, then complete separate default and explicit-model tasks through real TUI and headless agent commands.
• Use the shared dedicated-workspace cuj fixture, authenticated SDK client, and isolated local sessions. Catalog reads use SDK GET calls; workspace configuration remains read-only.
• Run through the existing dedicated-cuj CI job. Remove the custom catalog lane and bespoke integration-runner mode; shared workflow and conftest match current main.

Stack

Rebased onto current main through #968. MCP registration #990 is the child consumer stack; its changes are not included here.

Validation

• 133 focused runner/evidence/model-discovery/provider-catalog tests passed.
• All 10 catalog cases collect; consumer Ruff lint/format and git diff --check pass.
• No new live inference pass. Fresh SP credentials and a clean disposable runner are still required.
• Previously observed Claude default-banner/picker failures remain unresolved; migration does not weaken those assertions.
• Claude model evidence is response-reported; Codex evidence joins its client-selected model to the completed turn. Neither proves the gateway's backing destination.

Skills discovery remains deferred.

[github-actions]

UG review

This test-only change exercises scoped catalogs, native model pickers, configured defaults, and completed-task evidence. The main risk is making live CI assertions stricter than the successful agent behavior they intend to verify.

Major

• Allow recovered inference retries — tests/e2e_cuj/test_catalog_discovery.py:134: can we accept a successful retry instead of requiring every task request to return 200, so an agent that recovers from a 429/503 doesn't fail this catalog test?

Some patches were unavailable or truncated to fit the review context. Treat this as a partial review.

Automated advisory review of c8de99811443 using Lilly's UG review rubric. It does not approve or block this PR.

[david-siqi-liu]

Reviewed against main's tests/e2e_cuj/AGENTS.md (#1000) and the CUJ 3 plan in the UG Configure E2E doc.

P0

1. Machine-wide agent files are never cleaned up. The test launches AgentTerminal five times (test_catalog_discovery.py lines 114, 160, 190, 211, 235) but never runs a terminal ug revert or checks MANAGED_PATHS. On the current head the E2E CUJs job fails all 10 tests: after the first launch, every later configure hits Non-interactive CUJ configure unexpectedly created machine-wide settings (https://github.com/databricks/unity-gateway/actions/runs/37476090218/job/112311886645). The same leftovers make #990's MCP tests error at setup with Existing machine-wide agent settings, and once this lands on main they would break any CUJ that runs after it in the shared job. #911 adds a terminal ug revert to the cuj fixture teardown, but cuj is class scoped, so this class still needs to revert, or to finish all configures before its first terminal launch.
2. The recorder is unpacked but unused (session, workspace, _recorder = cuj, line 103), so model evidence is native records only. AGENTS.md: "Model evidence combines observed gateway requests/responses with native completed-turn records; neither routing banners nor native records alone prove an applied decision." recorder.configure_session(...) plus recorder.requests_after(checkpoint) checked against INFERENCE_PATHS and the request model covers it.
3. AgentTerminal.wait_for_task (line 133) answers a recognized find permission prompt with Enter (tests/integration/utils/terminal.py 352-369). AGENTS.md requires rejecting unexpected permission prompts. main's helpers/terminal.py Terminal.task does that, and so does #990's _wait_for_mcp_task.
4. Coverage: the plan's ug mcp list and ug status checks aren't in this stack (#990's body lists them as a follow-up). They don't depend on the skills blocker, so I'd include them before calling CUJ 3 covered.

The first failing test in that run also reports Unmatched or ambiguous claude picker row 'Default (recommended) Use the default model (currently Opus 5.5 (1M context)) · $4/$20 per Mtok': []. That one looks separate from the cleanup failures.

P1

• Only the existence of exactly one published config is checked, not its contents (schema location, defaults). A read-only check on workspace.config() up front would turn a misconfigured workspace into a clear failure. #990 needs the same check, so this could be one shared helper.
• The MCP check uses two hardcoded services instead of enumerating ug_e2e.tools, so the plan's "every MCP in ug_e2e.tools" would miss a newly added one.

Skills discovery and invocation are deferred per the PR bodies; that's the same Files API blocker CUJ 2 and CUJ 6 hit, so I'm treating it as a shared follow-up.

[david-siqi-liu]

Rechecked at 2803a02: the earlier blockers are fixed (per-test revert, recorder plus native-turn evidence, Terminal.task). All 10 catalog tests pass; the red E2E CUJs here is CUJ 2's Claude inference test, which passed on the same code in #990's run. ug status coverage stays as a follow-up in AIGTWY-4972.