AIGTWY-4878 [3/3] Test MCP registration and tool use in Claude and Codex by tt-le · Pull Request #990 · databricks/unity-gateway

tt-le · 2026-10-05T19:16:09Z

Summary

Stacked on #965. Adds real MCP registration, connection, schema-exclusion, listing, and read-only tool-use journeys for Claude and Codex in the CUJ3 workspace.

Configure installed UG using the shared cuj fixture and inspect Claude's MCP menu and Codex's /mcp verbose inventory.
Require ug_e2e.tools.fixture_reader and fixture_metadata to be connected and expose their expected tools.
Require the bounded inventories to omit ug_e2e.other_tools and decoy_status.
Have each agent call both read-only tools with a fresh nonce. Verify receipts in a completed parent assistant answer; expected receipts are withheld from the prompt. Codex approvals are limited to the expected server, tool, and nonce using per-call Allow.
Check ug mcp list reports both managed fixtures for claude, codex with aggregate connected status and no out-of-scope servers. Actual Codex connectivity is established by its TUI/tool journey, not the aggregate label.
Use shared session/auth/terminal scaffolding and the existing dedicated-cuj CI job. No MCP-specific workflow or CI-gate contract-test changes remain.

Known issue: ug status

ug status can show global system.ai models for Codex instead of the managed ug_e2e.models catalog, and Claude's inventory can collapse to its configured default. This affects the status model inventory; it is separate from these native MCP inventory and tool-use checks.

The standalone production fix is #1019, targeting main outside this test stack. ug status CUJ coverage remains deferred; this PR does not change status implementation or claim to test it.

Validation

Three MCP cases and all 13 combined catalog/MCP cases collect.
Focused stack evidence/model-discovery/contract tests: 163 passed.
Consumer Ruff lint/format, git diff --check, and stack ancestry checks passed after restacking.
Live passes are required to establish coverage; current results are tracked by CI. Local live runs require credentials and a clean disposable runner; no policy bypass is used.

Scope

Test-only consumer: MCP CUJ, inventory helper, marker, and documentation. Shared workflow/conftest and parent agent helpers are unchanged in this PR. Fixtures are provisioned externally. Skills remain deferred. No fixture scripts or credentials are committed. The unrelated scrollable-picker README note remains uncommitted.

github-actions · 2026-10-06T14:40:52Z

UG review

This test-only change exercises managed MCP inventories, tool receipts, and cleanup in Claude and Codex. The Codex approval handler can stall valid repeated tool calls.

Major

Permit repeated per-call MCP approvals — tests/e2e_cuj/test_mcp_registration.py:113: can we reset deduplication after an approval menu closes so a retry with the same server, tool, and nonce gets its own per-call approval instead of timing out?

Automated advisory review of 1df7a6e07c0f using Lilly's UG review rubric. It does not approve or block this PR.

david-siqi-liu

Reviewed against main's tests/e2e_cuj/AGENTS.md (#1000) and the CUJ 3 plan in the UG Configure E2E doc. Coverage gaps for the whole CUJ 3 stack are in my comment on #965.

P0

On the current head (ee92991) both MCP tests error at setup with Existing machine-wide agent settings; use a clean disposable runner (https://github.com/databricks/unity-gateway/actions/runs/37477191394/job/112315997247). The files come from #965's terminal launches earlier in the same run (see my comment on #965), so this clears once #965 cleans up after itself.

P1

The Codex journey depends on #1006: at this head managed_mcp_entry emits only command and args, and #1006 adds the DATABRICKS_BEARER / DATABRICKS_BEARER_COMMAND env_vars. The PR body already says so. Flagging it because #1006 targets main and isn't in this stack, so it has to merge first, then this stack rebased.
The test never checks that the published config uses unity_catalog_location == "ug_e2e.tools" or that the decoy MCP exists. If the decoy is missing or the config names the two fixtures explicitly, the "ug_e2e-other_tools" not in inventory and "decoy_status" not in inventory assertions pass without testing schema discovery. A read-only SDK GET for the decoy plus a check on workspace.config() would close it.

david-siqi-liu

Rechecked at 57deeec: per-test revert and the mcp list case are in, and E2E CUJs passes with all CUJ 3 tests (10 catalog, 3 MCP registration).

…cks#968) ## Summary Behavior-preserving extraction only: move existing Claude/Codex answer parsing, session paths, and child-session detection into per-agent modules. Shared evidence callers and existing journeys retain their behavior and names. Removed `tests/test_integration_agent_helpers.py`: the existing public evidence tests already cover these paths. No new model attribution, catalog requests, HTTP helpers, picker behavior, managed-config readers, or CUJ CI changes belong to this PR. ## Stack Targets main; rebased onto current main. Catalog discovery databricks#965 targets this branch, and MCP registration databricks#990 targets databricks#965. Provisioning databricks#964 remains parked separately. ## Validation - 50 existing evidence/contract tests passed on this parent after removal. - `git diff --check` passed; commit/push secret scans passed. - No live inference run.

…abricks#965) ## Summary Catalog discovery consumer, stacked on the behavior-preserving helper extraction databricks#968. Uses the preconfigured CUJ3 workspace; provisioning databricks#964 remains separate. - Require Claude's Sonnet/Haiku/Kimi set and Codex's GPT Luna/Kimi set under `ug_e2e.models`; exclude Gemini and `ug_e2e.other_models` decoys from scoped discovery and native pickers. - Inspect gateway catalogs and native pickers, then complete separate default and explicit-model tasks through real TUI and headless agent commands. - Use the shared dedicated-workspace `cuj` fixture, authenticated SDK client, and isolated local sessions. Catalog reads use SDK GET calls; workspace configuration remains read-only. - Run through the existing `dedicated-cuj` CI job. Remove the custom catalog lane and bespoke integration-runner mode; shared workflow and conftest match current main. ## Stack Rebased onto current main through databricks#968. MCP registration databricks#990 is the child consumer stack; its changes are not included here. ## Validation - 133 focused runner/evidence/model-discovery/provider-catalog tests passed. - All 10 catalog cases collect; consumer Ruff lint/format and `git diff --check` pass. - No new live inference pass. Fresh SP credentials and a clean disposable runner are still required. - Previously observed Claude default-banner/picker failures remain unresolved; migration does not weaken those assertions. - Claude model evidence is response-reported; Codex evidence joins its client-selected model to the completed turn. Neither proves the gateway's backing destination. Skills discovery remains deferred.

tt-le requested review from AarushiShah-db, lilly-luo and rohita5l as code owners October 5, 2026 19:16

tt-le added the ai-assisted label Oct 5, 2026

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from eb5054c to 669c938 Compare October 5, 2026 20:08

tt-le added this pull request to stack #969 October 6, 2026 13:21

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from 36f6544 to abe9ac4 Compare October 6, 2026 13:44

This was referenced Oct 6, 2026

AIGTWY-4878 [1/3] Extract existing native transcript helpers #968

Merged

AIGTWY-4878 [2/3] Test catalog discovery and configured defaults #965

Merged

tt-le changed the title ~~AIGTWY-4878 Test MCP registration and tool use in Claude and Codex~~ AIGTWY-4878 [3/3] Test MCP registration and tool use in Claude and Codex Oct 6, 2026

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from abe9ac4 to d732fc0 Compare October 6, 2026 14:05

tt-le mentioned this pull request Oct 6, 2026

AIGTWY-4878 Forward external Databricks auth to Codex MCP proxies #1006

Closed

david-siqi-liu added the ug-review Run the automated UG review label Oct 6, 2026

This was referenced Oct 6, 2026

[AIGTWY-4880] CUJ5: Verify budget recommendations and agent startup #911

Merged

[AIGTWY-4877] Add MPS and MCP two-agent CUJ #993

Merged

david-siqi-liu reviewed Oct 6, 2026

View reviewed changes

masonc08 mentioned this pull request Oct 6, 2026

[AIGTWY-4877] Exercise CUJ2 MCP tool use in both agents #1015

Draft

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from ee92991 to 209f748 Compare October 7, 2026 02:57

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch 2 times, most recently from 6989ec9 to 6bb0c60 Compare October 7, 2026 04:01

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from 9e68088 to ee77684 Compare October 7, 2026 05:13

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from ee77684 to 0f847a1 Compare October 7, 2026 05:28

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from 0f847a1 to 57deeec Compare October 7, 2026 05:31

david-siqi-liu approved these changes Oct 7, 2026

View reviewed changes

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from 57deeec to aa02815 Compare October 7, 2026 12:46

Base automatically changed from tt-le/aigtwy-4878-cuj3-model-tests to main October 7, 2026 13:27

tt-le added 7 commits October 7, 2026 09:27

AIGTWY-4878 Test MCP registration and read-only tool use in agent TUIs

5586380

AIGTWY-4878 Fix MCP discovery CI and forward Codex proxy auth

a6d1c37

Use shared dedicated CUJ framework for MCP registration

63b76f5

Move Codex MCP auth fix into separate prerequisite PR

91ba621

Revert managed MCP test settings after each agent journey

8cc7a28

Test managed MCP listing for both agents

0e93af3

Approve expected Codex MCP fixture calls through the TUI

1df7a6e

tt-le force-pushed the tt-le/aigtwy-4878-mcp-registration branch from aa02815 to 1df7a6e Compare October 7, 2026 13:27

tt-le added this pull request to the merge queue Oct 7, 2026

Merged via the queue into main with commit 79b04ca Oct 7, 2026
49 of 52 checks passed

tt-le deleted the tt-le/aigtwy-4878-mcp-registration branch October 7, 2026 13:52

david-siqi-liu mentioned this pull request Oct 7, 2026

[AIGTWY-4882] Add fresh-launch journeys and dedicated CUJ7 coverage #930

Merged

Navigation Menu

Repository navigation

AIGTWY-4878 [3/3] Test MCP registration and tool use in Claude and Codex - #990

AIGTWY-4878 [3/3] Test MCP registration and tool use in Claude and Codex#990
tt-le merged 7 commits into
mainfrom
tt-le/aigtwy-4878-mcp-registration

tt-le commented Oct 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

david-siqi-liu left a comment

Uh oh!

david-siqi-liu left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tt-le commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Known issue: ug status

Validation

Scope

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

UG review

Major

Uh oh!

david-siqi-liu left a comment

Choose a reason for hiding this comment

Uh oh!

david-siqi-liu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tt-le commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 6, 2026 •

edited

Loading