Skip to content

[AIGTWY-4877] Add MPS and MCP two-agent CUJ - #993

Merged
masonc08 merged 19 commits into
mainfrom
mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp
Oct 6, 2026
Merged

masonc08 merged 19 commits into
mainfrom
mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp

Conversation

@masonc08

@masonc08 masonc08 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

What did you change, and why?

Added the dedicated CUJ2 journey for a real published CodingAgentConfig: Codex uses ug_e2e.providers.openai with gpt-5-nano, Claude uses ug_e2e.providers.anthropic with claude-haiku-4-5-20251001, and both invoke system.ai.sandbox through the real agent TUIs. The runner and managed CI lane now execute the serialized workspace-backed CUJ with service-principal auth, preserve/restore any prior config, redact credentials, and install the pinned SDK through the authenticated internal Python proxy.

Skill coverage is intentionally deferred because the workspace Skill backend currently has a storage-path failure. This CUJ contains no Skill configuration, download, invocation, or assertion path.

How do you know it works?

  • 60 focused runner/contract tests pass.
  • Dedicated CUJ collection, Ruff/format, type checks, runner help, and diff checks pass.
  • Live read-only workspace checks confirm both exact MPS targets plus system.ai.sandbox and the negative system.ai.web_search fixture.
  • Full unit run: 3,170 passed and 6 skipped; four unrelated installed-agent tests fail because this developer host has machine-wide Claude/Codex policy and newer agent binaries.
  • The full service-principal two-agent run is wired into the managed CI lane and will run on this PR using repository secrets.

This PR was created with GitHub MCP.

@masonc08
masonc08 force-pushed the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch 3 times, most recently from d887243 to 132fea4 Compare October 6, 2026 00:32
@masonc08
masonc08 changed the base branch from main to test/cuj-smart-routing October 6, 2026 00:53
@masonc08
masonc08 changed the base branch from test/cuj-smart-routing to main October 6, 2026 00:53
@masonc08
masonc08 changed the base branch from main to test/cuj-smart-routing October 6, 2026 00:56
@masonc08
masonc08 changed the base branch from test/cuj-smart-routing to main October 6, 2026 00:56
@masonc08
masonc08 force-pushed the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch from 132fea4 to 20baee9 Compare October 6, 2026 00:58
@masonc08
masonc08 changed the base branch from main to test/cuj-smart-routing October 6, 2026 01:00
@masonc08
masonc08 force-pushed the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch 6 times, most recently from 2539627 to 31bb970 Compare October 6, 2026 01:43
@lilly-luo
lilly-luo force-pushed the test/cuj-smart-routing branch 5 times, most recently from 2bacf37 to 60f93ce Compare October 6, 2026 06:29
@lilly-luo
lilly-luo changed the base branch from test/cuj-smart-routing to main October 6, 2026 06:32
@lilly-luo
lilly-luo changed the base branch from main to test/cuj-smart-routing October 6, 2026 06:33
@masonc08
masonc08 force-pushed the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch from 1ac1c0d to 320af2d Compare October 6, 2026 12:45
@masonc08
masonc08 marked this pull request as ready for review October 6, 2026 12:47
@masonc08
masonc08 changed the base branch from test/cuj-smart-routing to main October 6, 2026 12:47
@masonc08
masonc08 force-pushed the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch from 320af2d to 1fa74e1 Compare October 6, 2026 12:50
@masonc08
masonc08 enabled auto-merge October 6, 2026 14:31
@david-siqi-liu david-siqi-liu added the ug-review Run the automated UG review label Oct 6, 2026
Comment thread tests/e2e_cuj/test_cuj2_mps_mcp.py Outdated
@masonc08
masonc08 force-pushed the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch from 133d9f3 to f187532 Compare October 6, 2026 17:47
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

UG review

The PR changes Codex MCP credential forwarding and workspace discovery URL construction, and adds two-agent workspace CUJs with teardown cleanup.

Major

  • Exercise Codex MCP auth forwarding — src/ucode/agents/codex.py:637: can we add a local Codex stdio MCP test verifying these variables reach the proxy? TOML snapshots and name-only listings still pass when auth forwarding fails.

Automated advisory review of f18753296923 using Lilly's UG review rubric. It does not approve or block this PR.

@david-siqi-liu david-siqi-liu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. One P1 to follow up on, not blocking:

P1

  • The MPS listing, MPS get, and Anthropic models calls now build URLs from normalize_workspace_url(workspace), which only adds a scheme and strips a trailing /. ug configure --workspace stores the URL as typed, so a workspace pasted as https://<host>/?o=123 turns the listing URL into https://<host>/?o=123/api/2.1/.... Before this change, https://{hostname} dropped the query. #1004 adds workspace_origin (scheme plus netloc, keeps the recorder port), and #987 carries the same hunks verbatim. Using it here avoids three PRs editing these lines three different ways.

@david-siqi-liu

Copy link
Copy Markdown
Collaborator

Follow-up after checking the test against the CUJ 2 plan in the UG Configure E2E doc. My approval stands, but I think these are must-fix before this counts as CUJ 2 coverage:

P0 (coverage)

  • MCP is only checked by name in mcp list (system.ai.sandbox present, web_search absent), and the agent prompt says "Do not use tools". The plan's MCP checks aren't exercised: the fixture MCP authenticating, initializing and listing its tool in each agent, the tool executing with its returned value in the final answer, and transcript or service evidence of the actual tool call. The plan's fixture is ug_e2e.tools.fixture_reader; AIGTWY-4878 [3/3] Test MCP registration and tool use in Claude and Codex #990 has a tool-call pattern (per-run nonce plus sha256 receipts checked in the native transcript) that could be reused here.

P1

  • Provider discovery is checked through the SDK MPS get/list, not through ug's or the agents' discovery output.

Skills are deferred per the PR body; that's the same Files API blocker CUJ 3 and CUJ 6 hit, so I'm treating it as a shared follow-up rather than blocking this PR.

@masonc08
masonc08 disabled auto-merge October 6, 2026 21:38
@masonc08
masonc08 added this pull request to the merge queue Oct 6, 2026
@masonc08

masonc08 commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up after checking the test against the CUJ 2 plan in the UG Configure E2E doc. My approval stands, but I think these are must-fix before this counts as CUJ 2 coverage:

P0 (coverage)

  • MCP is only checked by name in mcp list (system.ai.sandbox present, web_search absent), and the agent prompt says "Do not use tools". The plan's MCP checks aren't exercised: the fixture MCP authenticating, initializing and listing its tool in each agent, the tool executing with its returned value in the final answer, and transcript or service evidence of the actual tool call. The plan's fixture is ug_e2e.tools.fixture_reader; AIGTWY-4878 [3/3] Test MCP registration and tool use in Claude and Codex #990 has a tool-call pattern (per-run nonce plus sha256 receipts checked in the native transcript) that could be reused here.

P1

  • Provider discovery is checked through the SDK MPS get/list, not through ug's or the agents' discovery output.

Skills are deferred per the PR body; that's the same Files API blocker CUJ 3 and CUJ 6 hit, so I'm treating it as a shared follow-up rather than blocking this PR.

Thanks for the review @david-siqi-liu I think initially we skipped running MCPs and skills because there was some issue with the workspace setup. I'm going to merge this PR and see what we can fix here as a followup.

Merged via the queue into main with commit 4d8ff8d Oct 6, 2026
47 of 52 checks passed
@masonc08
masonc08 deleted the mason-cao_data/masonc08/aigtwy-4877-cuj2-mps-mcp branch October 6, 2026 21:42
david-siqi-liu added a commit that referenced this pull request Oct 7, 2026
list_model_services and the coding-agent config, recommendation, and
external-price calls rebuilt the URL as https://{hostname}, so the TUI request
recorder at http://127.0.0.1:<port> was dialed at port 443. Use
normalize_workspace_url, the same pattern as inference requests, #999, and #993.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted ug-review Run the automated UG review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants