Repository navigation
Conversation
This was referenced Oct 7, 2026
rohita5l
pushed a commit
to djcelis/unity-gateway
that referenced
this pull request
Oct 8, 2026
The CUJ plan's skills checks for CUJ 2, 3 and 6 were skipped because Default Storage rejects skill uploads. With skill fixtures on a schema backed by customer S3 storage, the journeys can now cover them. Changes: - CUJ 2 (`skills.names = ["ug_e2e.skills.fixture-summary"]`): - only that bundle downloads, byte-identical to the live fixture; - both agents list it in `/skills`, and the schema's other skills are absent; - both agents invoke it and answer with a token that exists only in the bundle's reference file. - `_assert_cuj2_config` now expects the skills selector. - CUJ 3 (`skills.unity_catalog_location = "ug_e2e.skills"`): - every skill in the schema downloads and is listed; - the `ug_e2e.other_skills` decoy stays absent; - `ug status` counts the skills per agent; - both agents invoke a downloaded skill. - CUJ 6: - Phase A publishes the named skill. It downloads once to both skill roots and is unchanged by a repeat configure. - Phase A runs no terminal launch: one would make ug own `/etc`, and phase B's headless configure would then refuse. - Phases B and C drop the selector, and the skill disappears from disk and from each agent's `/skills`, including after relaunch. - The hand-authored user skill must render in the same screen, so the absence checks can't pass on an unloaded list. - The user's Claude settings are seeded with the theme the Claude TUI saves on first launch, so only ug's writes show as a diff. - Shared helpers live in `tests/e2e_cuj/helpers/skills.py`. The `tests/README.md` coverage row is updated. - UC skill names reject underscores, so the plan's `fixture_summary` is `fixture-summary`. One-time workspace setup, done outside the tests on 2026-10-07: - CUJ 6 metastore (ca-central-1): - `ug_e2e.skills` sits on S3-backed storage; - the CUJ SP has `READ_VOLUME` on it, which `skill_authz.jsonnet` requires for skill reads and downloads; - workspace A publishes the Phase A skills selector. - CUJ 2/3 metastore (us-east-1): - the same S3-backed storage credential and an external location; - `ug_e2e.skills` and `ug_e2e.other_skills` recreated on that storage, holding the four fixture skills; - CUJ 2 publishes `skills.names`, and CUJ 3 already published `unity_catalog_location = ug_e2e.skills`. This overlaps Mason's open databricks#1014 and databricks#1015 in `test_cuj2_mps_mcp.py`, where this PR only adds one key to `_assert_cuj2_config`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
sunishsheth2009
pushed a commit
to erifyc1/unity-gateway
that referenced
this pull request
Oct 8, 2026
…atabricks#1040) The e2e CUJs grew one copy of each helper per journey. This consolidates the shared helpers so CUJ 7 (databricks#930) and databricks#1014 / databricks#1015 can build on one version, and fixes three recurring CUJ flakes found in recent CI runs. Shared helpers (behavior preserving; every assertion, timeout and threshold is unchanged unless listed under behavior changes): - `UserSession.revert_machine_wide` replaces the class teardown revert and CUJ 3's two per-test copies (same terminal names and messages). - `base.make_workspace_client` and `base.bearer` replace the per-CUJ client and bearer code. CUJ 6 now honours `CLIENT_ID_ENV` / `CLIENT_SECRET_ENV`. - `Terminal` drives bare `ug` (`agent=`), and `Terminal.wait_until` is the one permission-rejecting wait for CUJ 2, 3 and 6. CUJ 1 keeps `AgentTerminal.wait_for_task`, which approves the read-only `find` prompt Claude emits in CI. - `evidence.assert_models` / `assert_served` replace CUJ 1's native-model check and the CUJ 2 / CUJ 3 served-response checks. Per-CUJ thresholds and matchers stay local. - `Workspace.agent_configs` splits a published config by agent for CUJ 4, 5 and 6. CUJ 5 reads the config through `workspace.config()` instead of a raw GET. - `helpers/poll.py` is the one bounded poll for the CUJ 1 and CUJ 6 trace waits (same budgets) and the `ug mcp list` retry. Trace SQL stays on `tests/integration/utils/sql.py`. - `helpers/mcp.py` (`registered_name`), `evidence.claude_file_task`, and shared model and header constants in `helpers/constants.py`. Reliability fixes: - MCP registration (Claude), failing 3 of 13 recent runs: Claude returned correct receipts but keyed them `mcp__server__tool`, and on this PR's first run echoed each raw `{"result": "<sha>"}` tool result. The prompt names the exact keys, and the check accepts exactly those two native shapes. The receipt values and key set must still match exactly. - `ug mcp list` health row (`claude:failed`, 1 of 13): Claude's single-shot probe raced the proxy cold start. The test polls up to 90s until the rows match, then runs the same exact assertion. - CUJ 4 Claude tasks use the absolute-path prompt CUJ 3 already used, so Claude no longer searches with `find` and hits a permission prompt. - Teardown leaks: a class that leaves `/etc` agent files still fails, and later classes now name that class and those paths instead of a generic dirty-runner error. - The CUJ bearer is refreshed per test (classes such as catalog discovery can outlive the M2M token). Every bearer a session installs stays redacted in artifacts. - ug review: `max_output_tokens` 4,000 to 16,000. High-effort reasoning consumed the old budget, leaving "returned no output text". The error now includes `incomplete_details.reason`. CI: - E2E CUJs run one runner per file, checked as `E2E CUJs · <CUJ_NAME>` (for example `E2E CUJs · CUJ 3 · UC model discovery`, names from the CUJ plan doc). CUJs share machine-wide `/etc` agent settings, so they cannot run concurrently on one runner. Each file still runs against its own read-only workspace, with `fail-fast: false`, and `All integration tests` requires every shard. - A small plan job discovers every `tests/e2e_cuj/test_*.py` and reads its module-level `CUJ_NAME`, so adding a CUJ needs no workflow edit (and no `.github/` owner approval). A file without `CUJ_NAME` fails the plan job; a unit contract test runs the same discovery script and also enforces unique names and the `test_cuj<N>_<topic>.py` / `CUJ <N> ·` convention. - The CUJ runners set `UV_HTTP_TIMEOUT=120` and `UV_HTTP_RETRIES=6`. All of them install from PyPI at once, and one shard already failed on a metadata fetch timeout. - CUJ files are renamed to `test_cuj<N>_<topic>.py`: - CUJ 1: `test_cuj1_explicit_models.py` (was `test_ug_managed_sentinel.py`) - CUJ 2: `test_cuj2_mps_mcp.py`, `test_cuj2_skills.py` - CUJ 3: `test_cuj3_models.py` (was `test_catalog_discovery.py`), `test_cuj3_mcp.py` (was `test_mcp_registration.py`), `test_cuj3_skills.py` - CUJ 4: `test_cuj4_smart_routing.py`; CUJ 5: `test_cuj5_budget_defaults.py`; CUJ 6: `test_cuj6_repeated_config.py` Behavior changes worth a look: - CUJ 2 Claude now rejects "Do you want to proceed?" like every other CUJ (it previously waited out the timeout). - The CUJ 6 Codex span poll's last query lands at the 300s deadline rather than up to 15s after it. Testing: - `tests/test_e2e_cuj_helpers.py` adds offline tests for the new helpers. Terminal tests run the real driver on a PTY against a fake `ug`. - Full unit suite and `--collect-only` (78 CUJ tests) pass locally. The E2E CUJs lane is the live check. CI timing, measured from job start and end times (main run 37821776097 vs this PR's run 37822258143): - E2E CUJs: 19.5 min as one job, now 7.8 min wall across 9 parallel shards. Total runner time is about the same (21.9 min). - Longest shard: CUJ 6 at 7.8 min. All others take 1.2 to 3.0 min. - Whole CI run: 20.0 min, now 18.7 min. `Full journeys · Claude` (about 13 to 15 min) is now the critical path, so cutting CUJ time further won't shorten CI. Not included: the workspace URL normalization in `src/ucode` from AIGTWY-4972 (a product behavior change, separate PR). This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What did you change, and why?
Extended the CUJ2 journeys from #993 beyond name-only MCP registration. Both Claude and Codex now inspect their native MCP inventory, require the connected
ug_e2e.tools.fixture_readerservice andread_fixturetool, call it with a fresh nonce, and verify the nonce-derived SHA-256 receipt plus the actual tool-call record in the parent transcript.The terminal inventory helper follows the pattern in #990. Its duplicate helper file can disappear when this branch rebases after that PR merges.
How do you know it works?
Blocker
The CUJ2 workspace currently publishes
system.ai.sandbox. This draft expects the read-only workspace fixture to publishmcp_servers.unity_catalog_location = ug_e2e.tools. CI cannot pass until that external workspace configuration is provisioned; the test intentionally does not mutate or repair it.