Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
9d720ec
AIGTWY-4878 Add standalone CUJ3 model-service fixtures
tt-le Oct 2, 2026
7188e2b
AIGTWY-4878 Test schema-scoped agent model discovery and defaults
tt-le Oct 2, 2026
3a8548b
AIGTWY-4878 Harden CUJ3 model evidence and simplify test helpers
tt-le Oct 2, 2026
bc8a647
AIGTWY-4878 Use Databricks SDK for model fixture provisioning
tt-le Oct 2, 2026
42a4165
AIGTWY-4878 Simplify catalog discovery journeys and share agent helpers
tt-le Oct 2, 2026
8a03084
AIGTWY-4878 Make catalog discovery tests independent of fixture provi…
tt-le Oct 2, 2026
c373f72
AIGTWY-4878 Extract reusable native agent integration helpers
tt-le Oct 2, 2026
d1b4bb6
AIGTWY-4878 Simplify catalog discovery coverage and supporting helpers
tt-le Oct 2, 2026
fcba0c1
AIGTWY-4878 Remove unrelated catalog HTTP hardening
tt-le Oct 2, 2026
4a9319c
AIGTWY-4878 Split catalog discovery into focused user journeys
tt-le Oct 2, 2026
880ec3a
AIGTWY-4878 Prune generic component validation from catalog journey
tt-le Oct 2, 2026
e9664e2
Fix Claude catalog journey model evidence and file prompts
tt-le Oct 2, 2026
292fac4
Migrate catalog discovery to dedicated-workspace CUJ scaffolding
tt-le Oct 5, 2026
02745db
Use shared dedicated CUJ framework for catalog discovery
tt-le Oct 6, 2026
3f1ec82
Remove obsolete provider catalog refactor and tests
tt-le Oct 6, 2026
b10f021
Isolate catalog CUJs and verify live inference evidence
tt-le Oct 7, 2026
758a34d
Match Claude catalog display labels and default picker control
tt-le Oct 7, 2026
3d0e560
Correlate catalog inference with exact user prompts
tt-le Oct 7, 2026
c8de998
Remove unused agent catalog fetch aliases
tt-le Oct 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions tests/e2e_cuj/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Catalog discovery CUJ

The dedicated workspace must already publish a CodingAgentConfig enabling both agents
with `ug_e2e.models` as their model source. Claude is the default agent; its default and
Sonnet-family model are `claude_sonnet`, and Codex defaults to `gpt_luna`. Smart routing
and tracing are disabled. Tests neither provision nor validate fixture definitions.

| Service under `ug_e2e.models` | Claude Code | Codex |
| --- | --- | --- |
| `gpt_luna` | Excluded | Included; default |
| `claude_haiku` | Included | Excluded |
| `claude_sonnet` | Included; default | Excluded |
| `kimi` | Included | Included |
| `gemini_flash` | Excluded | Excluded |

Independent gateway catalogs, Codex's native model/list, and exact TUI picker inventories must agree.
Claude's built-in Default action is not a catalog entry; banners use catalog display names,
while gateway inference checks require the exact service/alias.
Compatible `ug_e2e.other_models.claude_decoy` and `codex_decoy` must be discoverable in their
own schema but absent from the scoped picker. Claude discovery aliases must be enabled;
Kimi uses `anthropic-aigw-<8-character SHA-256 prefix>-<service FQN>`.

Ten independently configured cases cover picker discovery, default launches, and explicit models.
Picker cases dismiss the menu without changing selection, then complete a task on the default.
Separate cases cover bare `ug`, `ug claude`, and `ug codex` TUI first tasks and Claude print/Codex
exec defaults. Each additional compatible model gets its own headless task case.
Success requires a completed native task and successful recorded gateway inference for the
expected service/alias, not generated settings or catalog caches. Defaults omit model overrides;
expected answers are withheld from prompts. Claude headless results also require nonzero output
tokens in `modelUsage`; Codex joins the completed answer to its client-selected turn model.
Claude transcript model IDs name the backing model, not the service. TUI tasks reject unexpected
permission prompts. These checks do not prove the gateway's backing destination. Only live passes
establish coverage.

The test class selects the CUJ3 workspace, `https://dbc-bbdd5508-648e.cloud.databricks.com`.
The shared `cuj` fixture supplies its authenticated SDK client and isolated local session;
workspace configuration remains read-only. Each case cleans up with public `ug revert` before
the next case configures, including after failures. Set `UG_CUJ_SP_CLIENT_ID` and
`UG_CUJ_SP_CLIENT_SECRET` before a live run.

CI discovers these tests through the shared `dedicated-cuj` job, which installs both
pinned agents. No catalog-specific workflow or workspace secret is needed. Locally,
install ug, both agents, and the Databricks CLI on a clean POSIX host, then run:

```bash
uv run --with pexpect==4.9.0 --with pyte==0.8.2 pytest \
--confcutdir=tests/e2e_cuj tests/e2e_cuj/test_catalog_discovery.py -v
```

MCP/skills discovery remains separate.

Collection only (no authentication or inference):

```bash
uv run pytest -c tests/e2e_cuj/pytest.ini --confcutdir=tests/e2e_cuj \
--collect-only tests/e2e_cuj
```
11 changes: 11 additions & 0 deletions tests/e2e_cuj/catalog_discovery_expectations.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
"""Model fixtures owned by the catalog discovery journey."""

MODEL_SCHEMA = "ug_e2e.models"
OTHER_MODEL_SCHEMA = "ug_e2e.other_models"
CLAUDE_DEFAULT = f"{MODEL_SCHEMA}.claude_sonnet"
CODEX_DEFAULT = f"{MODEL_SCHEMA}.gpt_luna"
GEMINI_MODEL = f"{MODEL_SCHEMA}.gemini_flash"
CLAUDE_MODELS = frozenset({CLAUDE_DEFAULT, f"{MODEL_SCHEMA}.claude_haiku", f"{MODEL_SCHEMA}.kimi"})
CODEX_MODELS = frozenset({CODEX_DEFAULT, f"{MODEL_SCHEMA}.kimi"})
CLAUDE_DECOY = f"{OTHER_MODEL_SCHEMA}.claude_decoy"
CODEX_DECOY = f"{OTHER_MODEL_SCHEMA}.codex_decoy"
11 changes: 11 additions & 0 deletions tests/e2e_cuj/pytest.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[pytest]
testpaths = .
pythonpath = ../..
addopts = --strict-markers --tb=short
markers =
managed: requires an admin CodingAgentConfig published on the selected workspace
catalog_discovery: managed schema-scoped model discovery and inference
workspace_isolated: reads preconfigured CUJ workspace state with isolated local sessions
tui: real interactive terminal and completed task
claude: only runs when Claude Code is explicitly selected
codex: only runs when Codex is explicitly selected
Loading
Loading