Parallelize smoke, full journeys, and agent CI - #606
Merged
Merged
Conversation
rohita5l
added this pull request to stack #607
September 14, 2026 16:24
lilly-luo
force-pushed
the
rohit/integration-stack-4-ci
branch
from
September 14, 2026 18:32
5f5c074 to
6bef98f
Compare
lilly-luo
reviewed
Sep 14, 2026
lilly-luo
approved these changes
Sep 14, 2026
rohita5l
force-pushed
the
rohit/integration-stack-4-ci
branch
4 times, most recently
from
September 15, 2026 02:13
4573b20 to
44a2ed1
Compare
rohita5l
force-pushed
the
rohit/integration-stack-4-ci
branch
from
September 15, 2026 02:16
44a2ed1 to
53c97ca
Compare
rohita5l
added a commit
that referenced
this pull request
Sep 15, 2026
## Summary Replace the six serial full-integration shards with two parallel agent lanes: one Claude VM and one Codex VM. Each lane installs its agent once and runs every case for that agent serially. The last successful #606 CI run took **14m18s**, including **10m50s** from the first full-journey job starting to the last one finishing: https://github.com/databricks/unity-gateway/actions/runs/34920527726 The first revision passed all checks in **8m19s**: **5m59s / 42% less time** than that baseline, retaining all 39 live cases: https://github.com/databricks/unity-gateway/actions/runs/34921778894 The follow-up also starts integration immediately alongside unit/agent e2e tests, removing the remaining agent-e2e startup barrier. The final required merge gate still waits for both suites and requires both to succeed. The new revision's CI will validate the additional speedup and increased request overlap. ## Coverage and safety - Preserve all **39 live cases**: Claude 15, Codex 24. A local collection audit verified that each lane exactly equals the union of its previous three shards, and that the lanes are disjoint and cover every live case. - Keep all four smoke cases, all three installation checks, and manual TUI selection of all four TUI cases. - Unit tests, agent e2e, and integration start independently. Keep integration's smoke → full barrier, `fail-fast: false`, per-case isolation/cleanup, evidence uploads, and the required `test` / `e2e` merge gates. - At most one full-suite job per agent within a run. Agent e2e and other PRs still share workspace capacity; this is not a global quota guarantee. - No test removals, new skips, retries, weakened assertions, sandbox changes, production-code changes, or dependency changes. - Update job/artifact names and local reproduction documentation for the two lanes. ## Validation - Clean local Linux ordinary suite: **2,185 passed, 47 existing skips**. No live integration pass is claimed by that run. - Exact pytest collection comparison: **39/39 live cases retained**, plus unchanged four-case smoke/TUI selections. - Ruff check and format check passed. - Actionlint with ShellCheck passed. - `git diff --check` passed. - Local workflow audit: immediate startup confirmed; all 16 success/failure/skipped/cancelled combinations checked against the final gate's actual shell command. Only two successful suites pass. - First revision live CI: all checks and all 39 full-suite cases passed in **8m19s**. - Immediate-start revision live CI and timing: pending.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs smoke and the full installed-product suite on relevant same-repository PRs and main pushes. Smoke covers four Hosted configure/TUI and headless cases in two agent jobs. All 45 full journeys run across eight independent runners. The existing e2e suite is split into seven gateway/agent jobs, with each agent job installing its required CLI.
Both matrices disable fail-fast and report aggregate results. Check names are Unit tests, Gateway API tests, Agent launch tests · Agent, Smoke journeys · Agent, and Full journeys · Agent · Group. Full coverage does not depend on a label or manual request. Small
testande2ecompatibility gates preserve the exact contexts required by current repository rules, without changing those rules.Validation:
The final tracked tree matches #568 plus the requested CI changes byte-for-byte across all 39 changed files. CI will supply live gateway results; none are claimed from local collection. The existing tracing test retains its pre-existing skip.
Live CI at
5f5c074:test_ug_configure_codex_openai_mps: the existing CI workspace returns HTTP 404 /ENDPOINT_NOT_FOUNDfor Codex model discovery and sayscodex/v1/models is not enabled for this workspace. No test retry, skip, or weaker assertion was added.Layer 4 of 4. Merge the stack from bottom to top after this layer's full CI passes. Required-check compatibility is preserved for the lower stack layers and other active PRs.