Repository navigation
ci: run the test suites on every PR - #12
Merged
Merged
Conversation
The repo had no test CI. release.yml and runtime-image.yml only fire on tags and image changes, so nothing ever ran the suites on a pull request — which is why the fake-provider E2E regression (#11) survived the pi 0.82.1 bump unnoticed for weeks, quietly making live Anthropic calls until it timed out. Three jobs, all hermetic (no API keys, no model-provider network): go — builds agentctl and runs go test ./..., which includes the schema-sync test that catches drift between schemas/ and the embedded copy under cli/internal/adl/schemas/. node — matrix over all four adapters. Each adapter's own `npm test` runs its build first, so dist/ is never stale. Node is pinned to 22.19.0, runtime/'s engines floor and the strictest of the four. e2e — the two tiers that need no credentials (codex, claude). The codex tier uses the repo's fake-codex.sh shim; the claude tier exercises the compile-time and adapter-startup rejection paths. The pi and opencode tiers stay manual: both need a live key, and opencode needs its CLI on PATH. e2e/run.sh calls `npm run build` but never `npm install`, so the e2e job installs the adapter it builds first — otherwise tsc is missing on a fresh checkout. Verified by deleting runtime-claude/node_modules and running the tier from a clean `npm ci`. Current state gated by this: 422 JS tests across the four adapters (115 + 103 + 109 + 95), the Go suite, and both hermetic e2e tiers — all passing.
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The repo has no test CI.
release.ymlfires on tags,runtime-image.ymlon image changes — nothing runs the suites on a pull request.That's not theoretical: it's why the fake-provider E2E regression (#11) survived the pi
0.82.1bump for weeks. The "hermetic" suite was quietly making live Anthropic calls and timing out, and no PR ever surfaced it.Now that the suites are green and genuinely hermetic (the whole
runtimesuite runs in ~1.2s with no credentials), gating on them is finally practical.What runs
goagentctl, runsgo test ./...— includes the schema-sync test that catches drift betweenschemas/and the embedded copynodenpm testbuilds first, sodist/is never stalee2ecodex(via the repo'sfake-codex.shshim) andclaude(compile-time + adapter-startup rejection paths)Node pinned to
22.19.0—runtime/'s engines floor, the strictest of the four. Go pinned to1.25, matchingcli/go.mod.The pi and opencode e2e tiers stay manual: both need a live API key, and opencode needs its CLI on
PATH. Per AGENTS.md they're a no-op withoutAGENT_CONTROLLER_RUN_LIVE=1anyway.Gotcha this had to work around
e2e/run.shcallsnpm run buildbut nevernpm install, so on a fresh checkouttscis missing. The e2e job installs the adapter it builds first. Verified the hard way — deletedruntime-claude/node_modulesand ran the tier from a cleannpm ci; it passes.Also confirmed
npm cisucceeds for all four adapters, so no lockfile is out of sync with itspackage.json.What this gates today
All green as of this branch:
runtime115 ·runtime-opencode103 ·runtime-codex109 ·runtime-claude95 — 422 JS testscligo test ./...codex+claude, both hermetic🤖 Generated with Claude Code