Skip to content

ci: run the backend, frontend and edge test suites on every pull request - #147

Open
kai-openswarm wants to merge 5 commits into
openswarm-ai:devfrom
kai-openswarm:b8/ci-lane
Open

ci: run the backend, frontend and edge test suites on every pull request#147
kai-openswarm wants to merge 5 commits into
openswarm-ai:devfrom
kai-openswarm:b8/ci-lane

Conversation

@kai-openswarm

Copy link
Copy Markdown

What

Nothing in CI ran the backend pytest suite (235 files), the renderer's 22 node:test files, or the edge suite — they were run by hand, one file at a time, so a regression only surfaced when someone happened to run the right one. This adds three small workflows, all hosted ubuntu, path-filtered, read-only token, workflow_dispatch for by-hand runs, triggered on every pull request (so dev PRs are gated) and on pushes to main/dev:

  • backend-tests — pytest on Python 3.13 from the locked requirements (--require-hashes) + requirements-dev.txt, plus a completion assertion: --collect-only count must equal the junit testcase count. A test process that dies mid-run can exit 0 with no summary (a hard-exit shutdown path did exactly that once and silently skipped ~42% of the suite), and this makes green mean green rather than "green as far as it got".
  • frontend-teststsc --noEmit, then node scripts/run-tests.mjs: node:test via tsx over src/**/*.test.ts(x). The runner is the one the test files already name in their headers; it did not exist.
  • edge-tests — pytest for openswarm-edge.

Separate first commit: shared/config.ts and shared/backendConnection.ts both touched window at import time (port/host derivation, the fetch interceptor install, the debug handle), so any node:test file importing a reducer that imports API_BASE died with window is not defined before its first assertion — fetchSessionsStrip.test.ts had been red that way since the resilience work landed. In a renderer nothing changes (same port/host, same interceptor, same handle); without a window the module answers with the defaults and installs nothing.

Verified

Against this exact tree, on GitHub-hosted ubuntu (the same workflow files):

  • backend-tests: 2966 tests collected2951 passed, 15 skipped in 4 min 15 s; the assertion printed collected=2966 ran=2966.
  • frontend-tests: tsc clean; # tests 143 / # pass 143 / # fail 0 (22 files). Before the import-safety commit: 139/140.
  • edge-tests: 14 passed.
  • The renderer is unaffected by the import guards: a packaged 1.7.7 build with them passes smoke.spec.ts and ten fetch-heavy specs.

Scope, on purpose

This is the smallest change that gives dev PRs a real test gate. Left out deliberately, each a candidate for its own PR: a windows-latest pytest leg (the suite currently carries a dozen Windows-portability assumptions in tests — charmap reads, POSIX-only asserts, native separators in API paths — which want fixing first), lint on dev (a policy question: dev is red on the linter today), an aggregate required-status job for branch protection, and e2e (the packaged app currently cold-boots past e2e.yml's budgets on hosted Windows).

… a window, so reducer tests run under node:test

Both modules touched window at import time (port/host derivation, the fetch
interceptor install, the debug handle), so any node:test file that imports a
reducer importing API_BASE died with 'window is not defined' before its first
assertion; fetchSessionsStrip.test.ts has been red that way since the
resilience work landed, unnoticed because nothing runs these tests in CI. In a
renderer (window present) nothing changes: same port/host, same interceptor,
same handle. Without one the module answers with the defaults and installs
nothing.
Nothing ran any of them in CI: the 235-file backend pytest suite, the 22
renderer node:test files and the edge suite were run by hand, one file at a
time, so a regression only surfaced when someone happened to run the right
one. Three small workflows, hosted ubuntu, path-filtered, read-only token:

- backend-tests: pytest on Python 3.13 from the locked requirements, plus a
  completion assertion (junit testcase count == collect-only count) so a test
  process that dies mid-run can never read as green
- frontend-tests: tsc --noEmit + node:test via tsx over src/**/*.test.ts(x),
  through frontend/scripts/run-tests.mjs (the runner the tests already name)
- edge-tests: pytest for openswarm-edge

All three are green on the current tree: 2951 backend tests, 143 frontend
tests across 22 files, 14 edge tests.
Two of five hosted runs of the backend suite stalled at 99% until the job cap
with no summary and no junit: one test blocked forever on a bare
ws.receive_json() (fixed on its own in a separate change). A CI lane should
never depend on every test being unable to hang, so add pytest-timeout to the
dev requirements and run the suite with --timeout=300. On Linux the default
signal method fails just the offending test and the run continues, so the
report and the "every collected test ran" assertion stay meaningful.
test_ws_endpoint_streams_a_full_turn_end_to_end read the socket with a bare
ws.receive_json() in a 40-iteration loop and broke only on the assistant reply.
When the loop ends early for any reason (fewer than 40 events, no reply), the
next receive blocks forever and the whole pytest run stalls at 99% until the
job cap. On hosted runners it does exactly that intermittently, on Linux and
Windows alike: the turn path's configure_provider_env decides whether 9Router
needs reviving from provider evidence earlier tests may leave behind, and that
revival spawns/installs the router behind a module-level asyncio.Lock; the
background turn-label aux call reaches the same machinery. Neither is part of
this test's contract ("SDK and WS auth mocked, everything else real").

Pin both out with monkeypatch, bound every receive at 5s (a regression now
fails this test instead of hanging the runner), and wait for the turn's
completed status before asserting on session.messages so the assertion cannot
race the loop's tail.
@kai-openswarm

Copy link
Copy Markdown
Author

Two follow-up commits, both surfaced by running this lane repeatedly against dev on hosted runners:

  • ci: cap each backend test at 300s so a stall fails by namepytest-timeout added to backend/requirements-dev.txt, suite run with --timeout=300. Two of five runs stalled at 99% until the job cap with no summary and no junit; with the cap the offending test fails by name, the rest of the suite still runs, and the "every collected test ran" step now also runs on a red suite so a failure report says whether the run was complete.
  • tests: the WS end-to-end turn test cannot hang the suite — the stall itself. test_ws_endpoint_streams_a_full_turn_end_to_end read the socket with a bare ws.receive_json() in a 40-iteration loop, so any early end of the turn blocked forever; on hosted runners the turn path could wander into 9Router revival (configure_provider_envrouter_availableensure_running, serialized on a module-level lock) from provider evidence earlier tests leave behind. Pinned out along with the background turn-label aux call (neither is in the test's "SDK and WS auth mocked" contract), every receive bounded at 5 s, and the test waits for the turn's completed status before asserting on session.messages.

Lane on hosted ubuntu with both: collected=2966 ran=2966, 2951 passed / 15 skipped, 4 min 22 s.

Same class as the config/backendConnection change: safeMode.ts read `window`
at import, and dashboardLayoutSlice imports it, so any reducer test that
imports the slice died under node:test before it ran. Guard the read; in a
renderer nothing changes.
@kai-openswarm

Copy link
Copy Markdown
Author

One more of the same kind: renderer: shared/safeMode is import-safe without a windowsafeMode.ts read window at import and dashboardLayoutSlice imports it, so any reducer test importing the slice died under node:test before running. Guarded; nothing changes in a renderer. (Both god-file split PRs that follow base on this lane's tip.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant