Skip to content

tests/packages: run the package suites side by side, the slow ones first - #410

Merged
derekross merged 2 commits into
derekross:mainfrom
robotosaith:omamobile-tests-parallel
Nov 4, 2026
Merged

derekross merged 2 commits into
derekross:mainfrom
robotosaith:omamobile-tests-parallel

Conversation

@robotosaith

Copy link
Copy Markdown
Contributor

What and why

CI's test job takes about 15 minutes, and most of it is three package suites running one after another: on #389 the job took 934 s, of which omamobile-audio took 316 s, omamobile-shell 263 s and omamobile-modem-sit 138 s, against about 217 s for the other suites together. The runner has four CPUs, so tests/packages now runs the suites side by side, up to OMAMOBILE_TEST_JOBS at once (default: the CPU count, at most 8; 1 keeps the old order), starting audio, shell and modem-sit first so the run doesn't end waiting on one of them. Each suite's output is kept together and printed when it ends, with the same ok / not ok lines and summary, so the CI log reads as before; no test changes. The package part should drop from about 12 minutes to about 5 (the audio suite alone), the whole job to about 8-9.

A bigger step, if you want it: on pull requests, run only the suites of the packages a PR touches (plus those that depend on them, and everything when shared tools or tests change), and keep the full run on pushes to main. Most PRs would then take 2-4 minutes. That's a policy call, so it isn't in here; happy to send it separately.

Device and build

  • Phone: none: test tooling only
  • Kernel release (uname -r): n/a
  • Packages changed, with versions: none

Checks (no phone needed)

  • tools/pr-check origin/main passes
  • tests/all passes (this PR's CI run is the one that counts: the suites need python-dbus and python-gobject, which the CI container installs)
  • tests/packages passes (or the packages I touched: none; locally, with OMAMOBILE_TEST_JOBS=4, every suite ran in 308 s, and the four that failed there failed the same way one at a time, for those missing Python modules, none because of the others)
  • Each changed package has its version bumped, and the new version is in the commit subject
  • Commits have area prefixes and bodies that say why; any AI help is a Co-Authored-By: or Assisted-by: trailer, never a Signed-off-by:

On the phone

  • I ran this on my own phone, booted into this build: n/a, no phone involved
  • Runbook steps passed (plan, step, result): n/a
  • What I did not test: the timing on the GitHub runner itself (this PR's own CI run will show it)
  • Nothing in this change writes outside slot B and userdata, weakens the storage guard, or reads registers from userspace (AGENTS.md §1)

🤖 Generated with Claude Code

CI's test job took 934 s on derekross#389, and three suites ran most of it in a
row: omamobile-audio 316 s, omamobile-shell 263 s and omamobile-modem-sit
138 s, against about 217 s for the other suites together. The runner has
four CPUs. The suites now run up to OMAMOBILE_TEST_JOBS at once (default:
the CPUs, at most 8; 1 is the old order), audio, shell and modem-sit
first. Each suite's output is kept together and printed when it ends,
with the same ok / not ok lines and summary as before.

Locally, with four jobs, every package suite took 308 s; the four that
failed failed the same way one at a time (python-dbus and python-gobject
missing on that machine), none because of the others.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekross

Copy link
Copy Markdown
Owner

Thanks for this. Halving the test job is a real win for everyone who opens a PR, and keeping each suite's output in one block means the CI log reads as before. I checked that the suites don't share anything when they run side by side: each makes its own temp directory, there are no fixed ports or sockets, and the only fixed /tmp path (libfprint's ninja.log) is in a script tests/packages doesn't run. Two small fixes in the runner before I merge:

  1. echo $? > "$out/$n.rc" (line 38, and line 36) creates the file before it writes the number. If report runs at that moment for another suite, it reads an empty result, prints not ok, and removes the file, so the suite's real result is lost. Writing to "$n.rc.tmp" and then mv it to "$n.rc" closes that window.
  2. If a suite's subshell dies before it writes its result, nothing reports it and the run passes. Before this change, a killed suite always counted as failed. Could you check at the end that every name got a result, and report not ok - <name> (no result) for any that didn't?

Optional, but useful when something hangs: the output is now held until a suite ends, so if a suite hangs until the job's timeout, CI shows nothing from it. A timeout per suite, or a trap that prints the tail of each unfinished log, would bring that back. One small wording point: with OMAMOBILE_TEST_JOBS=1 the slow suites still go first, so "runs them one at a time" is more accurate than "the old order". I'll re-run the job a couple of times before merging, since the audio watchdog tests use short real-time timeouts and now run next to other suites.

On the bigger step, running only the suites a PR touches: I'd like to see it as a separate PR. I'd want the full run to stay on main and on any change under tools/ or tests/.

…me limit per suite

Review on derekross#410: `echo $? > NAME.rc` created the file before the number
was in it, so a report running at that moment read an empty result,
printed not ok and removed it; the result is now written aside and moved
into place. A suite whose subshell died before writing a result passed
silently, where the old runner counted it as failed; every name is now
checked at the end and reported "not ok - NAME (no result)". And since
output is held until a suite ends, a hung suite showed nothing in CI: each
suite now stops after OMAMOBILE_TEST_TIMEOUT seconds (default 1800, the
slowest takes about 320 s) and fails with its log printed. With
OMAMOBILE_TEST_JOBS=1 the slow suites still go first, so the comment says
"one at a time" rather than "as before".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@robotosaith

Copy link
Copy Markdown
Contributor Author

Thanks for checking the suites for shared state. Both fixes are in (7eb4ff8):

  1. Each suite writes its result to NAME.rc.tmp and moves it to NAME.rc, so report only ever sees a complete number.
  2. At the end, every name without a result is reported as not ok - NAME (no result) and fails the run. I checked it with a suite that kills its own runner subshell.

I took the optional one too: each suite runs under timeout (OMAMOBILE_TEST_TIMEOUT, default 1800 s against about 320 s for the slowest), and a suite that hits it fails with its log and a "stopped after" line. The comment now says "one at a time". The first commit's message still says "the old order", so please reword that line in the squash message.

Running only the suites a PR touches can wait for a separate PR, with the full run kept on main and for tools/ and tests/.

@derekross
derekross merged commit d1fb7b6 into derekross:main Nov 4, 2026
2 checks passed
@robotosaith
robotosaith deleted the omamobile-tests-parallel branch November 4, 2026 14:01
@derekross

Copy link
Copy Markdown
Owner

Thank you, both fixes are in, and the timeout too. I ran the new runner in a scratch tree against fake suites: one that passes, one that fails, one that kills its own parent, one that runs past the time limit, and one that leaves a background process behind. Each came out in one block with the right result, the run failed with all four bad ones named, and nothing was left running afterwards.

CI passed on this head and I've merged it, with a squash message that doesn't say "the old order"; I'll watch the next few runs on main for the audio watchdog timing. If you're back in the file at some point, a name given twice on the command line still runs twice into the same log, and deduping the arguments the way the default list does would cover it. Not needed for this PR.

robertzas pushed a commit to robertzas/omarchy-mobile that referenced this pull request Nov 14, 2026
…rst (derekross#410)

CI's test job took 934 s on derekross#389, three suites most of it in a row
(omamobile-audio 316 s, omamobile-shell 263 s, omamobile-modem-sit 138 s).
The suites now run up to OMAMOBILE_TEST_JOBS at once (default: the CPUs,
at most 8; 1 runs them one at a time), the slow ones first, each suite's
output printed together when it ends.

A result is written aside and moved into place, so a report never reads
it half written; a suite that ends without a result fails as
"not ok - NAME (no result)"; and each suite stops after
OMAMOBILE_TEST_TIMEOUT seconds (default 1800) and fails with its log
printed, since output is held until a suite ends.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants