Repository navigation
tests/packages: run the package suites side by side, the slow ones first - #410
Conversation
CI's test job took 934 s on derekross#389, and three suites ran most of it in a row: omamobile-audio 316 s, omamobile-shell 263 s and omamobile-modem-sit 138 s, against about 217 s for the other suites together. The runner has four CPUs. The suites now run up to OMAMOBILE_TEST_JOBS at once (default: the CPUs, at most 8; 1 is the old order), audio, shell and modem-sit first. Each suite's output is kept together and printed when it ends, with the same ok / not ok lines and summary as before. Locally, with four jobs, every package suite took 308 s; the four that failed failed the same way one at a time (python-dbus and python-gobject missing on that machine), none because of the others. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for this. Halving the test job is a real win for everyone who opens a PR, and keeping each suite's output in one block means the CI log reads as before. I checked that the suites don't share anything when they run side by side: each makes its own temp directory, there are no fixed ports or sockets, and the only fixed
Optional, but useful when something hangs: the output is now held until a suite ends, so if a suite hangs until the job's timeout, CI shows nothing from it. A On the bigger step, running only the suites a PR touches: I'd like to see it as a separate PR. I'd want the full run to stay on main and on any change under |
…me limit per suite Review on derekross#410: `echo $? > NAME.rc` created the file before the number was in it, so a report running at that moment read an empty result, printed not ok and removed it; the result is now written aside and moved into place. A suite whose subshell died before writing a result passed silently, where the old runner counted it as failed; every name is now checked at the end and reported "not ok - NAME (no result)". And since output is held until a suite ends, a hung suite showed nothing in CI: each suite now stops after OMAMOBILE_TEST_TIMEOUT seconds (default 1800, the slowest takes about 320 s) and fails with its log printed. With OMAMOBILE_TEST_JOBS=1 the slow suites still go first, so the comment says "one at a time" rather than "as before". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for checking the suites for shared state. Both fixes are in (7eb4ff8):
I took the optional one too: each suite runs under Running only the suites a PR touches can wait for a separate PR, with the full run kept on main and for |
|
Thank you, both fixes are in, and the timeout too. I ran the new runner in a scratch tree against fake suites: one that passes, one that fails, one that kills its own parent, one that runs past the time limit, and one that leaves a background process behind. Each came out in one block with the right result, the run failed with all four bad ones named, and nothing was left running afterwards. CI passed on this head and I've merged it, with a squash message that doesn't say "the old order"; I'll watch the next few runs on main for the audio watchdog timing. If you're back in the file at some point, a name given twice on the command line still runs twice into the same log, and deduping the arguments the way the default list does would cover it. Not needed for this PR. |
…rst (derekross#410) CI's test job took 934 s on derekross#389, three suites most of it in a row (omamobile-audio 316 s, omamobile-shell 263 s, omamobile-modem-sit 138 s). The suites now run up to OMAMOBILE_TEST_JOBS at once (default: the CPUs, at most 8; 1 runs them one at a time), the slow ones first, each suite's output printed together when it ends. A result is written aside and moved into place, so a report never reads it half written; a suite that ends without a result fails as "not ok - NAME (no result)"; and each suite stops after OMAMOBILE_TEST_TIMEOUT seconds (default 1800) and fails with its log printed, since output is held until a suite ends.
What and why
CI's test job takes about 15 minutes, and most of it is three package suites running one after another: on #389 the job took 934 s, of which omamobile-audio took 316 s, omamobile-shell 263 s and omamobile-modem-sit 138 s, against about 217 s for the other suites together. The runner has four CPUs, so
tests/packagesnow runs the suites side by side, up toOMAMOBILE_TEST_JOBSat once (default: the CPU count, at most 8;1keeps the old order), starting audio, shell and modem-sit first so the run doesn't end waiting on one of them. Each suite's output is kept together and printed when it ends, with the sameok/not oklines and summary, so the CI log reads as before; no test changes. The package part should drop from about 12 minutes to about 5 (the audio suite alone), the whole job to about 8-9.A bigger step, if you want it: on pull requests, run only the suites of the packages a PR touches (plus those that depend on them, and everything when shared tools or tests change), and keep the full run on pushes to main. Most PRs would then take 2-4 minutes. That's a policy call, so it isn't in here; happy to send it separately.
Device and build
uname -r): n/aChecks (no phone needed)
tools/pr-check origin/mainpassestests/allpasses (this PR's CI run is the one that counts: the suites need python-dbus and python-gobject, which the CI container installs)tests/packagespasses (or the packages I touched: none; locally, withOMAMOBILE_TEST_JOBS=4, every suite ran in 308 s, and the four that failed there failed the same way one at a time, for those missing Python modules, none because of the others)versionbumped, and the new version is in the commit subjectCo-Authored-By:orAssisted-by:trailer, never aSigned-off-by:On the phone
userdata, weakens the storage guard, or reads registers from userspace (AGENTS.md §1)🤖 Generated with Claude Code