QVAC-23665 infra: make mobile integration tests dispatch-only with device + test selection - #3908
QVAC-23665 infra: make mobile integration tests dispatch-only with device + test selection#3908tobi-legan wants to merge 25 commits into
Conversation
…hared actions) Detach mobile integration tests from PRs and move them to on-demand workflow_dispatch with a mandatory, validated device selection. - schedule-test-run: add additive `manual-devices` mode (one run per test-spec x device model); existing modes/callers unchanged. - new validate-devices action: fail-fast if a dispatched run names no device or a device that does not exist on Device Farm. - tts mobile workflow: add platform/device/tests dispatch inputs, a fail-fast validate job, a single-platform dispatch matrix, and thread the selection through upload + schedule. workflow_call path untouched. - on-pr-tts-ggml: remove the mobile job and drop it from Merge Guard.
… selection Same pattern as tts-ggml: detach mobile from PRs, add validated manual device/test selection on workflow_dispatch, drop from Merge Guard needs. workflow_call (benchmark/on-merge) path unchanged.
…e selection Roll the dispatch-only mobile pattern to bci-whispercpp (dual-flagship) and decoder-audio (single-pool): validated manual device/test selection, detach from PRs, drop from Merge Guard. workflow_call paths unchanged.
…sification + model-fit Correctness fix for the dispatch-only mobile pattern: - Detect a direct manual run via a dispatch-only input (`inputs.platform`), not `github.event_name`. Inside a reusable (workflow_call) workflow the event name is the CALLER's event, so a dispatched benchmark caller would otherwise be misread as a manual run. `inputs.platform` is empty on workflow_call and set on direct dispatch, so it discriminates reliably. - Give build-and-test a skip-tolerant `if` (`!cancelled() && validate-devices success||skipped`). A skipped `needs:` job otherwise propagates a skip, which would have skipped the mobile run on benchmark/weekend/on-merge callers. - Roll the pattern to classification-ggml and model-fit (both single-pool), detaching them from PRs and dropping them from Merge Guard.
…ction Same dispatch-only pattern; build-and-test keeps its seed-models dependency and tolerates the skipped validate-devices on workflow_call. Detach from PRs, drop mobile from combine-perf-reports + Merge Guard needs.
…or sharded addons - schedule-test-run: let manual-devices win over the sharded branch, so a manual multi-spec run (OCR/LLM shards, TTS functional set) fans across the chosen device(s) instead of hitting the pool. No existing caller sets manual-devices, so other paths are unchanged. - ocr-ggml: dispatch-only with validated device/test selection; a manual test filter collapses the shard fan-out to a single spec. Detach from PRs; drop mobile from combine-perf-reports + Merge Guard.
Multi-spec dual-flagship lane (whisper + parakeet): dispatch-only manual device/test selection, single-platform dispatch matrix, manual filter takes precedence over benchmark/auto test-groups. Detach from PRs; drop mobile from combine-unified-performance-report + Merge Guard.
VLA (multi-spec, keeps its authorize/authorize-dispatch gates): dispatch-only validated device/test selection, manual filter collapses shard fan-out, build-and-test tolerates skipped validate-devices. Detach from PRs + Merge Guard.
…e selection Adds the manual-dispatch UI (platform, searchable device dropdown + multi-value free-text override, model-match operator, optional test-name filter) plus fail-fast device validation to the mobile workflow, and disables the automatic PR trigger on the mobile-only on-pr wrapper. Gating machinery is preserved so the PR path can be re-enabled later.
…ction Adds the manual-dispatch UI (platform, searchable device dropdown + multi-value free-text override, model-match operator, optional test-name filter) and fail-fast device validation to the mobile workflow; the workflow_call benchmark sweep is unchanged. Removes the PR mobile lane and drops mobile from the merge gate and benchmark summary needs.
…ection Adds the manual-dispatch UI (platform, searchable device dropdown + multi-value free-text override, model-match operator, optional test-name filter) and fail-fast device validation ahead of seed-models/build; the workflow_call benchmark-batch path is unchanged. Removes the PR mobile lane and drops it from merge-guard needs.
…tion Adds the manual-dispatch UI (platform, searchable device dropdown + multi-value free-text override, model-match operator, optional test-name filter) and fail-fast device validation to the sharded LLM mobile workflow. A manual test filter collapses the shard set to one grepped spec; the workflow_call platforms/device_model/benchmark paths are unchanged. Removes the PR mobile lane and drops it from combine-perf-reports and merge-guard needs.
Documents on the run_mobile output that standalone per-addon mobile suites are now on-demand (workflow_dispatch) only and no longer auto-run on PRs, while the flag stays wired for the on-device co-load smoke.
Adds docs/ci/MOBILE-ON-DEMAND.md covering how to run per-addon mobile tests via workflow_dispatch (platform, dropdown + free-text device selection, device validation, and the name-based test filter) and what changed on PRs / Merge Guard. Cross-links from the run-mobile-addon-tests label reference.
Extends docs/ci/MOBILE-ON-DEMAND.md with a known-good Device Farm model table, how to discover the live fleet (bogus-device validate-devices run or aws devicefarm list-devices), and where each addon's test-runner names live (test-groups.json / run* functions). Updates the device, devices_custom and tests input descriptions across all 14 mobile workflows to point there so dispatchers pass valid values.
Add a workflow-level concurrency guard to all 14 integration-mobile-test-* workflows so a manual workflow_dispatch never stacks a second Device Farm run: a fresh dispatch cancels the in-flight one, grouped per (workflow, branch). The workflow_call path (benchmarks / weekend / on-merge) is keyed per run (run_id) with cancel-in-progress disabled, so parallel/serial reusable calls are never cancelled or serialized. llm-llamacpp additionally keys the call path on artifact_suffix because benchmark-vlm-model-comparison fans several parallel legs from one run. Replaces the three ad-hoc concurrency blocks (asr, vla, inference-addon-cpp) that keyed on github.ref with unconditional cancel-in-progress, which could cancel workflow_call runs.
Wire each dispatchable addon's `package` input into the mobile setup action so a manual run can test a specific build instead of always the published @latest: - Empty/default (@qvac/<addon>@latest) -> latest published release (unchanged). - @qvac/<addon>@x.y.z -> that exact published npm version. - @tetherto/<addon>@<dev> -> the branch's GPR dev build, so unpublished native branch changes can be tested on-device. The pin is gated on `inputs.platform != ''` (the dispatch discriminator), so the workflow_call paths (benchmarks / weekend / on-merge) keep their existing artifact-first / prebuild_package behaviour untouched (QVAC-21879 safety). Relax the dispatch package-scope check from @qvac-only to @Qvac|@tetherto so the GPR branch build is a valid target (matches asr-ggml). Not changed: inference-addon-cpp already compiles its .bare in-run from ref; decoder-audio rides on bare-ffmpeg (skip-prebuilds), so neither takes a pin. Document build selection + the per-branch concurrency guard in MOBILE-ON-DEMAND.md.
A live dispatch surfaced that the fleet uses manufacturer-prefixed model names
(e.g. "Google Pixel 9", "Samsung Galaxy S25 Ultra", "Apple iPhone 17"). The old
dropdown values ("Pixel 9", "Samsung S25 Ultra", "iPhone 17") only worked with the
CONTAINS operator, and "Samsung S25 Ultra" was not even a substring of the real
name, so it failed outright; EQUALS rejected all of them.
Switch every dropdown to exact fleet names (valid with both CONTAINS and EQUALS)
and clarify the CONTAINS-vs-EQUALS matching + prefixed names in MOBILE-ON-DEMAND.md.
Document the intended team workflow now that dispatch stays one-platform-per-run: (1) broad coverage — run the pool per platform (Android + iOS as two dispatches), (2) narrow after a failure — re-run the single failing device with just the failing test via the `tests` grep. Includes copy-paste `gh workflow run` commands.
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
License compliance — cleanNo new dependency license findings in this PR. Warn-only (shadow) mode — this check does not block merges yet. Updated automatically by the canonical license compliance workflow. NOTICE presence (advisory)Missing NOTICE (advisory, does not block):
|
Workflow security (shadow mode)zizmor found 1114 finding(s) in Findings are annotated inline on the changed files and listed in the job summary. Reproduce locally: pipx run zizmor==1.27.0 --offline .github/ |
Three medium findings from security-review and bugbot on the dispatch-only
mobile lane:
1. WDIO config injection via the free-text `tests` dispatch input. The value
was substituted unescaped into the generated WDIO config (grep: '...') and
evaluated on the Device Farm host. Now:
- the sink JSON-encodes the grep (upload-to-devicefarm) so it is a proper
quoted JS string for any grep source, and
- the manual_tests step allowlists mocha-safe characters and fails fast on
anything else (defense-in-depth, all 14 workflows).
2. seed-models ran in parallel with validate-devices, so a bad/empty manual
device selection still burned the up-to-120-min model seed. seed-models is
now gated on validate-devices (skip-tolerant for workflow_call) in
llm-llamacpp, embed-llamacpp, and diffusion-cpp.
3. run_rtf_benchmarks was a dead workflow_dispatch input on asr-ggml and
audiogen-ggml: a manual run always sets `platform`, which wins in the build
matrix, so a dispatched benchmark flag was silently ignored. Removed the
dispatch input (benchmarks stay on the workflow_call path); documented why.
workflow_call (benchmarks / weekend / on-merge) behaviour is unchanged.
Review pass (Bugbot + Security Review) — 3 medium findings, all fixed in
|
- decoder-audio: accept @tetherto/* package scope on dispatch (match siblings, so GPR dev-build pinning documented in MOBILE-ON-DEMAND.md is not rejected) - inference-addon-cpp: gate per-platform prebuilds on the dispatched platform and on validate-devices, and require only the chosen platform's prebuild in the build job so a single-platform manual run is not blocked by the other platform (uses inputs.platform, not matrix, which is unavailable at job-level if) - audiogen-ggml: on manual dispatch, check out trusted composite actions from the dispatched ref so the manual-devices scheduler is present; workflow_call still pulls them from the default branch, preventing a silent fallback to the pool
gianni-cor
left a comment
gianni-cor
left a comment
There was a problem hiding this comment.
This keeps the fix loop at a single device + single test (the cheapest possible Device Farm spend) instead of re-billing the whole pool on every push.
will the wf download only the models needed by the selected test for any addon?
… vla) The security fix JSON-encoded the wdio `grep` value, switching it from single- to double-quoted. llm-llamacpp and vla-ggml extract that grep from the generated wdio config with a single-quote-only regex to pre-stage ONLY the grepped test's model(s) onto the device; the double quotes silently broke the match, so GREP came back empty and the prestage fell back to staging the FULL manifest on the (billed) Device Farm device — defeating the point of a single-test dispatch and inflating device-minutes on the two heaviest-model addons. - Match either quote style in the grep extractor regex (llm + vla). - Flip the vla test fixture to the production double-quoted form so a single-quote-only regex can never silently regress this again. tts/asr read the grep from /tmp/qvacShardGrep.txt (quote-agnostic) and are unaffected; ocr/embed/nmt stage their small model sets in full regardless.
|
@gianni-cor short answer: not automatically for every addon. It depends on how each addon stages its models:
So the |
|
also worth adding: the models now come from our own S3 bucket (presigned), not huggingface. so the download + prestage on the device is way faster and more reliable — that alone cuts the time (and cost) of the whole model setup on every run, not just the filtered ones. |
|
|
||
| ```bash | ||
| # Android — across the pool phones (one run per exact model) | ||
| gh workflow run integration-mobile-test-tts-ggml.yml --ref <branch> \ |
There was a problem hiding this comment.
P1 — this command does not test the branch passed to --ref.\n\nThe manual workflows default their separate ref input to main and then check out inputs.ref || github.ref. Because the input is never empty, --ref <branch> selects the workflow definition from that branch but the addon source still comes from main. That can produce a green run without testing the PR's code.\n\nPlease default the input to an empty string so github.ref is used, or add -f ref=<branch> to every documented command and make the UI guidance explicit. Could you take a look?
| # Manual (workflow_dispatch) runs must name a device that exists on Device | ||
| # Farm. This gate fails fast BEFORE the (~90 min) build or any Device Farm | ||
| # run, so a typo or an empty selection costs nothing. Skipped for workflow_call. | ||
| validate-devices: |
There was a problem hiding this comment.
P2 — an invalid selection still starts the host build.\n\nvalidate-devices has no dependency relationship with host-test, so GitHub schedules both jobs at once. A typo or empty device value will fail validation, but the host build and tests can still consume their full 20-minute timeout. That defeats this workflow's fail-fast/no-wasted-build intent (even though the downstream Device Farm jobs are blocked).\n\nPlease make host-test depend on validate-devices and use the same success || skipped guard as the prebuild jobs, so workflow_call continues to work unchanged. Could you take a look?
| # dispatched ref so new scheduler code (e.g. manual-devices) is present; | ||
| # otherwise the default-branch copy silently ignores manual-devices and | ||
| # falls back to the dual-flagship pool, breaking the pin guarantee. | ||
| ref: ${{ inputs.platform != '' && (inputs.ref || github.ref) || github.event.repository.default_branch }} |
There was a problem hiding this comment.
P1 — this executes composite actions from a caller-selected ref in a release job.\n\nFor a manual run, inputs.platform is set, so this resolves to inputs.ref || github.ref rather than the default branch. The subsequent ./trusted-actions/... steps then execute that ref's action code with the job's release environment, OIDC credentials, and GitHub token. This removes the default-branch trust boundary the workflow previously used, and the trust-policy test currently detects the regression.\n\nPlease keep the action checkout pinned to the default branch, or introduce a reviewed/explicitly constrained mechanism for testing action changes. Could you take a look?
| SCHEDULING_STARTED=1 | ||
|
|
||
| if [ "$SPEC_COUNT" -gt 1 ] && { [ "$SCHEDULING_MODE" != "dual-flagship" ] || [ "$MULTI_SPEC_DUAL_FLAGSHIP" != "true" ]; }; then | ||
| if [ "$SCHEDULING_MODE" != "manual-devices" ] && [ "$SPEC_COUNT" -gt 1 ] && { [ "$SCHEDULING_MODE" != "dual-flagship" ] || [ "$MULTI_SPEC_DUAL_FLAGSHIP" != "true" ]; }; then |
There was a problem hiding this comment.
P1 — the always-run trust-policy suite is red after this condition changed.\n\n.github/scripts/test/ci-trust-policy.test.mjs still asserts the old sharding condition verbatim, and this PR also changes the TTS/ASR matrix shape plus AudioGen's action checkout. Running node --test .github/scripts/test/ci-trust-policy.test.mjs currently gives 60 passing and 4 failing tests, which is also blocking the PR's policy-tests check.\n\nPlease update the policy assertions to cover the new manual-devices path while retaining checks for automatic sharding, the original workflow_call matrices, and the default-branch action trust boundary. Could you take a look?
| PLAT_UPPER=$([ "$PLATFORM" = "Android" ] && echo ANDROID || echo IOS) | ||
| # Split the comma-separated list, trimming surrounding whitespace and | ||
| # dropping empty entries (e.g. a trailing comma). | ||
| MODELS_JSON=$(printf '%s' "$DEVICE_MODELS" | jq -Rc 'split(",") | map(gsub("^\\s+|\\s+$";"")) | map(select(length > 0))') |
There was a problem hiding this comment.
P2 — duplicate device names create duplicate billed runs.\n\nThe list is trimmed and empty entries are dropped, but it is not deduplicated or bounded. For example, devices_custom="Google Pixel 9,Google Pixel 9" passes validation and schedules the same spec twice; for LLM/OCR shards that multiplies into many redundant Device Farm runs, each with a 120-minute ceiling.\n\nPlease canonicalize the parsed list with unique before counting/looping, reject a reasonable maximum device count and total spec × device runs, and apply the same normalization in validate-devices so validation matches scheduling. Could you take a look?
- default the dispatch `ref` to blank so `--ref <branch>` tests the branch's code (github.ref) instead of silently falling back to main (13 workflows) - gate inference-addon-cpp host-test on validate-devices (skip-tolerant) so a bad/empty device selection fails fast before the ~20 min host build - dedupe and bound the manual device list (unique, max 10 devices, max 20 spec x device runs) in both schedule-test-run and validate-devices so a duplicated/oversized list cannot schedule redundant billed runs - restore audiogen's default-branch trust boundary: both build-and-test and the new validate-devices job source their composite actions from the default branch, never a caller-selected ref (manual-devices is only exercisable there once merged, by design) - update ci-trust-policy.test.mjs to cover the new manual-devices path, the single-platform dispatch matrices, the manual test-filter override, and the per-branch dispatch concurrency, while retaining the automatic-sharding, workflow_call-matrix, and default-branch-action-trust assertions (64/64 green) - docs: clarify that `ref` defaults to blank so a plain `--ref <branch>` tests the branch
|
@gianni-cor thanks — all five addressed in P1 — P1 — audiogen ran composite actions from a caller-selected ref in a release job (L192). Reverted. Both P1 — trust-policy suite red (L271). Updated P2 — invalid selection still started the host build (inference-addon-cpp L93). P2 — duplicate device names → duplicate billed runs (L356). The parsed list is now |
2 participants
Summary
Mobile integration tests used to fire on every PR (and via the
run-mobile-addon-testslabel), which meant every push spun up AWS Device Farm runs across the fleet — the single biggest driver of our Device Farm bill. This makes them on-demand only: they now run viaworkflow_dispatchwith a mandatory device selection and an optional single-test filter, so you pay for exactly the device(s) and test(s) you ask for.Applied uniformly across all 14 mobile addons.
What changed
platform—AndroidoriOS(one platform per run; each has its own fleet).device— searchable dropdown of common fleet phones, or(custom).devices_custom— free-text, accepts multiple comma-separated models (e.g.Google Pixel 9, Samsung Galaxy S25 Ultra).device_model_operator—EQUALS(exact fleet name) orCONTAINS.tests— optional mocha--grepon the test runner name; empty = full suite.validate-devicesjob — fail-fast against the live Device Farm fleet before provisioning. A run with no device, or an unknown device, errors out immediately (and lists what's available) — no wasted Device Farm minutes.seed-modelsaddon (see below).workflow_callpaths are never cancelled by this.package/package_specinput pins which prebuild goes on the phone, so you can test an unpublished branch build (GPR dev version) instead of the published@latest. (inference-addon-cppcompiles its.barein-run;decoder-audiohas no native prebuild of its own.)run-mobile-addon-testslabel kept but announced as no-op for standalone mobile suites (still gates the on-device co-load smoke), so we can re-enable auto-runs later without rewiring.docs/ci/MOBILE-ON-DEMAND.md(inputs, valid device names, where to find test names, worked flow) +docs/ci/LABELS.mdnote.What did NOT change
The
workflow_callpaths — benchmarks, weekend runs, on-merge — are untouched and still run automatically. All new behavior is gated behind the dispatch discriminator (inputs.platform != '').How to run
Proposed flow for running mobile against a PR
Because mobile no longer fires automatically, treat it as a manual gate you trigger against the PR branch whenever a change could affect on-device behavior. The recommended loop:
--ref <pr-branch>), pinning the fleet phones you care about:-f package=@tetherto/<addon>@<dev-version>(seedocs/ci/MOBILE-ON-DEMAND.md→ Which build gets tested).Validation (live runs)
Covering the different addon shapes — standard, sharded (filter collapses shards), and
seed-models(heavy model seeding under a filter) — plus a multi-device pool run per platform:runAddonTestrunDoctrBasicTestrunClassifyrunApiBehaviorTestseed-modelsstages full manifest under a filterAlso confirmed by inspection that all 14 mobile workflows carry the identical machinery (the
testsinput, the manual-filter resolver, thevalidate-devicesjob,devices_custom, the dispatch concurrency guard, and themanual_tests → test-groupswiring).Related
docs/ci/MOBILE-ON-DEMAND.md