Repository navigation
test(testbed): Playwright's page tests in the testbed suite - #1505
Conversation
🟢 e2e web: 91 passedAll 91 tests in 41 files
Tested by e2e 0.22.0 · 1m 19s · web · run artifacts |
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 87.6k tokens (85% cached) · $0.0053 All 33 tests in 5 files
Tested by e2e 0.22.0 · 3m 4s · web · run artifacts |
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 40 model calls · 232.8k tokens (79% cached) · $0.0172 All 20 tests in 1 file
Tested by e2e 0.22.0 · 5m 53s · android-emulator · run artifacts |
There was a problem hiding this comment.
All reported issues were addressed across 122 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
🟢 e2e android: 57 passed, 1 skippedAll 58 tests in 7 files
Tested by e2e 0.22.0 · 9m 16s · android-emulator · run artifacts |
🟢 e2e ios: 55 passed, 1 skippedAll 56 tests in 7 files
Tested by e2e 0.22.0 · 6m 5s · ios-simulator · run artifacts |
🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped25 agent steps · 12 replayed from cache · 62 model calls · 472.0k tokens (81% cached) · $0.0340
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped | 25 steps · 62 calls | 9m 0s | |
| 🟢 | act completes Login Form | 1 step | 41.0s |
| 🟢 | act completes Infinite Scroll List | 1 step | 30.8s |
| 🟢 | act completes Modal Flow | 1 step | 10.5s |
| 🟢 | act completes Bottom Tabs | 1 step | 7.9s |
| 🟢 | act completes Text Input Variations | 1 step | 14.9s |
| 🟢 | act completes Broken Accessibility | 2 steps · 8 calls | 43.8s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 32.0s |
| act completes Gestures (1 failed attempt first) | 1 step · 11 calls | 46.2s | |
| 🟢 | act completes Async States | 1 step · 10 calls | 36.9s |
| 🟢 | act completes Debounced Search | 1 step · 1 call | 27.4s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 29.3s |
| 🟢 | act completes Flattened Login | 2 steps · 10 calls | 54.4s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 10.5s |
| 🟢 | act completes WebView Accessibility | 1 step | 12.2s |
| 🟢 | act completes Permission Prompt | 1 step | 11.5s |
| 🟢 | act completes Photo Picker | 1 step · 2 calls | 30.8s |
| 🟢 | act completes Product Catalog | 1 step | 10.1s |
| 🟢 | act completes Error Recovery | 1 step | 10.2s |
| 🟢 | act completes Choice Controls | 1 step · 5 calls | 33.6s |
| 🟢 | act completes Stripe PaymentSheet | 1 step · 8 calls | 34.3s |
| 🟢 | act completes Apple Pay | 1 step | 11.9s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
Tested by e2e 0.22.0 · 10m 18s · ios-simulator · run artifacts
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
10d6184 to
b9422fc
Compare
…same-server redirects, trusted CORS origins
…ns, /ws handshake, pdf type, backslash-safe redirects, cleanup in finally, fixed PARITY_PORT
…ected-gaps.txt One config, chromium and firefox targets on one server; browserName is a fixture read from the app URL path. e2eGap skips become tests-parity/expected-gaps.txt, checked by scripts/parity.ts: unlisted failure, closed gap, stale line, or TODO fails. Runs in the check job; separate parity job dropped.
Ported specs live in tests/ tagged playwright, served by the playground (app/playwright/), chromium only. Known gaps are test.fail; four that fail as engine failures are skipped. Drops the parity config, gap file, script, and CI step, and 10 testbed tests the ported ones cover.
- gaps refused as INVALID_ARGUMENT move from fail to skip; mask-in-parallel awaits both screenshots and names the real failure - ports about:blank null origin; click.e2e.ts lists the tests left out - restores testbed tests for onDialog shorthands, frame fill, download.path - cleans upload temp dirs; caps the slow expected failure at 5s - CI runs every Playwright test on a pull request; lightpanda and mixed leave them out
b9422fc to
22d080f
Compare
What and why
tests/page+tests/library/download.spec.tsatv1.64.1, ourplaywright-corepin), rewritten against the e2e public API, as ordinary testbed tests inapps/testbed/tests/. 32 files, ~500 tests: click, fill, check, select, keyboard, locators, actionability and scroll into view, frames, dialogs, route/fulfill/continue/fallback, waitForURL/Response, network idle, screenshots + masks, downloads, file chooser.playwright(--tag playwrightruns only them). Each file header names its source and lists the tests left out, with the reason.fail: 'reason'(feat(e2e): test.fail for tests expected to fail #1539), so a closed gap fails the run until its mark goes: 16 tests. 6 fail with an errorfaildoes not cover (an engine failure, or a refused argument), so they areskip: 'reason'.apps/testbed/app/playwright/, withLICENSE+NOTICE; assets unchanged, ignored by lint and fallow). One app, one config, chromium only.playwrighttests out of the model-filtered run and a second step runs all of them (23s locally); a push to main runs everything once.operation timed out in browser.waitForResponse "foo.css"), and url-pattern steps quote string patterns.Marked
fail(open gaps unless noted):visibility: hiddeniframe read as visible (3 tests)uncheckrefusesrole=radio;inputValueon a non-input reads""instead of throwingroute.fulfillbody string-only; header overrides can't delete a header?in route globs matches one character (Playwright: literal)Map/Arraybreaks evaluate (main world)selectOptionmatches label only,textContent()is rendered text,isChecked()on a non-checkable reads falseMarked
skip: multipleset-cookiein a fulfill (2), screenshot while navigating, screenshot with a stalled subframe,setInputFiles([]),timeout: 0.CodeQL on the routes (
apps/testbed/app/playwright/routes.mjs, test-only): the app listens on127.0.0.1only; delays and sizes are clamped, redirects stay on the app's two origins, credentialed CORS goes only to those origins, the multipart parse is linear.Issue
None. #1503 stacks on this.
Verified
Ran it locally: yes
Checklist