Skip to content

feat(core): add launch, SessionState, and createAgent to @e2e-dev/core - #1734

Open
kdy1 wants to merge 7 commits into
mainfrom
kdy1/e2e-core
Open

kdy1 wants to merge 7 commits into
mainfrom
kdy1/e2e-core

Conversation

@kdy1

@kdy1 kdy1 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Stack 3/3: session-half split → move into @e2e-dev/core → this PR.

2026-11-15: rebased onto main 235986078 and the package renamed from e2e-core to @e2e-dev/core. Command output quoted below predates the rename.

@e2e-dev/core drives a browser or a native app from plain code. It is the engines, locators, agent, and replay cache, with no test runner, config file, or report. e2e is now built on it.

import { launch } from '@e2e-dev/core';
import { createAgent } from '@e2e-dev/core/agent';
import { web } from '@e2e-dev/web/engine';

await using shop = await launch(web(), { app: { url: 'https://shop.example.com' }, secrets: { PASSWORD: process.env.PASSWORD! } });
await using session = await shop.newSession();
await session.app.open('/login');
await session.screen.getByLabel('Password').fill(shop.secret('PASSWORD'));

const agent = createAgent(session, { model, cache: { dir: '.bot/cache' } });
await agent.act('add the item and apply the best coupon', { cacheKey: 'cart' });

Engines without e2e

  • @e2e-dev/web/engine and @e2e-dev/mobile/engine export the factories and their types without test, so a program never loads e2e.
  • The engines' e2e peer becomes optional, needed only by the main entry that extends test.
  • @e2e-dev/core gains its @e2e-dev/core and @e2e-dev/core/agent entries. @e2e-dev/core/agent also exports what e2e/agent does.

What @e2e-dev/core adds

  • Hosts and sessions: launch(engine, options) returns a host, and host.newSession() returns an isolated session with screen, app, platform, and the engine's fixture.
    • EngineHandle<Fixtures> makes launch(web()) type session.browser. defineEngine<Fixtures>() requires a factory in fixtures for each fixture it names (besides screen, app, platform), returning that fixture, and a union of fixture sets needs one member's factories.
    • A session publishes nothing to globalThis and has no session deadline.
    • Each operation has a default timeout, set with setDefaultTimeout.
    • launch and newSession take a signal, and the host closes on Ctrl-C.
  • Saved state: session.saveState() returns an in-memory SessionState.
    • It leaves the process only through state.export({ key }), encrypted with AES-256-GCM as a new session-state-1 document (schema and fixtures in packages/e2e/schema).
    • The state carries the secret values the session knew.
  • Secrets: pass values to launch({ secrets }) and get handles from host.secret(name). There is no process-wide registry.
  • Agent: createAgent(session, options) in @e2e-dev/core/agent.
    • With cache, every act names its recording with cacheKey, enforced in types and at runtime.
  • Device links: app.linkBase plus a new optional engine hook session.openLink (protocol method session/openLink) make app.open(path) open a link in a device app. @e2e-dev/mobile implements the hook.

Docs: a new docs/core.mdx, plus updates to the engine, config, errors, security, and agent reference pages.

Verified

Ran it locally: yes

  • @e2e-dev/core without e2e: a scratch project installed only the packed @e2e-dev/core and @e2e-dev/web, and node_modules held no e2e. It ran a script on real Chromium: a login page with a host.secret() fill, saveState(), export({ key }) and SessionState.import, then a new session restored from that state.

    e2e installed? 0
    e2e-core copies: 1
    price: 39,000
    browser fixture: object
    restored session signed in
    
  • Install smoke: scripts/install-smoke.ts now packs @e2e-dev/core too and fails on a second copy of it. It passes with pnpm 11 and pnpm 12, a single @e2e-dev/core copy after both the fresh and the frozen-lockfile install.

  • Built CLI: testbed default suite 588 passed. Web benchmark tests/ 93 passed.

  • Agentic benchmark (CI=1 test:agent --strict-cache, no model key): the branch matches main, so no cache key moved. The 8 failures on both are steps that need a model call.

    main this branch
    cache 26 replayed · 3 handed off · 6 missed 26 replayed · 3 handed off · 6 missed
    failed tests (need a model) 8 8, the same tests
  • Review: a fresh-context review (GPT-6.1 Sol) found three secret-handling gaps in session state, all fixed with regression tests that fail without the fix:

    • a restored state lost redaction of the launch's own secrets;
    • saveState() errors bypassed the sanitized error boundary;
    • a malformed IV or tag threw a raw crypto error instead of SESSION_INVALID.
  • Fixes from cubic, each with a test that fails without it:

    • mobile openLink uses the input budget, so a timed-out open is ACTION_MAY_HAVE_COMMITTED;
    • linkBase refuses an empty ? or # and surrounding whitespace;
    • a recorded space press reads back (already broken on main for press keys);
    • agent operations follow setDefaultTimeout;
    • a failed close on Ctrl-C is reported on stderr (new SIGINT test on a child process);
    • isRefusedLinkScheme is shared from @e2e-dev/core/engine with the mobile engine.
  • defineEngine<Fixtures> typing (added after the split, maintainer's choice on the cubic thread): tests/types/sdk-types.ts asserts that a missing factory, a factory returning the wrong type, and a union with no member's factories fail to compile, and that narrowing app needs none. Each assertion fails without the change. web() and mobile() compile unchanged.

  • Since the split: this PR's code is the head reviewed and verified above, plus that typing change and the changesets split across the stack. A second fresh-context review (GPT-6.1 Sol) of the whole stack found the union bypass, fixed here.

Left for later

  • The agent guard (stop, deny, check) and ActResult.status were removed from this PR in 85af095ab. They can come back as their own PR.
  • @e2e-dev/core's tests still live in packages/e2e/tests and import it through @e2e-dev/core/lib/*.
  • Integrations (@e2e-dev/kernel, eas, testmu, and others) still peer on e2e, so they don't yet work with @e2e-dev/core alone.
  • @e2e-dev/core's types.ts still declares the test API types. They are type-only and add nothing at runtime.
  • session-state-1 copies session-1's 64 MiB cap without enforcing it on export, as session-1 does not; one rule for both formats would be its own change.

🤖 Generated with Claude Code

@kdy1
kdy1 requested a review from okwasniewski as a code owner November 13, 2026 05:01
@github-actions github-actions Bot added the size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 93 passed

All 93 tests in 42 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 576ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 425ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 151ms
🟢 tests/async-states.e2e.ts · 2 passed 7.0s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.2s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 245ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 129ms
🟢 canvas only › a shape out of order paints the error and resets progress 116ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 336ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 200ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 136ms
🟢 tests/cart-totals.e2e.ts · 2 passed 323ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 191ms
🟢 cart totals › a quantity never drops below one 132ms
🟢 tests/checkout-review.e2e.ts · 1 passed 166ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 166ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 476ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 247ms
🟢 context menu trap › the decoy menu items refuse and keep the file 229ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.3s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 216ms
🟢 control inventory › check picks a radio the pick replaces with its summary 139ms
🟢 control inventory › a long press and a double click are told apart from a tap 960ms
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 201ms
🟢 control inventory › setInputFiles attaches files named from the project root 143ms
🟢 control inventory › the details view is left through the browser history 151ms
🟢 control inventory › scrollIntoView brings the footnote into view 237ms
🟢 control inventory › every exercise done shows the success message 1.3s
🟢 tests/css-content-ui.e2e.ts · 2 passed 313ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 177ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 136ms
🟢 tests/date-picker.e2e.ts · 2 passed 561ms
🟢 date picker › navigates three months ahead and books the required date 311ms
🟢 date picker › confirming without the exact date shows the error 250ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 593ms
🟢 div soup › a wrong code is rejected 260ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 333ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 599ms
🟢 drag and drop › an out-of-order item resets the dropzone 300ms
🟢 drag and drop › dropping Banana then Cherry passes 299ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 210ms
🟢 filter deep link › filtering writes the query to the URL 119ms
🟢 filter deep link › opening a shared link restores the filter 91ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 502ms
🟢 gift card purchase › refuses to buy without an amount 137ms
🟢 gift card purchase › refuses a recipient without an at sign 181ms
🟢 gift card purchase › sends the gift card 184ms
🟢 tests/hover-menu.e2e.ts · 2 passed 416ms
🟢 hover menu › a decoy leaf is refused 185ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 231ms
🟢 tests/iframe-form.e2e.ts · 2 passed 532ms
🟢 iframe form › a wrong code is refused by the host page 263ms
🟢 iframe form › the code read from one frame applies through the other 269ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 479ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 110ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 174ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 195ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.7s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.7s
🟢 tests/kanban-board.e2e.ts · 3 passed 735ms
🟢 kanban board › pointer drags move both cards into the goal layout 371ms
🟢 kanban board › a swipe along a path drags a card as well 148ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 216ms
🟢 tests/login-form.e2e.ts · 4 passed 764ms
🟢 login form › rejects a malformed email 184ms
🟢 login form › rejects a short password 180ms
🟢 login form › rejects an unknown account 170ms
🟢 login form › signs in and logs out 230ms
🟢 tests/lying-labels.e2e.ts · 4 passed 596ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 131ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 98ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 184ms
🟢 lying labels › visible text drives the payment through 183ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 433ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 171ms
🟢 native dialogs › dismissing the confirm cancels the order 130ms
🟢 native dialogs › a wrong gift code is rejected 132ms
🟢 tests/new-tab-flow.e2e.ts · 2 passed 466ms
🟢 new tab flow › reads the code in the tab the app opens and enters it back in the original tab 228ms
🟢 new tab flow › a tap that opens a tab leaves the screen on the original tab, and activate shows the new one 238ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.4s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 245ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 330ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.1s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.3s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.4s
🟢 tests/order-history.e2e.ts · 2 passed 1.5s
🟢 order history › overview and settings render their panels 194ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.3s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 467ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 147ms
🟢 otp auto-advance › a wrong code clears every box and starts over 773ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 644ms
🟢 continues only after the banner and the overlay are gone 644ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 541ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 260ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 281ms
🟢 tests/price-sorting.e2e.ts · 3 passed 393ms
🟢 price sorting › opens in the featured order with sale prices struck through 120ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 135ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 138ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.3s
🟢 buys a deal once the promo chrome is out of the way 1.3s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 440ms
🟢 rich text editor › bolding exactly the required word passes the check 224ms
🟢 rich text editor › bold that spills into the previous word fails the check 216ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 551ms
🟢 saved addresses › adds an address and clears the form 192ms
🟢 saved addresses › ignores an add with an empty field 162ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 197ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 533ms
🟢 shadow DOM form › rejects a wrong access code 266ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 267ms
🟢 tests/smoke.e2e.ts · 1 passed 396ms
🟢 home lists the scenarios and opens one 396ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 183ms
🟢 accepts the terms once the button clears the sticky footer 183ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 14.6s
🟢 scrolls the windowed table to the Golden Row and claims it 14.6s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 249ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

Tested by e2e 0.23.1 · 1m 1s · web · run artifacts

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 34 passed

35 agent steps · 26 replayed from cache · 27 model calls · 127.6k tokens (88% cached) · $0.0069

All 34 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.2s
🟢 control inventory › check sets the checkbox and the radio 1 step 597ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 230ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 871ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.5s
🟢 control inventory › upload attaches the named project files 1 step 2.4s
🟢 control inventory › back returns from the details view to the inventory 1 step 232ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 329ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 323ms
🟢 act signs in with the benchmark account 1 step 323ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 13.9s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 7.7s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.7s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 4.5s
🟢 tests-agent/scenarios.e2e.ts · 22 passed 22 steps · 19 calls 1m 16s
🟢 act completes shadow-dom-form 1 step · 5 calls 7.4s
🟢 act completes lying-labels 1 step · 3 calls 5.1s
🟢 act completes iframe-form 1 step · 3 calls 4.1s
🟢 act completes infinite-scroll 1 step 3.2s
🟢 act completes virtualized-table 1 step 15.5s
🟢 act completes overlay-trap 1 step 827ms
🟢 act completes sticky-chrome 1 step 891ms
🟢 act completes async-states 1 step · 3 calls 6.7s
🟢 act completes debounced-search 1 step 2.2s
🟢 act completes hover-menu 1 step 448ms
🟢 act completes drag-and-drop 1 step 400ms
🟢 act completes kanban-board 1 step 585ms
🟢 act completes context-menu-trap 1 step 2.6s
🟢 act completes stale-dom 1 step 417ms
🟢 act completes otp-auto-advance 1 step 637ms
🟢 act completes date-picker 1 step 714ms
🟢 act completes rich-text-editor 1 step 10.1s
🟢 act completes playbook-cleanup 1 step 509ms
🟢 act completes promo-storefront 1 step 2.2s
🟢 act completes new-tab-flow 1 step · 5 calls 8.9s
🟢 act completes gift-card-purchase 1 step 2.4s
🟢 act filters the list and the URL carries the filter 1 step 195ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 238ms
🟢 act adds an address with a run-unique label 1 step 238ms

Tested by e2e 0.23.1 · 1m 38s · web · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/mobile/src/engine.ts Resolved
Comment thread docs/examples/core/checkout.ts Resolved
Comment thread docs/reference/security.mdx Resolved
Comment thread packages/core/src/config/app.ts Resolved
Comment thread docs/examples/core/prices.ts Resolved
Comment thread packages/core/src/cache/recorder.ts Resolved
Comment thread docs/reference/config.mdx Resolved
Comment thread packages/mobile/README.md Resolved
Comment thread .changeset/e2e-core.md Resolved
Comment thread packages/core/src/core/signals.ts Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 688 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Note: This PR contains a large number of files. cubic selects up to 200 of the highest-priority eligible files for this review, so some files may not have been reviewed.
Requires human review: Auto-approval blocked because this review re-detected 10 unresolved issues already reported by Cubic.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/core/src/agent/step-cache.ts Resolved
Comment thread packages/core/src/engine/manifest.ts Resolved
Comment thread packages/mobile/src/provider.ts Resolved
Comment thread docs/reference/agent.mdx Resolved
Comment thread packages/core/src/config/app.ts Resolved
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 1 flaky, 19 passed

24 agent steps · 15 replayed from cache · 40 model calls · 328.6k tokens (79% cached) · $0.0260

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:266

ACTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 37.4s and 14 model calls

agent.act failed: The cached long-press had already advanced to Step 2. I attempted several leftward swipes across the visible “Swipe me left” card using both screenshot and scaled screen coordinates, but the screen remained on Step 2 of 3; the required swipe did not complete.

  • Turn 11: screenshot({}) → [screenshot] The screen changed substantially since revision b14. Current screen (revision b16, path dev.e2e.benchmark, 18 nodes):
  • Turn 12: swipe({"from":{"x":70,"y":140},"to":{"x":30,"y":140}}) → [swipe] Swiped from (70, 140) to (30, 140).
  • Turn 13: screenshot({}) → [screenshot] The screen changed substantially since revision b16. Current screen (revision b18, path dev.e2e.benchmark, 18 nodes):
  • Screen: dev.e2e.benchmark

Evidence: screenshot, log

All 20 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 19 passed 24 steps · 40 calls 5m 55s
🟢 act completes Login Form 1 step 24.3s
🟢 act completes Infinite Scroll List 1 step 20.1s
🟢 act completes Modal Flow 1 step 11.6s
🟢 act completes Bottom Tabs 1 step 7.7s
🟢 act completes Text Input Variations 1 step 7.1s
🟢 act completes Broken Accessibility 2 steps · 5 calls 13.8s
🟢 act completes Vision Only 2 steps · 1 call 6.6s
⚠️ act completes Gestures (1 failed attempt first) 1 step · 9 calls 29.1s
🟢 act completes Async States 1 step · 13 calls 33.8s
🟢 act completes Debounced Search 1 step 6.5s
🟢 act completes Huge Virtualized List 1 step 1m 49s
🟢 act completes Flattened Registration Form 2 steps · 1 call 8.3s
🟢 act completes Flattened Login 2 steps · 6 calls 15.1s
🟢 act completes Sticky Chrome Target 1 step 8.6s
🟢 act completes WebView Accessibility 1 step · 5 calls 13.6s
🟢 act completes Permission Prompt 1 step 6.1s
🟢 act completes Product Catalog 1 step 7.9s
🟢 act completes Error Recovery 1 step 8.7s
🟢 act completes Choice Controls 1 step 8.0s
🟢 act completes Sequential Onboarding 1 step 9.3s

Tested by e2e 0.23.1 · 6m 36s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 12 replayed from cache · 55 model calls · 342.4k tokens (82% cached) · $0.0227

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 55 calls 8m 28s
🟢 act completes Login Form 1 step 37.6s
🟢 act completes Infinite Scroll List 1 step 38.6s
🟢 act completes Modal Flow 1 step 13.2s
🟢 act completes Bottom Tabs 1 step 8.7s
🟢 act completes Text Input Variations 1 step 13.5s
🟢 act completes Broken Accessibility 2 steps · 5 calls 24.5s
🟢 act completes Vision Only 2 steps · 6 calls 29.6s
🟢 act completes Gestures 1 step · 7 calls 32.9s
🟢 act completes Async States 1 step · 9 calls 32.7s
🟢 act completes Debounced Search 1 step · 1 call 27.2s
🟢 act completes Flattened Registration Form 2 steps · 1 call 26.3s
🟢 act completes Flattened Login 2 steps · 10 calls 54.6s
🟢 act completes Sticky Chrome Target 1 step 13.5s
🟢 act completes WebView Accessibility 1 step 13.0s
🟢 act completes Permission Prompt 1 step 12.2s
🟢 act completes Photo Picker 1 step · 2 calls 29.9s
🟢 act completes Product Catalog 1 step 10.1s
🟢 act completes Error Recovery 1 step 10.2s
🟢 act completes Choice Controls 1 step · 6 calls 36.7s
🟢 act completes Stripe PaymentSheet 1 step · 8 calls 31.6s
🟢 act completes Apple Pay 1 step 11.0s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.1 · 8m 28s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 21 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/core/src/cache/trace-read.ts Resolved
Comment thread docs/reference/config.mdx Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 18 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 58 passed, 1 skipped

All 59 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 27s
🟢 control inventory › a label query resolves a labeled text field 19.5s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 10.6s
🟢 control inventory › a placeholder query and attribute reads find the field 4.6s
🟢 control inventory › a secure field takes a fill and denies value reads 5.8s
🟢 control inventory › press sends a character and Enter through the soft keyboard 6.5s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 7.1s
🟢 control inventory › toBeFocused follows the focus 7.1s
🟢 control inventory › a switch checks, unchecks, and reports its state 7.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 6.6s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 9.6s
🟢 control inventory › a tab role answers getByRole 3.5s
🟢 control inventory › a radio group reports the checked option 7.2s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.4s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 5.7s
🟢 control inventory › a header answers its accessible name 3.3s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 4.9s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 4.9s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 6.6s
🟢 control inventory › a double tap lands inside the 300 ms window 6.0s
🟢 control inventory › an ambiguous locator fails at once 4.6s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 5.9s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.7s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 13s
🟢 device fixture › setOrientation rotates the window the app measures 4.7s
🟢 device fixture › home leaves the app, openApp brings it back where it was 3.7s
🟢 device fixture › foregroundApp sees the home screen after home() 3.4s
🟢 device fixture › closeApp then openApp relaunches on the home list 5.6s
🟢 device fixture › closeApp leaves the next commands on the same device 5.6s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.1s
🟢 device fixture › openLink opens the scenario a deep link names 4.2s
🟢 device fixture › device.back pops the scenario 3.9s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 3.6s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 14.9s
🟢 device fixture › setLocation and clearLocation change what the app reads 19.6s
🟢 tests/device.e2e.ts · 5 passed 36.4s
🟢 permission prompt › allowing the system dialog enables the microphone 5.3s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 10.9s
🟢 sequential onboarding echoes the exact values 9.1s
🟢 onboarding rejects a malformed email before advancing 5.1s
🟢 dismissing the keyboard over choice controls leaves them as they were 5.9s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 2s
🟢 infinite scroll reaches the target item 26.9s
🟢 product catalog adds exactly two of the right variant 8.0s
🟢 gestures: long-press, then swipe left 7.5s
🟢 gestures: double-tap completes the flow 9.1s
🟢 sticky chrome: accept below the fold, then continue 10.2s
🟢 async states: load, pull to refresh, claim through the toast 12.4s
🟢 huge virtualized list reaches row 512 1m 47s
🟢 tests/flows.e2e.ts · 7 passed 45.5s
🟢 text inputs unlock the submit button 5.7s
🟢 filled text inputs report their values 4.6s
🟢 modal flow confirms through the native alert 8.0s
🟢 bottom tabs act inside the last tab 6.1s
🟢 error recovery retries, then confirms the delete 7.8s
🟢 choice controls place the exact order 5.4s
🟢 debounced search selects the target once results arrive 7.9s
🟢 tests/login-form.e2e.ts · 3 passed 25.1s
🟢 login form › rejects a malformed email 6.5s
🟢 login form › rejects an unknown account 6.7s
🟢 login form › signs in and logs out 11.8s
🟢 tests/smoke.e2e.ts · 1 passed 4.7s
🟢 home lists the scenarios and opens one 4.7s

Tested by e2e 0.23.1 · 8m 34s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 56 passed, 1 skipped

All 57 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 2m 59s
🟢 control inventory › a label query resolves a labeled text field 11.8s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 15.9s
🟢 control inventory › a placeholder query and attribute reads find the field 5.4s
🟢 control inventory › a secure field takes a fill and denies value reads 7.7s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.4s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.7s
🟢 control inventory › toBeFocused follows the focus 9.6s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.0s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.5s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.5s
🟢 control inventory › a radio group reports the checked option 10.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.7s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.5s
🟢 control inventory › a header answers its accessible name 4.2s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.0s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.5s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.3s
🟢 control inventory › pointer verbs land at the point asked for 9.1s
🟢 control inventory › an ambiguous locator fails at once 6.1s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 10.5s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 515ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 12 passed 1m 53s
🟢 device fixture › setAppearance flips the color scheme the app reads 12.0s
🟢 device fixture › setOrientation rotates the window the app measures 10.3s
🟢 device fixture › the clipboard reads back what was written 8.3s
🟢 device fixture › home leaves the app, openApp brings it back where it was 8.1s
🟢 device fixture › closeApp then openApp relaunches on the home list 9.4s
🟢 device fixture › closeApp leaves the next commands on the same device 12.0s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.4s
🟢 device fixture › openLink opens the scenario a deep link names 14.4s
🟢 device fixture › device.back pops the scenario 6.8s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 4.2s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.1s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 11.3s
🟢 tests/device.e2e.ts · 3 passed 47.2s
🟢 sequential onboarding echoes the exact values 26.5s
🟢 onboarding rejects a malformed email before advancing 9.7s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.1s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 58s
🟢 infinite scroll reaches the target item 36.7s
🟢 product catalog adds exactly two of the right variant 9.8s
🟢 gestures: long-press, then swipe left 8.5s
🟢 sticky chrome: accept below the fold, then continue 12.3s
🟢 async states: load, pull to refresh, claim through the toast 11.4s
🟢 huge virtualized list reaches row 512 2m 27s
🟢 web view coupon form applies the code shown on the page 12.3s
🟢 tests/flows.e2e.ts · 7 passed 1m 7s
🟢 text inputs unlock the submit button 12.0s
🟢 filled text inputs report their values 10.2s
🟢 modal flow confirms through the native alert 9.6s
🟢 bottom tabs act inside the last tab 7.3s
🟢 error recovery retries, then confirms the delete 9.2s
🟢 choice controls place the exact order 7.3s
🟢 debounced search selects the target once results arrive 11.5s
🟢 tests/login-form.e2e.ts · 3 passed 44.7s
🟢 login form › rejects a malformed email 12.5s
🟢 login form › rejects an unknown account 12.7s
🟢 login form › signs in and logs out 19.5s
🟢 tests/smoke.e2e.ts · 1 passed 7.8s
🟢 home lists the scenarios and opens one 7.8s

Tested by e2e 0.23.1 · 6m 39s · ios-simulator · run artifacts

@kdy1

kdy1 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

[claude-opus-5-5] responding on behalf of kdy1

Left for a human:

  • engine/manifest.ts thread: whether defineEngine<Fixtures>() should infer its fixture type from spec.fixtures instead of taking it explicitly. That is a public type contract choice.
  • The nightly examples workflow's new e2e-core override (.github/actions/example-packages/action.yml) only runs on the nightly; it could not be run locally.
  • e2e-core needs a first manual npm publish before Trusted Publishing can be set up for it.

@kdy1 kdy1 changed the title feat(core): add e2e-core and build e2e on it feat(core): add launch, SessionState, and createAgent to e2e-core Nov 13, 2026
@kdy1
kdy1 changed the base branch from main to kdy1/e2e-core-2-move November 13, 2026 06:58
@kdy1
kdy1 force-pushed the kdy1/e2e-core branch 2 times, most recently from a6e56df to ddc9313 Compare November 13, 2026 07:06
@kdy1

kdy1 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

[claude-opus-5-5] responding on behalf of kdy1

This PR is now layer 3 of a stack: #1736 → #1737 → #1734. It supersedes my earlier handoff comment here:

@okwasniewski okwasniewski changed the title feat(core): add launch, SessionState, and createAgent to e2e-core feat(core): add launch, SessionState, and createAgent to @e2e-dev/core Nov 15, 2026
@okwasniewski
okwasniewski force-pushed the kdy1/e2e-core branch 2 times, most recently from 500cd14 to 8efa596 Compare November 15, 2026 18:18
Base automatically changed from kdy1/e2e-core-2-move to main November 15, 2026 18:32
kdy1 and others added 3 commits November 15, 2026 18:32
@e2e-dev/core becomes usable on its own. launch(engine) opens a core
attempt on the standalone worker (nothing published to globalThis, no
process secret registry, no trace);
sessions save and restore session-state-1 state, AES-256-GCM encrypted
under a caller's key, and take secrets from launch(). createAgent from
@e2e-dev/core/agent adds the agent with a replay cache keyed by cacheKey and
a guard the dispatcher checks before every action, replayed ones
included. @e2e-dev/web/engine and @e2e-dev/mobile/engine export the
factories without test, so the engines' e2e peer becomes optional.
Device targets take app.linkBase, opened through session.openLink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… names

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main changed AgentDeviceSurface.inputBudget to take the operation's signal
and timeout instead of a number of milliseconds; the openLink hook added in
this branch still passed operation.timeoutMs, which no longer typechecks
after the rebase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, sessions, and the agent cache

Guard
- Reads the screen the turn's earlier actions left before it names what an
  action lands on, so a batched turn cannot walk past a stop rule.
- A `key` rule matches the key under any modifier, and typed text holding a
  line break counts as Enter. A misspelled `key` or `actions` name is
  INVALID_ARGUMENT instead of a rule that never matches.
- A stop that `check` decided is asked again when its recording replays.

Sessions
- A caught agent failure drops only the recordings it found unconfirmed, and
  a close before a check evicts nothing.
- `launch({ timeout })` and `setDefaultTimeout` bound app opens and restarts.
- SIGHUP ends the program though the engine listens for it.
- Thrown errors are classified and redacted; a closed session says so.
- A fixture may not take the name of a session method.
- A recording's key no longer depends on the working directory.
- A session keeps its newest 500 step records.

Types
- `createAgent` takes a `Session`, and options held in a variable give an
  agent whose `act` takes either form.
- The agent entry exports the executor, store, and agent config types.
- New reference page: launch options, host, session, saved state,
  createAgent, cache, guard, and the act result.
- Guide: Node 22 form without `await using`, the state flow closes the
  session first, what secrets are redacted from, error handling, default
  timeouts, and the install line for the agent.
- The checkout example no longer checks an error code a program never sees.
A `unique symbol` is unique to the declaration file that declares it. A
project that updates `e2e` or an engine alone holds two copies of
@e2e-dev/core, and `engine: web()` then failed to typecheck against the
runner's `EngineHandle`, as did `expect(locator)` and `expect(browser)`.
The runtime keys stay the registered symbols, which every copy shares.
Removes guard, ActResult.status and stoppedAt, and the recorded stop
from the trace. The agent loop and trace format are back to main's.
Kept for a follow-up.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants