Skip to content

feat(mcp): open a session without a config by naming the engine - #1665

Merged
kdy1 merged 3 commits into
mainfrom
krzysztof/feat/mcp-without-config
Nov 10, 2026
Merged

kdy1 merged 3 commits into
mainfrom
krzysztof/feat/mcp-without-config

Conversation

@KrzysztofMoch

@KrzysztofMoch KrzysztofMoch commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

e2e mcp only worked next to an e2e.config.ts. A coding agent in a fresh directory got CONFIG_NOT_FOUND and had to set a project up before it could look at an app. Now open_session can name the engine instead.

Fixes ENG-1191.

open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}

open_session {
  engine: {package: "@e2e-dev/mobile", options: {
    platform: "ios",
    device: {package: "@e2e-dev/limrun", export: "limrun", options: {region: "eu"}}
  }},
  app: {bundleId: "com.example.app"}
}
  • engine is a package name, or the factory call a config would write as {package, export?, options?}. The second call above is mobile({ platform: 'ios', device: limrun({ region: 'eu' }) }).
  • An option value with only those keys is called first and replaced by its result. That is how an engine gets a device or browser provider, which JSON can't carry. export names any export, so providers already on npm work without a release.
  • app takes url, bundleId, and appPath only.
  • Plain open_session with no config still fails. The error now shows both shapes, which is how an agent finds the option.
  • engine is refused where a config exists, and together with target, config, or test.
  • @e2e-dev/web and @e2e-dev/mobile default-export their factory. mobile() without a platform now says which platforms it takes.

Where a package comes from

flowchart LR
  A[package name] --> B{in the project?}
  B -- yes --> L[load it]
  B -- no --> C{beside e2e?}
  C -- yes --> L
  C -- no --> D{in ~/.e2e/engines?}
  D -- yes --> L
  D -- no --> E[ENGINE_NOT_INSTALLED<br/>names the npm command]
Loading
  • The server installs nothing. The agent runs the command the error prints, in its own shell.
  • An engine that brings its own copy of e2e works. Only the handle's brand and contract version cross.
  • Core still names no engine: packages/e2e/src/mcp/default-engine.ts imports a resolved file, never a specifier.

Security

  • This is the first MCP argument that makes the server load code the agent named. docs/reference/security.mdx says so.
  • Only installed packages load. A path, a URL, a .. segment, and a Node builtin name are refused with INVALID_ARGUMENT.
  • A __proto__ key in options stays an own property.

Open for a decision

  • A nested factory call is recognized by shape: package plus nothing but export and options. A third-party engine option that looks like that would be read as a call. No shipped engine or provider has one. The fresh-context reviewer suggested a reserved key such as {$call: {...}}. I kept the plain shape, made the options object itself never a call, and made the not-installed error say how the value was read.

Verified

Ran it locally: yes

Surface: the built e2e mcp server in directories with no config.

Final build, driven over stdio (limrun here is the repo's, with #1663):

>>> open_session {}
CONFIG_NOT_FOUND: no e2e.config.ts, e2e.config.mts, e2e.config.js, or e2e.config.mjs found in <project> or its parent directories; run e2e init to create one, or pass engine to open_session to open a session without one: {engine: "@e2e-dev/web", app: {url: "<app URL>"}}, or {engine: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, app: {bundleId: "<app>"}}

>>> open_session {"engine":"util"}
INVALID_ARGUMENT: open_session: engine names an installed package such as "@e2e-dev/web", got "util"

>>> open_session {"engine":"@e2e-dev/mobile","app":{"bundleId":"com.apple.mobilesafari"}}
INVALID_CONFIG: mobile() needs platform: 'ios' or 'android', got no platform

>>> open_session {"engine":{"package":"@e2e-dev/mobile","options":{"platform":"ios","device":{"package":"@e2e-dev/absent-cloud"}}},"app":{"bundleId":"com.apple.mobilesafari"}}
ENGINE_NOT_INSTALLED: an option value with only package, export, and options keys is read as a factory call, and package @e2e-dev/absent-cloud is not installed in <project>, beside e2e, or in ~/.e2e/engines. Install it once for every project with: npm install --prefix "~/.e2e/engines" @e2e-dev/absent-cloud; then call open_session again

>>> open_session {"engine":{"package":"@e2e-dev/mobile","options":{"platform":"ios","device":{"package":"@e2e-dev/limrun","export":"limrun","options":{"hardTimeout":"15m"}}}},"app":{"bundleId":"com.apple.mobilesafari"}}
Session 01a1c478-7df0-770e-ab87-b38dcb0a1590 open on target "ios" (platform ios, engine mobile 0.12.6), headless; no config file: one target on @e2e-dev/mobile, every other setting at its default. run_tests and open_session {test} need an e2e.config.ts (npx e2e init writes one).
App: com.apple.mobilesafari, launched fresh.

>>> close_session {}
Session 01a1c478-7df0-770e-ab87-b38dcb0a1590 closed (closed by the agent); 1 tool calls ran.

Real clients calling the tools, on earlier builds of this branch:

  • Codex, GPT-6.1 Sol (before the last review-fix commit). The server was added with codex mcp add. Codex showed the model engine?: string | { package, export?, options? }. Sol opened e2e.com with the string form, tapped [ docs ], read test_lines, then opened /docs/quickstart with the object form. Every step passed.
  • Claude Code (string form). A fresh session with the server registered by claude mcp add opened e2e.com, went to the docs, and returned a four-line draft test.
  • Two copies of e2e. The runner alone in one tree, @e2e-dev/web installed with the printed command into a separate ~/.e2e/engines: a session opened, run_code passed a tap and an assertion.
  • iOS simulator (string form). @e2e-dev/mobile drove Safari on a local simulator to e2e.com and the docs.
Web session, docs page Limrun device, nested provider
web limrun

Not verified: Android, and an engine installed with pnpm or Bun into ~/.e2e/engines.

Review

  • Fresh-context review, seven findings. Fixed: import-only packages failed to load, an installed package with no built entry was reported as not installed, engine: "@e2e-dev/mobile" failed with a raw TypeError, a Node builtin name became a file import, two untested hunks, a stale comment. Kept with changes: the shape rule above.
  • Bot review on the PR, eight findings. Fixed six: a skill example with no app.url value, a __proto__ test that would pass if the key were dropped, "no e2e.config.ts" where any config name counts, one debugging-table row covering two error codes, the string shorthand missing from AGENTS.md, and a security sentence that read as covering more than engine. Declined two: offering engine when a lookalike config file exists (that branch predates this PR and names the rename as the fix), and wildcard subpaths of import-only packages (the import fallback is deliberately narrow and fails with an explicit error).
  • Reviewed with sol-loop: 2 rounds with gpt-6.1-sol. Sol verdict: approved. That was before the review-fix commits.

Docs pass: docs/mcp-server.mdx 199 to 212 lines, 1065 to 1105 prose words. docs/writing-an-engine.mdx 306 to 326 lines, 1606 to 1665 words. Both gain one short section; nothing was cut.

Written by Claude Fable 5.2 in T3 Code, on the Claude Code harness.

View guided diff Turn on auto-fix

open_session takes engine, a package name or the factory call a config
would write as {package, export, options}, and app. An option can be
another such call, which is how an engine gets a device or browser
provider. Packages are found in the project, beside e2e, or in
~/.e2e/engines; ENGINE_NOT_INSTALLED names the install command.
@e2e-dev/web and @e2e-dev/mobile default-export their factory.
Review fixes: resolve a package by its node_modules directory, so an
installed package without a loadable entry is reported as such; fall
back to the import condition; refuse Node builtin names; never read
the options object itself as a factory call; mobile() names the
platforms it takes when called without one.
@github-actions github-actions Bot added the size:XL Up to 999 changed lines, not counting tests, fixtures, or the lockfile label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 91 passed

All 91 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 640ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 471ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 169ms
🟢 tests/async-states.e2e.ts · 2 passed 7.1s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.3s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 234ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 126ms
🟢 canvas only › a shape out of order paints the error and resets progress 108ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 386ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 235ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 151ms
🟢 tests/cart-totals.e2e.ts · 2 passed 342ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 175ms
🟢 cart totals › a quantity never drops below one 167ms
🟢 tests/checkout-review.e2e.ts · 1 passed 146ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 146ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 483ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 253ms
🟢 context menu trap › the decoy menu items refuse and keep the file 230ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.4s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 294ms
🟢 control inventory › check picks a radio the pick replaces with its summary 143ms
🟢 control inventory › a long press and a double click are told apart from a tap 973ms
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 219ms
🟢 control inventory › setInputFiles attaches files named from the project root 137ms
🟢 control inventory › the details view is left through the browser history 161ms
🟢 control inventory › scrollIntoView brings the footnote into view 250ms
🟢 control inventory › every exercise done shows the success message 1.3s
🟢 tests/css-content-ui.e2e.ts · 2 passed 318ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 178ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 140ms
🟢 tests/date-picker.e2e.ts · 2 passed 739ms
🟢 date picker › navigates three months ahead and books the required date 414ms
🟢 date picker › confirming without the exact date shows the error 325ms
🟢 tests/debounced-search.e2e.ts · 2 passed 3.0s
🟢 debounced search › a decoy one character off is refused 1.5s
🟢 debounced search › picking the exact item adds it to the cart 1.5s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 707ms
🟢 div soup › a wrong code is rejected 299ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 408ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 740ms
🟢 drag and drop › an out-of-order item resets the dropzone 414ms
🟢 drag and drop › dropping Banana then Cherry passes 326ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 276ms
🟢 filter deep link › filtering writes the query to the URL 154ms
🟢 filter deep link › opening a shared link restores the filter 122ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 571ms
🟢 gift card purchase › refuses to buy without an amount 136ms
🟢 gift card purchase › refuses a recipient without an at sign 217ms
🟢 gift card purchase › sends the gift card 218ms
🟢 tests/hover-menu.e2e.ts · 2 passed 452ms
🟢 hover menu › a decoy leaf is refused 213ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 239ms
🟢 tests/iframe-form.e2e.ts · 2 passed 534ms
🟢 iframe form › a wrong code is refused by the host page 268ms
🟢 iframe form › the code read from one frame applies through the other 266ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 453ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 118ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 135ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 200ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.7s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.7s
🟢 tests/kanban-board.e2e.ts · 3 passed 719ms
🟢 kanban board › pointer drags move both cards into the goal layout 364ms
🟢 kanban board › a swipe along a path drags a card as well 137ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 218ms
🟢 tests/login-form.e2e.ts · 4 passed 747ms
🟢 login form › rejects a malformed email 183ms
🟢 login form › rejects a short password 166ms
🟢 login form › rejects an unknown account 182ms
🟢 login form › signs in and logs out 216ms
🟢 tests/lying-labels.e2e.ts · 4 passed 627ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 136ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 110ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 199ms
🟢 lying labels › visible text drives the payment through 182ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 419ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 136ms
🟢 native dialogs › dismissing the confirm cancels the order 150ms
🟢 native dialogs › a wrong gift code is rejected 133ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.6s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 290ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 331ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.2s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.3s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.4s
🟢 tests/order-history.e2e.ts · 2 passed 1.4s
🟢 order history › overview and settings render their panels 180ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.2s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 490ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 160ms
🟢 otp auto-advance › a wrong code clears every box and starts over 779ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 698ms
🟢 continues only after the banner and the overlay are gone 698ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 481ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 252ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 229ms
🟢 tests/price-sorting.e2e.ts · 3 passed 420ms
🟢 price sorting › opens in the featured order with sale prices struck through 118ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 147ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 155ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 429ms
🟢 rich text editor › bolding exactly the required word passes the check 210ms
🟢 rich text editor › bold that spills into the previous word fails the check 219ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 532ms
🟢 saved addresses › adds an address and clears the form 170ms
🟢 saved addresses › ignores an add with an empty field 161ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 201ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 531ms
🟢 shadow DOM form › rejects a wrong access code 265ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 266ms
🟢 tests/smoke.e2e.ts · 1 passed 409ms
🟢 home lists the scenarios and opens one 409ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.1s
🟢 taps the moving target three times across rebuilds of the list 2.1s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 200ms
🟢 accepts the terms once the button clears the sticky footer 200ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 14.7s
🟢 scrolls the windowed table to the Golden Row and claims it 14.7s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 230ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

Tested by e2e 0.23.0 · 1m 2s · web · run artifacts

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 33 passed

34 agent steps · 26 replayed from cache · 20 model calls · 88.2k tokens (94% cached) · $0.0036

All 33 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.4s
🟢 control inventory › check sets the checkbox and the radio 1 step 689ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 253ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 886ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.5s
🟢 control inventory › upload attaches the named project files 1 step 2.4s
🟢 control inventory › back returns from the details view to the inventory 1 step 267ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 370ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 355ms
🟢 act signs in with the benchmark account 1 step 355ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 10.5s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 5.4s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.6s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 3.5s
🟢 tests-agent/scenarios.e2e.ts · 21 passed 21 steps · 12 calls 1m 5s
🟢 act completes shadow-dom-form 1 step · 3 calls 3.8s
🟢 act completes lying-labels 1 step · 3 calls 5.1s
🟢 act completes iframe-form 1 step · 3 calls 3.6s
🟢 act completes infinite-scroll 1 step 3.4s
🟢 act completes virtualized-table 1 step 17.8s
🟢 act completes overlay-trap 1 step 860ms
🟢 act completes sticky-chrome 1 step 938ms
🟢 act completes async-states 1 step · 3 calls 6.0s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 485ms
🟢 act completes drag-and-drop 1 step 450ms
🟢 act completes kanban-board 1 step 613ms
🟢 act completes context-menu-trap 1 step 2.6s
🟢 act completes stale-dom 1 step 445ms
🟢 act completes otp-auto-advance 1 step 696ms
🟢 act completes date-picker 1 step 785ms
🟢 act completes rich-text-editor 1 step 10.0s
🟢 act completes playbook-cleanup 1 step 579ms
🟢 act completes promo-storefront 1 step 2.3s
🟢 act completes gift-card-purchase 1 step 2.4s
🟢 act filters the list and the URL carries the filter 1 step 196ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 253ms
🟢 act adds an address with a run-unique label 1 step 253ms

Tested by e2e 0.23.0 · 1m 24s · web · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 28 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread skills/e2e/SKILL.md Resolved
Comment thread packages/e2e/src/config/load.ts Resolved
Comment thread packages/e2e/src/mcp/server.ts Resolved
Comment thread AGENTS.md Resolved
Comment thread packages/e2e/src/mcp/default-engine.ts Resolved
Comment thread docs/reference/security.mdx Resolved
… the __proto__ key

Review fixes: a concrete app.url in the skill example, the string
shorthand in AGENTS.md, the engine-only wording in the security note,
and a test assertion that the __proto__ key survives as an own property.
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 11 replayed from cache · 68 model calls · 427.9k tokens (83% cached) · $0.0281

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 68 calls 9m 33s
🟢 act completes Login Form 1 step · 6 calls 1m 21s
🟢 act completes Infinite Scroll List 1 step 36.9s
🟢 act completes Modal Flow 1 step 10.7s
🟢 act completes Bottom Tabs 1 step 7.4s
🟢 act completes Text Input Variations 1 step 14.0s
🟢 act completes Broken Accessibility 2 steps · 8 calls 39.4s
🟢 act completes Vision Only 2 steps · 6 calls 28.1s
🟢 act completes Gestures 1 step · 7 calls 31.3s
🟢 act completes Async States 1 step · 10 calls 31.9s
🟢 act completes Debounced Search 1 step · 1 call 28.7s
🟢 act completes Flattened Registration Form 2 steps · 1 call 31.0s
🟢 act completes Flattened Login 2 steps · 10 calls 52.5s
🟢 act completes Sticky Chrome Target 1 step 14.4s
🟢 act completes WebView Accessibility 1 step 13.5s
🟢 act completes Permission Prompt 1 step 12.9s
🟢 act completes Photo Picker 1 step · 2 calls 31.9s
🟢 act completes Product Catalog 1 step 10.4s
🟢 act completes Error Recovery 1 step 9.8s
🟢 act completes Choice Controls 1 step · 7 calls 37.1s
🟢 act completes Stripe PaymentSheet 1 step · 10 calls 35.5s
🟢 act completes Apple Pay 1 step 14.2s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.0 · 9m 33s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 11 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 39 model calls · 215.0k tokens (86% cached) · $0.0126

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 39 calls 5m 45s
🟢 act completes Login Form 1 step 22.6s
🟢 act completes Infinite Scroll List 1 step 21.3s
🟢 act completes Modal Flow 1 step 9.7s
🟢 act completes Bottom Tabs 1 step 7.5s
🟢 act completes Text Input Variations 1 step 5.4s
🟢 act completes Broken Accessibility 2 steps · 6 calls 16.4s
🟢 act completes Vision Only 2 steps · 1 call 6.4s
🟢 act completes Gestures 1 step · 7 calls 18.9s
🟢 act completes Async States 1 step · 12 calls 27.8s
🟢 act completes Debounced Search 1 step 6.2s
🟢 act completes Huge Virtualized List 1 step 1m 53s
🟢 act completes Flattened Registration Form 2 steps · 1 call 7.6s
🟢 act completes Flattened Login 2 steps · 6 calls 17.4s
🟢 act completes Sticky Chrome Target 1 step 8.9s
🟢 act completes WebView Accessibility 1 step · 6 calls 15.1s
🟢 act completes Permission Prompt 1 step 6.3s
🟢 act completes Product Catalog 1 step 8.1s
🟢 act completes Error Recovery 1 step 9.0s
🟢 act completes Choice Controls 1 step 8.2s
🟢 act completes Sequential Onboarding 1 step 9.6s

Tested by e2e 0.23.0 · 5m 46s · android-emulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 35s
🟢 control inventory › a label query resolves a labeled text field 22.7s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 10.8s
🟢 control inventory › a placeholder query and attribute reads find the field 4.8s
🟢 control inventory › a secure field takes a fill and denies value reads 5.7s
🟢 control inventory › press sends a character and Enter through the soft keyboard 7.5s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 7.5s
🟢 control inventory › toBeFocused follows the focus 7.5s
🟢 control inventory › a switch checks, unchecks, and reports its state 7.8s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 6.8s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 10.3s
🟢 control inventory › a tab role answers getByRole 3.7s
🟢 control inventory › a radio group reports the checked option 7.3s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.4s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.4s
🟢 control inventory › a header answers its accessible name 3.5s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.3s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.0s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 6.8s
🟢 control inventory › a double tap lands inside the 300 ms window 6.1s
🟢 control inventory › an ambiguous locator fails at once 4.7s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 6.3s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.3s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 11s
🟢 device fixture › setOrientation rotates the window the app measures 4.9s
🟢 device fixture › home leaves the app, openApp brings it back where it was 3.9s
🟢 device fixture › foregroundApp sees the home screen after home() 3.8s
🟢 device fixture › closeApp then openApp relaunches on the home list 5.9s
🟢 device fixture › closeApp leaves the next commands on the same device 5.9s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.0s
🟢 device fixture › device.back pops the scenario 4.3s
🟢 device fixture › unstable_raw reaches the test session through agent-device 3.6s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.5s
🟢 device fixture › setLocation and clearLocation change what the app reads 19.4s
🟢 tests/device.e2e.ts · 5 passed 38.0s
🟢 permission prompt › allowing the system dialog enables the microphone 5.1s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 11.2s
🟢 sequential onboarding echoes the exact values 9.4s
🟢 onboarding rejects a malformed email before advancing 5.6s
🟢 dismissing the keyboard over choice controls leaves them as they were 6.7s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 16s
🟢 infinite scroll reaches the target item 29.9s
🟢 product catalog adds exactly two of the right variant 8.8s
🟢 gestures: long-press, then swipe left 8.5s
🟢 gestures: double-tap completes the flow 9.8s
🟢 sticky chrome: accept below the fold, then continue 11.6s
🟢 async states: load, pull to refresh, claim through the toast 12.2s
🟢 huge virtualized list reaches row 512 1m 55s
🟢 tests/flows.e2e.ts · 7 passed 46.4s
🟢 text inputs unlock the submit button 5.6s
🟢 filled text inputs report their values 5.2s
🟢 modal flow confirms through the native alert 7.9s
🟢 bottom tabs act inside the last tab 6.2s
🟢 error recovery retries, then confirms the delete 8.0s
🟢 choice controls place the exact order 5.7s
🟢 debounced search selects the target once results arrive 7.9s
🟢 tests/login-form.e2e.ts · 3 passed 26.7s
🟢 login form › rejects a malformed email 7.3s
🟢 login form › rejects an unknown account 7.5s
🟢 login form › signs in and logs out 11.9s
🟢 tests/smoke.e2e.ts · 1 passed 5.7s
🟢 home lists the scenarios and opens one 5.7s

Tested by e2e 0.23.0 · 9m 0s · android-emulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 7s
🟢 control inventory › a label query resolves a labeled text field 11.0s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 18.0s
🟢 control inventory › a placeholder query and attribute reads find the field 5.3s
🟢 control inventory › a secure field takes a fill and denies value reads 10.4s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.8s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.8s
🟢 control inventory › toBeFocused follows the focus 9.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.9s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 9.0s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 13.3s
🟢 control inventory › a radio group reports the checked option 9.4s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.3s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.2s
🟢 control inventory › a header answers its accessible name 4.3s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.4s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.7s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.2s
🟢 control inventory › pointer verbs land at the point asked for 9.5s
🟢 control inventory › an ambiguous locator fails at once 5.4s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 10.2s
🟢 refusals › verbs a device never declares are refused before the engine 3ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 605ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 44s
🟢 device fixture › setAppearance flips the color scheme the app reads 14.4s
🟢 device fixture › setOrientation rotates the window the app measures 8.7s
🟢 device fixture › the clipboard reads back what was written 6.4s
🟢 device fixture › home leaves the app, openApp brings it back where it was 7.1s
🟢 device fixture › closeApp then openApp relaunches on the home list 9.7s
🟢 device fixture › closeApp leaves the next commands on the same device 13.3s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.9s
🟢 device fixture › device.back pops the scenario 6.5s
🟢 device fixture › unstable_raw reaches the test session through agent-device 5.7s
🟢 device fixture › setLocation and clearLocation change what the app reads 13.1s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 11.7s
🟢 tests/device.e2e.ts · 3 passed 1m 8s
🟢 sequential onboarding echoes the exact values 42.9s
🟢 onboarding rejects a malformed email before advancing 14.0s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.2s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 23s
🟢 infinite scroll reaches the target item 36.6s
🟢 product catalog adds exactly two of the right variant 10.7s
🟢 gestures: long-press, then swipe left 9.0s
🟢 sticky chrome: accept below the fold, then continue 12.6s
🟢 async states: load, pull to refresh, claim through the toast 12.4s
🟢 huge virtualized list reaches row 512 2m 48s
🟢 web view coupon form applies the code shown on the page 13.0s
🟢 tests/flows.e2e.ts · 7 passed 1m 12s
🟢 text inputs unlock the submit button 14.1s
🟢 filled text inputs report their values 11.6s
🟢 modal flow confirms through the native alert 9.9s
🟢 bottom tabs act inside the last tab 7.6s
🟢 error recovery retries, then confirms the delete 9.5s
🟢 choice controls place the exact order 7.7s
🟢 debounced search selects the target once results arrive 11.7s
🟢 tests/login-form.e2e.ts · 3 passed 49.5s
🟢 login form › rejects a malformed email 14.1s
🟢 login form › rejects an unknown account 14.4s
🟢 login form › signs in and logs out 21.0s
🟢 tests/smoke.e2e.ts · 1 passed 7.9s
🟢 home lists the scenarios and opens one 7.9s

Tested by e2e 0.23.0 · 7m 15s · ios-simulator · run artifacts

@KrzysztofMoch

Copy link
Copy Markdown
Member Author

[claude-fable-5-2] responding on behalf of KrzysztofMoch

Ready for review at 5835a98. Three things are left for a human:

  • A decision on how a nested factory call is written. An option value with only package, export, and options keys is read as a call. A reviewer suggested a reserved key such as {$call: {...}} instead, so an engine option that happens to have that shape can never be misread. The plain shape is kept, the options object itself is never read as a call, and the not-installed error says how the value was read. See "Open for a decision" in the description.
  • Not verified: Android, and an engine installed with pnpm or Bun into ~/.e2e/engines.
  • Declined in review: wildcard subpaths of packages that export only to import, and offering engine when a lookalike config file such as e2e.config.json exists. Both replies are in the resolved threads.

@okwasniewski

Copy link
Copy Markdown
Member

ENG-1191

@kdy1 kdy1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings. I checked out the branch at 5835a980 and went through the parts that carry risk.

Checked

  • Two copies of e2e. isEngineHandle tests the brand Symbol.for('e2e.engine.v1') (internal/brands.ts:7), so a handle from an engine that installed its own e2e into ~/.e2e/engines still passes. loadEngine also checks spiVersion.
  • __proto__ test. I swapped Object.fromEntries for plain assignment in resolveEntries. The test failed, so it pins the own-property behavior. I restored the file afterward.
  • Unit tests. mcp-default-engine.test.ts passes, 17 of 17.
  • configSource refusals. engine with target, config, or test, and app without engine, are each covered. So are an engine session next to a config session (CONFIG_IN_USE) and engine where a config file exists.
  • Secrets in options. @e2e-dev/limrun reads LIMRUN_API_KEY from the environment, and @e2e-dev/browserstack reads BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY the same way. Neither takes a key as an option, so an agent has no reason to put a secret literal in options. A third-party provider that takes one would not reach the secret ledger.
  • importEntry. It runs only after Node has parsed package.json and answered ERR_PACKAGE_PATH_NOT_EXPORTED, so the JSON.parse there cannot see a broken manifest.
  • Security doc. The security.mdx row says the server loads installed packages the agent names, calls the exports it names, and runs that code with the server's full authority. That matches the code. run_code already runs arbitrary test statements, so this does not widen what an agent can do.

Not checked

  • I did not run the integration tests (mcp-server, mcp-session-host) locally. I relied on CI, where every check passed.
  • Android, and an engine installed with pnpm or Bun into ~/.e2e/engines. The PR body lists both as unverified.
  • Two engine copies side by side. I read the brand check but did not reproduce the install.

Open decision

isFactoryRef reads an option value as a factory call when it has a string package and no key besides export and options. A third-party option with that exact shape would be called instead of passed through. The tests pin that {package, bundleId} is left alone and that the options object itself is never a call, and the ENGINE_NOT_INSTALLED message says how the value was read, so a misread does not fail quietly.

I would keep the plain shape. A reserved key such as {$call: ...} would be a breaking change later, so decide it now if you want one.

I did not approve, since AGENTS.md says agents never approve. This is a comment review.

@kdy1 kdy1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified this PR with help of Sonnet 5.5
I left the verification history above

@kdy1
kdy1 added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 3c9de80 Nov 10, 2026
34 checks passed
@kdy1
kdy1 deleted the krzysztof/feat/mcp-without-config branch November 10, 2026 11:41
kblok added a commit to hardkoded/e2e-dotnet that referenced this pull request Nov 10, 2026
* Document MCP open_session without a config (#1509) as unported

Upstream's open_session {engine} (e2e-inc/e2e#1665) names an engine
package to load. The port builds engines in code from a target's platform
and loads no packages, so the feature is recorded as not ported in
COMPATIBILITY.md, with the reason.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

* List every skipped upstream part of the MCP engine session (#1509)

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

* Gate the sloppy fixture's late rejections on the test's release (#1509)

The rejection timers of Leaves_a_promise_behind (50 ms) and
Leaves_a_promise_for_after_the_close (1 s) raced the run's own work: on a
slow runner the 50 ms rejection landed inside the turn and failed the run.
Each rejection now waits for the test to release it once its run has
returned or its session has closed, so it lands at the same point every
run. The assertions are unchanged.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Haiku 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:XL Up to 999 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants