Skip to content

fix(limrun): let tests run on Limrun devices again - #1663

Merged
kdy1 merged 1 commit into
mainfrom
krzysztof/fix/limrun-lease-session
Nov 10, 2026
Merged

kdy1 merged 1 commit into
mainfrom
krzysztof/fix/limrun-lease-session

Conversation

@KrzysztofMoch

@KrzysztofMoch KrzysztofMoch commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

What and why

  • Every limrun() lease failed seconds after Limrun created the device, so no test or e2e mcp session could run on one (@e2e-dev/limrun 0.2.4 with @e2e-dev/mobile 0.12.6):

    ENGINE_FAILURE: device provider "Limrun" could not lease a device: returned a lease without an id and a daemon baseUrl or JSON client configuration (no `session` or daemon keys) while preparing engine mobile for target "ios"
    
  • fix(mobile): name the agent-device session for device-cloud leases #1513 put the agent-device session on the lease's client, so the install and heartbeat clients that reuse that object would name one. The mobile engine reserves that key (RESERVED_CLIENT_KEYS in packages/mobile/src/bindings.ts) and rejects a lease that sets it.

  • The driver now leaves session off the client it returns. Its allocate and install clients and the heartbeat add it themselves from leaseSession(stateDir), the way packages/mobile/src/lease/ does.

  • DriverBinding.client is typed as DeviceClientConfig, so a session in that literal no longer compiles.

  • leaseSession is a local copy of mobile's five lines. @e2e-dev/mobile/lease does not export it, and exporting it would raise Limrun's peer floor on mobile.

  • The driver unit test asserted that session was on the binding. It now asserts the exact client the engine receives, and fails on main.

Issue

No e2e issue. This is a regression from #1513, found on 2026-11-10 with the published packages.

Verified

Ran it locally: yes

  • A scratch project on the built packages, against a real Limrun iOS simulator. The target is mobile({ platform: 'ios', device: limrun({ hardTimeout: '15m' }) }) with app: { bundleId: 'com.apple.mobilesafari' }, run with e2e run tests/safari.e2e.ts --video. The test opens Safari, opens https://example.com, closes Safari's first-launch tip, and expects the page text.
main this branch
Lease rejected, no test executed, exit 3 Lease accepted, 1 test passed, exit 0
no device to show Safari on the Limrun simulator

main:

ℹ leasing 1 ios device(s) from Limrun
ℹ Limrun (1 of 1): created ios_euna_01m723ptnafjqaby9advh0rnmp
ℹ Limrun: released 1 device(s)

 ERROR  infrastructure error ENGINE_FAILURE (launch)
device provider "Limrun" could not lease a device: returned a lease without an id and a daemon baseUrl or JSON client configuration (no `session` or daemon keys) while preparing engine mobile for target "ios"

 Test Files  no test files
      Tests  no tests executed
     Errors  1 error

this branch:

ℹ leasing 1 ios device(s) from Limrun
ℹ Limrun (1 of 1): created ios_euna_01m724he11fya8hmab1bjqgwm3
ℹ Limrun: leased ios_euna_01m724he11fya8hmab1bjqgwm3
ℹ booting limrun:ios:1e6bf23c551b5a69ed8fbb04f9baeb0f (1 of 1)
 ✓ mobile engine for target "ios" prepared 1.42s
   ✓ |ios| tests/safari.e2e.ts > Safari opens a page on a Limrun device 7.10s
ℹ Limrun: released 1 device(s)

 Test Files  1 passed (1)
      Tests  1 passed (1)
  • A second test idles 90s before driving Safari, past the lease's 60s TTL. It passed and logged no driver lease renewal failed line. I ran no control without the heartbeat.
  • After the runs the account has no instance in creating, assigned, or ready.
  • Not run: Android on Limrun, the appPath install on a live device (Safari is preinstalled, so that client is unit-tested only), and e2e mcp (it reaches the same engine prepare).
  • A fresh-context review of the diff found nothing to fix.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: not applicable.
  • Docs updated in the same PR: no change needed, the docs do not describe the lease client's session.
  • Contract change: not applicable.
  • Engine contract change: not applicable.

Written by Claude Fable 5.2 in Claude Code, running in T3 Code.

limrun-ios-after.mp4

View guided diff Turn on auto-fix

The driver put the agent-device session on the lease's client (#1513), a
key the mobile engine reserves, so the engine rejected every Limrun lease
with ENGINE_FAILURE. The session now stays on the driver's and the
heartbeat's own clients and off the client the engine receives.
@github-actions github-actions Bot added the size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile label Nov 10, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 47 model calls · 299.3k tokens (78% cached) · $0.0232

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 47 calls 6m 5s
🟢 act completes Login Form 1 step 21.7s
🟢 act completes Infinite Scroll List 1 step 20.1s
🟢 act completes Modal Flow 1 step 10.4s
🟢 act completes Bottom Tabs 1 step 7.1s
🟢 act completes Text Input Variations 1 step 5.9s
🟢 act completes Broken Accessibility 2 steps · 5 calls 14.6s
🟢 act completes Vision Only 2 steps · 1 call 6.3s
🟢 act completes Gestures 1 step · 9 calls 26.4s
🟢 act completes Async States 1 step · 20 calls 46.7s
🟢 act completes Debounced Search 1 step 6.9s
🟢 act completes Huge Virtualized List 1 step 1m 54s
🟢 act completes Flattened Registration Form 2 steps · 1 call 7.8s
🟢 act completes Flattened Login 2 steps · 6 calls 14.6s
🟢 act completes Sticky Chrome Target 1 step 8.7s
🟢 act completes WebView Accessibility 1 step · 5 calls 13.6s
🟢 act completes Permission Prompt 1 step 6.1s
🟢 act completes Product Catalog 1 step 8.1s
🟢 act completes Error Recovery 1 step 8.8s
🟢 act completes Choice Controls 1 step 8.2s
🟢 act completes Sequential Onboarding 1 step 9.5s

Tested by e2e 0.23.0 · 6m 6s · android-emulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 0s
🟢 control inventory › a label query resolves a labeled text field 11.8s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 16.4s
🟢 control inventory › a placeholder query and attribute reads find the field 4.7s
🟢 control inventory › a secure field takes a fill and denies value reads 8.3s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.7s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.3s
🟢 control inventory › toBeFocused follows the focus 8.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.7s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.4s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.9s
🟢 control inventory › a radio group reports the checked option 8.6s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.5s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 8.1s
🟢 control inventory › a header answers its accessible name 4.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.2s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 7.1s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.3s
🟢 control inventory › pointer verbs land at the point asked for 10.1s
🟢 control inventory › an ambiguous locator fails at once 6.0s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 11.1s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 397ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.2s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 33s
🟢 device fixture › setAppearance flips the color scheme the app reads 15.3s
🟢 device fixture › setOrientation rotates the window the app measures 8.6s
🟢 device fixture › the clipboard reads back what was written 6.2s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.4s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.2s
🟢 device fixture › closeApp leaves the next commands on the same device 11.2s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 5.5s
🟢 device fixture › device.back pops the scenario 6.3s
🟢 device fixture › unstable_raw reaches the test session through agent-device 4.3s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.4s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 10.9s
🟢 tests/device.e2e.ts · 3 passed 1m 8s
🟢 sequential onboarding echoes the exact values 41.8s
🟢 onboarding rejects a malformed email before advancing 14.3s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.7s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 13s
🟢 infinite scroll reaches the target item 34.2s
🟢 product catalog adds exactly two of the right variant 10.4s
🟢 gestures: long-press, then swipe left 9.1s
🟢 sticky chrome: accept below the fold, then continue 12.8s
🟢 async states: load, pull to refresh, claim through the toast 10.5s
🟢 huge virtualized list reaches row 512 2m 43s
🟢 web view coupon form applies the code shown on the page 12.2s
🟢 tests/flows.e2e.ts · 7 passed 1m 10s
🟢 text inputs unlock the submit button 13.8s
🟢 filled text inputs report their values 10.5s
🟢 modal flow confirms through the native alert 9.7s
🟢 bottom tabs act inside the last tab 7.4s
🟢 error recovery retries, then confirms the delete 9.2s
🟢 choice controls place the exact order 7.5s
🟢 debounced search selects the target once results arrive 11.9s
🟢 tests/login-form.e2e.ts · 3 passed 47.4s
🟢 login form › rejects a malformed email 13.7s
🟢 login form › rejects an unknown account 13.7s
🟢 login form › signs in and logs out 20.0s
🟢 tests/smoke.e2e.ts · 1 passed 7.4s
🟢 home lists the scenarios and opens one 7.4s

Tested by e2e 0.23.0 · 6m 54s · ios-simulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 37s
🟢 control inventory › a label query resolves a labeled text field 21.5s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 11.2s
🟢 control inventory › a placeholder query and attribute reads find the field 4.4s
🟢 control inventory › a secure field takes a fill and denies value reads 6.0s
🟢 control inventory › press sends a character and Enter through the soft keyboard 7.0s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 8.1s
🟢 control inventory › toBeFocused follows the focus 7.6s
🟢 control inventory › a switch checks, unchecks, and reports its state 8.2s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 6.2s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 10.7s
🟢 control inventory › a tab role answers getByRole 3.8s
🟢 control inventory › a radio group reports the checked option 7.4s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.7s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.2s
🟢 control inventory › a header answers its accessible name 3.8s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.5s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.3s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.1s
🟢 control inventory › a double tap lands inside the 300 ms window 6.3s
🟢 control inventory › an ambiguous locator fails at once 4.7s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 6.9s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.3s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 12s
🟢 device fixture › setOrientation rotates the window the app measures 4.2s
🟢 device fixture › home leaves the app, openApp brings it back where it was 4.7s
🟢 device fixture › foregroundApp sees the home screen after home() 3.9s
🟢 device fixture › closeApp then openApp relaunches on the home list 5.8s
🟢 device fixture › closeApp leaves the next commands on the same device 6.3s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.2s
🟢 device fixture › device.back pops the scenario 4.4s
🟢 device fixture › unstable_raw reaches the test session through agent-device 3.6s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.1s
🟢 device fixture › setLocation and clearLocation change what the app reads 19.8s
🟢 tests/device.e2e.ts · 5 passed 38.3s
🟢 permission prompt › allowing the system dialog enables the microphone 5.3s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 12.0s
🟢 sequential onboarding echoes the exact values 9.4s
🟢 onboarding rejects a malformed email before advancing 5.3s
🟢 dismissing the keyboard over choice controls leaves them as they were 6.3s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 10s
🟢 infinite scroll reaches the target item 28.7s
🟢 product catalog adds exactly two of the right variant 8.1s
🟢 gestures: long-press, then swipe left 7.8s
🟢 gestures: double-tap completes the flow 9.2s
🟢 sticky chrome: accept below the fold, then continue 10.5s
🟢 async states: load, pull to refresh, claim through the toast 11.4s
🟢 huge virtualized list reaches row 512 1m 55s
🟢 tests/flows.e2e.ts · 7 passed 48.3s
🟢 text inputs unlock the submit button 5.7s
🟢 filled text inputs report their values 5.2s
🟢 modal flow confirms through the native alert 8.6s
🟢 bottom tabs act inside the last tab 6.3s
🟢 error recovery retries, then confirms the delete 8.3s
🟢 choice controls place the exact order 5.6s
🟢 debounced search selects the target once results arrive 8.6s
🟢 tests/login-form.e2e.ts · 3 passed 26.0s
🟢 login form › rejects a malformed email 7.0s
🟢 login form › rejects an unknown account 6.9s
🟢 login form › signs in and logs out 12.1s
🟢 tests/smoke.e2e.ts · 1 passed 5.4s
🟢 home lists the scenarios and opens one 5.4s

Tested by e2e 0.23.0 · 8m 58s · android-emulator · run artifacts

@github-actions

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 12 replayed from cache · 67 model calls · 513.1k tokens (78% cached) · $0.0402

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 67 calls 9m 42s
🟢 act completes Login Form 1 step 38.2s
🟢 act completes Infinite Scroll List 1 step 37.9s
🟢 act completes Modal Flow 1 step 9.8s
🟢 act completes Bottom Tabs 1 step 7.3s
🟢 act completes Text Input Variations 1 step 13.3s
🟢 act completes Broken Accessibility 2 steps · 5 calls 24.0s
🟢 act completes Vision Only 2 steps · 6 calls 28.7s
🟢 act completes Gestures 1 step · 8 calls 37.7s
🟢 act completes Async States 1 step · 10 calls 32.6s
🟢 act completes Debounced Search 1 step · 1 call 27.1s
🟢 act completes Flattened Registration Form 2 steps · 1 call 28.4s
🟢 act completes Flattened Login 2 steps · 22 calls 2m 15s
🟢 act completes Sticky Chrome Target 1 step 13.9s
🟢 act completes WebView Accessibility 1 step 12.9s
🟢 act completes Permission Prompt 1 step 11.7s
🟢 act completes Photo Picker 1 step · 2 calls 30.3s
🟢 act completes Product Catalog 1 step 10.3s
🟢 act completes Error Recovery 1 step 9.6s
🟢 act completes Choice Controls 1 step · 4 calls 31.5s
🟢 act completes Stripe PaymentSheet 1 step · 8 calls 31.7s
🟢 act completes Apple Pay 1 step 10.5s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.0 · 9m 43s · ios-simulator · run artifacts

@kdy1 kdy1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did human review with help of Sonnet 5.5

@kdy1
kdy1 added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit 322f00e Nov 10, 2026
32 checks passed
@kdy1
kdy1 deleted the krzysztof/fix/limrun-lease-session branch November 10, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:M Up to 99 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants