Headed mode + iOS-from-Windows (TCP daemon + MJPEG viewer)#1
Closed
jackulau wants to merge 30 commits into
Closed
Conversation
…ecycle + IPC tests (33 new, 242 total)
…ecycle + IPC integration tests (no device) - tests/_mock_iphone_daemon.py + tests/_mock_android_daemon.py: stub daemons mirror real IPC contract minus Appium - tests/test_ipc.py: 20 tests cover ping/identify/connect/request, malformed JSON, server-side raise, name validation, namespace separation, socket file cleanup - tests/test_daemon_lifecycle.py: 13 tests cover ensure_daemon spawn + idempotency + Appium-handshake restart, restart_daemon kill + pid cleanup, stale socket/pid handling, double-spawn race - iphone_harness/admin.py + android_harness/admin.py: add IPH_DAEMON_MODULE / ANH_DAEMON_MODULE env override (test-only escape hatch) so tests can inject mock daemon module
…ale() + SIGKILL escalation in restart_daemon (5 new tests, 247 total)
…connect/sleep/lock recovery
- iphone_harness/helpers.py + android_harness/helpers.py: add DeviceDisconnectError, is_locked(), wake_device(), @retry_on_disconnect(max_attempts, backoff) decorator
- is_locked uses appium("mobile: isLocked") with deviceInfo fallback; both platforms safe under appium-not-running (returns False)
- wake_device tries mobile:unlock first, falls back to power+menu/swipe on iOS and power+menu keycodes on Android
- @retry_on_disconnect catches RuntimeError matching disconnect patterns (unreachable, session, stale, timed out, wda/uiautomator) — restarts daemon + wakes device between attempts, raises DeviceDisconnectError with --doctor pointer on exhaustion
- tests/test_recovery.py: 14 tests cover decorator behaviour (succeeds-on-retry, exhausts, ignores-unrelated-errors, preserves metadata) + is_locked/wake_device on both platforms
…g helper - mobile_use/ios_wda.py: check_wda_signing() parses installed .mobileprovision files via `security cms`, matches by WDA bundle id, returns (signed|expired|not_signed|unknown, details). Picks profile with latest ExpirationDate when multiple match. - find_wda_project() walks the three known appium-xcuitest-driver install paths (npm, brew x2). open_in_xcode() shells out to `open -a Xcode`. - SIGNING_STEPS prints the 6 manual steps (target → Signing tab → team → build → trust on device → re-check). Free vs paid expiry guidance inline. - mobile-use ios sign-wda CLI subcommand wired in cli.py: --check prints status (exit 0 always — caller wants info), bare invocation opens Xcode + prints steps. - tests/test_ios_wda.py: 15 tests cover signing-state classification, multi-profile latest-expiry selection, non-macOS handling, CLI exit codes — all without real WDA install.
…ort for the Android path
- mobile_use/bootstrap.py: _linux_pkg_manager() detects apt/dnf/pacman via /etc/os-release ID + ID_LIKE, falls back to PATH lookup. _linux_adb_install_cmd / _linux_node_install_cmd emit the right argv per distro.
- plan() on Linux emits Linux-native install steps for adb + node (not mac_only) and keeps iOS steps as mac_only so they SKIP cleanly with a useful hint ("iOS requires macOS + Xcode").
- run() prints distro-specific manual install commands when no supported package manager is detected on Linux.
- tests/test_bootstrap.py: 11 new tests cover pkg-manager detection (ubuntu/fedora/arch), install-cmd dispatch, plan composition on Linux, and that iOS steps still appear but are mac_only. All without touching the real system.
- SETUP.md already has Linux section from prior work.
…een() via Appium API (iOS + Android, 11 tests)
…lay tap sequences - mobile_use/record_replay.py: start_recording(path, helpers, fn_names=...) wraps the listed helper functions (tap, swipe, type_text, press_*, scroll, etc.) with a journaling trampoline that records timing + args + kwargs and forwards to the original. stop_recording() restores originals and writes a runnable .py script. replay(path) execs it. - Default RECORDED_HELPERS covers the 15 most common interactions. Custom fn_names override per call. - Generated scripts are plain Python — import the helpers, time.sleep between actions (only for >0.05s gaps), call site exactly as recorded. Editable by hand before replay. - tests/test_record_replay.py: 13 tests cover record→script→replay roundtrip, helper-restore on stop, error on double-start, custom fn_names filtering, and JSON-formatted args.
…(WDA signing, battery, screen wakefulness)
…ice smoke (13 tests)
…sh (SETUP troubleshooting tree, README runtime helpers, .env.example required/optional)
- _pid_alive: reject bool (isinstance(True, int) is True → would resolve to PID 1) - cleanup_stale: tolerate UnicodeDecodeError + PermissionError on PID file read - retry_on_disconnect: validate max_attempts >= 1 and backoff >= 0 - wake_device: return False on unlock failure; confirm post-state via is_locked - record_screen: validate duration > 0, reject directory paths, create parent dirs - _check_battery (iOS+Android): WARN-only on low battery (don't fail doctor); tolerate non-integer 'level' values from adb - bootstrap: _sudo_prefix() drops sudo when root, returns None when sudo missing - record_replay: reject non-module helpers; restore on partial-wrap failure; add recording() context manager 42 new hardening tests, 374 total (was 332), 3x flakiness clean.
…os UX - mobile-use ios <action>: explicit help for no/unknown action - mobile-use ios sign-wda --help: prints usage (was running sign action) - mobile-use ios sign-wda --check: exit code reflects signing state - mobile-use doctor: subcommand alias for --doctor flag - CLI -c exec: clean error handling (no cli.py traceback noise) - Mock daemons: cover all real RPC methods (click_element, send_keys, set_value, pick_wheel, active_app) - test_cli_dispatch.py: 16 tests (subcommand routing, mock contract, bootstrap idempotency, exit codes) - SETUP.md: daemon log paths in debugging section 392 tests, no regressions.
Defense in depth — caps incoming IPC data to prevent unbounded memory growth from malfunctioning/compromised daemon. Triggered by: - Test in test_ipc.py with cap lowered to verify enforcement - 64MB default covers any realistic iPhone screenshot/video payload
…, press_back, press_recents)
… (--reload, build-wda)
…ork with App Store + xcode-select remediation)
…-autostart-appium
…OS, Android, env-error paths)
… tests green 5x in a row (parent-pkg attr binding)
…ld-wda, --autostart-appium)
…ctor Centralize sys.platform / Linux pkg manager detection in mobile_use/_platform.py. Supports apt/dnf/pacman + zypper + apk (extends the prior apt/dnf/pacman-only detection with openSUSE and Alpine). bootstrap.py keeps legacy _linux_pkg_manager/_sudo_prefix/_linux_*_install_cmd symbols as back-compat shims so existing tests continue to work. Adds new _linux_libimobiledevice_install_cmd shim plus LINUX_LIBIMOBILEDEVICE_PKGS mapping for D2/D4. Verify: 29/29 tests/test_platform.py + 25/25 tests/test_bootstrap.py + 18/18 tests/test_hardening.py green. 2 pre-existing failures (test_cli_dispatch + test_e2e_no_device) confirmed unrelated on main.
… remote-daemon scaffolding
iphone_harness/admin.py + android_harness/admin.py:
- run_doctor() now emits platform-correct install hints (no hardcoded
'brew install' on Linux).
- _check_libimobiledevice() (new in iphone) uses idevice_id-on-PATH
instead of asking Homebrew.
- _check_adb() (new in android) is distro-neutral.
- _check_xcode / _check_wda_signing skip with 'OK' on Linux instead
of FAILing — Xcode/WDA codesigning are macOS-only.
mobile_use/_platform.py: install_hint() helper does the platform dispatch
for doctor remediation strings.
iphone_harness/_ipc.py + android_harness/_ipc.py + admin.py: TCP endpoint
support (IPH_CONNECT/ANH_CONNECT=tcp://...) — enables the remote-daemon
flow that D6 will polish.
tests/test_doctor_linux.py: 18 tests across install_hint, iphone+android
admin checks, and full-doctor output assertions (no 'brew install' in
Linux output).
Verify: 491 passed; 2 pre-existing failures unrelated.
…on endpoint 17 tests cover parse_endpoint, sock_addr_tcp, transport_serve_warns_on_non_loopback, and unix-still-works regression. Pairs with the IPH_CONNECT/ANH_CONNECT TCP support added in D2 admin.py + _ipc.py changes; D6 builds on this for the iOS-on-Linux via remote macOS Appium flow.
Both iphone_harness.helpers.ocr() and android_harness.helpers.ocr() now raise OCRNotAvailableError on non-macOS hosts BEFORE attempting the Vision/Foundation import (was: silent ImportError → RuntimeError). The new error message points at SETUP.md#ocr-on-linux for Tesseract setup instead of misleadingly suggesting 'pip install pyobjc-framework-Vision' (which won't work on Linux). OCRNotAvailableError subclasses RuntimeError so existing exception handlers don't break. Verify: 6/6 tests/test_ocr_platform.py.
…ent mode
D1 — TCP transport in IPC (both iphone_harness + android_harness):
- parse_endpoint('unix:/...' | 'tcp://host:port') → kind + components
- bind_endpoint / connect_endpoint resolve from IPH_BIND/IPH_CONNECT
(and ANH_BIND/ANH_CONNECT) env vars; default = AF_UNIX (no behavior change)
- serve() dispatches to asyncio.start_unix_server OR start_server
- connect() picks AF_UNIX vs socket.create_connection
- cleanup_endpoint / cleanup_stale skip socket-file unlink in TCP mode
- Security: non-loopback bind prints WARNING to stderr (RPC is unauth)
D2 — Windows / Linux client-only mode for iOS:
- mobile_use/_platform.py: is_windows(), needs_remote_mac_for_ios(),
windows_ios_setup_hint() (multi-line operator guidance)
- iphone_harness.admin.is_remote_daemon() — true when IPH_CONNECT is a
TCP endpoint. Same for android_harness.admin.is_remote_daemon().
- ensure_daemon() in client-only mode NEVER spawns locally — pings,
raises remediation checklist on failure (ssh tunnel hint, lsof check)
- CLI: --remote-daemon <URI> flag (sets IPH_CONNECT/ANH_CONNECT after
parse_endpoint validation); --headed / --headless flags (MOBILE_USE_HEADED)
- HELP rewritten with 'iOS FROM WINDOWS / LINUX' section
Tests:
- tests/test_ipc_tcp.py: 16 tests (parse, sock_addr, TCP roundtrip per
platform, no-unix-file, shutdown, non-loopback warning, default-unix)
- tests/test_remote_daemon.py: 14 tests (is_remote_daemon policy, client
mode no-spawn via Popen spy, CLI flag wiring, _platform helpers)
- 39 IPC tests + 53 combined pass; no regressions in existing tests
…aded CLI + docs
D3 — Screen stream RPC (iphone_harness + android_harness):
- Daemon class: _stream_task/_stream_frame/_stream_fps/_stream_quality/_stream_max_dim state
- _m_screen_stream_start(fps, quality, max_dim): spawn capture loop, idempotent
- _m_screen_stream_frame(): pull latest JPEG (base64) — single-consumer
- _m_screen_stream_stop(): cancel loop, clear buffer
- PIL lazy-imported (already a project dep via pillow>=10)
- Frame loop logs+continues on transient errors; doesn't kill stream
- Helpers: screen_stream_start/frame/stop wrappers (drop conn first to avoid stale cache)
- Mock daemons: synthetic 67-byte JPEG stub so tests don't need PIL
D4 — HTTP MJPEG viewer sidecar (mobile_use/viewer/):
- server.py:ViewerServer(platform, port, fps, quality, max_dim)
- Routes: GET / (embedded HTML), /stream (multipart/x-mixed-replace),
/still (single JPEG), /healthz (JSON status)
- Threaded HTTP server (daemon threads) so /healthz doesn't block behind /stream
- Lifecycle: start() spawns thread + tells daemon to start capture loop;
stop() shuts server down, releases port, stops daemon capture
- Stdlib only — no new pip deps
D5 — CLI --headed / --headless flag wiring:
- main(): parses flags, sets MOBILE_USE_HEADED env (with try/finally restore
so in-process tests don't leak state to siblings)
- _maybe_start_viewer(platform): used by _run_ios + _run_android + agent_loop;
opens browser via webbrowser.open (silent fallback if unavailable); soft
warning + None on failure (viewer never blocks the -c script)
- HELP text: 'iOS FROM WINDOWS / LINUX' section; --headed/--headless documented
D6 — End-to-end smoke tests (tests/test_e2e_headed.py):
- e2e_headed_ios: mock daemon → ViewerServer → /stream returns multipart JPEG
- e2e_headed_android: parity
- e2e_remote_iphone: TCP daemon endpoint + client-only mode + RPC works
- e2e_stream_loop_progresses: frame_no strictly increasing across polls
D7 — Docs (SETUP.md + README.md):
- SETUP.md: 'Part A* — iOS from Windows / Linux (remote Mac bridge)' section
with Mac-side setup, daemon TCP launch, Windows/Linux client connect,
SSH tunnel pattern, security caveat, troubleshooting checklist
- README.md: 'Headed mode' section (--headed launches MJPEG viewer in
browser); 'iOS from Windows / Linux' quick-start pointing at SETUP.md
Tests: 38 new (12 screen_stream + 12 viewer_mjpeg + 7 cli_headed + 4 e2e + 3 sanity);
full suite: 572 passed, 1 pre-existing Linux test failure (008-linux-device-support).
Owner
Author
|
Replaced by rebased branch goal/008-headed-rebased — original branch had 30-commit divergence after PR #2 (Linux) squash-merged. New PR contains the same headed-mode + Windows-iOS work cleanly rebased on top of current main. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two user-visible features in one branch (shared scope across CLI + both harness daemons + IPC):
--headedflag spins up a local browser viewer that mirrors the live device screen via MJPEG at ~6 fps. Stdlib-only, no new pip deps. Read-only viewer (it shows what the device is doing; doesn't take input).IPH_BIND=tcp://...on the Mac;--remote-daemon tcp://...on the client). Windows/Linux hosts can drive a Mac that's running Appium + WebDriverAgent without needing Xcode locally. SSH-tunnel pattern documented in SETUP.md.The two features are intentionally bundled: both touch CLI + daemon IPC, both fall out of the same TCP transport seam.
Deliverables (all verified)
parse_endpoint('unix:/...' | 'tcp://host:port'),bind_endpoint/connect_endpointfrom env. Default = AF_UNIX (zero behavior change). Non-loopback bind prints security warning.mobile_use/_platform.pygainsis_windows()+needs_remote_mac_for_ios()+windows_ios_setup_hint().admin.is_remote_daemon()in both harnesses;ensure_daemon()skips local spawn in client-only mode, raises remediation checklist with ssh-tunnel hint. CLI--remote-daemon <URI>flag (parse_endpoint-validated)._m_screen_stream_start/frame/stop. Frame loop captures via existing get_screenshot_as_png, JPEG-encodes via PIL (existing dep). Single-consumer pull model.mobile_use/viewer/server.py— HTTP MJPEG sidecar. Routes:/(HTML),/stream(multipart),/still(single JPEG),/healthz(JSON). Threaded server. ViewerServer lifecycle manages daemon-side capture loop.--headed/--headlessflag wired into_run_ios,_run_android, andagent_loop.run_agent. Browser auto-open via webbrowser.open (silent fallback on headless boxes). env-var restored on exit to avoid pytest leakage.Test results
Try it
Test plan
mobile-use --helpadvertises --remote-daemon, --headed, --headlessSecurity notes