Headed mode + iOS-from-Windows (TCP daemon + MJPEG viewer)

.env.example

@@ -2,49 +2,63 @@
 # YOUR device's values. Both files are .gitignore'd by default.
 #
 # You only need to fill in the section for the platform you're using.
+# Run `mobile-use init` for a guided fill — auto-detects connected devices.
 
 # ============================================================================
 # iOS (iphone-harness)
 # ============================================================================
 
-# ---- Required ----
-
-# Your iPhone's UDID. Find it with:
-#   idevice_id -l                 (libimobiledevice — `brew install libimobiledevice`)
-#   xcrun xctrace list devices    (Apple's tool, also lists offline devices)
+# required — iPhone UDID. Find with: `idevice_id -l` or `xcrun xctrace list devices`
 IPH_UDID=YOUR-IPHONE-UDID-HERE
 
-# Your Apple Developer Team ID (10 chars). Find at developer.apple.com → Membership.
+# required — Apple Developer Team ID (10 chars).
+# Find at developer.apple.com → Membership, or Keychain Access → your Apple Development cert → Get Info → Organizational Unit.
 IPH_XCODE_ORG_ID=YOURTEAMID
 
-# A bundle id YOU own/can sign, used for WebDriverAgent on the device.
-# Example: com.<your-handle>.mobile-use.wda
+# required — A reverse-DNS bundle id you can sign. Example: com.<your-handle>.mobile-use.wda
+# This becomes the bundle ID of WebDriverAgent on your device.
 IPH_WDA_BUNDLE_ID=com.example.mobile-use.wda
 
-# ---- Optional (iOS) ----
-
+# optional — pinned iOS version (Appium usually auto-detects)
 # IPH_PLATFORM_VERSION=18.3.2
+
+# optional — device display name (visible in Appium logs)
 # IPH_DEVICE_NAME=My iPhone
+
+# optional — non-default Appium server URL
 # IPH_APPIUM_URL=http://127.0.0.1:4723
+
+# optional — daemon instance name (for multibox / multiple devices)
 # IPH_NAME=default
+
+# optional — enable domain-specific skills (per-app playbooks)
 # IPH_DOMAIN_SKILLS=1
+
+# optional — seconds Appium waits between agent calls before killing session
 # IPH_NEW_COMMAND_TIMEOUT=600
 
+
 # ============================================================================
 # Android (android-harness)
 # ============================================================================
 
-# ---- Required ----
-
-# Your Android device serial. Find it with:
-#   adb devices
+# required — Android device serial. Find with: `adb devices`
 ANH_UDID=YOUR-ANDROID-SERIAL-HERE
 
-# ---- Optional (Android) ----
-
+# optional — pinned Android version
 # ANH_PLATFORM_VERSION=14.0
+
+# optional — device display name
 # ANH_DEVICE_NAME=Android
+
+# optional — non-default Appium server URL
 # ANH_APPIUM_URL=http://127.0.0.1:4723
+
+# optional — daemon instance name (for multibox)
 # ANH_NAME=default
+
+# optional — enable domain-specific skills
 # ANH_DOMAIN_SKILLS=1
+
+# optional — seconds Appium waits between agent calls before killing session
 # ANH_NEW_COMMAND_TIMEOUT=600

README.md

@@ -37,10 +37,38 @@ If anything fails:
 ```bash
 mobile-use --doctor               # numbered checks with one-line remediations
 iphone-harness --reload           # nuke the daemon (rare but kills weird stale state)
+mobile-use ios sign-wda           # iOS: re-sign WebDriverAgent (the #1 setup blocker)
+mobile-use ios build-wda          # iOS: build the WDA test target (first-run setup)
+mobile-use quickstart --autostart-appium   # spawn Appium server in background
 ```
 
-See [`SETUP.md`](SETUP.md) for the manual / per-step appendix, or skip to the
-sections below for usage.
+See [`SETUP.md`](SETUP.md) for the manual / per-step appendix, including a
+[troubleshooting decision tree](SETUP.md#part-c--troubleshooting). Linux users:
+the Android path works out-of-the-box on apt/dnf/pacman systems (iOS still needs
+macOS + Xcode).
+
+### Runtime helpers (no device pain)
+
+```python
+from iphone_harness.helpers import wake_device, retry_on_disconnect, record_screen
+
+wake_device()                              # screen-off / locked? wake it.
+
+@retry_on_disconnect(max_attempts=3)        # USB blip / WDA crash → auto-restart + retry
+def run_script():
+    tap(find(label="Compose"))
+    type_text("hello")
+
+record_screen(duration=10)                  # save mp4 to /tmp (XCUITest + UIAutomator2)
+
+# record/replay a tap sequence:
+from mobile_use import record_replay
+import iphone_harness.helpers as h
+record_replay.start_recording("flow.py", helpers=h)
+# ... your taps/swipes/typing ...
+record_replay.stop_recording()              # writes runnable flow.py
+record_replay.replay("flow.py")             # play it back
+```
 
 ### Manual setup (skip if `mobile-use bootstrap` worked)
 
@@ -169,6 +197,53 @@ pool.broadcast_android(lambda d: d.press_home())
 
 Each device gets its own named daemon instance (`IPH_NAME` / `ANH_NAME`) with separate sockets, so they don't collide.
 
+## Headed mode — watch the device while it runs
+
+By default mobile-use is **headless**: scripts run, the daemon talks to the
+device, you see no UI. Add `--headed` to spin up a local MJPEG viewer in
+your browser and watch the live device screen mirror while the script runs:
+
+```bash
+mobile-use --ios --headed -c 'tap_at_xy(100, 200); time.sleep(2)'
+# → opens http://127.0.0.1:<random-port>/ in your default browser
+# → live mirror at ~6 fps, JPEG quality 60 (knobs in mobile_use/viewer/server.py)
+```
+
+The viewer is read-only — it shows what the device is doing; it doesn't
+take input. Use `--headless` (or omit the flag) to skip it. Works on iOS
+and Android.
+
+Quality knobs (via Python API, when running in agent mode):
+
+```python
+from mobile_use.viewer.server import ViewerServer
+v = ViewerServer(platform="ios", fps=12, quality=80, max_dim=1200)
+v.start(); print(v.url)
+# ...
+v.stop()
+```
+
+## iOS from Windows / Linux
+
+Windows hosts can't build WebDriverAgent (no Xcode). Drive iOS via a Mac
+on the network running the daemon over TCP:
+
+```bash
+# On the Mac (one time): full Part A in SETUP.md
+# On the Mac (each session):
+IPH_BIND=tcp://127.0.0.1:8763 iphone-harness -c 'pass'
+
+# On Windows / Linux:
+ssh -L 8763:127.0.0.1:8763 user@mac.local           # SSH tunnel (recommended)
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 -c 'print(active_app())'
+
+# Add --headed to also see the live screen mirror in your local browser:
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 --headed -c '...'
+```
+
+Full walkthrough + security caveat: SETUP.md → "iOS from Windows / Linux
+(remote Mac bridge)".
+
 ## Skills
 
 ### iOS Interaction Skills
@@ -339,6 +414,13 @@ paste_text(text, ...)
 # Device
 unlock()
 
+# Navigation (both platforms — Android native buttons, iOS gesture equivalents)
+press_home()                             # both — go to home screen
+press_back()                             # Android: back key; iOS: swipe-from-left edge
+press_recents()                          # Android: recents; iOS: app switcher
+swipe_back()                             # iOS: explicit edge-swipe (alias for press_back on iOS)
+open_app_switcher()                      # iOS: swipe up + pause
+
 # iOS-only
 native_screenshot()                      # saves to iPhone Photos
 set_assistive_touch(on=True)
@@ -349,9 +431,6 @@ start_screen_recording()
 stop_screen_recording()
 
 # Android-only
-press_back()
-press_home()
-press_recents()
 open_notifications()
 close_notifications()
 grant_permission(package, permission)

SETUP.md

@@ -112,11 +112,127 @@ iphone-harness -c 'print(active_app())'
 
 ---
 
+# Part A* — iOS from Windows / Linux (remote Mac bridge)
+
+Windows and Linux cannot build or sign WebDriverAgent locally — `xcodebuild`
+and Apple codesigning are macOS-only. The path is to keep one Mac on the
+network as the "iOS bridge" and drive it remotely via TCP. The mobile-use
+client running on Windows/Linux talks to the daemon on the Mac; the Mac
+talks to the iPhone via Appium + WDA exactly as in Part A.
+
+### One-time setup on the Mac
+
+Follow Part A above on the Mac (Xcode, libimobiledevice, Appium, WDA signing,
+.env with IPH_UDID/IPH_XCODE_ORG_ID/IPH_WDA_BUNDLE_ID).
+
+Verify it works on the Mac itself before adding network in the mix:
+
+```bash
+iphone-harness -c 'print(active_app())'
+```
+
+### Each session on the Mac
+
+Start the daemon bound to TCP loopback (preferred — pair with SSH tunnel) OR
+to all interfaces (faster setup, less secure — see security caveat below).
+
+Loopback + SSH tunnel (recommended):
+
+```bash
+# On the Mac:
+IPH_BIND=tcp://127.0.0.1:8763 iphone-harness -c 'pass'  # starts daemon, exits client
+# Daemon keeps running. Re-running 'iphone-harness -c' attaches to it.
+```
+
+All-interfaces (skip SSH, faster — security warning printed to stderr):
+
+```bash
+IPH_BIND=tcp://0.0.0.0:8763 iphone-harness -c 'pass'
+```
+
+### On Windows or Linux
+
+```bash
+pip install mobile-use   # Android-only deps; iOS daemon never runs locally
+
+# Open SSH tunnel in another terminal (skip if Mac is bound to 0.0.0.0):
+ssh -L 8763:127.0.0.1:8763 user@mac.local
+
+# Drive iOS:
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 -c 'print(active_app())'
+
+# Or with the headed viewer in the browser:
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 --headed -c 'print(active_app())'
+```
+
+### How it works
+
+- `IPH_BIND=tcp://...` on the Mac switches the daemon's IPC from AF_UNIX
+  to TCP. AF_UNIX is unchanged when IPH_BIND is unset.
+- `--remote-daemon tcp://...` on the client sets `IPH_CONNECT` and flips
+  the harness into **client-only mode**: `ensure_daemon` never tries to
+  spawn a local daemon; it pings the remote, raises a remediation
+  checklist if unreachable.
+- The daemon over TCP serves the same JSON-line RPC protocol as over
+  AF_UNIX — no new methods, no protocol break. Everything that works
+  locally works remotely.
+
+### Security caveat
+
+The RPC is **unauthenticated**. Anything that can connect to the daemon's
+port can drive the phone. Mitigations:
+
+- Bind 127.0.0.1 on the Mac and use SSH tunnels (encrypts + authenticates
+  via SSH). This is the recommended pattern.
+- If you must bind 0.0.0.0, put a firewall rule (`pf` on macOS) in front
+  that only allows your specific Windows/Linux IP.
+- Future: HMAC token in `IPH_CONNECT_TOKEN` env (tracked as a follow-up).
+
+### Troubleshooting from the client
+
+```
+iphone-harness: remote daemon unreachable at tcp://127.0.0.1:8763
+```
+
+Means: client can't reach the daemon. On the Mac, check:
+
+```bash
+pgrep -fa iphone_harness.daemon          # daemon process alive?
+lsof -iTCP -sTCP:LISTEN | grep python    # bound to TCP?
+```
+
+On Windows: `Test-NetConnection -ComputerName <mac> -Port 8763`.
+
+---
+
 # Part B — Android Setup
 
 Android setup is significantly simpler than iOS — no signing, no Xcode, no provisioning.
+**Works on macOS and Linux.** iOS requires macOS+Xcode; Android does not.
+
+## B1. System tools
+
+### Linux (Ubuntu/Debian/Fedora/Arch)
+
+```bash
+# Ubuntu / Debian / Mint:
+sudo apt install -y android-tools-adb nodejs npm
+
+# Fedora / RHEL / Rocky:
+sudo dnf install -y android-tools nodejs npm
+
+# Arch / Manjaro:
+sudo pacman -S --noconfirm android-tools nodejs npm
+
+# Then (any Linux):
+npm i -g appium
+appium driver install uiautomator2
+pip install -e .
+```
+
+`mobile-use bootstrap` autodetects the Linux package manager via `/etc/os-release`.
 
-## B1. System tools (Mac side)
+### macOS
 
 ### Install Android SDK Platform Tools
 
@@ -222,6 +338,61 @@ for el in ui_tree(visible_only=True)[:5]:
 
 # Part C — Troubleshooting
 
+## Decision tree — most common failures
+
+```
+Something broke.
+│
+├── First: run `mobile-use --doctor` — shows the bad check + a one-line fix.
+│
+├── "daemon unreachable" / "stale session"
+│   └── → `iphone-harness --reload` (or `android-harness --reload`).
+│       Restart Appium too if the issue persists.
+│
+├── iOS: "xcodebuild failed with code 65"
+│   └── → WDA signing. Run `mobile-use ios sign-wda`.
+│       Free Apple account? Re-sign weekly (profile expires every 7 days).
+│
+├── iOS: "Tunnel registry port not found"
+│   └── → xcuitest 11.x bug. Pin to 10.43.1:
+│       `appium driver install --source=npm appium-xcuitest-driver@10.43.1`
+│
+├── Android: "device not found" / "unauthorized"
+│   └── → unplug, replug, tap **Allow** on the phone.
+│       Verify with `adb devices`.
+│
+├── "USB disconnect during script"
+│   └── → Wrap your script with `@retry_on_disconnect(max_attempts=3)`.
+│       Plug into a powered USB hub instead of a laptop port if it keeps happening.
+│
+├── "device locked" / "screen off"
+│   └── → Call `wake_device()` at the top of your script.
+│
+├── Tests pass but real run fails
+│   └── → Daemon is stuck. `rm /tmp/iph-*.sock /tmp/iph-*.pid` (iOS) or
+│       `rm /tmp/anh-*.sock /tmp/anh-*.pid` (Android), then re-run.
+│
+└── Everything else
+    └── → `mobile-use --doctor` then read SETUP.md for that step.
+```
+
+## Daemon logs (debugging)
+
+When `iphone-harness -c '...'` or `android-harness -c '...'` fails after running for a while, check the daemon logs:
+
+```bash
+# iOS (one log per IPH_NAME, default "default"):
+tail -50 /tmp/iph-default.log
+
+# Android:
+tail -50 /tmp/anh-default.log
+
+# Live tail while running another shell:
+tail -f /tmp/iph-default.log
+```
+
+The daemon writes `connecting to Appium…`, `session ok`, `stale session, reconnecting`, and any `fatal:` lines here. If the log is empty or missing, the daemon never started — run `mobile-use --doctor`.
+
 ## iOS
 
 - **`xcodebuild failed with code 65`**: check Appium server log — usually untrusted cert, missing provisioning, or missing device support files.