Headed mode + iOS-from-Windows (TCP daemon + MJPEG viewer)

README.md

@@ -227,6 +227,53 @@ pool.broadcast_android(lambda d: d.press_home())
 
 Each device gets its own named daemon instance (`IPH_NAME` / `ANH_NAME`) with separate sockets, so they don't collide.
 
+## Headed mode — watch the device while it runs
+
+By default mobile-use is **headless**: scripts run, the daemon talks to the
+device, you see no UI. Add `--headed` to spin up a local MJPEG viewer in
+your browser and watch the live device screen mirror while the script runs:
+
+```bash
+mobile-use --ios --headed -c 'tap_at_xy(100, 200); time.sleep(2)'
+# → opens http://127.0.0.1:<random-port>/ in your default browser
+# → live mirror at ~6 fps, JPEG quality 60 (knobs in mobile_use/viewer/server.py)
+```
+
+The viewer is read-only — it shows what the device is doing; it doesn't
+take input. Use `--headless` (or omit the flag) to skip it. Works on iOS
+and Android.
+
+Quality knobs (via Python API, when running in agent mode):
+
+```python
+from mobile_use.viewer.server import ViewerServer
+v = ViewerServer(platform="ios", fps=12, quality=80, max_dim=1200)
+v.start(); print(v.url)
+# ...
+v.stop()
+```
+
+## iOS from Windows / Linux
+
+Windows hosts can't build WebDriverAgent (no Xcode). Drive iOS via a Mac
+on the network running the daemon over TCP:
+
+```bash
+# On the Mac (one time): full Part A in SETUP.md
+# On the Mac (each session):
+IPH_BIND=tcp://127.0.0.1:8763 iphone-harness -c 'pass'
+
+# On Windows / Linux:
+ssh -L 8763:127.0.0.1:8763 user@mac.local           # SSH tunnel (recommended)
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 -c 'print(active_app())'
+
+# Add --headed to also see the live screen mirror in your local browser:
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 --headed -c '...'
+```
+
+Full walkthrough + security caveat: SETUP.md → "iOS from Windows / Linux
+(remote Mac bridge)".
+
 ## Skills
 
 ### iOS Interaction Skills

SETUP.md

@@ -112,6 +112,99 @@ iphone-harness -c 'print(active_app())'
 
 ---
 
+# Part A* — iOS from Windows / Linux (remote Mac bridge)
+
+Windows and Linux cannot build or sign WebDriverAgent locally — `xcodebuild`
+and Apple codesigning are macOS-only. The path is to keep one Mac on the
+network as the "iOS bridge" and drive it remotely via TCP. The mobile-use
+client running on Windows/Linux talks to the daemon on the Mac; the Mac
+talks to the iPhone via Appium + WDA exactly as in Part A.
+
+### One-time setup on the Mac
+
+Follow Part A above on the Mac (Xcode, libimobiledevice, Appium, WDA signing,
+.env with IPH_UDID/IPH_XCODE_ORG_ID/IPH_WDA_BUNDLE_ID).
+
+Verify it works on the Mac itself before adding network in the mix:
+
+```bash
+iphone-harness -c 'print(active_app())'
+```
+
+### Each session on the Mac
+
+Start the daemon bound to TCP loopback (preferred — pair with SSH tunnel) OR
+to all interfaces (faster setup, less secure — see security caveat below).
+
+Loopback + SSH tunnel (recommended):
+
+```bash
+# On the Mac:
+IPH_BIND=tcp://127.0.0.1:8763 iphone-harness -c 'pass'  # starts daemon, exits client
+# Daemon keeps running. Re-running 'iphone-harness -c' attaches to it.
+```
+
+All-interfaces (skip SSH, faster — security warning printed to stderr):
+
+```bash
+IPH_BIND=tcp://0.0.0.0:8763 iphone-harness -c 'pass'
+```
+
+### On Windows or Linux
+
+```bash
+pip install mobile-use   # Android-only deps; iOS daemon never runs locally
+
+# Open SSH tunnel in another terminal (skip if Mac is bound to 0.0.0.0):
+ssh -L 8763:127.0.0.1:8763 user@mac.local
+
+# Drive iOS:
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 -c 'print(active_app())'
+
+# Or with the headed viewer in the browser:
+mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 --headed -c 'print(active_app())'
+```
+
+### How it works
+
+- `IPH_BIND=tcp://...` on the Mac switches the daemon's IPC from AF_UNIX
+  to TCP. AF_UNIX is unchanged when IPH_BIND is unset.
+- `--remote-daemon tcp://...` on the client sets `IPH_CONNECT` and flips
+  the harness into **client-only mode**: `ensure_daemon` never tries to
+  spawn a local daemon; it pings the remote, raises a remediation
+  checklist if unreachable.
+- The daemon over TCP serves the same JSON-line RPC protocol as over
+  AF_UNIX — no new methods, no protocol break. Everything that works
+  locally works remotely.
+
+### Security caveat
+
+The RPC is **unauthenticated**. Anything that can connect to the daemon's
+port can drive the phone. Mitigations:
+
+- Bind 127.0.0.1 on the Mac and use SSH tunnels (encrypts + authenticates
+  via SSH). This is the recommended pattern.
+- If you must bind 0.0.0.0, put a firewall rule (`pf` on macOS) in front
+  that only allows your specific Windows/Linux IP.
+- Future: HMAC token in `IPH_CONNECT_TOKEN` env (tracked as a follow-up).
+
+### Troubleshooting from the client
+
+```
+iphone-harness: remote daemon unreachable at tcp://127.0.0.1:8763
+```
+
+Means: client can't reach the daemon. On the Mac, check:
+
+```bash
+pgrep -fa iphone_harness.daemon          # daemon process alive?
+lsof -iTCP -sTCP:LISTEN | grep python    # bound to TCP?
+```
+
+On Windows: `Test-NetConnection -ComputerName <mac> -Port 8763`.
+
+---
+
 # Part B — Android Setup
 
 Android setup is significantly simpler than iOS — no signing, no Xcode, no provisioning.

android_harness/daemon.py

@@ -92,6 +92,13 @@ def __init__(self):
         self.driver = None
         self.stop = None
         self._loop = None
+        # Screen-stream state — populated by screen_stream_start RPC.
+        self._stream_task = None
+        self._stream_frame = None
+        self._stream_frame_no = 0
+        self._stream_fps = 6.0
+        self._stream_quality = 60
+        self._stream_max_dim = 800
 
     async def _drive(self, fn):
         return await self._loop.run_in_executor(None, fn)
@@ -177,6 +184,79 @@ async def _m_screenshot(d, params):
     return {"path": path, "bytes": len(png)}
 
 
+# ---- live screen stream (powers --headed viewer) --------------------------
+
+async def _stream_loop(d):
+    """Capture frames at d._stream_fps; JPEG-encode; store latest."""
+    import io
+    try:
+        from PIL import Image
+    except ImportError:
+        log("stream: Pillow not installed — install via `pip install pillow`")
+        return
+    while True:
+        period = 1.0 / max(0.1, d._stream_fps)
+        try:
+            png = await d._drive(d.driver.get_screenshot_as_png)
+            img = Image.open(io.BytesIO(png))
+            if d._stream_max_dim and max(img.size) > d._stream_max_dim:
+                img.thumbnail((d._stream_max_dim, d._stream_max_dim))
+            buf = io.BytesIO()
+            img.convert("RGB").save(buf, format="JPEG", quality=d._stream_quality)
+            d._stream_frame = buf.getvalue()
+            d._stream_frame_no += 1
+        except asyncio.CancelledError:
+            raise
+        except Exception as e:
+            log(f"stream: capture failed: {e}")
+        await asyncio.sleep(period)
+
+
+async def _m_screen_stream_start(d, params):
+    """Start (or reconfigure) capture. params: {fps, quality, max_dim}."""
+    fps = float(params.get("fps", 6))
+    quality = max(1, min(95, int(params.get("quality", 60))))
+    max_dim = int(params.get("max_dim", 800))
+    d._stream_fps = fps
+    d._stream_quality = quality
+    d._stream_max_dim = max_dim
+    if d._stream_task is not None and not d._stream_task.done():
+        return {"running": True, "updated": True, "fps": fps,
+                "quality": quality, "max_dim": max_dim}
+    d._stream_frame_no = 0
+    d._stream_task = asyncio.create_task(_stream_loop(d))
+    return {"running": True, "started": True, "fps": fps,
+            "quality": quality, "max_dim": max_dim}
+
+
+async def _m_screen_stream_frame(d, params):
+    """Return latest captured frame as base64 JPEG. Single-consumer pull model."""
+    import base64
+    if d._stream_frame is None:
+        return {"ready": False, "frame_no": 0}
+    return {
+        "ready": True,
+        "frame_no": d._stream_frame_no,
+        "jpeg_b64": base64.b64encode(d._stream_frame).decode("ascii"),
+        "fps": d._stream_fps,
+        "quality": d._stream_quality,
+    }
+
+
+async def _m_screen_stream_stop(d, params):
+    """Cancel capture loop and clear buffered frame. Idempotent."""
+    if d._stream_task is None:
+        return {"running": False}
+    d._stream_task.cancel()
+    try:
+        await d._stream_task
+    except (asyncio.CancelledError, Exception):
+        pass
+    d._stream_task = None
+    d._stream_frame = None
+    return {"running": False, "stopped": True}
+
+
 async def _m_page_source(d, params):
     return await d._drive(lambda: d.driver.page_source)
 
@@ -280,6 +360,10 @@ def _do():
     "send_keys":      _m_send_keys,
     "set_value":      _m_set_value,
     "active_app":     _m_active_app,
+    # Live screen mirror — powers `mobile-use --headed`.
+    "screen_stream_start":  _m_screen_stream_start,
+    "screen_stream_frame":  _m_screen_stream_frame,
+    "screen_stream_stop":   _m_screen_stream_stop,
 }
 
 

android_harness/helpers.py

@@ -227,6 +227,33 @@ def screenshot(path=None):
     return r["path"]
 
 
+# ---- live screen stream (consumed by viewer/server.py) --------------------
+
+def screen_stream_start(fps=6, quality=60, max_dim=800):
+    """Start the daemon's screen-capture loop. Returns daemon's reply dict."""
+    _drop_conn()
+    r = _send({
+        "method": "screen_stream_start",
+        "params": {"fps": fps, "quality": quality, "max_dim": max_dim},
+    })
+    return r.get("result", {"running": False})
+
+
+def screen_stream_frame():
+    """Pull the latest JPEG frame from the daemon. Drops cached socket first
+    since the daemon closes the conn per reply (silent empty otherwise)."""
+    _drop_conn()
+    r = _send({"method": "screen_stream_frame", "params": {}}, timeout=10.0)
+    return r.get("result", {"ready": False, "frame_no": 0})
+
+
+def screen_stream_stop():
+    """Cancel the daemon's capture loop. Idempotent."""
+    _drop_conn()
+    r = _send({"method": "screen_stream_stop", "params": {}})
+    return r.get("result", {"running": False})
+
+
 def window_size():
     """Logical screen size: {'width': W, 'height': H}.
     These are the units tap_at_xy(x, y) expects.

iphone_harness/daemon.py

@@ -102,6 +102,13 @@ def __init__(self):
         # Pool a single thread for blocking driver calls so Appium's HTTP
         # client doesn't fight asyncio's event loop. One driver, one worker.
         self._loop = None
+        # Screen-stream state — populated by screen_stream_start RPC.
+        self._stream_task = None
+        self._stream_frame = None       # latest JPEG bytes
+        self._stream_frame_no = 0       # increments per capture; viewer detects drops
+        self._stream_fps = 6.0
+        self._stream_quality = 60
+        self._stream_max_dim = 800      # largest side in px; thumbnailed
 
     async def _drive(self, fn):
         """Run a blocking driver callable in the default executor."""
@@ -199,6 +206,81 @@ async def _m_screenshot(d, params):
     return {"path": path, "bytes": len(png)}
 
 
+# ---- live screen stream (powers --headed viewer) --------------------------
+# Producer (frame loop) lives in the daemon; consumer (HTTP MJPEG sidecar)
+# pulls one frame at a time via screen_stream_frame. Single-consumer for v1.
+
+async def _stream_loop(d):
+    """Capture frames at d._stream_fps; JPEG-encode; store latest. Log + continue on errors."""
+    import io
+    try:
+        from PIL import Image
+    except ImportError:
+        log("stream: Pillow not installed — install via `pip install pillow`")
+        return
+    while True:
+        period = 1.0 / max(0.1, d._stream_fps)
+        try:
+            png = await d._drive(d.driver.get_screenshot_as_png)
+            img = Image.open(io.BytesIO(png))
+            if d._stream_max_dim and max(img.size) > d._stream_max_dim:
+                img.thumbnail((d._stream_max_dim, d._stream_max_dim))
+            buf = io.BytesIO()
+            img.convert("RGB").save(buf, format="JPEG", quality=d._stream_quality)
+            d._stream_frame = buf.getvalue()
+            d._stream_frame_no += 1
+        except asyncio.CancelledError:
+            raise
+        except Exception as e:
+            log(f"stream: capture failed: {e}")
+        await asyncio.sleep(period)
+
+
+async def _m_screen_stream_start(d, params):
+    """Start (or reconfigure) the capture loop. params: {fps, quality, max_dim}."""
+    fps = float(params.get("fps", 6))
+    quality = max(1, min(95, int(params.get("quality", 60))))
+    max_dim = int(params.get("max_dim", 800))
+    d._stream_fps = fps
+    d._stream_quality = quality
+    d._stream_max_dim = max_dim
+    if d._stream_task is not None and not d._stream_task.done():
+        return {"running": True, "updated": True, "fps": fps,
+                "quality": quality, "max_dim": max_dim}
+    d._stream_frame_no = 0
+    d._stream_task = asyncio.create_task(_stream_loop(d))
+    return {"running": True, "started": True, "fps": fps,
+            "quality": quality, "max_dim": max_dim}
+
+
+async def _m_screen_stream_frame(d, params):
+    """Return latest captured frame as base64 JPEG. Single-consumer pull model."""
+    import base64
+    if d._stream_frame is None:
+        return {"ready": False, "frame_no": 0}
+    return {
+        "ready": True,
+        "frame_no": d._stream_frame_no,
+        "jpeg_b64": base64.b64encode(d._stream_frame).decode("ascii"),
+        "fps": d._stream_fps,
+        "quality": d._stream_quality,
+    }
+
+
+async def _m_screen_stream_stop(d, params):
+    """Cancel the capture loop and clear buffered frame. Idempotent."""
+    if d._stream_task is None:
+        return {"running": False}
+    d._stream_task.cancel()
+    try:
+        await d._stream_task
+    except (asyncio.CancelledError, Exception):
+        pass
+    d._stream_task = None
+    d._stream_frame = None
+    return {"running": False, "stopped": True}
+
+
 async def _m_page_source(d, params):
     """Raw XML UI tree from WebDriverAgent. Helpers parse it client-side."""
     return await d._drive(lambda: d.driver.page_source)
@@ -327,6 +409,10 @@ def _do():
     "send_keys":      _m_send_keys,
     "set_value":      _m_set_value,
     "pick_wheel":     _m_pick_wheel,
+    # Live screen mirror — powers `mobile-use --headed`.
+    "screen_stream_start":  _m_screen_stream_start,
+    "screen_stream_frame":  _m_screen_stream_frame,
+    "screen_stream_stop":   _m_screen_stream_stop,
 }