Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
85a6599
goal/004-fully-working-no-device-issues: ✓ deliverable 1 — daemon lif…
jackulau May 24, 2026
d3d5b25
goal/004-fully-working-no-device-issues: ✓ deliverable 1 — daemon lif…
jackulau May 24, 2026
05c4507
goal/004-fully-working-no-device-issues: ✓ deliverable 2 — cleanup_st…
jackulau May 24, 2026
979b5e3
goal/004-fully-working-no-device-issues: ✓ deliverable 3 — device dis…
jackulau May 24, 2026
9c7c2fe
goal/004-fully-working-no-device-issues: ✓ deliverable 4 — WDA signin…
jackulau May 24, 2026
92da2b4
goal/004-fully-working-no-device-issues: ✓ deliverable 5 — Linux supp…
jackulau May 24, 2026
7568611
goal/004-fully-working-no-device-issues: ✓ deliverable 6 — record_scr…
jackulau May 24, 2026
3a75dcd
goal/004-fully-working-no-device-issues: ✓ deliverable 7 — record/rep…
jackulau May 24, 2026
1e61c25
goal/004-fully-working-no-device-issues: ✓ deliverable 8 — doctor v2 …
jackulau May 24, 2026
fd82763
goal/004-fully-working-no-device-issues: ✓ deliverable 9 — e2e no-dev…
jackulau May 24, 2026
264d25b
goal/004-fully-working-no-device-issues: ✓ deliverable 10 — docs poli…
jackulau May 24, 2026
89fc8a5
goal/005-verify-and-harden: fix bugs found by edge-case + coverage audit
jackulau May 24, 2026
6ba4885
goal/006-final-hardening: CLI dispatch fixes + mock daemon parity + i…
jackulau May 24, 2026
a0b6454
goal/006-final-hardening: IPC response size cap (64MB)
jackulau May 24, 2026
fd7aeec
goal/007-fully-working: ✓ D1 — iOS nav parity (press_home, swipe_back…
jackulau May 24, 2026
a24ca28
goal/007-fully-working: ✓ D2 — CLI HELP rewrite + ios subcommand hint…
jackulau May 24, 2026
8e84c48
goal/007-fully-working: ✓ D4 — mobile-use ios build-wda (automates SE…
jackulau May 24, 2026
deda0be
goal/007-fully-working: ✓ D3 — bootstrap Xcode preflight (gates iOS w…
jackulau May 24, 2026
a8f1b23
goal/007-fully-working: ✓ D5 — quickstart Appium server preflight + -…
jackulau May 24, 2026
910bf78
goal/007-fully-working: ✓ D6 — init Team ID guidance + .env preflight…
jackulau May 24, 2026
3dff752
goal/007-fully-working: ✓ D7 — agent loop smoke test (14 tests, mock …
jackulau May 24, 2026
85f0c7e
goal/007-fully-working: ✓ D8 — -c e2e via subprocess + mock daemon (i…
jackulau May 24, 2026
f075c44
goal/007-fully-working: ✓ D9 — patch agent loop mock injection so 446…
jackulau May 24, 2026
29cc236
goal/007-fully-working: ✓ D10 — README accuracy (iOS nav helpers, bui…
jackulau May 24, 2026
b56c47d
goal/008-linux-device-support: D1 — _platform module + bootstrap refa…
jackulau May 25, 2026
6c977c0
goal/008-linux-device-support: D2 — Linux remediation in doctor + iOS…
jackulau May 25, 2026
402b9a0
goal/008-linux-device-support: add test_ipc_tcp coverage for TCP daem…
jackulau May 25, 2026
02bcb99
goal/008-linux-device-support: D3 — OCR Linux fallback
jackulau May 25, 2026
6898a4c
goal/008-headed-mode: ✓ D1+D2 — TCP IPC transport + remote-daemon cli…
jackulau May 25, 2026
21f4477
goal/008-linux-device-support: D4 — bootstrap on Linux (Android path …
jackulau May 25, 2026
740c730
goal/008-linux-device-support: D5 — CLI Linux behavior (init/quicksta…
jackulau May 25, 2026
711dee2
goal/008-linux-device-support: D6 — iOS on Linux via remote macOS Appium
jackulau May 25, 2026
f8fe5a3
goal/008-linux-device-support: D7 — GitHub Actions CI matrix
jackulau May 25, 2026
56f34bc
goal/008-linux-device-support: D8 — Dockerfile.linux-test + os-releas…
jackulau May 25, 2026
e5f7f87
goal/008-linux-device-support: D9 — README + SETUP Linux sections
jackulau May 25, 2026
a07a2c1
goal/008-linux-device-support: D11 — macOS regression sweep (and re-a…
jackulau May 25, 2026
e66a16d
goal/008-linux-device-support: fix Ubuntu CI — bypass D5 short-circui…
jackulau May 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.git
.github
.claude-workspace
.venv
__pycache__
*.egg-info
.pytest_cache
.ruff_cache
agent-workspace
sessions
training-data
docs/demos/*.png
*.pyc
.DS_Store
46 changes: 30 additions & 16 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,63 @@
# YOUR device's values. Both files are .gitignore'd by default.
#
# You only need to fill in the section for the platform you're using.
# Run `mobile-use init` for a guided fill — auto-detects connected devices.

# ============================================================================
# iOS (iphone-harness)
# ============================================================================

# ---- Required ----

# Your iPhone's UDID. Find it with:
# idevice_id -l (libimobiledevice — `brew install libimobiledevice`)
# xcrun xctrace list devices (Apple's tool, also lists offline devices)
# required — iPhone UDID. Find with: `idevice_id -l` or `xcrun xctrace list devices`
IPH_UDID=YOUR-IPHONE-UDID-HERE

# Your Apple Developer Team ID (10 chars). Find at developer.apple.com → Membership.
# required — Apple Developer Team ID (10 chars).
# Find at developer.apple.com → Membership, or Keychain Access → your Apple Development cert → Get Info → Organizational Unit.
IPH_XCODE_ORG_ID=YOURTEAMID

# A bundle id YOU own/can sign, used for WebDriverAgent on the device.
# Example: com.<your-handle>.mobile-use.wda
# required — A reverse-DNS bundle id you can sign. Example: com.<your-handle>.mobile-use.wda
# This becomes the bundle ID of WebDriverAgent on your device.
IPH_WDA_BUNDLE_ID=com.example.mobile-use.wda

# ---- Optional (iOS) ----

# optional — pinned iOS version (Appium usually auto-detects)
# IPH_PLATFORM_VERSION=18.3.2

# optional — device display name (visible in Appium logs)
# IPH_DEVICE_NAME=My iPhone

# optional — non-default Appium server URL
# IPH_APPIUM_URL=http://127.0.0.1:4723

# optional — daemon instance name (for multibox / multiple devices)
# IPH_NAME=default

# optional — enable domain-specific skills (per-app playbooks)
# IPH_DOMAIN_SKILLS=1

# optional — seconds Appium waits between agent calls before killing session
# IPH_NEW_COMMAND_TIMEOUT=600


# ============================================================================
# Android (android-harness)
# ============================================================================

# ---- Required ----

# Your Android device serial. Find it with:
# adb devices
# required — Android device serial. Find with: `adb devices`
ANH_UDID=YOUR-ANDROID-SERIAL-HERE

# ---- Optional (Android) ----

# optional — pinned Android version
# ANH_PLATFORM_VERSION=14.0

# optional — device display name
# ANH_DEVICE_NAME=Android

# optional — non-default Appium server URL
# ANH_APPIUM_URL=http://127.0.0.1:4723

# optional — daemon instance name (for multibox)
# ANH_NAME=default

# optional — enable domain-specific skills
# ANH_DOMAIN_SKILLS=1

# optional — seconds Appium waits between agent calls before killing session
# ANH_NEW_COMMAND_TIMEOUT=600
60 changes: 60 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: ci

on:
push:
branches: [main]
pull_request:
branches: [main]

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
name: pytest (${{ matrix.os }} / py${{ matrix.python }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
cache: pip

- name: Install package
run: |
python -m pip install --upgrade pip
pip install -e .
pip install pytest

- name: Linux — install adb (smoke-test the bootstrap-recommended path)
if: runner.os == 'Linux'
run: |
sudo apt-get update
sudo apt-get install -y android-tools-adb

- name: Run tests
run: python -m pytest -q
env:
PYTHONDONTWRITEBYTECODE: "1"

lint:
name: ruff
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- run: pip install ruff
- run: ruff check mobile_use iphone_harness android_harness || true
# ruff currently advisory — surfaces warnings without blocking merge.
# Tighten after the codebase is fully ruff-clean.
11 changes: 11 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,17 @@ Built by @jackulau — github.com/jackulau/mobile_use.
## Shared
- `SKILL.md` tells agents how to use either harness and CLI.
- `SETUP.md` tells agents how to install, attach a device, and troubleshoot.
- `mobile_use/_platform.py` — single source of truth for host detection
(is_linux/is_macos/linux_pkg_manager/install_hint). Touch this before
sprinkling new `sys.platform` checks.

## Hosts
- macOS: full iOS + Android. `brew` for system deps.
- Linux: full Android (apt/dnf/pacman/zypper/apk). iOS only via remote macOS
(IPH_CONNECT TCP or IPH_APPIUM_URL pointing at a Mac). See SETUP.md
"iOS from Windows / Linux".
- Windows: same iOS-via-remote-Mac pattern as Linux. Android works if adb is
on PATH.
- `agent-workspace/` — agent-editable helpers + per-app domain skills
- `interaction-skills/` — iOS-specific UI mechanics
- `android-interaction-skills/` — Android-specific UI mechanics
Expand Down
54 changes: 54 additions & 0 deletions Dockerfile.linux-test
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# System deps: python, pip, build tools, adb (the canonical Linux Android tool),
# node + npm (for the Appium CLI if the user wants to install it later),
# and libimobiledevice tools (idevice_id — for iOS pairing if the user ever
# bridges to a remote Mac).
RUN apt-get update && apt-get install -y --no-install-recommends \
python3 python3-pip python3-venv \
android-tools-adb \
nodejs npm \
libimobiledevice6 libimobiledevice-utils usbmuxd \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /workspace
COPY pyproject.toml ./
COPY mobile_use mobile_use
COPY iphone_harness iphone_harness
COPY android_harness android_harness
COPY tests tests
COPY docs docs
COPY README.md SETUP.md AGENTS.md ./

# Editable install picks up our package + sets up CLI entry points.
# --break-system-packages: Ubuntu 24.04's python is PEP 668 marked-external;
# this is a single-purpose throwaway test container, so loosening that is fine.
RUN pip install --break-system-packages -e . \
&& pip install --break-system-packages pytest

# Sanity: prove imports work on Linux without pyobjc.
RUN python3 -c "from mobile_use._platform import is_linux, host_os_label; \
assert is_linux(); print('Container host:', host_os_label())"

# Linux-focused unit set — proves the platform abstraction, doctor remediation,
# OCR fallback, bootstrap, CLI, and remote-Appium plumbing all work on Linux
# without macOS deps. Override with e.g.:
# docker run --rm mobile-use-linux-test:goal008 python3 -m pytest -q
# docker run --rm mobile-use-linux-test:goal008 mobile-use --doctor
CMD ["python3", "-m", "pytest", "-q", \
"tests/test_platform.py", \
"tests/test_bootstrap.py", \
"tests/test_bootstrap_linux.py", \
"tests/test_doctor_linux.py", \
"tests/test_ocr_platform.py", \
"tests/test_cli_linux.py", \
"tests/test_ios_remote_linux.py", \
"tests/test_ipc_tcp.py", \
"tests/test_remote_daemon.py", \
"tests/test_hardening.py", \
"tests/test_imports.py"]
72 changes: 67 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,68 @@ If anything fails:
```bash
mobile-use --doctor # numbered checks with one-line remediations
iphone-harness --reload # nuke the daemon (rare but kills weird stale state)
mobile-use ios sign-wda # iOS: re-sign WebDriverAgent (the #1 setup blocker)
mobile-use ios build-wda # iOS: build the WDA test target (first-run setup)
mobile-use quickstart --autostart-appium # spawn Appium server in background
```

See [`SETUP.md`](SETUP.md) for the manual / per-step appendix, or skip to the
sections below for usage.
See [`SETUP.md`](SETUP.md) for the manual / per-step appendix, including a
[troubleshooting decision tree](SETUP.md#part-c--troubleshooting).

### Linux

Android-on-Linux is a first-class target. `mobile-use bootstrap` auto-detects
your package manager (apt, dnf, pacman, zypper, apk) and installs `adb`, `node`,
and the Appium uiautomator2 driver natively — no Homebrew required.

```bash
# Linux host (any apt/dnf/pacman/zypper/apk distro):
pip install -e .
mobile-use bootstrap --android-only
mobile-use init --android-only
mobile-use quickstart --android
```

**iOS on Linux** requires a Mac somewhere in the loop (Xcode + Apple
codesigning are macOS-only by Apple). Two patterns:

- **Remote daemon (TCP)** — Linux runs zero daemon locally; talks to a
remote `iphone-harness` daemon on a Mac via TCP:
```bash
# On the Mac (one shot):
IPH_BIND=tcp://127.0.0.1:8763 iphone-harness -c 'pass'
# On Linux (in another shell):
ssh -L 8763:127.0.0.1:8763 <mac-host>
mobile-use --ios --remote-daemon tcp://127.0.0.1:8763 -c 'print(active_app())'
```
- **Remote Appium URL** — `IPH_APPIUM_URL=http://<mac>:4723` lets a local
iphone-harness on Linux talk to a Mac running just Appium+WDA.

See [`SETUP.md` → "iOS from Windows / Linux"](SETUP.md#ios-from-windows--linux)
for the full walkthrough.

### Runtime helpers (no device pain)

```python
from iphone_harness.helpers import wake_device, retry_on_disconnect, record_screen

wake_device() # screen-off / locked? wake it.

@retry_on_disconnect(max_attempts=3) # USB blip / WDA crash → auto-restart + retry
def run_script():
tap(find(label="Compose"))
type_text("hello")

record_screen(duration=10) # save mp4 to /tmp (XCUITest + UIAutomator2)

# record/replay a tap sequence:
from mobile_use import record_replay
import iphone_harness.helpers as h
record_replay.start_recording("flow.py", helpers=h)
# ... your taps/swipes/typing ...
record_replay.stop_recording() # writes runnable flow.py
record_replay.replay("flow.py") # play it back
```

### Manual setup (skip if `mobile-use bootstrap` worked)

Expand Down Expand Up @@ -339,6 +397,13 @@ paste_text(text, ...)
# Device
unlock()

# Navigation (both platforms — Android native buttons, iOS gesture equivalents)
press_home() # both — go to home screen
press_back() # Android: back key; iOS: swipe-from-left edge
press_recents() # Android: recents; iOS: app switcher
swipe_back() # iOS: explicit edge-swipe (alias for press_back on iOS)
open_app_switcher() # iOS: swipe up + pause

# iOS-only
native_screenshot() # saves to iPhone Photos
set_assistive_touch(on=True)
Expand All @@ -349,9 +414,6 @@ start_screen_recording()
stop_screen_recording()

# Android-only
press_back()
press_home()
press_recents()
open_notifications()
close_notifications()
grant_permission(package, permission)
Expand Down
Loading
Loading