Anan iPhone Use

English · 简体中文

An iPhone MCP server built for M5Stack StopWatch (ESP32-S3), with USB and Bluetooth mouse and keyboard input.

The client observes the screen, decides what to do and checks the result; StopWatch executes taps, drags, scrolling and keyboard input. The firmware runs no model and provides no phone screenshots.

Integrated with Anan (an entirely client-side agent, coming soon as open source). Other MCP agents and ESP32 boards are welcome; the anan-iphone-use skill guides setup and hardware-specific adaptation.

Demos and setup

Draw in Notes	Check Xiaohongshu followers	Do Not Disturb and timer
Prompt: Open Notes and draw a little dog.	Prompt: Open Xiaohongshu and check how many followers I have.	Prompt: Turn on Do Not Disturb, then start a 25-minute timer.
iphone-use-notes-dog.mp4	iphone-use-red.mp4	iphone-use-mode-count.mp4

After flashing, choose a connection:

• Wireless: scan the StopWatch Wi-Fi QR, open http://192.168.4.1, confirm MCP pairing, then select your 2.4 GHz Wi-Fi. Copy the MCP configuration before closing setup; rejoin your usual Wi-Fi, import it, then select BLE input and pair Anan StopWatch on iPhone.
• USB: connect StopWatch to iPhone, configure USB Ethernet (192.168.7.2, mask 255.255.255.0, no router/DNS), then open http://192.168.7.1 to pair and import MCP.

Enable AssistiveTouch for either route, then tap Start control on StopWatch: white eyes on black react to input, tilt and touch, with colored cursors and keycaps during operations. Press either front key to stop and return Home. Wireless setup is illustrated below; full setup and recovery covers both routes.

Avatar credit: the expressive eyes are adapted from Trentct/m5stack-stopwatch-avatar, with MCP input feedback added here.

1 · Join the setup hotspot	2 · Confirm MCP pairing	3 · Copy config before leaving	4 · Pair BLE and start control

Static illustrations rendered from the firmware layout, not device screenshots. Codes, Wi-Fi password, IP and battery values are synthetic examples; MCP pairing is separate from Bluetooth pairing.

How it works

A phone or computer agent reaches StopWatch's MCP over USB networking or Wi-Fi; StopWatch sends input to iPhone over USB or experimental BLE. Network access and input selection are independent, so Wi-Fi MCP can also control a USB-connected phone.

What you need

Item	Requirement
Device	M5Stack StopWatch (ESP32-S3); computer and data cable for initial flashing
Phone	iPhone with AssistiveTouch enabled; USB input requires a USB-C connection
Client	Anan, or a phone/computer HTTP MCP client with access to the USB subnet or Wi-Fi LAN
Screen observation	Supplied by the client; Anan's system screen sharing requires iOS 27+

Quick start

git clone https://github.com/ananhq/anan-iphone-use.git
cd anan-iphone-use

Open your agent in the repository directory and enter:

Use $anan-iphone-use to identify my board, flash the firmware and set up iPhone MCP.
If the skill is not discovered, first read .agents/skills/anan-iphone-use/SKILL.md.

The skill guides flashing and connection setup, or minimal adaptation for another ESP32 board. Update through flash.sh / flash.py to preserve BLE pairing and Wi-Fi/MCP settings; see flashing instructions.

MCP interface

Default endpoint: http://stopwatch.local/mcp over USB/Wi-Fi; USB fallback: http://192.168.7.1/mcp. Import the exported configuration, including its Bearer header; switching USB/BLE input does not require a new import.

Method	Purpose	Parameters
device_status	Read connection, control grant and input-busy status	None
tap	Single click, double-click or long press	x, y: position count: 1 click or 2 clicks holdMs: long-press duration intervalMs: double-click interval
drag	Swipe, drag or draw a continuous stroke	points: ordered coordinates; 2 for a straight line, more for a path durationMs: movement duration holdMs: hold before moving
press_key	Keyboard, editing and system/media keys	key: name, e.g. tab, copy, home, search repeat: navigation repeat count pressMs, gapMs: press duration and repeat interval
type_ascii	Type ASCII text into focused input	text: text to enter; no Chinese/Unicode
scroll	Scroll a chosen region	x, y: target position vertical, horizontal: signed wheel amounts, not pixels repeat, gapMs: repeat count and interval
stop	Interrupt input and revoke control	None

Coordinates are portrait-normalized 0…1; times are in milliseconds. Screen observation comes from the client. See tool reference for full parameters/key names and client integration for connection details. Refresh tool discovery after updating firmware.

Current limitations

• Wi-Fi uses 2.4 GHz and an accessible local network; the setup hotspot pauses BLE and is not an Internet router. Up to five Wi-Fi networks are remembered.
• Portrait, one pointer and US-layout ASCII input only; no direct Chinese text input.
• BLE input remains experimental. Pointer accuracy, reconnection, client interoperability and endurance need device acceptance; build/offline results are recorded separately in validation.

Comparison and credits

Special thanks to Trentct and the contributors to m5stack-stopwatch-avatar. The control face adapts their expression engine, including expression tables, blinking and transitions. This repository ports it to native ESP-IDF and adds MCP activity animations and rendering/motion adjustments. The source origin record lists the pinned commit and modifications; upstream AGPL-3.0-or-later notices are retained.

Thanks also to actell for the USB NCM/HID source reference, and esp32-iphone-agent for exploring iPhone hardware automation.

Project	Observation and agent	Action interface	iPhone input
Anan iPhone Use + Anan	Phone-side screen sharing and agent	USB/Wi-Fi MCP	USB / experimental BLE absolute positioning and keyboard
actell	Phone-side agent, CoreDevice display stream	CoreDevice	CoreDevice taps and drags
esp32-iphone-agent	Computer-side AirPlay and Python agent	Wi-Fi TCP	USB relative mouse and keyboard

actell also retains the USB NCM/HID route referenced here; the table describes its CoreDevice route. esp32-iphone-agent's MIT license and Copyright (c) 2026 Henry Yuan are preserved in the complete notice. Exact versions, file origins and modifications are recorded in THIRD_PARTY.

License

Existing code retains GPL-3.0-only. The control avatar incorporates the AGPL-3.0-or-later engine from Trentct/m5stack-stopwatch-avatar; the combination follows GPLv3/AGPLv3 section 13. The setup page offers this build’s source and build instructions.

Upstream copyright notices and applicable licenses for third-party code and fonts are retained in NOTICE and THIRD_PARTY.