onehumanbeing/esp32-iphone-agent

Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak

Summary

ESP32 iPhone Agent lets a vision model drive a real iPhone using a small ESP32-S3 board plugged into the phone, with no Mac, developer account, or jailbreak required at run time. The board acts as both an AirPlay receiver (eyes) and a USB HID mouse and keyboard (hands), while an iPhone app runs the model and decides what to do. The target is engineers experimenting with phone-driving agents on a roughly $7 board.

What it is useful for

• Building AI agents that interact with iOS apps through screen observation and pointer/keyboard input, similar to computer-use agents but on a phone
• Prototyping automation on a real device without a tethered computer, jailbreak, or paid Apple Developer account
• Testing any OpenAI-compatible vision endpoint (vLLM, SGLang, llama.cpp, Ollama, cloud APIs) against live iOS UI
• Reviewing runs: every run is stored on the phone and can be replayed step by step with taps and swipes drawn on top
• A self-contained reference for BLE-based device control, an AirPlay receiver, and USB HID emulation on ESP32-S3

How engineers can use it

Hardware documented in the README: a Seeed Studio XIAO ESP32S3 (plain, 8 MB PSRAM, about $7), an iPhone on iOS 18 or later with cellular data, and an OpenAI-compatible vision endpoint. A Mac with Xcode is only needed once, to flash the board and build the app.

Setup steps from the README:

1. Flash the board: install arduino-cli, the ESP32 core 3.3.x, and [email protected], then run hardware/flash.sh (the script applies the BLE buffer sizes in hardware/hid_bridge/nimble_cfg.h).
2. Build the app: copy ios/Config/Local.xcconfig.example to ios/Config/Local.xcconfig, set DEVELOPMENT_TEAM and IA_BUNDLE_ID, open ios/IPhoneAgent.xcodeproj in Xcode, and run the IPhoneAgent scheme on the iPhone. A free Apple ID can sign it, but the app must be re-signed every 7 days.
3. In-app setup walks through pairing over Bluetooth, joining the board's hotspot and enabling Screen Mirroring to ESP32 Agent, turning on AssistiveTouch, and entering a model endpoint and API key (stored in the Keychain).
4. Type a goal such as "Open Settings and go to Wi-Fi" and press Start. The agent runs in the background; the Dynamic Island shows progress and a stop button. Siri, Shortcuts, widgets, and Control Center can launch saved tasks.

Prompts are plain text in prompts/ and bundled with the app, so planner and actor behavior can be edited and rebuilt.

Documented limitations: BLE bandwidth caps mirroring at about 480×1040 and ≤5 fps, so each step takes a few seconds; iOS won't mirror passcode, Apple Pay, or DRM screens, and mirroring stops when the phone locks; the phone's Wi-Fi is the board's hotspot during runs, so the model is reached over cellular data; keyboard input is ASCII only; landscape apps are not yet supported. The firmware is MIT-licensed, but hardware/hid_bridge/src/airplay/ is GPL-3.0 (ported from UxPlay), which makes the combined firmware binary GPL-3.0. The app also uses a private iOS API (suspend), which the README notes is fine for a self-built app but would need replacing before App Store distribution.