Choose theme
AI Resources
ESP32 iPhone Agent
ESP32 iPhone Agent connects a vision model to an iPhone through screen mirroring and a USB input board.
A separate macOS or Linux computer receives the screen, runs the agent, and sends input commands to an ESP32-S3 plugged into the phone. The project uses AirPlay and AssistiveTouch, without installing a test-runner app on the iPhone. Use this as a first read, not a recommendation. Open the original project before trusting details like terms, limits, privacy, cost, setup, or safety.
What it is
A hardware-assisted phone agent
The repository supplies ESP32-S3 firmware and a Python server with a browser interface. A planner turns a goal and screen image into a plan; an actor uses subsequent screenshots to choose taps, swipes, typing, drawing strokes, or completion.
Why it stands out
Screen images and ordinary input
UxPlay receives the AirPlay screen stream, while the board presents itself to the phone as a USB mouse and keyboard. The model works from pixels and pointer actions, so this route does not require an Apple developer account, Xcode, or a jailbreak.
Availability
Source, firmware, and setup instructions
The project declares an MIT license and documents board flashing, server installation, phone settings, and calibration. It is an early project for technical setup, with tests in the repository and portrait-only screen control.
Why it matters
What makes it useful
For a builder experimenting with an agent on a physical phone, the project brings screen capture, model decisions, and input delivery into one loop. Its demonstrations include opening a whiteboard and drawing with continuous strokes, and setting a repeating alarm in Clock. The browser interface also offers manual taps, swipes, typing, and a live view. Those examples show the intended use; they do not establish reliable completion across apps.
What to know
Where it fits
The computer and phone share a network for AirPlay mirroring. The board plugs into the phone and receives commands from the computer over Wi-Fi; AssistiveTouch turns its mouse clicks and drags into screen input. The agent calls a configured vision-model endpoint, which can be a cloud service or a self-hosted model server. The board handles input rather than running the model, so an ESP32 by itself does not replace the host computer.
Notable points
What stands out
Mouse movement is relative, so a requested tap depends on knowing where the pointer really is. The project describes moving the pointer briefly to locate it in the mirrored image, correcting its position, and refining calibration. It also compares screens after actions and can re-plan when actions repeat. The step wall keeps the screenshot used for each decision with the tap or stroke drawn over it, making the action sequence available to inspect. These feedback mechanisms do not guarantee that an app changed in the intended way.
Before using
What to review
The documented setup uses an ESP32-S3 board whose connector reaches its native USB, with the plain Seeed XIAO ESP32S3 named as the tested board. USB-C iPhones connect directly; Lightning phones require a camera adapter. Flashing uses Arduino tooling.
Screen coordinates and calibration currently assume portrait orientation. Keyboard input is ASCII-only, mirroring stops when the phone locks, and some screens such as passcode entry and Apple Pay are not mirrored.
The chosen model endpoint receives screen images and task prompts. Cloud processing depends on that provider; selecting a self-hosted endpoint is a separate setup choice. Optional debug traces save model calls and screenshots on the host.
The included web server listens on all host network interfaces by default, and its phone-control API routes do not include a sign-in layer. The web interface exposes live screen and input controls, so access to the host interface matters.
The project asks users to supervise the agent and keep its emergency stop available. The interface includes a stop control and an Esc shortcut; this is not a per-action approval workflow.
Reader fit
Who may find it relevant
Relevant to developers and hardware hobbyists exploring vision-driven interaction with their own phone. Expect firmware flashing, a host server, model configuration, and pointer calibration. It is a source project to assemble and inspect, rather than a finished iPhone app to download. The proposal to move the whole loop onto a small Linux board is a future direction, not part of the current setup.
Editorial note
Why LifeHubber lists it
We list ESP32 iPhone Agent because its step history connects a model decision to the screen it saw and the input it attempted. For someone exploring why a phone task went off course, that record offers a way to examine whether the problem began with interpreting the screen or delivering the action. A final screenshot alone would not show that sequence. The project makes those intermediate steps visible while leaving the actual app result for the person running it to check.
Source links
Source materials
Reader note
Before relying on this entry
LifeHubber lists entries to help readers inspect AI projects, not to endorse them or prove they are safe, suitable, accurate, maintained, or right for a specific use. We do not verify every entry in depth. Before relying on anything listed, review the original materials, terms, privacy practices, limits, and risks that matter for your situation.
More in AI Agents
Keep browsing this category
Explore more AI agent projects.
Paperclip
paperclipai/paperclip
A self-hosted server and dashboard for coordinating agent teams through companies, goals, roles, issues, heartbeats, budgets, approvals, and persistent activity records.
Universal Modder
rehan-remade/universal-modder
Agent skills and a Python CLI for PC game modding, with engine research, generated assets, in-game checks, and shared field notes. Setup and compatibility depend on the game and workflow; fal generation requires a separate API key.
DeepTutor
HKUDS/DeepTutor
A tutoring workspace with study materials, Mastery Path, editable learner memory, web and CLI access, and configured image and video generation tools.
For project maintainers
Listed here? You can use the badge.
If you maintain a project with a current LifeHubber listing, you may add the optional “Listed on LifeHubber AI Resources” badge to its README, docs, or website. No introduction or permission request is needed.
