How AI agents control an Android phone

A language model cannot tap a screen. To drive an Android phone it needs a harness that turns the screen into something the model can read, and turns the model's decisions into touch and key events. Research harnesses, test tools and products that book or order things for their users all use some version of the same loop.

This article walks through that loop step by step, with the ADB commands underneath it, then covers where MCP fits and the failures that show up in every real project: popups, permissions, dates, long lists and screens the agent simply cannot read.

The loop

Every step of an agent on a phone has four parts:

1. Observe. Capture the current screen: a screenshot, the UI tree, or the two together.
2. Decide. Send the goal, a short history of earlier steps and the current observation to the model. The model answers with one structured action: tap this element, type this text, swipe, press Back, open an app, wait, or finish with an answer.
3. Act. The harness turns that action into input events on the phone.
4. Check. Observe again and confirm the screen changed the way the action intended.

In code, the skeleton is short:

steps = 0
while steps < max_steps:
    obs = phone.observe()              # screenshot + elements with text and bounds
    action = model.next_action(goal, history, obs)
    if action.kind == "done":
        return action.answer
    phone.act(action, seq=obs.seq)     # refuse if the screen changed since obs
    history.append((obs.summary, action))
    steps += 1

The hard work hides inside observe and act, and in what you put into history. Keep the history compact: a line per step with the action and what changed is enough. Full screenshots of every earlier step burn context without helping the model decide.

Seeing the screen: pixels and the UI tree

There are two ways to show a model the screen, and most working agents combine them.

With plain ADB the raw material looks like this:

adb exec-out screencap -p > screen.png
adb shell uiautomator dump /sdcard/ui.xml && adb pull /sdcard/ui.xml

The dump gives every node with its text, resource-id, content-desc and bounds. Tapping the center of a node's bounds is far more reliable than asking a vision model for raw coordinates. One catch: a UI dump waits for the screen to go idle, and screens that never stop moving, such as a playing video or a ticking timer, can make it fail. A harness should fall back to the screenshot when that happens.

Acting: taps, text and timing

The basic input commands:

adb shell input tap 540 1650
adb shell input swipe 540 1600 540 600 300
adb shell input text "hello%sworld"
adb shell input keyevent KEYCODE_BACK

Note the %s in input text: it stands for a space, and the command does not handle most non-ASCII characters. Harnesses that need other alphabets put the text on the clipboard and paste it.

Three details separate a reliable agent from a flaky one:

• Focus before typing. Tap the field first, then type. The keyboard then covers roughly the lower half of the screen, so observe again before the next tap.
• Settle, do not sleep. After an action, wait until the screen stops changing, with a cap of a few seconds. Fixed sleeps either waste time or act before a page has loaded.
• Refuse stale actions. If the screen changed after the observation the model used, the action may hit the wrong element. A good harness tags each observation with a sequence number and rejects an action computed from an older one, which forces a fresh look instead of a guess.

Where MCP fits

The Model Context Protocol is a standard way to give a model tools. A phone MCP server exposes tools such as listing devices, observing a screen and acting on it, so any MCP client (Claude Code, Codex, Cursor and others) can drive a phone without custom glue code. The loop does not change; MCP only standardizes the calls.

There are two levels of control. At the low level your agent calls observe and act and decides every step itself. At the high level it hands a goal to an agent that lives next to the phone: "open the app, add a 2 kg bag of rice to the cart, report the total". Clousd supports each level: observe and act through the REST API and the MCP server, and Tasks, where you give a phone a goal in plain words and Clousd's agent does the steps and returns the answer. The agents page shows the two side by side.

What breaks agents

Popups and interruptions

Permission dialogs, "rate this app" prompts, update notices, tooltips, cookie banners inside WebViews and "app isn't responding" dialogs all appear at random moments. An agent that follows a plan blindly taps the wrong thing. Put a rule in the prompt: before continuing a plan, close any dialog that is not part of the task. In test setups, grant permissions in advance with adb shell pm grant com.example.app android.permission.POST_NOTIFICATIONS.

Permissions that differ by Android version

Android 13 added a runtime permission for notifications. Android 14 added partial photo access, where the user picks "Select photos" instead of allowing all of them. Location has "While using the app" and "Only this time". The same task produces different dialogs on different versions, so test an agent on more than one.

Dates and time

"Add a meeting next Tuesday" depends on the phone's clock and time zone, while the model's sense of today comes from its training data. Put the device date, time and time zone into every observation. Test suites that check dated results need the phone clock set the way the task expects.

Long text and long lists

Typing several paragraphs eats the step budget. Counting items across a list that needs five scrolls, or finding duplicates in it, requires the agent to remember what it already saw. These are the tasks where agents fail most often today, and the AndroidWorld results on our benchmark page break the failures down by type.

Screens the agent cannot read

Apps that set the secure flag on a window, common in banking and password screens, come out black in screenshots. Games that draw on a canvas have no UI tree. Plan tasks around these limits instead of letting the agent loop on an empty screen.

State that leaks between runs

Logged-in sessions, cached data and notifications from an earlier run change what the agent sees in the next one. For evaluations, reset the phone to a fresh state between runs so every attempt starts the same way.

Building an evaluation that means something

• A fixed set of tasks, each with an end state you can check from outside: a row in a database, a file, the top activity (adb shell dumpsys activity top).
• One attempt per task with a fixed step budget. Retries hide real failure rates.
• A screenshot and the action for every step, so you can read why a task failed.
• Two or three phone models and Android versions. An agent that works on one screen size often fails on another.

For checks that need more than the screen, the same phone can be reached over ADB through a secure relay.