← Portfolio03 / AGENT EVALUATION · 2025

DailyDroid

We tested whether adding screenshots helps Android agents complete everyday tasks, then examined the failures that a success rate alone would hide.

DailyDroid seminar slide showing how models, apps, interfaces and frameworks contribute to mobile agent reliability.

What DailyDroid tests

We selected five apps in each of five categories and wrote a simple, medium and hard task for each app. The resulting 75 tasks cover everyday interactions with forms, search, media controls and system settings.

Benchmark coverage from Table 2. Each category contains five apps and 15 tasks.
CategoryExample apps
Productivity & toolsCalendar, Email, Notes
Utilities & system tasksClock, Settings, Calculator
Information accessWikipedia, Chrome, Maps
Media & entertainmentSpotify, YouTube, Books
CommunicationFacebook, Reddit, Messages

See the full benchmark table

Two views of the same screen

We extended AutoTask to capture a screenshot at each step. The text-only setting used the Android UI tree converted to simplified HTML. The multimodal setting added a screenshot from the same step. Both models ran all 75 tasks in both settings, with the same emulator configuration and a ten-step cap. We reviewed the logs to verify each outcome.

Simplified HTML for Google Maps beside a screenshot of the same screen centred on the Eiffel Tower.
Figure 2. Screen text exposes element labels and structure. The screenshot also shows layout, icons and map content. View in the paper

What screenshots changed

Selected metrics from Table 4. Each condition had 75 trials. Mean time and model cost refer to successful runs.
Model and inputCompletedMean timeMean cost (USD)
GPT-4o
Screen text
20/75
26.7%
141.92 s$0.0947
GPT-4o
Text + screenshot
24/75
32.0%
144.86 s$2.4593
o4-mini
Screen text
22/75
29.3%
204.97 s$0.0077
o4-mini
Text + screenshot
25/75
33.3%
220.39 s$0.1906

Adding screenshots yielded four more completions for GPT-4o and three for o4-mini. In this setup, the average model cost per successful task was about 25–26 times higher with screenshots. See the full results table

Where the runs broke down

The Failure Handbook distinguishes system-level blockers from errors the agent makes while it can still act. Failure to retrieve the UI was the most frequent system-level issue, affecting 23–25 of the 75 trials in each condition.

Selected categories from Table 3, the Failure Handbook
FailureWhat happens
UI retrievalThe system fails to fetch the app’s interface.
UI parsingThe interface is retrieved, but the system cannot identify the controls needed to continue.
ExecutionThe intended action is identified but cannot be performed.
Model prediction or reflectionThe agent picks the wrong action or misjudges whether a step worked.
Step limitThe run exhausts its action budget, sometimes after repeating the same attempts.
Three Android screenshots highlight a reading-progress indicator, a red hang-up button and a search result summary.
Figure 3. Three visual cues that the text-only input missed. View in the paper

A screenshot can reveal a control such as the red hang-up button. The agent still needs a way to act on it, so an accessibility or execution failure can block the task even when the control is visible. Read the full Failure Handbook

Two recorded runs

The recordings show different tasks and model settings. The tables above compare the four settings on the same tasks.

Success · Clear YouTube’s cache

GPT-4o with screen text · 167 seconds. An Android settings task completed in the text-only setup.

Open in Google Drive
Failure · Find a Domino’s phone number

o4-mini with screen text and screenshots · 460 seconds. The run ended without completing the lookup.

Open in Google Drive
← Back to portfolioTalk about this project