Large Language Model Agents for Smartphone Interaction: Introducing DailyDroid and Understanding Mobile Agent Failures
My University of Melbourne seminar introduces DailyDroid and shows how we traced mobile agent failures back to the interface, the automation framework, and the model.
Jeremy Shiquan Zhang · University of Melbourne
Open in Google DriveWhat the talk covers
The seminar opens with a simple-sounding task: ordering a specific coffee for delivery. The agent can get the temperature wrong, choose the wrong shop, or default to pickup. These examples motivate a broader question about how to identify where everyday smartphone tasks break down.
From benchmark to failure analysis
DailyDroid evaluates 75 tasks across 25 Android apps and three difficulty levels. Using an open-source mobile agent framework, we compared GPT-4o and o4-mini with screen text alone and with screen text plus screenshots across 300 trials.
The talk uses examples such as finding a nearby Domino’s phone number and planning a walking route in Maps to illustrate different failure modes. It closes with implications for accessible UI design, agent action spaces, and how to choose and evaluate the information an agent receives.
Explore the DailyDroid case study for the results table and two individual run recordings.