DailyDroid
What happens when AI agents try to use our everyday apps?
Seeing more isn’t the whole story.
DailyDroid investigates smartphone automation using screen text versus text plus screenshots. The benchmark covers everyday tasks across Android apps and three difficulty levels.
Make the failures visible.
The study evaluates GPT-4o and o4-mini, comparing task success across input modalities. It looks beyond aggregate scores to understand where agents get stuck.
Do LLMs need to see everything?
What we learned.
Text-only and multimodal inputs showed comparable performance, with a marginal advantage for multimodal inputs. Failure analysis highlighted issues in UI accessibility, input modalities and LLM/app design.
The work contributes a benchmark and a handbook of common failures to help inform the design of mobile agents and interfaces.
Read the research.
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots.
Shiquan Zhang and collaborators · arXiv preprint, 2026.