Paper notes
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
First arXiv submission: · ICLR 2025
Provides a functional Android environment with parameterized tasks and programmatic checks of device state.
Contribution and setting
Combines task initialization, success checking, and teardown with variable task instances, making mobile-agent evaluation reproducible and less dependent on a fixed test set.
- Model output
- Android observations and task rewards.
- Action interface
- Agent actions executed on an Android device environment.
- State and memory
- The Android system and apps maintain state; evaluators inspect that state directly.
Relationship to NeuralOS
Provides Android tasks and outcome checks for evaluating mobile agents.
Scope and limitations
The benchmark covers Android apps with parameterized tasks and programmatic success checks.
Related terms: AndroidWorld, mobile agents, GUI evaluation, task success.
Cite this paper
Published version
Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. International Conference on Learning Representations, 2025.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2405.14573, initial submission 2024-05-23.