# AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

Authors: Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva.

Provides a functional Android environment with parameterized tasks and programmatic checks of device state.

First public: 2024-05-23.
ArXiv initial submission: 2024-05-23.

[Original paper](https://arxiv.org/abs/2405.14573) · [Reading-list entry](https://neural-os.com/reading-list/papers/2405.14573/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2405.14573/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2405.14573/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2405.14573/record.json)

**Contribution:** Combines task initialization, success checking, and teardown with variable task instances, making mobile-agent evaluation reproducible and less dependent on a fixed test set.

**Model output:** Android observations and task rewards.

**Action interface:** Agent actions executed on an Android device environment.

**State and memory:** The Android system and apps maintain state; evaluators inspect that state directly.

**Relationship to NeuralOS:** Provides Android tasks and outcome checks for evaluating mobile agents.

**Scope and limitations:** The benchmark covers Android apps with parameterized tasks and programmatic success checks.

[Relationship source](https://arxiv.org/abs/2405.14573)

Related terms: AndroidWorld, mobile agents, GUI evaluation, task success.

## Published version

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva. International Conference on Learning Representations, 2025.
[Publication](https://proceedings.iclr.cc/paper_files/paper/2025/hash/01a83bc2f2732a58e6aa731e659e7101-Abstract-Conference.html) · [BibTeX](https://neural-os.com/reading-list/papers/2405.14573/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2405.14573/publication.csl.json)

## Approaches

- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.