# Environments and evaluation Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements. Papers are ordered by first public date. For example: After an agent saves a file, inspect the resulting file and its contents instead of judging success only from a plausible screenshot. Version 1.0.1; literature checked through 2026-10-19. [Topic guide](https://neural-os.com/reading-list/guides/evaluation/) · [JSON](https://neural-os.com/reading-list/data/topics/evaluation.json) ## Papers — oldest first ## OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments Authors: Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu. Provides real computer environments and reproducible tasks spanning desktop applications, websites, and file operations. First public: 2024-04-11. ArXiv initial submission: 2024-04-11. [Original paper](https://arxiv.org/abs/2404.07972) · [Reading-list entry](https://neural-os.com/reading-list/papers/2404.07972/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2404.07972/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2404.07972/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2404.07972/record.json) **Contribution:** Pairs task setup with execution-based evaluators, allowing agents to complete workflows through actual software rather than matching a fixed action sequence. **Model output:** Observations and evaluation results from real computer environments. **Action interface:** Computer interaction through the benchmark's supported action interfaces. **State and memory:** State persists in the real operating system and applications used by the environment. **Relationship to NeuralOS:** Provides desktop tasks and outcome checks for evaluating computer-use agents. **Scope and limitations:** Evaluates workflows in real desktop applications, websites, and files. [Relationship source](https://arxiv.org/abs/2404.07972) Related terms: OSWorld, computer-use agents, execution-based evaluation, desktop benchmark. ### Published version OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu. Advances in Neural Information Processing Systems, 2024. [Publication](https://proceedings.neurips.cc/paper_files/paper/2024/hash/5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.html) · [BibTeX](https://neural-os.com/reading-list/papers/2404.07972/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2404.07972/publication.csl.json) ### Approaches - [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements. Reading-list version 1.0.1; literature checked through 2026-10-19. ## Diffusion for World Modeling: Visual Details Matter in Atari Authors: Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, François Fleuret. Introduces DIAMOND, an agent trained in a diffusion world model, and studies how preserving visual detail affects control. First public: 2024-05-20. ArXiv initial submission: 2024-05-20. [Original paper](https://arxiv.org/abs/2405.12399) · [Reading-list entry](https://neural-os.com/reading-list/papers/2405.12399/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2405.12399/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2405.12399/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2405.12399/record.json) **Contribution:** Connects diffusion-based observation prediction with Atari policy learning and demonstrates an interactive Counter-Strike simulator. **Model output:** Predicted visual observations; auxiliary reward and termination predictions support reinforcement learning. **Action interface:** Actions of the modeled game environment. **State and memory:** Visual dynamics condition on a finite history of observations and actions. **Relationship to NeuralOS:** NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. **Scope and limitations:** Evaluated on games; NeuralOS adapts and evaluates it for desktop screen prediction. [Relationship source](https://arxiv.org/html/2507.08800v2#A1) Related terms: DIAMOND, diffusion world model, neural game engine, model-based reinforcement learning. ### Method comparisons **A game world model evaluated on desktop interactions** DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction. Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1) Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) ### Approaches - [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions. - [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements. Reading-list version 1.0.1; literature checked through 2026-10-19. ## AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Authors: Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva. Provides a functional Android environment with parameterized tasks and programmatic checks of device state. First public: 2024-05-23. ArXiv initial submission: 2024-05-23. [Original paper](https://arxiv.org/abs/2405.14573) · [Reading-list entry](https://neural-os.com/reading-list/papers/2405.14573/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2405.14573/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2405.14573/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2405.14573/record.json) **Contribution:** Combines task initialization, success checking, and teardown with variable task instances, making mobile-agent evaluation reproducible and less dependent on a fixed test set. **Model output:** Android observations and task rewards. **Action interface:** Agent actions executed on an Android device environment. **State and memory:** The Android system and apps maintain state; evaluators inspect that state directly. **Relationship to NeuralOS:** Provides Android tasks and outcome checks for evaluating mobile agents. **Scope and limitations:** The benchmark covers Android apps with parameterized tasks and programmatic success checks. [Relationship source](https://arxiv.org/abs/2405.14573) Related terms: AndroidWorld, mobile agents, GUI evaluation, task success. ### Published version AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva. International Conference on Learning Representations, 2025. [Publication](https://proceedings.iclr.cc/paper_files/paper/2025/hash/01a83bc2f2732a58e6aa731e659e7101-Abstract-Conference.html) · [BibTeX](https://neural-os.com/reading-list/papers/2405.14573/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2405.14573/publication.csl.json) ### Approaches - [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements. Reading-list version 1.0.1; literature checked through 2026-10-19. ## NeuralOS: Towards Simulating Operating Systems via Neural Generative Models Authors: Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng. Studies neural operating systems through learned, interactive GUI simulation: a model produces desktop screens in response to mouse and keyboard events, opening a path toward generative computer interfaces. First public: 2025-07-11. ArXiv initial submission: 2025-07-11. [Original paper](https://arxiv.org/abs/2507.08800) · [Reading-list entry](https://neural-os.com/reading-list/papers/2507.08800/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2507.08800/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2507.08800/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2507.08800/record.json) **Contribution:** Formalizes OS-interface generation and develops state tracking, precise cursor conditioning, staged training, and data collection. The 2026 revision also demonstrates learning a new simulated application from synthesized training data. **Model output:** Autoregressively generated desktop screen frames. **Action interface:** Cursor coordinates, left/right clicks, and keyboard press/release events. **State and memory:** A hierarchical recurrent model carries state across frames and conditions a diffusion renderer. **Relationship to NeuralOS:** Introduces NeuralOS and its formulation of learned OS-interface simulation. **Scope and limitations:** Fine-grained typing and long-horizon state retention remain difficult. The simulated interface does not provide the underlying filesystem, processes, or application services of a conventional OS. [Relationship source](https://arxiv.org/html/2507.08800v1) Dated versions: [Original version](https://arxiv.org/html/2507.08800v1) · [Revised version](https://arxiv.org/html/2507.08800v2) [Code](https://github.com/yuntian-group/neural-os) Related terms: NeuralOS, neural operating system, learned computer simulator, generative interface, GUI world model, desktop world model, action-conditioned screen generation. ### Method comparisons **A game world model evaluated on desktop interactions** DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction. Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1) Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) **Direct frame generation and renderable-code prediction** NeuralOS predicts desktop frames using learned state and a diffusion renderer. Code2World predicts a UI through code that is then rendered into a screen, and tests its usefulness for mobile navigation. Code2World cites NeuralOS in Section 2.2. Sources: [Original NeuralOS paper](https://arxiv.org/html/2507.08800v1) · [Code2World Section 2.2](https://arxiv.org/html/2602.09856v1#S2.SS2) Paper details: [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) · [Code2World: A GUI World Model via Renderable Code Generation](https://neural-os.com/reading-list/papers/2602.09856/) Date note: Initial arXiv submission on July 11, 2025; revised on March 12, 2026 and published at ICLR 2026. [Source](https://arxiv.org/abs/2507.08800). ### Published version NeuralOS: Towards Simulating Operating Systems via Neural Generative Models. Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng. International Conference on Learning Representations, 2026. [Publication](https://proceedings.iclr.cc/paper_files/paper/2026/hash/2174a6749c3f769358cd21ade1221b34-Abstract-Conference.html) · [BibTeX](https://neural-os.com/reading-list/papers/2507.08800/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2507.08800/publication.csl.json) ### Approaches - [Neural operating systems](https://neural-os.com/reading-list/guides/neural-operating-systems/): Study how learned generative models can supply computer interaction, interface behavior, and adaptable user experiences. The research direction includes conceptual proposals and learned interface simulators. - [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions. - [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements. Reading-list version 1.0.1; literature checked through 2026-10-19.