# OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Authors: Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu.

Provides real computer environments and reproducible tasks spanning desktop applications, websites, and file operations.

First public: 2024-04-11.
ArXiv initial submission: 2024-04-11.

[Original paper](https://arxiv.org/abs/2404.07972) · [Reading-list entry](https://neural-os.com/reading-list/papers/2404.07972/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2404.07972/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2404.07972/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2404.07972/record.json)

**Contribution:** Pairs task setup with execution-based evaluators, allowing agents to complete workflows through actual software rather than matching a fixed action sequence.

**Model output:** Observations and evaluation results from real computer environments.

**Action interface:** Computer interaction through the benchmark's supported action interfaces.

**State and memory:** State persists in the real operating system and applications used by the environment.

**Relationship to NeuralOS:** Provides desktop tasks and outcome checks for evaluating computer-use agents.

**Scope and limitations:** Evaluates workflows in real desktop applications, websites, and files.

[Relationship source](https://arxiv.org/abs/2404.07972)

Related terms: OSWorld, computer-use agents, execution-based evaluation, desktop benchmark.

## Published version

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu. Advances in Neural Information Processing Systems, 2024.
[Publication](https://proceedings.neurips.cc/paper_files/paper/2024/hash/5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.html) · [BibTeX](https://neural-os.com/reading-list/papers/2404.07972/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2404.07972/publication.csl.json)

## Approaches

- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.