Paper notes
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
First arXiv submission: · NeurIPS 2024
Provides real computer environments and reproducible tasks spanning desktop applications, websites, and file operations.
Contribution and setting
Pairs task setup with execution-based evaluators, allowing agents to complete workflows through actual software rather than matching a fixed action sequence.
- Model output
- Observations and evaluation results from real computer environments.
- Action interface
- Computer interaction through the benchmark's supported action interfaces.
- State and memory
- State persists in the real operating system and applications used by the environment.
Relationship to NeuralOS
Provides desktop tasks and outcome checks for evaluating computer-use agents.
Scope and limitations
Evaluates workflows in real desktop applications, websites, and files.
Related terms: OSWorld, computer-use agents, execution-based evaluation, desktop benchmark.
Cite this paper
Published version
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Advances in Neural Information Processing Systems, 2024.
Publication BibTeX · Publication CSL-JSON
ArXiv preprint
arXiv:2404.07972, initial submission 2024-04-11.