# Neural Operating Systems and GUI World Models

Reading list curated by Yuntian Deng. Version 1.0.1.

[Introduction](https://neural-os.com/reading-list/guides/neural-operating-systems/) · [Downloads and citations](https://neural-os.com/reading-list/data/)

Papers are ordered by first arXiv submission date; publication years and later revisions are separate.

# Action-Conditional Video Prediction using Deep Networks in Atari Games

Authors: Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, Satinder Singh.

Learns to predict Atari frames from past observations and control inputs, connecting visual generation to an agent's actions.

First public: 2015-07-31.
ArXiv initial submission: 2015-07-31.

[Original paper](https://arxiv.org/abs/1507.08750) · [Reading-list entry](https://neural-os.com/reading-list/papers/1507.08750/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/1507.08750/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/1507.08750/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/1507.08750/record.json)

**Contribution:** Develops feedforward and recurrent architectures with action-dependent transformations and evaluates long visual rollouts and their usefulness for control.

**Model output:** Predicted game frames.

**Action interface:** Discrete Atari control actions.

**State and memory:** Past observations; the recurrent variant carries a hidden state through time.

**Relationship to NeuralOS:** Earlier action-conditioned visual-prediction work cited in NeuralOS's introduction and references.

**Scope and limitations:** The experiments use Atari games.

[Relationship source](https://arxiv.org/html/2507.08800v1)

Related terms: action-conditioned video prediction, visual dynamics, Atari, learned simulator.

## Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# World Models

Authors: David Ha, Jürgen Schmidhuber.

Combines learned visual compression and recurrent dynamics with a controller, including training an agent inside a generated environment.

First public: 2018-03-27.
ArXiv initial submission: 2018-03-27.

[Original paper](https://arxiv.org/abs/1803.10122) · [Reading-list entry](https://neural-os.com/reading-list/papers/1803.10122/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/1803.10122/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/1803.10122/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/1803.10122/record.json)

**Contribution:** Demonstrates how a compact learned environment representation can support policy learning and transfer from a simulated dream to the original task.

**Model output:** Predicted latent observations, with an image decoder and task-dependent termination predictions.

**Action interface:** Actions from the controller in the studied reinforcement-learning environments.

**State and memory:** A recurrent model summarizes the history of latent observations and actions.

**Relationship to NeuralOS:** Cited by NeuralOS as background for learned simulation and agent training.

**Scope and limitations:** The model compresses observations into a latent representation for control.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: world model, recurrent dynamics, latent imagination, model-based reinforcement learning.

## Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Learning to Simulate Dynamic Environments with GameGAN

Authors: Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, Sanja Fidler.

Learns an interactive visual game simulator from gameplay and keyboard actions using a generative adversarial network.

First public: 2020-05-25.
ArXiv initial submission: 2020-05-25.

[Original paper](https://arxiv.org/abs/2005.12126) · [Reading-list entry](https://neural-os.com/reading-list/papers/2005.12126/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2005.12126/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2005.12126/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2005.12126/record.json)

**Contribution:** Introduces a memory module for revisited locations and separates static and dynamic image components, enabling visual consistency and component recombination.

**Model output:** The next game-screen image.

**Action interface:** Game controls represented by keyboard actions.

**State and memory:** Recurrent dynamics plus an external memory module that stores an internal environmental map.

**Relationship to NeuralOS:** An earlier interactive neural simulator discussed in NeuralOS's related work.

**Scope and limitations:** The simulator learns from recorded games, with memory organized as a spatial map.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: GameGAN, neural game engine, interactive simulation, visual memory.

## Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models

Authors: Gabriele Tolomei, Cesare Campagnano, Fabrizio Silvestri, Giovanni Trappolini.

Proposes an operating-system interaction paradigm in which connected generative models interpret user intentions and mediate or replace conventional application interfaces.

First public: 2023-10-07.
ArXiv initial submission: 2023-10-07.

[Original paper](https://arxiv.org/abs/2310.04875) · [Reading-list entry](https://neural-os.com/reading-list/papers/2310.04875/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2310.04875/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2310.04875/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2310.04875/record.json)

**Contribution:** Sets out a broader vision for natural-language, personalized computer interaction, including language and diffusion models as interface components.

**Model output:** Envisioned context-dependent responses and generated interfaces.

**Action interface:** Natural-language user requests, including text and speech.

**State and memory:** Proposes personalization and contextual interaction.

**Relationship to NeuralOS:** NeuralOS cites Prompt-to-OS as background for generative computer interaction.

**Scope and limitations:** A conceptual proposal for generative computer interfaces.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: Prompt-to-OS, generative interface, neural operating system, human-computer interaction.

## Approaches

- [Neural operating systems](https://neural-os.com/reading-list/guides/neural-operating-systems/): Study how learned generative models can supply computer interaction, interface behavior, and adaptable user experiences. The research direction includes conceptual proposals and learned interface simulators.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Genie: Generative Interactive Environments

Authors: Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, Tim Rocktäschel.

Learns controllable visual environments from videos without ground-truth action labels, using a learned latent action space.

First public: 2024-02-23.
ArXiv initial submission: 2024-02-23.

[Original paper](https://arxiv.org/abs/2402.15391) · [Reading-list entry](https://neural-os.com/reading-list/papers/2402.15391/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2402.15391/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2402.15391/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2402.15391/record.json)

**Contribution:** Combines video tokenization, a dynamics model, and latent-action learning so users can interact with worlds initialized from images.

**Model output:** Action-conditioned future video frames.

**Action interface:** Discrete latent actions inferred during training rather than recorded physical input events.

**State and memory:** A spatiotemporal dynamics model conditions predictions on previous visual tokens and actions.

**Relationship to NeuralOS:** Cited in NeuralOS's related work. Genie learns its action space from video; NeuralOS takes mouse and keyboard events as inputs.

**Scope and limitations:** Controls use learned latent actions inferred from video.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: Genie, generative interactive environments, latent actions, foundation world model.

## Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Authors: Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu.

Provides real computer environments and reproducible tasks spanning desktop applications, websites, and file operations.

First public: 2024-04-11.
ArXiv initial submission: 2024-04-11.

[Original paper](https://arxiv.org/abs/2404.07972) · [Reading-list entry](https://neural-os.com/reading-list/papers/2404.07972/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2404.07972/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2404.07972/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2404.07972/record.json)

**Contribution:** Pairs task setup with execution-based evaluators, allowing agents to complete workflows through actual software rather than matching a fixed action sequence.

**Model output:** Observations and evaluation results from real computer environments.

**Action interface:** Computer interaction through the benchmark's supported action interfaces.

**State and memory:** State persists in the real operating system and applications used by the environment.

**Relationship to NeuralOS:** Provides desktop tasks and outcome checks for evaluating computer-use agents.

**Scope and limitations:** Evaluates workflows in real desktop applications, websites, and files.

[Relationship source](https://arxiv.org/abs/2404.07972)

Related terms: OSWorld, computer-use agents, execution-based evaluation, desktop benchmark.

## Published version

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu. Advances in Neural Information Processing Systems, 2024.
[Publication](https://proceedings.neurips.cc/paper_files/paper/2024/hash/5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets_and_Benchmarks_Track.html) · [BibTeX](https://neural-os.com/reading-list/papers/2404.07972/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2404.07972/publication.csl.json)

## Approaches

- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Diffusion for World Modeling: Visual Details Matter in Atari

Authors: Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, François Fleuret.

Introduces DIAMOND, an agent trained in a diffusion world model, and studies how preserving visual detail affects control.

First public: 2024-05-20.
ArXiv initial submission: 2024-05-20.

[Original paper](https://arxiv.org/abs/2405.12399) · [Reading-list entry](https://neural-os.com/reading-list/papers/2405.12399/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2405.12399/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2405.12399/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2405.12399/record.json)

**Contribution:** Connects diffusion-based observation prediction with Atari policy learning and demonstrates an interactive Counter-Strike simulator.

**Model output:** Predicted visual observations; auxiliary reward and termination predictions support reinforcement learning.

**Action interface:** Actions of the modeled game environment.

**State and memory:** Visual dynamics condition on a finite history of observations and actions.

**Relationship to NeuralOS:** NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision.

**Scope and limitations:** Evaluated on games; NeuralOS adapts and evaluates it for desktop screen prediction.

[Relationship source](https://arxiv.org/html/2507.08800v2#A1)

Related terms: DIAMOND, diffusion world model, neural game engine, model-based reinforcement learning.

## Method comparisons

**A game world model evaluated on desktop interactions**

DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction.

Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1)

Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/)


## Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.
- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

Authors: Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva.

Provides a functional Android environment with parameterized tasks and programmatic checks of device state.

First public: 2024-05-23.
ArXiv initial submission: 2024-05-23.

[Original paper](https://arxiv.org/abs/2405.14573) · [Reading-list entry](https://neural-os.com/reading-list/papers/2405.14573/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2405.14573/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2405.14573/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2405.14573/record.json)

**Contribution:** Combines task initialization, success checking, and teardown with variable task instances, making mobile-agent evaluation reproducible and less dependent on a fixed test set.

**Model output:** Android observations and task rewards.

**Action interface:** Agent actions executed on an Android device environment.

**State and memory:** The Android system and apps maintain state; evaluators inspect that state directly.

**Relationship to NeuralOS:** Provides Android tasks and outcome checks for evaluating mobile agents.

**Scope and limitations:** The benchmark covers Android apps with parameterized tasks and programmatic success checks.

[Relationship source](https://arxiv.org/abs/2405.14573)

Related terms: AndroidWorld, mobile agents, GUI evaluation, task success.

## Published version

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. Chris Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, Oriana Riva. International Conference on Learning Representations, 2025.
[Publication](https://proceedings.iclr.cc/paper_files/paper/2025/hash/01a83bc2f2732a58e6aa731e659e7101-Abstract-Conference.html) · [BibTeX](https://neural-os.com/reading-list/papers/2405.14573/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2405.14573/publication.csl.json)

## Approaches

- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Diffusion Models Are Real-Time Game Engines

Authors: Dani Valevski, Yaniv Leviathan, Moab Arar, Shlomi Fruchter.

Presents GameNGen, a diffusion model that interactively simulates DOOM from previous frames and player actions.

First public: 2024-08-27.
ArXiv initial submission: 2024-08-27.

[Original paper](https://arxiv.org/abs/2408.14837) · [Reading-list entry](https://neural-os.com/reading-list/papers/2408.14837/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2408.14837/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2408.14837/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2408.14837/record.json)

**Contribution:** Uses agent-collected gameplay, conditioning augmentation, and decoder fine-tuning to sustain autoregressive visual interaction.

**Model output:** The next rendered game frame.

**Action interface:** DOOM player actions.

**State and memory:** Conditioning on a sequence of recent frames and actions; continued play feeds generated frames back into the model.

**Relationship to NeuralOS:** Interactive-simulation background cited by NeuralOS; NeuralOS also refers to its perceptual-evaluation setup.

**Scope and limitations:** The simulator is trained and evaluated on DOOM.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: GameNGen, neural game engine, real-time simulation, diffusion renderer.

## Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

Authors: Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, Jinyoung Yeo.

Predicts the effects of candidate web actions as natural-language descriptions of important changes, then uses those predictions for action selection.

First public: 2024-10-17.
ArXiv initial submission: 2024-10-17.

[Original paper](https://arxiv.org/abs/2410.13232) · [Reading-list entry](https://neural-os.com/reading-list/papers/2410.13232/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2410.13232/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2410.13232/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2410.13232/record.json)

**Contribution:** Introduces a transition-focused abstraction that reduces the burden of predicting long, repetitive web observations and supports world-model-augmented planning.

**Model output:** Natural-language descriptions of changes in the next web observation.

**Action interface:** Candidate web-navigation actions.

**State and memory:** The planner uses web context and predicted descriptions of state changes.

**Relationship to NeuralOS:** Connects environment prediction to agent decisions in web navigation.

**Scope and limitations:** The paper evaluates web navigation using textual descriptions of state changes.

[Relationship source](https://arxiv.org/abs/2410.13232)

Related terms: web world model, counterfactual planning, transition prediction, WMA.

## Approaches

- [GUI prediction and planning](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning): Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# ViMo: A Generative Visual GUI World Model for App Agents

Authors: Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye Hao, Jun Wang, Kun Shao.

Predicts future mobile-app screens to help agents compare possible actions, separating graphic content from textual content.

First public: 2025-04-15.
ArXiv initial submission: 2025-04-15.

[Original paper](https://arxiv.org/abs/2504.13936) · [Reading-list entry](https://neural-os.com/reading-list/papers/2504.13936/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2504.13936/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2504.13936/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2504.13936/record.json)

**Contribution:** Uses symbolic text placeholders to preserve layout while assigning graphical and text prediction to separate components.

**Model output:** Future mobile GUI images assembled from graphics and text predictions.

**Action interface:** Candidate mobile-app actions.

**State and memory:** Screen-and-action-conditioned prediction; symbolic text representation retains text locations and layout.

**Relationship to NeuralOS:** Visual GUI world-model research released in April 2025, before NeuralOS's July 2025 preprint.

**Scope and limitations:** The experiments predict mobile-app screens for agent planning.

[Relationship source](https://arxiv.org/abs/2504.13936)

Related terms: ViMo, GUI world model, next-screen prediction, mobile app simulation.

## Approaches

- [GUI prediction and planning](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning): Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

Authors: Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng.

Studies neural operating systems through learned, interactive GUI simulation: a model produces desktop screens in response to mouse and keyboard events, opening a path toward generative computer interfaces.

First public: 2025-07-11.
ArXiv initial submission: 2025-07-11.

[Original paper](https://arxiv.org/abs/2507.08800) · [Reading-list entry](https://neural-os.com/reading-list/papers/2507.08800/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2507.08800/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2507.08800/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2507.08800/record.json)

**Contribution:** Formalizes OS-interface generation and develops state tracking, precise cursor conditioning, staged training, and data collection. The 2026 revision also demonstrates learning a new simulated application from synthesized training data.

**Model output:** Autoregressively generated desktop screen frames.

**Action interface:** Cursor coordinates, left/right clicks, and keyboard press/release events.

**State and memory:** A hierarchical recurrent model carries state across frames and conditions a diffusion renderer.

**Relationship to NeuralOS:** Introduces NeuralOS and its formulation of learned OS-interface simulation.

**Scope and limitations:** Fine-grained typing and long-horizon state retention remain difficult. The simulated interface does not provide the underlying filesystem, processes, or application services of a conventional OS.

[Relationship source](https://arxiv.org/html/2507.08800v1)

Dated versions: [Original version](https://arxiv.org/html/2507.08800v1) · [Revised version](https://arxiv.org/html/2507.08800v2)
[Code](https://github.com/yuntian-group/neural-os)

Related terms: NeuralOS, neural operating system, learned computer simulator, generative interface, GUI world model, desktop world model, action-conditioned screen generation.

## Method comparisons

**A game world model evaluated on desktop interactions**

DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction.

Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1)

Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/)

**Direct frame generation and renderable-code prediction**

NeuralOS predicts desktop frames using learned state and a diffusion renderer. Code2World predicts a UI through code that is then rendered into a screen, and tests its usefulness for mobile navigation. Code2World cites NeuralOS in Section 2.2.

Sources: [Original NeuralOS paper](https://arxiv.org/html/2507.08800v1) · [Code2World Section 2.2](https://arxiv.org/html/2602.09856v1#S2.SS2)

Paper details: [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) · [Code2World: A GUI World Model via Renderable Code Generation](https://neural-os.com/reading-list/papers/2602.09856/)


Date note: Initial arXiv submission on July 11, 2025; revised on March 12, 2026 and published at ICLR 2026. [Source](https://arxiv.org/abs/2507.08800).

## Published version

NeuralOS: Towards Simulating Operating Systems via Neural Generative Models. Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng. International Conference on Learning Representations, 2026.
[Publication](https://proceedings.iclr.cc/paper_files/paper/2026/hash/2174a6749c3f769358cd21ade1221b34-Abstract-Conference.html) · [BibTeX](https://neural-os.com/reading-list/papers/2507.08800/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2507.08800/publication.csl.json)

## Approaches

- [Neural operating systems](https://neural-os.com/reading-list/guides/neural-operating-systems/): Study how learned generative models can supply computer interaction, interface behavior, and adaptable user experiences. The research direction includes conceptual proposals and learned interface simulators.
- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.
- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.


# Code2World: A GUI World Model via Renderable Code Generation

Authors: Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin.

Predicts a GUI's next visual state by generating renderable code, and evaluates whether the predicted outcomes help mobile-agent navigation.

First public: 2026-02-10.
ArXiv initial submission: 2026-02-10.

[Original paper](https://arxiv.org/abs/2602.09856) · [Reading-list entry](https://neural-os.com/reading-list/papers/2602.09856/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2602.09856/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2602.09856/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2602.09856/record.json)

**Contribution:** Constructs paired screen/action/code data and trains with rendered visual feedback to balance appearance and action consistency.

**Model output:** Renderable UI code, converted into the predicted next screen.

**Action interface:** A current GUI observation and a proposed interaction, represented for the model with action descriptions and visual cues.

**State and memory:** Prediction conditions on the current screen and action; renderable structure represents the output interface.

**Relationship to NeuralOS:** Code2World cites NeuralOS in Section 2.2 and its bibliography.

**Scope and limitations:** Uses generated code and a browser to predict screens for Android navigation.

[Relationship source](https://arxiv.org/html/2602.09856v1#S2.SS2)

Related terms: Code2World, GUI world model, renderable code, next-screen prediction, AndroidCode.

## Method comparisons

**Direct frame generation and renderable-code prediction**

NeuralOS predicts desktop frames using learned state and a diffusion renderer. Code2World predicts a UI through code that is then rendered into a screen, and tests its usefulness for mobile navigation. Code2World cites NeuralOS in Section 2.2.

Sources: [Original NeuralOS paper](https://arxiv.org/html/2507.08800v1) · [Code2World Section 2.2](https://arxiv.org/html/2602.09856v1#S2.SS2)

Paper details: [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) · [Code2World: A GUI World Model via Renderable Code Generation](https://neural-os.com/reading-list/papers/2602.09856/)


## Approaches

- [GUI prediction and planning](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning): Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code.

Reading-list version 1.0.1; literature checked through 2026-10-19.