# Interactive learned environments

Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Papers are ordered by first public date.

For example: A world model receives a movement action, predicts the next frame, and uses that generated frame when responding to the following action.

Version 1.0.1; literature checked through 2026-10-19.

[Topic guide](https://neural-os.com/reading-list/guides/games-to-computers/) · [JSON](https://neural-os.com/reading-list/data/topics/interactive-worlds.json)

## How the methods connect and differ

**A game world model evaluated on desktop interactions**

DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction.

Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1)

Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/)

## Papers — oldest first

## Action-Conditional Video Prediction using Deep Networks in Atari Games

Authors: Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, Satinder Singh.

Learns to predict Atari frames from past observations and control inputs, connecting visual generation to an agent's actions.

First public: 2015-07-31.
ArXiv initial submission: 2015-07-31.

[Original paper](https://arxiv.org/abs/1507.08750) · [Reading-list entry](https://neural-os.com/reading-list/papers/1507.08750/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/1507.08750/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/1507.08750/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/1507.08750/record.json)

**Contribution:** Develops feedforward and recurrent architectures with action-dependent transformations and evaluates long visual rollouts and their usefulness for control.

**Model output:** Predicted game frames.

**Action interface:** Discrete Atari control actions.

**State and memory:** Past observations; the recurrent variant carries a hidden state through time.

**Relationship to NeuralOS:** Earlier action-conditioned visual-prediction work cited in NeuralOS's introduction and references.

**Scope and limitations:** The experiments use Atari games.

[Relationship source](https://arxiv.org/html/2507.08800v1)

Related terms: action-conditioned video prediction, visual dynamics, Atari, learned simulator.

### Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


## World Models

Authors: David Ha, Jürgen Schmidhuber.

Combines learned visual compression and recurrent dynamics with a controller, including training an agent inside a generated environment.

First public: 2018-03-27.
ArXiv initial submission: 2018-03-27.

[Original paper](https://arxiv.org/abs/1803.10122) · [Reading-list entry](https://neural-os.com/reading-list/papers/1803.10122/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/1803.10122/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/1803.10122/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/1803.10122/record.json)

**Contribution:** Demonstrates how a compact learned environment representation can support policy learning and transfer from a simulated dream to the original task.

**Model output:** Predicted latent observations, with an image decoder and task-dependent termination predictions.

**Action interface:** Actions from the controller in the studied reinforcement-learning environments.

**State and memory:** A recurrent model summarizes the history of latent observations and actions.

**Relationship to NeuralOS:** Cited by NeuralOS as background for learned simulation and agent training.

**Scope and limitations:** The model compresses observations into a latent representation for control.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: world model, recurrent dynamics, latent imagination, model-based reinforcement learning.

### Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


## Learning to Simulate Dynamic Environments with GameGAN

Authors: Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, Sanja Fidler.

Learns an interactive visual game simulator from gameplay and keyboard actions using a generative adversarial network.

First public: 2020-05-25.
ArXiv initial submission: 2020-05-25.

[Original paper](https://arxiv.org/abs/2005.12126) · [Reading-list entry](https://neural-os.com/reading-list/papers/2005.12126/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2005.12126/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2005.12126/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2005.12126/record.json)

**Contribution:** Introduces a memory module for revisited locations and separates static and dynamic image components, enabling visual consistency and component recombination.

**Model output:** The next game-screen image.

**Action interface:** Game controls represented by keyboard actions.

**State and memory:** Recurrent dynamics plus an external memory module that stores an internal environmental map.

**Relationship to NeuralOS:** An earlier interactive neural simulator discussed in NeuralOS's related work.

**Scope and limitations:** The simulator learns from recorded games, with memory organized as a spatial map.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: GameGAN, neural game engine, interactive simulation, visual memory.

### Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


## Genie: Generative Interactive Environments

Authors: Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, Tim Rocktäschel.

Learns controllable visual environments from videos without ground-truth action labels, using a learned latent action space.

First public: 2024-02-23.
ArXiv initial submission: 2024-02-23.

[Original paper](https://arxiv.org/abs/2402.15391) · [Reading-list entry](https://neural-os.com/reading-list/papers/2402.15391/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2402.15391/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2402.15391/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2402.15391/record.json)

**Contribution:** Combines video tokenization, a dynamics model, and latent-action learning so users can interact with worlds initialized from images.

**Model output:** Action-conditioned future video frames.

**Action interface:** Discrete latent actions inferred during training rather than recorded physical input events.

**State and memory:** A spatiotemporal dynamics model conditions predictions on previous visual tokens and actions.

**Relationship to NeuralOS:** Cited in NeuralOS's related work. Genie learns its action space from video; NeuralOS takes mouse and keyboard events as inputs.

**Scope and limitations:** Controls use learned latent actions inferred from video.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: Genie, generative interactive environments, latent actions, foundation world model.

### Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


## Diffusion for World Modeling: Visual Details Matter in Atari

Authors: Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, François Fleuret.

Introduces DIAMOND, an agent trained in a diffusion world model, and studies how preserving visual detail affects control.

First public: 2024-05-20.
ArXiv initial submission: 2024-05-20.

[Original paper](https://arxiv.org/abs/2405.12399) · [Reading-list entry](https://neural-os.com/reading-list/papers/2405.12399/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2405.12399/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2405.12399/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2405.12399/record.json)

**Contribution:** Connects diffusion-based observation prediction with Atari policy learning and demonstrates an interactive Counter-Strike simulator.

**Model output:** Predicted visual observations; auxiliary reward and termination predictions support reinforcement learning.

**Action interface:** Actions of the modeled game environment.

**State and memory:** Visual dynamics condition on a finite history of observations and actions.

**Relationship to NeuralOS:** NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision.

**Scope and limitations:** Evaluated on games; NeuralOS adapts and evaluates it for desktop screen prediction.

[Relationship source](https://arxiv.org/html/2507.08800v2#A1)

Related terms: DIAMOND, diffusion world model, neural game engine, model-based reinforcement learning.

### Method comparisons

**A game world model evaluated on desktop interactions**

DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction.

Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1)

Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/)


### Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.
- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.


## Diffusion Models Are Real-Time Game Engines

Authors: Dani Valevski, Yaniv Leviathan, Moab Arar, Shlomi Fruchter.

Presents GameNGen, a diffusion model that interactively simulates DOOM from previous frames and player actions.

First public: 2024-08-27.
ArXiv initial submission: 2024-08-27.

[Original paper](https://arxiv.org/abs/2408.14837) · [Reading-list entry](https://neural-os.com/reading-list/papers/2408.14837/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2408.14837/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2408.14837/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2408.14837/record.json)

**Contribution:** Uses agent-collected gameplay, conditioning augmentation, and decoder fine-tuning to sustain autoregressive visual interaction.

**Model output:** The next rendered game frame.

**Action interface:** DOOM player actions.

**State and memory:** Conditioning on a sequence of recent frames and actions; continued play feeds generated frames back into the model.

**Relationship to NeuralOS:** Interactive-simulation background cited by NeuralOS; NeuralOS also refers to its perceptual-evaluation setup.

**Scope and limitations:** The simulator is trained and evaluated on DOOM.

[Relationship source](https://arxiv.org/html/2507.08800v2)

Related terms: GameNGen, neural game engine, real-time simulation, diffusion renderer.

### Approaches

- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.

Reading-list version 1.0.1; literature checked through 2026-10-19.


## NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

Authors: Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng.

Studies neural operating systems through learned, interactive GUI simulation: a model produces desktop screens in response to mouse and keyboard events, opening a path toward generative computer interfaces.

First public: 2025-07-11.
ArXiv initial submission: 2025-07-11.

[Original paper](https://arxiv.org/abs/2507.08800) · [Reading-list entry](https://neural-os.com/reading-list/papers/2507.08800/)
[Preprint BibTeX](https://neural-os.com/reading-list/papers/2507.08800/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2507.08800/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2507.08800/record.json)

**Contribution:** Formalizes OS-interface generation and develops state tracking, precise cursor conditioning, staged training, and data collection. The 2026 revision also demonstrates learning a new simulated application from synthesized training data.

**Model output:** Autoregressively generated desktop screen frames.

**Action interface:** Cursor coordinates, left/right clicks, and keyboard press/release events.

**State and memory:** A hierarchical recurrent model carries state across frames and conditions a diffusion renderer.

**Relationship to NeuralOS:** Introduces NeuralOS and its formulation of learned OS-interface simulation.

**Scope and limitations:** Fine-grained typing and long-horizon state retention remain difficult. The simulated interface does not provide the underlying filesystem, processes, or application services of a conventional OS.

[Relationship source](https://arxiv.org/html/2507.08800v1)

Dated versions: [Original version](https://arxiv.org/html/2507.08800v1) · [Revised version](https://arxiv.org/html/2507.08800v2)
[Code](https://github.com/yuntian-group/neural-os)

Related terms: NeuralOS, neural operating system, learned computer simulator, generative interface, GUI world model, desktop world model, action-conditioned screen generation.

### Method comparisons

**A game world model evaluated on desktop interactions**

DIAMOND learns visual game dynamics with a diffusion model. NeuralOS adapts DIAMOND to OS screen prediction as a baseline in Appendix A of its 2026 revision. NeuralOS adds a recurrent state representation and spatial cursor conditioning for desktop interaction.

Sources: [DIAMOND paper](https://arxiv.org/abs/2405.12399) · [NeuralOS Appendix A](https://arxiv.org/html/2507.08800v2#A1)

Paper details: [Diffusion for World Modeling: Visual Details Matter in Atari](https://neural-os.com/reading-list/papers/2405.12399/) · [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/)

**Direct frame generation and renderable-code prediction**

NeuralOS predicts desktop frames using learned state and a diffusion renderer. Code2World predicts a UI through code that is then rendered into a screen, and tests its usefulness for mobile navigation. Code2World cites NeuralOS in Section 2.2.

Sources: [Original NeuralOS paper](https://arxiv.org/html/2507.08800v1) · [Code2World Section 2.2](https://arxiv.org/html/2602.09856v1#S2.SS2)

Paper details: [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) · [Code2World: A GUI World Model via Renderable Code Generation](https://neural-os.com/reading-list/papers/2602.09856/)


Date note: Initial arXiv submission on July 11, 2025; revised on March 12, 2026 and published at ICLR 2026. [Source](https://arxiv.org/abs/2507.08800).

### Published version

NeuralOS: Towards Simulating Operating Systems via Neural Generative Models. Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng. International Conference on Learning Representations, 2026.
[Publication](https://proceedings.iclr.cc/paper_files/paper/2026/hash/2174a6749c3f769358cd21ade1221b34-Abstract-Conference.html) · [BibTeX](https://neural-os.com/reading-list/papers/2507.08800/publication.bib) · [CSL-JSON](https://neural-os.com/reading-list/papers/2507.08800/publication.csl.json)

### Approaches

- [Neural operating systems](https://neural-os.com/reading-list/guides/neural-operating-systems/): Study how learned generative models can supply computer interaction, interface behavior, and adaptable user experiences. The research direction includes conceptual proposals and learned interface simulators.
- [Interactive learned environments](https://neural-os.com/reading-list/guides/games-to-computers/): Learn how an environment changes after actions, then generate observations during continued interaction. Compare visual fidelity, control interfaces, memory, and the reliability of repeated predictions.
- [Environments and evaluation](https://neural-os.com/reading-list/guides/evaluation/): Test action response, visual quality, remembered state, and completed tasks separately. Real desktop and mobile environments provide execution-based checks that complement learned-simulator measurements.

Reading-list version 1.0.1; literature checked through 2026-10-19.