From neural game engines to GUI world models
An interactive visual model repeatedly answers a practical question: given the available history and the next action, what should the user see? Its own predictions become part of the next interaction. That feedback loop makes control, memory, and recovery from prediction errors central to the problem.
The interactive-simulation background
Oh and colleagues (2015) studied action-conditioned Atari video prediction, including recurrent models and longer visual rollouts. Ha and Schmidhuber (2018) connected learned visual representations and recurrent dynamics to policy learning inside generated environments. These works offer two complementary starting points: predicting observable consequences and learning useful behavior through a model. Action-conditional prediction · World Models.
GameGAN (2020) maintains a consistent environment when returning to previously visited locations through memory and separate representations of static and dynamic content. Genie (2024) learns latent actions from videos without recorded action labels. GameGAN · Genie.
DIAMOND and GameNGen, both first released in 2024, use diffusion models for interactive visual simulation. DIAMOND also trains reinforcement-learning agents in its generated environment. GameNGen focuses on sustained DOOM simulation. DIAMOND · GameNGen.
What changes on a desktop?
Desktop tasks can hinge on a small local change: a click activates a particular control, an application opens after a delay, or a character changes a command. An image can look nearly identical while the interaction meaning changes substantially. Useful comparisons should therefore identify the action representation and the state needed for the next response, rather than comparing image quality alone.
NeuralOS uses low-level inputs, spatial cursor conditioning, and recurrent state tracking. Its revised paper adapts DIAMOND to OS screen prediction and compares the two models on desktop interactions. NeuralOS baseline comparison.
GUI prediction and planning
GUI models also differ in what they predict. That choice changes how their predictions can be used:
- Textual changes: Web Agents with World Models predicts descriptions of important differences between observations, helping a planner compare web actions. Paper.
- Screen images with explicit text handling: ViMo separates graphics and text generation for mobile-app observations. Its April 2025 preprint predates NeuralOS’s July 2025 release. Paper.
- Renderable code: Code2World generates interface code and renders the predicted screen. It evaluates navigation benefits and cites NeuralOS in its related work. Paper.
- Continuing desktop-frame generation: NeuralOS maintains learned state while responding to successive user events. Original paper.
These representations serve different purposes. A planner uses predicted action outcomes to choose an action; an interactive simulator produces the next observation a user or agent will encounter.
For any pair of papers, compare the output, action interface, state representation, and evaluation setting. Those details appear in every paper record. Continue to evaluation for ways to measure interaction, memory, and task completion.