GUI world models: from video prediction to computer use

A learned environment answers a practical question: given the interaction history and the next action, what happens next? For a computer interface, the answer involves visible screens, precise input effects, and state that persists when a window is hidden. This guide connects that problem to video prediction, interactive simulation, and agent planning.

GUI world models · Generative GUI simulation · World models for computer use · Action-conditioned video prediction

GUI world models

A GUI world model predicts the consequences of actions in a graphical user interface. Its input includes the current observation or interaction history and an action; its output describes or renders the resulting interface. The prediction target determines what a user or agent receives:

These are different ways to model interface dynamics. Compare the output representation, action interface, remembered state, and evaluation setting before comparing image quality or task scores.

Generative GUI simulation

Generative GUI simulation produces the interface throughout an interaction. A generated screen becomes part of the next step’s history, so a useful simulator must sustain the effects of earlier actions. Opening a window, moving it, switching away, and returning tests more than the appearance of one predicted frame: it tests the relationship between input, visible response, and retained state.

NeuralOS, first released on July 11, 2025 by Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, and Yuntian Deng, studies this interaction loop as part of the broader neural-operating-system direction. It formalizes OS-interface generation and develops state tracking, spatial cursor conditioning, staged training, and data collection. Original formulation and method.

Generative UI also includes systems that create interactive applications through generated code. NeuralOS learns interface dynamics and renders successive screens directly. Both connect to the broader question of generating interfaces on the fly; their representations determine how actions, memory, and execution work. A click must affect the intended control, a keystroke must reach the active window, and earlier state must survive changes of view. Read the generative-interface research guide for the original NeuralOS vision and its connection to code-generated interfaces.

NeuralOS’s March 2026 revision also demonstrates learning a new simulated application from synthesized training data, connecting learned simulation to adaptable, generative interfaces. Revised paper. Read the research-direction guide for the wider connection to neural operating systems.

World models for computer use

World models for computer use predict action outcomes that an agent uses to choose what to do. The agent proposes actions and selects one; the world model predicts their consequences. For example, a planner compares the predicted results of two clicks before committing to one in the real application.

Web Agents with World Models studies this through textual transition predictions. ViMo predicts mobile screens for planning, while Code2World uses renderable code to produce predicted screens and evaluates downstream navigation. These papers connect prediction quality to the agent’s decisions. Web planning · ViMo · Code2World.

NeuralOS studies the learned environment: the model responds to a user’s or agent’s inputs with generated desktop observations.

For that evaluation, OSWorld supplies real desktop applications, task initialization, and execution-based success checks. It provides a way to assess actual computer-use outcomes alongside learned-simulator measurements. OSWorld paper. The evaluation guide separates visual quality, action fidelity, state retention, and completed tasks.

Action-conditioned video prediction

Action-conditioned video prediction models future frames from earlier observations and control inputs. The action is part of the prediction problem: the same starting screen followed by a click, a keypress, or no input calls for different continuations. NeuralOS applies this formulation to desktop screens, using cursor coordinates, mouse buttons, and keyboard events as controls. NeuralOS formulation.

Oh and colleagues (2015) studied action-conditional Atari video prediction, including recurrent models and longer visual rollouts. Ha and Schmidhuber (2018) connected learned visual representations and recurrent dynamics to policy learning inside generated environments. These works offer two complementary starting points: predicting observable consequences and learning behavior through a model. Action-conditional prediction · World Models.

GameGAN (2020) uses memory and separate representations of static and dynamic content to maintain consistency when returning to locations. Genie (2024) learns latent actions from videos without recorded action labels. Its learned controls differ from NeuralOS’s explicit mouse and keyboard events. GameGAN · Genie.

DIAMOND and GameNGen, both first released in 2024, use diffusion models for interactive visual simulation. DIAMOND also trains reinforcement-learning agents in its generated environment; GameNGen focuses on sustained DOOM simulation. NeuralOS’s revised paper adapts DIAMOND to OS screen prediction and compares the models on desktop interactions. DIAMOND · GameNGen · NeuralOS desktop comparison.

On a desktop, a small visual change has a specific operational meaning: a click activates a particular control, an application opens after a delay, or a character changes a command. This makes precise action conditioning and remembered interface state central to extending visual world models from games to computers.

Return to the reading-list overview for a short reading path, or browse the paper records for authors, dates, sources, and citation downloads.