# GUI prediction and planning Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code. Papers are ordered by first public date. For example: An agent compares predicted results of two clicks before choosing which action to execute in the real application. Version 1.0.1; literature checked through 2026-10-19. [Topic guide](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning) · [JSON](https://neural-os.com/reading-list/data/topics/gui-planning.json) ## How the methods connect and differ **Direct frame generation and renderable-code prediction** NeuralOS predicts desktop frames using learned state and a diffusion renderer. Code2World predicts a UI through code that is then rendered into a screen, and tests its usefulness for mobile navigation. Code2World cites NeuralOS in Section 2.2. Sources: [Original NeuralOS paper](https://arxiv.org/html/2507.08800v1) · [Code2World Section 2.2](https://arxiv.org/html/2602.09856v1#S2.SS2) Paper details: [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) · [Code2World: A GUI World Model via Renderable Code Generation](https://neural-os.com/reading-list/papers/2602.09856/) ## Papers — oldest first ## Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation Authors: Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, Jinyoung Yeo. Predicts the effects of candidate web actions as natural-language descriptions of important changes, then uses those predictions for action selection. First public: 2024-10-17. ArXiv initial submission: 2024-10-17. [Original paper](https://arxiv.org/abs/2410.13232) · [Reading-list entry](https://neural-os.com/reading-list/papers/2410.13232/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2410.13232/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2410.13232/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2410.13232/record.json) **Contribution:** Introduces a transition-focused abstraction that reduces the burden of predicting long, repetitive web observations and supports world-model-augmented planning. **Model output:** Natural-language descriptions of changes in the next web observation. **Action interface:** Candidate web-navigation actions. **State and memory:** The planner uses web context and predicted descriptions of state changes. **Relationship to NeuralOS:** Connects environment prediction to agent decisions in web navigation. **Scope and limitations:** The paper evaluates web navigation using textual descriptions of state changes. [Relationship source](https://arxiv.org/abs/2410.13232) Related terms: web world model, counterfactual planning, transition prediction, WMA. ### Approaches - [GUI prediction and planning](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning): Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code. Reading-list version 1.0.1; literature checked through 2026-10-19. ## ViMo: A Generative Visual GUI World Model for App Agents Authors: Dezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis, Jifei Song, Shaogang Gong, Jianye Hao, Jun Wang, Kun Shao. Predicts future mobile-app screens to help agents compare possible actions, separating graphic content from textual content. First public: 2025-04-15. ArXiv initial submission: 2025-04-15. [Original paper](https://arxiv.org/abs/2504.13936) · [Reading-list entry](https://neural-os.com/reading-list/papers/2504.13936/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2504.13936/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2504.13936/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2504.13936/record.json) **Contribution:** Uses symbolic text placeholders to preserve layout while assigning graphical and text prediction to separate components. **Model output:** Future mobile GUI images assembled from graphics and text predictions. **Action interface:** Candidate mobile-app actions. **State and memory:** Screen-and-action-conditioned prediction; symbolic text representation retains text locations and layout. **Relationship to NeuralOS:** Visual GUI world-model research released in April 2025, before NeuralOS's July 2025 preprint. **Scope and limitations:** The experiments predict mobile-app screens for agent planning. [Relationship source](https://arxiv.org/abs/2504.13936) Related terms: ViMo, GUI world model, next-screen prediction, mobile app simulation. ### Approaches - [GUI prediction and planning](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning): Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code. Reading-list version 1.0.1; literature checked through 2026-10-19. ## Code2World: A GUI World Model via Renderable Code Generation Authors: Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin. Predicts a GUI's next visual state by generating renderable code, and evaluates whether the predicted outcomes help mobile-agent navigation. First public: 2026-02-10. ArXiv initial submission: 2026-02-10. [Original paper](https://arxiv.org/abs/2602.09856) · [Reading-list entry](https://neural-os.com/reading-list/papers/2602.09856/) [Preprint BibTeX](https://neural-os.com/reading-list/papers/2602.09856/citation.bib) · [Preprint CSL-JSON](https://neural-os.com/reading-list/papers/2602.09856/citation.csl.json) · [JSON record](https://neural-os.com/reading-list/papers/2602.09856/record.json) **Contribution:** Constructs paired screen/action/code data and trains with rendered visual feedback to balance appearance and action consistency. **Model output:** Renderable UI code, converted into the predicted next screen. **Action interface:** A current GUI observation and a proposed interaction, represented for the model with action descriptions and visual cues. **State and memory:** Prediction conditions on the current screen and action; renderable structure represents the output interface. **Relationship to NeuralOS:** Code2World cites NeuralOS in Section 2.2 and its bibliography. **Scope and limitations:** Uses generated code and a browser to predict screens for Android navigation. [Relationship source](https://arxiv.org/html/2602.09856v1#S2.SS2) Related terms: Code2World, GUI world model, renderable code, next-screen prediction, AndroidCode. ### Method comparisons **Direct frame generation and renderable-code prediction** NeuralOS predicts desktop frames using learned state and a diffusion renderer. Code2World predicts a UI through code that is then rendered into a screen, and tests its usefulness for mobile navigation. Code2World cites NeuralOS in Section 2.2. Sources: [Original NeuralOS paper](https://arxiv.org/html/2507.08800v1) · [Code2World Section 2.2](https://arxiv.org/html/2602.09856v1#S2.SS2) Paper details: [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://neural-os.com/reading-list/papers/2507.08800/) · [Code2World: A GUI World Model via Renderable Code Generation](https://neural-os.com/reading-list/papers/2602.09856/) ### Approaches - [GUI prediction and planning](https://neural-os.com/reading-list/guides/games-to-computers/#gui-prediction-and-planning): Predict the consequences of interface actions so an agent can compare alternatives before execution. Prediction targets include textual changes, screen images, and renderable interface code. Reading-list version 1.0.1; literature checked through 2026-10-19.