Paper notes

Genie: Generative Interactive Environments

Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, Tim Rocktäschel

First arXiv submission: February 23, 2024 · arXiv

Learns controllable visual environments from videos without ground-truth action labels, using a learned latent action space.

Read paper ↗

Contribution and setting

Combines video tokenization, a dynamics model, and latent-action learning so users can interact with worlds initialized from images.

Model output

Action-conditioned future video frames.

Action interface

Discrete latent actions inferred during training rather than recorded physical input events.

State and memory

A spatiotemporal dynamics model conditions predictions on previous visual tokens and actions.

Relationship to NeuralOS

Cited in NeuralOS's related work. Genie learns its action space from video; NeuralOS takes mouse and keyboard events as inputs.

Source ↗

Scope and limitations

Controls use learned latent actions inferred from video.

Related terms: Genie, generative interactive environments, latent actions, foundation world model.

Read or reuse this entry

JSON record · Markdown · Find in the full list