Genie 4: Google DeepMind's World Model Adds Audio, Dialogue and an API

Google DeepMind’s Genie 4 world model generates video and audio together in real time, lets users talk to characters inside a scene, and is coming to the API for the first time. Raia Hadsell, Diego Rivas and Hang Qi of Google DeepMind demoed it live with host Logan Kilpatrick on Google for Developers’ Release Notes, covering what world models are, the path from Genie 1 to Genie 4, Gemini running in the loop, robotics, real-time safety and why evaluating world models is still unsolved.

TLDW

Genie 4 follows Genie 3, which the team says was released in January, and adds three things: audio generated by the same model as the video, much greater photorealism from building on Gemini’s Omni video foundation model, and freeform controls that go beyond navigation to picking things up, serving coffee and talking to people in the scene. A Gemini model, Gemini 3.8 Flash in the live demos, watches each frame, works out what the user intended and steers what happens next. The team sees robotics and education as the main applications, with a partner demo using a model called Argon to plan robot actions that Genie then renders. Genie 4 will be available in Project Genie and the API, with SynthID watermarks and a gradual rollout starting with trusted testers.

Thoughts

The most consequential claim in the episode is that a world model could control a robot directly, not just feed it training data. Hadsell says the original plan was to prompt Genie to look like a robotics lab and generate synthetic data, which already solves a real problem. Kilpatrick describes his visit to the robotics lab as a slow film set where every task takes setup and retakes. But she goes further: if Genie produces the robot’s actions, the same model that knows what happens when you drop a heavy cup versus a light one is doing the planning. That is closer to how she says humans move, constantly running a short prediction forward to stay fluid. The demo that followed should be read carefully, though. The Scrabble example was recorded by robotics partners, did not run in real time, and was rendered entirely inside Genie, with Argon writing instructions and Genie generating actions and pixels. It shows a simulated robot doing a task, not a physical one. Hadsell’s own caveat, “research is always research,” is the honest summary.

Hadsell asks the question that should worry anyone building world models: do we still need Genie if coding models can already generate an interactive 3D scene or game? Her answer is that the two will merge, and the split she draws is useful. Code gives you state that persists. Walk over the hill, come back, and the house is exactly where you left it, which Genie 3 struggled with once its context ran out. Pixels give you expressiveness, the dragon flying through, everything turning to watercolor, things no game engine was programmed to do. Code could also make worlds cheaper to run, longer-lasting and able to host several players at once. If that hybrid works, the world model stops being the whole product and becomes the imaginative layer on top of a structured one. That is a more defensible role than trying to out-render a game engine, and it suggests the most interesting Genie products will be built by developers who combine it with code, which is exactly why the API matters.

The exchange about frame rate reveals that latency, not raw compute, is the hard limit. Kilpatrick asks whether Genie 4 could hit 60 frames per second if every TPU were dedicated to him. Qi says they have never tried, and the team says they are happy at 24 to 30. The more telling point is why 24. Qi explains that it is the rate at which the model can stop, take in a user’s keyboard input and generate the next frames without the stream turning laggy or choppy. Smoother playback could be bought with more compute, but the loop between action and response cannot simply be scaled away, and Qi says getting the model to run in real time at all required trading off configurations. Hadsell frames the broader challenge the same way: keep adding capabilities while training a model small and efficient enough to stay reactive. For anyone hoping world models replace game engines soon, this is the constraint to watch.

Safety in a real-time photorealistic simulator is a harder problem than the team’s relaxed tone suggests. The team calls it uncharted territory: what happens when photorealism meets real time and anyone can simulate anything. Kilpatrick raises the real tension. Training first responders or recreating a disaster scenario needs exactly the content a general model would block. Hadsell’s answer is context: a filmmaker or an educator building a realistic battlefield has valid reasons a casual user does not, and she says questions that felt like stumbling blocks a couple of years ago now have answers based on who is using a model and how. That works inside a curated product. It is much harder to verify through an open API. The more novel idea is Qi’s: with Gemini already watching every frame and driving the scene’s logic, an unsafe scene can be steered away rather than simply cut off. Starting with trusted testers is the right call while that is still being worked out.

The closing admission that world model evaluation is unsolved deserves more attention than it got. Kilpatrick notes that a new benchmark for language models now takes 500 or 5,000 hours to build instead of 50, and asks whether world models are easier or harder to measure. The team’s answer is that the evals are never ready and keep changing as the model reveals new problems. The plan is to use DeepMind’s game-playing agents, the SIMA project, to explore Genie worlds automatically and evaluate them. That is clever, and it echoes how DeepMind started this work, building worlds to train agents. But it also means one AI system would judge another’s realism, so what counts as a good world gets defined by what an agent notices. A busy restaurant kitchen where the chefs look like twins and the sprinklers spray into pots, both moments from the live demo, is exactly the kind of flaw a human spots instantly and an automated eval may not.

Key Takeaways

Hadsell defines a world model as one that keeps an internal state of a grounded environment and predicts how actions and interactions change it, even in fantastical worlds.
Genie 1 generated 2D platformer worlds, Genie 2 moved to 3D a year later but was not real time, and Genie 3, which the team says was released in January, was the first real-time interactive version.
Genie 4 is built on Gemini’s Omni video foundation model, which drives a large jump in photorealism, especially in how people and hands look and move.
Audio is a new modality, and Genie 4 predicts video and audio together from the same model, including music, ambient sound and dialogue.
The release focuses on first-person views controlled with keyboard and mouse, plus camera controls, rather than Genie 3’s third-person characters.
Freeform controls let users act on the world, such as picking up or serving a coffee, with characters reacting and speaking in real time.
Gemini runs in the loop beside the diffusion model, reading each frame and the user’s intent to decide what should happen next. The live demos used Gemini 3.8 Flash for speed.
A partner robotics demo used a model called Argon to write instructions while Genie 4 generated a robot’s actions and pixels to spell words with Scrabble tiles. It did not run in real time.
The team sees filmmaking value mainly in pre-visualization and storyboarding, where fast, rough, interactive output is useful.
Genie 4 runs at roughly 24 to 30 frames per second, and 24 is tied to how often the model can take in user input without the stream turning choppy. The team has never tried 60.
Hadsell expects code and generated pixels to combine, with code holding persistent state and the world model supplying expressiveness.
Genie 4 will be available in Project Genie and, for the first time, in the API, with SynthID watermarks on outputs.
Rollout starts with trusted testers, and Gemini’s monitoring role means unsafe scenes can be steered away rather than cut off.
The team says world model evals are never ready, and plans to use DeepMind’s SIMA game-playing agents to explore and evaluate Genie worlds.

Chapters

1:02 What World Models Are and the Road From Genie 1

Hadsell contrasts Gemini, a thinking model that deals with knowledge and language, with a world model that understands a grounded environment and how interactions change it. DeepMind began with game environments because richer simulations train better agents, and Genie 1 was a high-risk project to generate endless new environments for training AGI. It produced 2D platformer worlds, Genie 2 moved to 3D without real-time interaction, and Genie 3 brought real-time, interactive realism. The team describes Genie 4 as a step further into usefulness while still very much a research model that makes mistakes.

5:10 Audio, Photorealism and Real-Time Prompting

Rivas shows a silent, game-like Genie 3 scene as a baseline, then the same scene in Genie 4, where a prompt brings a band to life and unmuting reveals music. He compares the arrival of audio to the move from Veo 2 to Veo 3, and points to the detail in characters and hand motion that comes from the Omni foundation. Qi explains that real-world video is always paired with synced audio, so modeling both together is modeling the same world. The demo interface lets the team pause the stream to type text captions that steer the model in real time, which may or may not ship to users.

11:03 First-Person Views and Talking to Characters

Genie 4 shifts focus from third-person characters to first-person views and camera controls. In a coffee shop demo, egocentric hands make coffee, the scene stays consistent while walking, and a character responds out loud when the user drops a cup. Qi says the emphasis has moved from navigating a scene to interacting with it, with people reacting to actions and holding dialogue. The team admits dialogue is still an active and imperfect research area.

15:10 Gemini in the Loop

The dialogue is driven by a Gemini model in the background, working hand in hand with Genie’s diffusion model to judge what happened in the last frame, what the user intended and what should come next. The demos use Gemini 3.8 Flash to keep things real time, while the team has also experimented with Argon for robotics. Qi sees Gemini’s knowledge and reasoning as the route to instructional and educational uses, and first-person footage of tasks as a bridge to robotics.

17:10 Turning the Set Into a Fantasy World

Rivas turns a photo of the studio set into a fantastical scene, adds a dragon and has it breathe fire while a crew voice comments on framing and lighting. He says filmmakers have found the most value in pre-visualization and storyboarding, where rough, rapid output helps a coarse idea become something they can see and iterate on. Being able to pause, move the camera and add elements makes even interim output useful.

19:01 Genie 4 and Robotics

Hadsell says humans carry a world model that predicts slightly forward and lets them move fluidly, and robots with one could be more dexterous and agile. The original idea was generating training data, but using Genie to produce the robot’s actions directly could unlock generalizable movement and skills. Rivas shows a partner demo in which Argon writes instructions and Genie 4 renders a robot arranging Scrabble tiles. Kilpatrick notes how slow physical robot data collection is, and Rivas imagines Argon and Genie running endless variations of spatial tasks.

24:11 Where Genie 4 Shines

Rivas names three gains. People in Genie 3 scenes were often static or behaved oddly, and Genie 4’s characters are far more realistic. Freeform controls move the model beyond navigation so users can do almost anything in the world, which he calls the shift from exploration to simulation. And audio, as with Veo 3, is the missing piece people did not know they needed.

25:55 Scaling, Frame Rate and Code Meets Pixels

Hadsell says early worlds blurred out over time, and the frontier now is keeping long scenes consistent while keeping the model small and fast. She asks whether Genie is still needed when coding models can build interactive games, and argues code and generated pixels will combine, with code holding persistent state and the world model supplying expressiveness. Qi explains that Genie 4 targets 24 to 30 frames per second because that rate matches how often user input can enter the generation without lag, and that 60 has never been tried. The team describes open-ended, adaptive experiences as a product direction to explore through the API.

34:42 API Access, SynthID and Real-Time Safety

Genie 4 will be available in Project Genie and the API, and all outputs carry SynthID. The team describes building guardrails that filter edge cases in real time without breaking latency. Hadsell says the approach mirrors Omni, where filmmakers and educators can have valid reasons for content a general user would not, and a gradual rollout through trusted testers is meant to surface edge cases. Qi adds that Gemini’s role watching the scene means it can steer away from unsafe content rather than terminate it, and Hadsell says Google will tailor deployments to different users such as directors who need precise camera control.

41:36 Go-To Prompts and Evaluating World Models

Rivas’s favorite test prompt is a busy photorealistic restaurant kitchen, which shows many people acting in sequence. In the demo the chef explains he is pan-searing salmon with herbs and garlic, the cooks look suspiciously like twins, and a sprinkler system sprays into the pots. Kilpatrick prefers non-photorealistic prompts like bumper cars shaped as animals or vegetables. The team says evals are never ready, and plans to use DeepMind’s SIMA agents to explore and evaluate Genie worlds automatically.

Notable Quotes

“It’s the obvious thing that you didn’t know you were missing.”
Diego Rivas, on adding audio to Genie 4

“And so it’s the shift from exploration to simulation almost.”
Diego Rivas, on freeform controls

“Research is always research. You never really know until you’ve really done it.”
Raia Hadsell, on using Genie to control robots

“So I guess the question is, do we still need Genie if we can just generate a world with code and have it be interactive?”
Raia Hadsell, on coding models that build games

“The power of a world model is that it’s so expressive.”
Raia Hadsell, on why code alone is not enough

“Sixty FPS, we never tried. I actually do not know.”
Hang Qi, on whether Genie 4 could run at 60 frames per second

“These questions felt like real stumbling blocks a couple of years ago and now it feels like oh, we’ve got some answers here.”
Raia Hadsell, on safety for real-time world models

Watch the full conversation here.

Genie 4: Google DeepMind’s World Model Adds Audio, Dialogue and an API