World models, video generators and simulators serve different jobs

A filmmaker exploring a camera move, an agent learning to navigate and a roboticist checking a grasp can all ask for a simulated world. They need different answers. The filmmaker may need a convincing sequence. The agent needs an environment that responds consistently to actions. The roboticist may need inspectable geometry, forces and timing. A visually impressive result does not establish all three.

DeepMind’s October 27 Genie 4 announcement describes an interactive world model with real-time steering and native audio. It is a research preview for trusted testers. DeepMind also identifies drift, spatial accuracy, complex physics and long-term consistency as continuing limitations. Its examples illustrate the company’s reported capabilities, rather than independent validation of physical accuracy or robot transfer.

The useful comparison is therefore a map of interfaces and evidence. This survey examines interactive generation alongside documented video-generation APIs, MuJoCo, CARLA, Isaac Sim and Gymnasium’s environment contract. It does not report new hands-on benchmarks. Its purpose is to identify which system exposes the controls and observations required to answer a particular question.

TL;DR Match the tool to the decision. Generated video can support visual ideation; an interactive world can support responsive exploration; a structured simulator can expose dynamics and measurements. Each still needs validation for the intended use.

Three jobs behind one attractive image

The categories overlap, but their interfaces emphasise different outputs. A video generator produces an audiovisual artifact. An interactive generative environment produces a continuing experience conditioned on user actions. A structured simulator evolves an explicit model and can render observations from it. A hybrid may combine all three, but the combination still needs an account of which component determines each result.

These are not rankings. A detailed dynamics model can be a poor fit for generating a rich cinematic setting quickly. A visually rich interactive scene can be a poor fit for estimating the force needed to move a specific mechanism. The right choice follows from the output the team intends to trust.

The word “simulation” is consequently insufficient as a specification. A team needs to say whether it is simulating appearance, interaction, mechanics, sensor readings, task logic or some combination. It also needs to name the horizon over which consistency matters. A few coherent seconds and a repeatable sequence of hundreds of control decisions are different requirements.

That framing makes demonstrations easier to assess. A demo can establish that a system produced a particular experience under selected conditions. It leaves open the distribution of outcomes, the unshown attempts and the boundaries of the interface. Those questions matter when moving from an impressive example to a dependable workflow.

A clip and a controllable environment

The current Gemini API video guide directs users to Gemini Omni Flash for video generation and conversational editing. It also records the shutdown of the Veo 3.1 preview in that API. This is a reminder that the name of a model, the endpoint exposing it and its availability status are separate facts.

A clip-generation request can provide detailed creative control. That does not automatically turn the clip into an environment an agent can operate. The relevant question is whether an action taken after observing a frame changes a continuing state, or whether the system returns another artifact in response to a new request. Both interactions can be useful, but they create different training and evaluation opportunities.

Consider a camera moving behind a pillar. A video can plausibly show the camera emerging on the other side. An environment used for navigation must also preserve the relationship between the pillar, the camera and other objects through the occlusion. If the user reverses direction, the earlier scene should remain consistent with the action history required by the task.

This is a proposed distinction to examine, not a test performed on the named models. It illustrates why a benchmark built only from attractive final clips can miss the properties an interactive application needs. The application cares about the consequences of actions that were not preselected for the demo.

The same distinction applies to editing. Asking for a different object in a regenerated scene is a creative operation. Moving an existing object while preserving its identity, position history and relationships is a state-management requirement. A product can support one well while leaving the other partially specified.

A team evaluating generated media should therefore record what the interface actually promises. Prompt adherence, artistic continuity and editable references can be sufficient for a storyboard. They do not imply an exposed state transition model suitable for control research.

State belongs to more than the visible frame

Two identical images can conceal different states. A rolling object may be moving toward the camera in one case and away in another. A joint can share the same position while carrying different velocity or actuator state. A controller that sees only one frame must infer those differences from history or receive additional observations.

The MuJoCo overview makes its representation explicit: the model description and changing simulation data occupy separate structures. The engine advances the dynamic state and provides quantities useful for control and analysis. That separation lets a researcher distinguish parameters defining the world from variables describing its present condition.

The analytical advantage is inspectability. If a trajectory changes, the team can ask whether the initial state, an applied control, a model parameter or the numerical procedure changed. The answer is still conditional on the chosen model. Explicit state does not make the model a complete description of reality.

A generative environment may instead keep much of its state inside a learned representation. This can support complex visual experiences without requiring every object and interaction to be authored explicitly. It also means that a task requiring exact object identity or a measurable physical variable needs a documented way to obtain or validate that information.

Observation history matters in both cases. A camera-based agent may not receive the simulator’s full internal state, even when the simulator exposes it to the evaluator. That can be intentional: the evaluator needs privileged measurements to assess performance while the agent receives only the observations available in its intended deployment.

Reports should state that boundary. Giving an agent exact coordinates is different from requiring it to infer coordinates from images. A comparison that changes this information advantage can still be useful, but it compares complete workflows rather than visual reasoning alone.

Actions need units and consequences

A control input can represent a desired position, a force, a camera movement or a sentence requesting a scene change. Those commands are not interchangeable. They differ in units, timing and the mechanism translating intention into the next state.

The MuJoCo computation guide describes how actuator inputs, activation dynamics and transmission determine forces applied to the model. It also separates the continuous-time dynamics from their numerical integration. This provides a concrete account of the chain between a control value and a simulated consequence.

A natural-language request can operate at a different level. “Put the cup on the table” describes a goal and permits many possible action sequences. A joint command describes an action within a specified control interface. A world that renders the goal being achieved has not necessarily exposed the intermediate control problem that a robot must solve.

This distinction is especially important when a model steers another model. A planner can suggest a sequence of scene changes while a generator makes those changes visible. The combination may be useful for demonstrating task structure or imagining outcomes. Evidence of successful rendering is different from evidence that a physical controller could execute the same sequence.

The appropriate evaluation follows the action interface. For camera exploration, the test may examine navigation response and scene persistence. For force control, it may examine motion, stability and contact. For high-level task planning, it may examine whether the plan’s prerequisites and intermediate outcomes are satisfied by an independent state representation.

A task description should identify which level the system controls and which level is supplied by the environment. Otherwise, an apparently autonomous success can include substantial assistance hidden in the transition from a requested outcome to the next rendered scene.

Visible geometry and collision geometry

The Isaac Sim fundamentals give a particularly clear example of why appearance cannot establish mechanics. Assets need physics properties and colliders; visual meshes do not by themselves determine collision behaviour. The documentation demonstrates how a convex collision approximation can fill the visible hole in a torus, and how a different approximation changes that behaviour.

This is a concrete modelling trade-off rather than a flaw unique to one simulator. A collision representation can simplify geometry to reduce computational cost. The simplification may be entirely adequate for one task and invalidate another. A broad surface may work for navigation while the same approximation fails for inserting an object through an opening.

The visible image alone cannot reveal which representation produced the contact. An evaluator needs the collider or an independently measured consequence. Increasing rendering quality does not repair an inappropriate collision model, because the two changes affect different parts of the system.

A generative scene introduces a related question: what determines whether two objects pass through one another, stop or deform? A convincing image sequence can suggest a plausible interaction. A mechanical application needs an account of the relevant constraints and their error across the conditions being used.

The practical implication is to validate the task’s contact events directly. If the decision concerns clearance, inspect clearance. If it concerns contact force, inspect force. If it concerns whether a camera can see an object, inspect visibility. “Looks realistic” combines those different questions into an impression and can conceal the one that determines success.

This is also why a visually simple simulator can be appropriate for a demanding control task. Its useful fidelity may lie in the variables the task depends on, rather than in the richness of its rendered surface.

Physics engines still contain assumptions

Explicit equations make assumptions easier to inspect; they do not remove approximations. The MuJoCo modelling guide describes contact and constraint parameters, including how stiffness, damping and impedance affect the solver’s behaviour. The engine offers choices that need to match the system and purpose being studied.

A controller can perform well in a perfectly repeatable simulation whose parameters poorly match the real mechanism. That result establishes performance in the specified model. It does not establish performance on the physical device. Reproducibility and external validity are different properties.

Calibration therefore needs an observable target. Matching a resting pose does not necessarily match response during impact. Matching motion under one load does not necessarily establish response under another. A parameter fit can be accurate in the measured regime while leaving important deployment conditions insufficiently constrained.

The same reasoning applies to numerical settings. A simulation step is an approximation to evolving dynamics. If changing integration or contact settings materially changes the outcome relevant to the task, that sensitivity belongs in the result. An apparently stable rendered animation can conceal a materially different force or trajectory.

A useful report identifies which quantities have been checked against a reference and over what range. That reference might be an analytical case, a measured physical trajectory or another validated instrument. It should be independent of the rendering quality being assessed.

Learned and structured models thus share a need for task-specific validation. Their strengths differ in how the world is represented and controlled, and in what can be inspected. Neither category earns a general claim of physical accuracy from its name. A well-scoped result states which consequences match the evidence and which remain outside it.

Three clocks in one simulation

Interactive systems combine wall-clock time, simulated time and observation timing. A user may experience smooth playback while the underlying simulation advances in different increments. An agent may receive a frame after the state that produced it has already changed.

The CARLA timing guide separates fixed simulation timesteps from client-server synchrony. It explains that synchronised execution lets the client control advancement, while fixed timesteps provide consistent simulated intervals. Its reproducibility guidance also includes world reloads and coordinated command handling. Determinism is conditional on configuration.

This matters when comparing two policies. If one takes longer to choose an action while the world continues advancing, it faces a different control situation. If the world pauses during each decision, the task measures a different interaction. Both modes can be appropriate, but the benchmark should say which one it uses.

Rendering frequency is another variable. Several physics steps can occur between displayed frames. Increasing the number of visible frames does not necessarily increase the precision of the physical integration, and increasing integration frequency does not necessarily give the agent more observations.

A useful timing record therefore attaches actions and observations to simulated state and time. Without that alignment, a failure can be misdiagnosed as poor reasoning when the policy was responding to a stale observation, or as poor physics when the command was applied later than assumed.

For an interactive world, end-to-end response latency remains important even when no physical mechanism is being modelled. The experience depends on how quickly an input affects what the user sees and hears. That is a user-facing timing property, distinct from the accuracy of the simulated duration represented by the scene.

Driving worlds expose a richer measurement layer

The CARLA core concepts distinguish the world, actors, maps and sensors. Its documented sensor types include cameras, collision detection and inertial measurements. That architecture illustrates how an environment can expose several views of the same underlying scenario.

A driving agent’s camera image and an evaluator’s collision record answer different questions. The image describes an observation available to the policy. The record can establish whether a specified event occurred according to the simulator. A successful-looking clip should not replace the event definition when the task is to avoid collisions.

Scenario control is similarly important. To study a particular interaction, a researcher needs to specify which actors appear, how they behave and what changes between trials. A world containing believable traffic is useful for exploration, but a comparison of policies also needs a defined distribution of scenarios.

Rare situations require special care. Generating many visually diverse scenes does not establish that the task-critical combinations are represented. Conversely, repeatedly testing one carefully specified difficult scenario provides a narrow stress test rather than a population-wide success rate.

The report should preserve that distinction between breadth and controlled difficulty. It should also identify which measurements come from the simulator’s internal state and which are inferred from rendered observations. If an external judge reads the images, its errors form another layer in the evaluation.

These principles transfer beyond driving. A robot scene, interactive game or generated training world becomes more assessable when it exposes the state and events needed to determine success independently of a persuasive visual story.

The environment contract shapes learning

The Gymnasium environment API provides a useful vocabulary for the interface: an environment accepts actions, returns observations and rewards, and distinguishes task termination from truncation. Resetting starts another episode. These are software semantics, independent of whether the underlying world uses rigid-body physics, a game engine or a learned generator.

That distinction reveals what a training workflow needs beyond a stream of images. It needs a defined action space, a way to initialise episodes and a success or reward rule. It also needs to distinguish a task ending from an episode stopping because a limit was reached.

A generated environment could expose such a contract. The existence of interactive visuals alone does not establish that it does. If success is assessed by a separate language model, the evaluator’s definition and reliability become part of the environment. If success comes from structured state, that state must remain consistent with the observations the agent receives.

Reset semantics are equally consequential. Starting from the same prompt is not necessarily starting from the same world state. A controlled comparison may require matching the actual initial state or sampling both policies from a declared distribution. A repeated phrase in a prompt is insufficient evidence that this requirement has been met.

Our guide to comparing evaluation frameworks explains the related need to separate task definition from scoring implementation. For simulated worlds, that separation includes the environment’s own transitions and failure conditions. The evaluation cannot be stronger than the event definitions it relies on.

Separating perception from control errors

A failed interaction can originate in several places. The agent may misread the scene, choose an inappropriate action, or receive an outcome inconsistent with the environment’s stated rules. A single success score combines those possibilities. Diagnosing them requires observations that identify where the chain broke.

A structured environment can support paired evaluations with different observation interfaces. One policy receives images; another receives selected state variables. This is a proposed diagnostic design, not a result measured here. The contrast can help reveal whether perception is the limiting factor, provided the action interface and scenario distribution remain comparable.

The interpretation still needs care. Supplying state variables may simplify more than perception by removing uncertainty that is intrinsic to the task. The result should therefore describe the additional information, rather than treating the state-based score as an upper bound achieved under identical conditions. The purpose is to localise a bottleneck and guide the next experiment.

A useful boundary for hybrid worlds

Hybrid systems can assign different jobs to different components. A structured model might own object positions and task logic while a learned model renders appearance. A planner might choose high-level actions while a simulator determines their physical consequences. The design becomes assessable when those responsibilities are explicit.

The difficult part is consistency between representations. If a rendered object appears somewhere other than its structured position, the agent and evaluator may disagree about the same action. If a generator changes an object’s identity while the state layer preserves it, a task involving memory can become ambiguous. The interface needs a way to detect and handle such disagreements.

A practical evaluation record therefore names the authoritative state, the available actions, the observation pipeline, the timing model and the success rule. It separately records visual consistency, dynamic accuracy and any real-world transfer evidence. These are fields in a specification, not a demand that every creative tool become a robotics simulator.

The appropriate conclusion is correspondingly specific. A system may be effective for interactive concept design without being validated for engineering prediction. A simulator may provide a useful control environment without matching every real-world contact. A successful policy may transfer under tested conditions without establishing unrestricted reliability.

Reading a model card as evidence helps preserve these boundaries when a release combines demonstrations, aspirations and measured results. The decisive question is what the system lets the user control and verify. World generation becomes a dependable tool when its evidence matches the job assigned to it.