A field guide to how machines build, run, and act inside models of reality — and why it matters for anyone building AI agents.
"The world is everything that is the case." — Ludwig Wittgenstein, Tractatus Logico-Philosophicus (1921)
Wittgenstein opened the Tractatus by insisting that the world is the totality of facts, not of things. A century later that line reads almost like a design spec for artificial intelligence. If we want machines that act competently in the physical world, they need an internal model of "everything that is the case" — a world model.
In her essay A Functional Taxonomy of World Models, Stanford's Fei-Fei Li — with the team at World Labs — offers the clearest map yet of what those models actually do. Instead of sorting world models by how they are built, she sorts them by their function: the question each one answers. What follows is our summary of that taxonomy, and why we think it is the most useful lens available for teams building agents today.
Every embodied agent runs the same loop. It perceives the world, updates an internal model of it, plans against that model, and acts — then perceives the consequences and goes around again. This is the classic perception–action loop, formalized decades ago in reinforcement learning (Sutton & Barto) and, further back, in Kenneth Craik's 1943 proposal that the brain carries a "small-scale model" of reality it can run experiments on before committing to action.
A world model is whatever fills the "internal model" slot in that loop. And here is Li's key insight: different systems fill that slot in fundamentally different ways, because they are answering different questions.
Li identifies three.
A renderer outputs observations: pixels, video, sensory streams. Given a state and an action, it predicts what the next frame looks like. Google's Genie 3 and the wave of "generate-a-playable-world" systems live here. Renderers are dazzling because their output is directly human-visible — but pixels are a consequence of the world, not the world itself.
A simulator outputs state: geometry, objects, physics, the persistent structure of a scene. World Labs' Marble, which turns a prompt or image into an explorable 3D world, is a simulator. It does not merely paint a picture of a room — it maintains the room, so the far wall is still there when you turn back around. This is the layer that holds ground truth.
A planner outputs actions. Vision-language-action (VLA) models that map a goal and a scene onto motor commands are planners. They consume a model of the world in order to choose behavior.
The elegant part of Li's argument is that these three are not peers — they are arranged around a hub. The simulator is the linchpin.
A renderer without an underlying simulator produces beautiful, inconsistent dreams: worlds that flicker and forget. A planner without a simulator is guessing about the consequences of its actions. It is the simulator — the persistent, physically grounded model of state — that gives the renderer something consistent to draw, and gives the planner something reliable to plan against. Get the simulator right and the other two functions finally have solid ground to stand on.
This is also why the economic stakes are so large. Industrial simulation platforms such as NVIDIA's Omniverse already anchor a market measured in the trillions of dollars, because "what is actually true, and what happens next?" is the question that robotics, autonomous vehicles, manufacturing, and scientific discovery all depend on.
Today these functions usually live in separate systems. Li's forecast is that they will not stay separate. The frontier is a unified world model — one system that can render what it simulates and plan over what it renders, closing the perception–action loop end to end. Renderer, simulator, and planner become three faces of a single capability rather than three research programs.
At Ejentic AI we build autonomous agents, so this taxonomy is more than academic housekeeping — it is a diagnostic. When an agent behaves unreliably, it helps to ask which function failed:
Most "the agent went off the rails" failures are really simulator failures: the system had no persistent, trustworthy model of the world it was operating in. Li's framing tells builders where to look first — and it matches what we see every day. The agents that hold up in production are the ones with the best internal model of their world, not the ones with the flashiest output.
This is Ejentic AI's summary and commentary. The original essay — with the full argument and examples — is by Fei-Fei Li and the World Labs team. Please read it in full here: A Functional Taxonomy of World Models.