PublishedCurated

Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

Abstract

Proposes Agent World Model (AWM), a fully synthetic environment generation pipeline scaled to 1,000 environments covering everyday scenarios, in which agents interact with rich toolsets and obtain high-quality observations. The environments are code-driven and backed by databases rather than simulated by an LLM, which the authors argue gives more reliable and consistent state transitions than LLM-simulated environments, and enables reliable reward functions because the database state is accessible. Experiments on three benchmarks show that training exclusively in synthetic environments yields strong out-of-distribution generalisation.

Simulation drift is the hidden cost

Training an agent inside an LLM-simulated environment means the world model and the agent are the same kind of thing. State transitions are plausible rather than correct, and the reward signal inherits every bias of the simulator.

AWM makes environments code-driven with a database behind them. Transitions are executed, not generated. State is queryable.

Why the database matters more than the scale

A thousand environments is the headline but the mechanism is the point: because the state is in a database, the reward function can read it. You are no longer asking a judge model to decide whether the agent succeeded, you are checking whether the row changed.

That is the difference between a reward that can be gamed by fluent output and one that cannot.

The transfer result

Training only in synthetic environments and evaluating on real benchmarks produced strong out-of-distribution generalisation. If that holds up, it changes the economics of agent training — the bottleneck has been environment availability, and this treats environment generation as the thing you scale rather than the thing you scavenge.

BibTeX

@inproceedings{wang2026agent,
  author    = {Zhaoyang Wang and Canwen Xu and Boyi Liu and Yite Wang and Siwei Han and Zhewei Yao and Huaxiu Yao and Yuxiong He},
  title     = {Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning},
  booktitle = {International Conference on Machine Learning (ICML) 2026},
  year      = {2026},
  eprint    = {2602.10090},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2602.10090}
}

Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.