WorldOracle, which its developers describe as the first benchmark for action-driven world models that uses real-world execution as the yardstick, launched on Oct. 16 and opened evaluation submissions to developers. The project was jointly completed by researchers from AGIBOT, NVIDIA, Princeton University, the University of Texas at Austin and the University of Toronto, combining world-model developers with benchmark researchers.
The platform addresses a question the industry has lacked a benchmark to answer directly: given a set of actions, how far is a model’s predicted future from what actually happens? WorldOracle defines this as “Execution Coherence.” The researchers said a reliable world model must withstand three stress tests: causal obedience, meaning it must follow the given action sequence, including wrong or failed operations; distribution shift, meaning it must handle actions from human teleoperation and from autonomous robot policies with pauses, jitter and corrections; and temporal endurance, meaning errors must remain controlled after 120 seconds of continuous operation, not just in the first 10 seconds.
The benchmark uses pre-collected real-world operation footage and action sequences. Models generate predictions under the same initial frames and action sequences through open-loop replay, and the platform compares them frame by frame with real execution. Failed operations such as missed grasps, drops and spilled water are preserved, so accurately predicting a failure also earns a high score. The team said WorldOracle is the only benchmark combining action conditioning, real data, failure samples, long-horizon evaluation, multiple embodiments, multiple views and policy-autonomous execution distribution.
The evaluation set contains 335 trajectories: 100 from the AgiBot G2 robot, 75 from the YAM robot, 100 from first-person human hand operation and 60 from G2 policy-autonomous execution. Each is about two minutes long. About 40% of interaction clips are failures or suboptimal executions. Tasks cover four physical forms: rigid-object fruit grasping, deformable towel hanging, fluid pouring and articulated drawer opening and closing. The team also collected 1,779 G2 training trajectories, including about 20% failure clips, and released the YAM ABC-130K dataset. Human-hand scenarios are evaluated zero-shot.
Annotation is automated with a VLM generating physical criteria for key result frames and SAM3 segmenting objects, robot arms and human hands, so evaluation focuses on interaction regions rather than being diluted by background. In human validation, 119 participants blind-reviewed 128 prediction pairs; four metrics showed 88.1%–93.2% agreement with human judgment and Spearman correlations of 0.72–0.82. Control experiments showed key-region metrics fell monotonically as distortion in interaction regions increased but barely moved when only the background changed.
WorldOracle evaluated seven representative action-driven world models: IRASim, Ctrl-World, DreamDojo, Cosmos 3, GE-Sim 2.0, JointControlVideo and OSCAR. The results exposed clear weaknesses. On AgiBot G2 human-operated trajectories, IRASim scored 78.8 for the robot arm but only 29.1 for the manipulated object; Cosmos 3 scored 89.4 versus 45.4; even the strongest GE-Sim 2.0 scored 100.0 for the arm but 62.8 for the object. The models replicated arm motion but failed to capture its effects on the world.
Long-horizon performance also deteriorated sharply. GE-Sim 2.0’s single-view composite score fell from 86.1 at 10 seconds to 63.4 at 120 seconds, while other models ended at 26.7–35.1. On full trajectories, DreamDojo fell from 50.56 to 27.08 and Cosmos 3 from 53.99 to 26.60. The researchers said short-horizon scores are poor predictors of long-horizon performance.
Spatial alignment and physical-outcome correctness were found to be independent capabilities. GE-Sim 2.0 achieved 96.96 multi-view consistency and 73.49 spatial alignment on full trajectories, but only 60.26 physical-state correctness. In one extreme case, a prediction scored 80.8 on spatial alignment but 0.0 on physical state: a drawer that was fully closed in reality was predicted as open. Under policy-autonomous execution, Ctrl-World’s physical-state correctness fell from 51.15 under human operation to 27.22. Human hands remained a blind spot: the best zero-shot model scored only 15.2 in object-region fidelity.
WorldOracle has published its leaderboard and submission guidelines and supports open-weight and closed models. After registration and review, developers can submit predicted videos by embodiment and track and view composite scores, sub-scores and segment-by-segment comparisons. The researchers said the benchmark is intended to test whether improvements make object motion more realistic, reduce misjudgments of failed operations and hold up over two-minute predictions as world models enter robot planning, policy evaluation and training pipelines.
Project page: https://worldoracle-benchmark.github.io/
Paper: https://arxiv.org/abs/2610.18826
AGIBOT
Figure AI
UBTECH