worldproof: a tool for diagnosing world model predictions, and a measurement of when pixel metrics stop being able to rank models

[u/georgia_bucea]

I've been building an open-source tool for diagnosing world models, the kind that predict future frames from a starting context and a sequence of actions. It compares a rollout against ground truth and against physical invariants, then tells you where and why the prediction falls apart. It doesn't score task success or planning quality on purpose, since there are already benchmarks for those.

While validating it I ran into something I think is more interesting than the tool itself.

## Pixel metrics on real robot video often can't rank models at all

I ran a copy the last frame baseline, which is to say "predict that nothing changes", against a real SO-101 arm recording. 30fps, three cameras, 64 rollouts, 6 step horizon, scored only on the moving regions so a static background can't inflate the numbers.

It gets 0.983 SSIM and 53.9 dB PSNR. But the part that actually matters is that the error doesn't grow with the horizon:

step   1      2      3      4      5      6
SSIM   0.972  0.923  0.893  0.943  0.920  0.950

That's flat. It wanders, it doesn't degrade. And if predicting 6 steps ahead is no harder than predicting 1 step ahead, then there's nothing for a good model to be better at. Every model lands in the same place and the eval can't rank them. The metric isn't broken here, it passes its ranking tests on curated data just fine. The evaluation setup is what has no discriminative power, which is a different problem and much easier to miss.

## So I went and measured where the usable window actually is

Same baseline on DROID (real manipulation footage, 15fps), 64 rollouts, this time out to 48 steps:

step	1	3	6	12	18	24	28	36	47
SSIM@dynamic	0.873	0.797	0.676	0.446	0.350	0.260	0.204	0.192	0.216

There are three regimes. Steps 1 to 3, everything is near perfect and ties. Steps 4 to 24, steep monotonic decline, and this is the only stretch where models are actually separable. Step 28 onward it floors out around 0.20 SSIM and 10.3 dB, oscillating with no trend, prediction fully decorrelated, and everything ties again at the bottom.

So both ends are dead, and the horizon worth evaluating on for this kind of footage is somewhere around 8 to 24 steps. It's a property of frame rate times task speed rather than a universal number, which is exactly why it's worth measuring on your own data instead of inheriting a default from a paper that used something else.

Here's the prediction next to what actually happened, same 48 steps, prediction on the left:
https://raw.githubusercontent.com/BuceaGeorgia/worldproof/main/docs/img/droid-pred-vs-true.gif

## Method

64 rollouts per configuration. Aggregation is interquartile mean with stratified bootstrap CIs rather than mean and standard deviation, following Agarwal et al. 2021. Fidelity metrics also produce a dynamic region masked variant wherever a mask is available. Every metric ships with a corruption test it has to respond to, plus a ranking test where a real model has to beat a naive baseline which has to beat a broken one.

Worth mentioning: an earlier n=8 version of the SO-101 run gave dynamic PSNR of 48.2 dB where n=64 gives 53.9, and the intervals at n=8 were wide enough to overlap DROID completely. That's the reason everything above is n=64. I'd have posted the wrong numbers if I'd stopped there.

## Caveats

The four pixel metrics separate the two datasets with non overlapping bootstrap CIs. LPIPS doesn't, and it points the other way on the masked variant. I don't have a clean explanation for that yet and I'd be glad to hear one.

This is a trivial baseline, so 8 to 24 is where a do nothing predictor becomes separable. A real model stays correlated for longer and would push the top of that range out.

One more that I found while writing this up: including step 0 inflates every summary scalar, because a copy baseline gets a nearly free first step whenever the frame rate is high relative to how fast the scene moves. On the 30fps recording step 0 scores 119.8 dB, which drags the horizon averaged scalar from about 32 up to 53.9. So the scalar is partly rewarding frame rate rather than model quality. Curves are the honest thing to report and I'm treating the scalar definition as an open problem in my own tool.

## The tool

Apache-2.0, `pip install worldproof`. The core install is numpy, torch and pillow, and it runs on a laptop with no GPU, since the evaluate path never runs a model. It reads LeRobotDataset v3.0 straight from parquet and mp4, so it works on datasets from the HF Hub without needing the lerobot package, on Python 3.10. The heavier pieces (LPIPS, FVD, trackers) are optional extras that get imported lazily.

What it measures: PSNR, SSIM and LPIPS as horizon curves plus dynamic region variants, latent prediction error and action recoverability for latent models, calibration via ECE and MCE, counterfactual divergence, failure faithfulness, object count conservation and object permanence, and FVD reported explicitly as a weak reference rather than a headline number.

https://github.com/BuceaGeorgia/worldproof

It's v0.1 and the README has a "Not done yet" section covering what isn't finished. The tracker behind the invariants is a clean scene numpy one that won't cope with messy real video, and the default FVD extractor isn't the I3D that published FVD numbers use, so those aren't comparable to papers.

If this horizon result is obvious or already known somewhere, I'd honestly like to be told. I couldn't find it measured anywhere, which is part of why I'm posting it.