A tripod is the hardest test I can give a video model

Here is a test that costs nothing to run and that no video generator passes.

Point a camera at a quiet room. Bolt it to the wall. Now ask a video model to produce the same shot — a fixed viewpoint, nothing moving that shouldn't move — and measure how long it manages before the room starts drifting.

The camera on my wall holds still for 28.8 minutes.

The best generated footage I have measured holds still for 20 seconds.

Why a tripod is nasty

Most video benchmarks bundle a dozen things together — subject consistency, motion smoothness, temporal flicker, aesthetic quality — and most need reference footage or human raters to score anything. They are expensive and they are arguable.

A tripod is neither. It asks one question:

The ground truth is physics. A camera screwed to a wall cannot pan. If the wall in the generated footage moves, that is model error, full stop. No reference video, no raters, no learned metric, nothing to argue about.

And there is nowhere to hide. Camera drift, geometry deformation, texture creep, accumulated conditioning error — every failure mode a long-video model has shows up as the same visible thing: the wall moved, and walls do not move.

The measurement

Solve a geometric transform between the first frame and each later frame — the standard computer-vision approach, feature matching with outlier rejection. If the scene is mostly static, the dominant transform is the camera. Report how far it moved and how long it took to move.

Then measure three more things separately, because a single number hides which failure happened: what the camera model cannot explain (the room deforming rather than moving), appearance drift after normalising brightness, and catastrophic collapse.

Three tiers came out, and they are cleanly separated:

Eighty-six times between reality and the frontier. About ten times between the frontier and a 2-billion-parameter model you can run at home.

The thing that nearly fooled me

My first instinct was the obvious metric: measure how much the static parts of the frame change, pixel by pixel. Wall shouldn't change. Measure the change.

It does not work, and it fails in the worst possible way — by looking like it works.

Real fixed-camera footage scored 1.19 on that metric. Generated footage scored 1.06. Indistinguishable, and if anything the real camera looked slightly worse.

The reason is that in real footage, the "change" in a static region is sensor noise and JPEG compression. In generated footage it is the camera genuinely drifting. Same number, opposite cause. A pixel-difference metric cannot tell an unstable camera from a slightly grainy one.

The geometric solve is immune, because noise produces no consistent transform. Random per-pixel wobble has no direction; a drifting camera does. That distinction is the whole test, and I only found it because I had a real camera to check against — which is the same property that makes the benchmark work at all.

Two ways to cheat, both of which I built before noticing

Generate a still image three hundred times. Perfect stillness, perfect score, zero value. So subject motion has to be reported alongside, and a clip where nothing moves is a null result rather than a win. Read the two together or don't use the test.

Reset constantly. Chunked generators periodically snap back to a reference frame. That means the maximum drift is lower if you use shorter chunks — the error never has time to build before the next reset. Measured on my own output: short chunks 24 pixels of drift, long chunks 98 pixels, while the long-chunk configuration is independently better on every other measure.

A model that reset every ten frames would score beautifully and look like a strobe light. So the headline metric counts a reset as a break in stillness, not a fix.

And one that caught me twice

Half my first batch of frontier clips scored exactly 12.5 seconds of stillness. Impressive consistency — until I noticed 12.5 seconds was the length of the clips. They had not passed the test. They had run out of video.

That is censoring, in the survival-analysis sense, and it is the difference between "this model holds still for at least 12.5 seconds" and "this model holds still for 12.5 seconds." Re-run at full length, two of them failed at 16 and 20 seconds and two were still censored.

It also breaks the nicest property of the test. I had written that it was cheap because generators fail fast. True of weak models. A frontier model needs half a minute of video to fail, and the real camera did not fail for 28.8 minutes. The cost of this test scales with how good the thing you are testing is — which is an awkward thing to admit about a benchmark you have just described as free.

Why I think it is worth having

Long-horizon video consistency is an active research problem, and most of the work on it modifies model internals — attention mechanisms, memory caches, retraining. That is the right place to attack it if you have access. Most people do not. They have an API and an image to condition on.

A tripod test measures what happens under those conditions, on a scale anchored to a physical object rather than to another model. And it is embarrassingly simple: a fixed prompt, a room where nothing should move, and four numbers.

The protocol and a single-file implementation are published. I can run two open models on the hardware I have, which gives distributions across scenes and seeds but a thin model axis. If you have access to models I do not — larger open weights, closed APIs — the interesting question is whether the 16-to-20-second ceiling is a property of one model family or of the current state of the art.

I would find either answer useful. I would find the second one more interesting.