The stillest reference made the camera move the most

Correction — 9 August 2026

I have been trying to get a small video model to do something that sounds trivial: film a room from a camera that does not move.

Not a fancy shot. A bolted-to-the-wall, hospital-corridor, nothing-ever-happens shot. The kind of footage a security camera produces by accident, all day, for free.

The model would not do it. It panned. It pushed in. Told in the prompt "static security camera, no camera motion, no pan, no zoom" — and told again in the negative prompt, which is where you put the things you specifically do not want — it zoomed anyway, in roughly half of everything it generated. Over a long enough clip it would push in from a wide shot to a close-up of the patient's face, and the room would simply leave the frame.

That is where this starts. It ends somewhere I did not expect: with the discovery that the advice I was following, which is also the advice the literature gives, has a hole in it that you can fall straight through without ever seeing it.

The fix that works, and the reason it works

There is a known trick for this. If you condition an image-to-video model on one reference frame, small models are unstable. NVIDIA's Cosmos work measures it directly: single-frame conditioning corrupts often; nine-frame conditioning is stable, with failure rates under 2%.

So I gave the model seventeen reference frames instead of one. The improvement was dramatic. At long clip lengths, generation failures went from 41.7% to zero. Camera drift dropped by three quarters. The room stopped leaving.

Wonderful. More frames, better output. A clean rule, published by people with far more compute than I have, confirmed on my hardware. I wrote it down as a design guideline and moved on.

Then I tested the rule properly

The thing about a rule like "more frames are better" is that it does not say which frames. I had been feeding the model seventeen consecutive frames from a clip where the subject was moving a little. What if the frames were more still? What if they were more active?

So I ran eight versions. Same model, same scene, same everything — seventeen conditioning frames in every single one. The only difference was how much motion those seventeen frames contained, measured beforehand, from almost nothing to quite a lot.

Then I measured two things separately, because measuring them together is a mistake I had already made twice: how much the subject moved in the output, and how much the camera moved. The camera measurement is easy and honest — you look at a strip of wall near the top of the frame. Wall does not move. If it changes, the camera moved.

Here is the camera column, ordered from the stillest reference window to the most active:

3.708 → 0.828 → 0.772 → 0.944 → 1.036 → 2.549 → 1.279 → 3.826

Read the first number again. The stillest reference frames produced the worst camera drift in the entire experiment. Worse than the most chaotic window I tried. The thing I was doing to hold the camera still, done harder, made the camera move more.

Seventeen copies of one photograph

The explanation is embarrassing in how simple it is, and I did not see it for two days.

If you hand the model seventeen frames in which nothing happens, those seventeen frames are nearly identical. They carry the information of one frame, repeated seventeen times.

The benefit was never the number of frames. It was the temporal information those frames carry, and a still window carries almost none.

Both edges are cliffs

So more motion in the reference is better? No — and this is the part that makes it a real rule rather than a correction.

Push the reference window's motion up and the camera gets worse again, for a completely different reason. Video models tend to move the camera when the subject moves — a known effect called motion entanglement, and one with a mundane cause: the footage these models learn from is overwhelmingly moving-camera footage, because cinematographers follow their subjects. Ask for subject motion and you get camera motion bundled in, because in the training data those two things almost always arrived together.

So the curve has a floor in the middle. Too still, and your seventeen frames collapse into one. Too active, and entanglement takes over. In between there is a band — for this model, reference windows with a specific moderate amount of motion — where the camera is quietest and nothing fails. Sitting in that band instead of at the still end gave me 43% more subject motion, zero generation failures, and identical compute cost. Free, purely from choosing better reference frames.

The part where I was wrong in public

I want to include this because leaving it out would make the story cleaner than the work was.

Having measured three points past the good band — 8% failure, then 100%, then back to 17% — I had already written, in a document, that I had located a sharp cliff between two specific values. I was pleased with it. It was a tidy result.

The fourth point destroyed it. Above the good band, failure rates go 8%, 100%, 17%, 100%, with no relationship to how much motion the reference contains. There is no cliff to locate. The honest description is not "the boundary is here" but "below this the model is reliable and above it, it is erratic" — which is a worse result, less quotable, and true.

That was the tenth measurement I got wrong in a week, and they were all the same error: I kept reporting something global as though it were something local. A drifting camera read as a moving subject. A deliberate change I made myself read as the model degrading. A ratio that got worse while both of its terms improved. Every one of them looked like a finding. None of them were caught by better statistics — they were caught by known-answer tests and by opening the image files and looking.

What to take from it

If you are conditioning a video model on multiple frames — and you should be, it is a large free improvement — then check that your frames actually differ from each other. A reference clip of someone asleep, or a locked-off shot of an empty room, or any window where the interesting thing has not started yet, is a single frame wearing a disguise. You will get the instability you were trying to avoid, and the fix will look like it is already applied.

There is a right amount of motion in a reference window. It is not zero, and it is not as much as possible. On the system I measured, both extremes were worse than the middle by a factor of four.

I do not know yet whether the exact band transfers to other models, other scenes, or other clip lengths — those experiments are cheap and I am running them. But the shape of the thing, the fact that there is an interior optimum at all, is not something I have found written down anywhere. Which either means it is new, or means I have not looked hard enough yet.

Given how the rest of this week went, I would not bet heavily on the first one.

What the rule reduces to

Check that your conditioning frames actually differ from each other. A reference window in which nothing happens is a single frame in disguise: you pay for multi-frame conditioning and get single-frame instability, with nothing in the output to indicate it.

That is what reproduced. Everything past it — where the optimum sits, whether there is one at all, what happens beyond it — was measured on one scene and did not survive contact with a second. Find your own operating point rather than adopting mine.