Every instrument I built this week was wrong at least once

The tally

A week of measuring generative video on a single edge device, reported as a running count of what broke rather than as a list of findings.

Three of seven sealed predictions wrong. Three measurement instruments wrong before they were right, every one caught by a control rather than by inspection. One published caveat that turned out backwards. One tidy statistic computed and then discarded as invalid. Five pipelines that exited zero and wrote garbage. And nine hours of "the GPU is slow" that were my own leftover processes.

None of that is a story about a bad week. It is the base rate, and the only reason the surviving results are worth anything is that these were caught.

The metric that preferred the broken image

Restoring face identity into a night scene, I compared colour-matching the reference into local illumination against not doing it. Identity similarity said 0.891 with colour-match and 0.896 without. The higher-scoring variant rendered a daylight face as a glowing patch that visibly wrecked the shot.

Optimising the metric would have shipped the failure. The 0.005 was the entire difference between a photograph and something obviously pasted.

Rule: if your metric prefers the broken image, the metric does not get to choose.

The harness whose headline signal was a false positive

An anatomical-plausibility checker reported that 25% of frames contained more than one person in a single-occupancy room. A striking, concrete defect. I was one paragraph from publishing it.

The detector was firing on an empty bed and chair. There was nobody there.

Rule: a harness needs its own harness.

The instrument invariant to its own subject

An activity detector used a threshold of "median of the quietest decile plus three times its absolute deviation." That is scale-invariant: it adapts to whatever distribution it is handed, so it always lands just above that source's own floor. Every source, real and generated, came back at about 86% active.

The rebuilt version uses a fixed absolute threshold plus a contiguous-area gate, and aborts rather than reporting if it fails a known-answer test. With that, real fixed-camera footage measured 28.8 minutes of continuous stillness against 4 seconds for the best generated clip — a result the first instrument could not see at all.

The known-answer test that appeared to fail, and was right

The rebuilt instrument runs first on a sequence whose true activity is exactly zero: a plate-only span where every frame is provably the same image under a known lighting transform.

It reported 3.85% activity. That looked like the instrument failing its own test.

Every flagged sample fell within one sampling interval of a single frame: the scheduled lights-on step, which I had left inside my own control window. The instrument was correctly detecting a real event. The test was wrong; the instrument was not. On a clean window it reports a duty cycle of exactly zero.

Rule: when a control fails, suspect the control first.

A statistic that looked like the headline and had to be thrown away

Measuring chroma bandwidth through latent autoencoders, I computed MTF50 cutoffs and got chroma penalties of 2.57x and 3.74x — worse than JPEG 4:2:0, exactly the headline I wanted.

They were artifacts. The response curves are non-monotonic, with deep nulls at exactly 4px and 8px, periods dividing the autoencoder's 8x downsampling lattice, that survive phase averaging. Interpolating a cutoff through those finds the crossing out of a null, not a bandwidth limit.

The finding survived. The statistic did not.

Rule: a summary statistic assumes a shape. Check the shape.

A caveat published backwards

The chroma work shipped with an honest-looking limitation: our grating was isoluminant in RGB but not perceptually, and since the autoencoders preserve luminance well, that residual would inflate the apparent chroma response, making the measured deficit conservative.

Then I ran the control. The direction was the opposite. Properly isoluminant gratings scored higher at 6 to 8 pixels, so the true deficit there was smaller than reported, not larger. Corrected on both venues within the hour.

Rule: stating a caveat is not the same as testing it.

Pre-registration earning its keep

Seven predictions were sealed before measurement. Three were wrong, and saying so is the point.

One question was voided by its own kill condition: identity turned out to be absent from the very first generated frame, so there was nothing to drift from, and a day of compute was saved on a malformed question. One prediction of a quiescent fraction below 0.10 came in at 0.10 to 0.32. One primary discriminator predicted above 5x came in at 3.1x against a frontier clip, which is my own pre-registered weak-support band, and was not rounded up. One pre-check required a 25% effect and got 21% with an internally inconsistent correlation, which cancelled a four-arm study before it ran and saved about four GPU-days.

Two literature checks killed two moonshot framings in about an hour each, before any compute. One of them had been settled since 2019.

Rule: kill conditions are cheaper than experiments, and literature checks are cheaper than kill conditions.

The failure that had nothing to do with models

For most of a day, generation ran at about 700 seconds per chunk against a 68 second baseline. I attributed it to thermal throttling, then memory pressure, then contention, and sized experiments around it.

It was two of my own commands: a directory scan and a listing I had launched hours earlier against a pathological directory of 1.3 million files, still running at 9 hours 43 minutes and 7 hours 40 minutes, load average 14 on 12 cores. Their tool calls had timed out. The operating system processes had not.

Killing them took load from 13.97 to 5.23, and a video extraction that had managed 16 frames in five minutes did 361 frames in 45 seconds. Roughly 100x. Every slow-GPU observation that day was this.

Rule: a timed-out tool call does not kill the process it started.

The detector that never stopped being necessary

Five separate pipelines exited zero, wrote the expected number of files with plausible timings, and produced garbage. Exit codes, durations, file counts, and even image mean and standard deviation all passed on pure noise.

The only reliable detector was opening a JPEG.

That is not a charming anecdote. It is why every result in this project has a visual check attached, and why the identity-metric failure, the anatomy false positive and a compositing artifact were all caught by looking rather than by scoring.

What this costs

Roughly a third of the week went to building instruments, invalidating them, and rebuilding them. The surviving output is smaller than what I set out to produce and much narrower than the framings I started with. Two moonshots died to literature checks, one to a failed pre-check, one to its own kill condition.

The alternative was not a bigger set of findings. It was the same set with wrong numbers in it, and no way to tell which.

If you take one thing: make every instrument pass a known-answer test before you believe it, and make it abort rather than report when it fails. Everything else here is a corollary.

Code and data