How not to believe your own instruments
A field guide from one week of measuring generative video on a single edge device. Not a guide to going faster — a guide to not being fooled by measurements you built yourself.
Joe Cox, Resoa
What this is
A field guide to the narrow and useful problem of not being fooled by your own measurements. That is where nearly all of our wasted effort went, and where nearly all of the saved effort came from.
The record it rests on: three of seven sealed predictions wrong. Three instruments wrong before they were right. One published caveat that was backwards. One headline statistic computed and discarded. Five pipelines that exited zero and wrote garbage. Nine hours of "the GPU is slow" that were our own leftover processes.
Nothing here required unusual skill. Everything here required distrusting a number that looked correct.
What actually cost the time
Almost nothing was lost to bugs in the ordinary sense. Implementations mostly worked the first time. The losses were epistemic.
| failure mode | what it cost |
|---|---|
| instruments confidently wrong | 3 rebuilt, roughly a third of the week |
| our own stale processes | 9 hours misattributed to a slow GPU |
| believing a first measurement | a caveat published backwards |
| pipelines exiting zero with garbage | 5 of them, caught by none of the automated signals |
One bias is worth naming because it was systematic rather than random: we over-predicted effect sizes in the direction we wanted, every time. Predicted a quiescent fraction below 0.05, measured 0.10 to 0.32. Predicted a 25% effect, measured 21% with an internally inconsistent correlation. The errors were consistently optimistic. The only reason that is visible at all is that the predictions were written down before the measurements.
Five practices, ranked by what they saved
### 1. Literature check before compute — one hour, every time
Two moonshot framings died to checks that took about an hour each. One had been settled since 2019. Another turned out to be a 2023 paper we had independently reimplemented.
The framing matters more than the search. Do not ask "is this taken." Ask: what is already established, what is the landmine, and what is the sharper still-open question.
That framing repeatedly returned more than a verdict. It handed us better problems than the ones we brought. Once it caught a dataset whose recording gaps were padded with a frozen test-pattern image — unscreened, that padding would have manufactured our own headline result as an encoding artifact, and it would have been invisible in the summary statistics, because a frozen frame and a genuinely still room are identical to the instrument.
### 2. Pre-registration with kill conditions
Not rigour theatre. Its real function is to make you specify what would change your mind while you are still cheap to persuade.
Kill conditions fired three times. One voided a question that turned out to be malformed — we were about to measure the drift of a quantity that was never present. One cancelled a four-arm study for roughly four GPU-days before it ran. One caught a comparison confounded by content rather than by the thing we claimed to measure.
Write the thresholds as numbers, before you look. If you cannot state a number that would falsify you, you do not yet have an experiment.
### 3. Known-answer tests
The highest-leverage technical practice we adopted.
An instrument that cannot report zero on a zero-signal control should not be permitted to report anything. Ours now runs on a provably-static sequence first, and aborts rather than emitting numbers if it fails.
The worked example is the part worth keeping. Our activity detector's first version used a threshold defined as the median of the quietest decile plus three times its absolute deviation. That is scale-invariant: it adapts to whatever distribution it is handed, so it always lands just above that source's own floor. Every source we measured, real and generated, came back at about 86% active.
The check that killed it: exposure-normalising the input changed the mean frame difference by 64x and moved the reported output by 0.002. A measurement invariant to a 64x change in its own subject is not a measurement.
The best evidence the rebuilt version worked came from an apparent failure. On its known-answer control it reported 3.85% activity where the truth was zero. Every flagged sample fell within one sampling interval of a single frame: a scheduled lighting change we had left inside our own control window. The instrument was right; the test was wrong.
When a control fails, suspect the control first.
### 4. Open the file
Five pipelines exited zero, wrote the expected number of files with plausible timings, and produced garbage. Exit codes, durations, file counts, and even image mean and standard deviation all passed on pure noise.
The only reliable detector was opening a JPEG. Three separate later failures — a metric that scored a visibly broken image higher than the good one, a detector firing on an empty bed, a compositing artifact — were all caught by looking, none by scoring.
### 5. Check your own leftovers before believing any performance claim
Generation ran at about 700 seconds per unit against a 68 second baseline for most of a day. We attributed it to thermal throttling, then memory pressure, then contention, and sized experiments around it.
It was two of our own commands: a directory scan and a listing launched hours earlier, still running at 9 hours 43 minutes and 7 hours 40 minutes. Their tool calls had timed out. The operating system processes had not. Killing them took load from 13.97 to 5.23, and a task that had managed 16 frames in five minutes did 361 in forty-five seconds.
A timed-out call does not kill the process it started.
Grounding as constraints, not as prompt flavour
The pattern that produced the most correction per unit of effort: encode domain truth as invariants that reject invalid outputs, rather than as descriptive text hoping to influence a model.
Two forms. Temporally, a schedule generator where "no patient goes unchecked for more than 2.5 hours" is a hard constraint, and sampled timelines that violate it are rejected and resampled — 7% were. Spatially, a room specification where a headwall must carry oxygen and vacuum outlets, nurse call, three distinct light types, and services on both sides of the bed, because a room with services on one side looks fine to a layperson and obviously wrong to a clinician.
The value is that constraints reject. Prompt text only suggests, and we measured that directly: text invariants held semantic facts such as layout and camera position perfectly, and failed completely on stylistic ones. Output posterised despite "posterized, flat colours, cartoon" sitting in the negative prompt the entire time.
Publish the scorecard, not just the wins
Three of seven sealed predictions wrong. We publish that number alongside the results, because it is the only thing that makes any individual result believable. A project reporting only successes gives a reader no way to calibrate how much to trust the next number.
It also disciplines the writing. It is much harder to round 3.1x up to "confirmed" when the pre-registered weak-support band is written down and public.
The staffing observation, which we did not expect
The highest-leverage contributions were domain knowledge and measurement discipline, not modelling skill.
Three interventions changed the direction of this work, and none of them were technical.
"Real cameras have auto low-light — it needs to be understood, not removed." This turned something we had written off as a defect into a measured, publishable rendering layer, and corrected published advice of ours that was too broad.
"This is an inpatient room, not an operating room." This caught a 13x error in our lighting model. We had exam lighting on for 19.9% of the day, which is a surgical theatre, not a ward. Corrected to 1.5%.
"We own a site with room standards." This produced a lighting standard that then independently confirmed a curve we had measured empirically from a camera. Two unrelated routes to the same fact is the strongest validation that layer has had.
For a small team on this kind of problem: one person who measures well, one with real domain truth, one who reads literature fast. The person who trains models is the least scarce input. The domain expert does not need to be technical. They need to be right about the world, and they will catch the errors that are invisible from inside the code.
Appendix: the literature-check prompt skeleton
The most portable artifact we have. Substitute your claim, keep the structure.
Thorough literature search. Blunt and skeptical — I would rather learn an idea is
taken than waste weeks.
THE CLAIM: <state it precisely, with your measured numbers>
Search specifically for:
1. <the exact mechanism, by its likely technical name>
2. <the adjacent field that may have solved it under different vocabulary>
3. <the standards/codec/classical literature — this is where "novel" ideas
usually already live>
4. <benchmarks or datasets that would let me evaluate, with LICENCES>
5. <anyone who has argued the opposite>
For each: ID/venue/year, one-line summary, SCOOPS / PARTIAL / OPEN.
Finish with:
(a) VERDICT — scooped / partial / open
(b) the 3 most threatening prior works
(c) the sharpest still-open reformulation
(d) the single experiment that would most quickly falsify the ideaItem 3 earns its place repeatedly. Our "novel" chroma restoration idea turned out to be standardised in a video codec. Our anchor-placement idea had a mature counter-argument in the codec I-frame literature. The classical and standards literature is where most apparently-new ideas are already sitting, under a different name.
Item (d) is what converts a check from a verdict into a plan.
What this is not
This is not a method for going faster. Roughly a third of the week went into building instruments, invalidating them, and rebuilding them. The surviving output is smaller and much narrower than what we set out to produce.
The alternative was not more findings. It was the same findings with wrong numbers in them, and no way to tell which.
Code and data
Full write-up and code: huggingface.co/resoajoe/loglens-longvideo-drift