I ran a two-day AI research sprint. Seven of my results died. The eighth was already on arXiv.
What actually happens when you let an AI agent run experiments — and hold it to real controls.
Joe Cox
The session started badly. A Jetson AGX Orin sitting in my office had filled its root disk to 100%, which broke authentication, which killed a long-running research session mid-sentence. The root cause turned out to be an NVIDIA ethernet driver debug flag someone had left enabled, dumping an idle DMA ring buffer to the kernel log every two seconds. Ten point six gigabytes of a network card saying nothing, repeatedly.
The previous safety net — a logrotate rule capped at 500 MB — had failed in an almost poetic way: logrotate runs daily, the flood produced ~3.5 GB per day, and once a disk is 100% full, logrotate can't run at all, because it can't write the compressed file it's rotating to. The tool designed to prevent the failure was disabled by the failure.
That set the tone for everything that followed.
The thing we were chasing
The research is about very small vision models — around 100,000 parameters, the kind you can run continuously on a camera without a datacenter. Small models fail a lot, and when they fail, you can't tell whether the task was impossible, the hyperparameters were wrong, or you had a bug.
There was an existing finding: a single number, computed from the task's images with no training at all, predicted whether the model would succeed. Rank correlation 0.988 across ten conditions that had been pre-registered before training. Good result. Nobody knew why it worked.
Two days of experiments later, we knew why. The untrained network already preserves — and amplifies — the same signal-to-clutter structure the number measures. The prediction works because the answer is, in a sense, already sitting in the network's random initialization.
Then we watched learning actually happen from the inside, and it isn't gradual. The model sits flat, doing nothing useful, for thousands of steps. Then an internal signal explodes — roughly 100,000× — and afterwards the accuracy moves. Sometimes that never happens at all, and the identical setup with a different random seed just sits there forever.
That's the good part. Here's the rest.
The seven deaths
"Initialization geometry decides which seed succeeds." It doesn't. The seed with the best starting geometry collapsed. Falsified by its own pre-registered control.
"The signal running away is what causes escape." One run had that signal explode to 185 — higher than any successful run — and it failed anyway. It turned out the metric was a ratio, and the denominator was collapsing toward zero. The number went up because the network was degenerating, not because it was learning.
"The compiler solved its last impossible task." This one had already been written into a status report before the crash. A single seed had rescued a task that had resisted everything. Running four more seeds showed it was luck: the "solution" worked on 2 of 6 seeds. Roughly a coin flip, reported as a discovery.
"The learn/fail boundary is a hard wall." Mostly it's a distribution of waiting times. One run "failed" at 8,000 steps, "failed" at 16,000 steps, and succeeded at 17,600. Most of our confident failures were just impatience.
"Upstream signals always predict, readout signals never do." A clean structural law, confirmed beautifully on two tasks — and flatly contradicted by the third. Pre-registration said all three had to agree. They didn't.
"We can steer training by pushing on the signal." This one looked spectacular: adding a term that pushed the internal signal made models learn 71% faster, on every seed, with no accuracy cost. Then the control arm came back — a deliberately meaningless signal, pushed the same way, got two-thirds of the same speedup. Most of the effect was just "extra pressure helps," not our mechanism.
"Suppressing the signal delays learning." Beautiful single-seed result: +1,800 steps of delay. Across five seeds, the mean was −560 steps. In the wrong direction. The clean result was noise.
There's a pattern here, and it's not subtle: every single result that looked exciting on one seed died on five. I flagged that caveat, out loud, most times — and still got fooled by it repeatedly. Knowing about a bias is not the same as being immune to it.
The eighth death, which was the useful one
Late in the session we found something that genuinely seemed new: a dissociation. The internal variable that reliably predicts the learning transition — several thousand steps early, with zero false alarms — is not the variable that controls it. Suppressing a different variable (feature norm) blocked learning completely, 7 out of 7 runs, while leaving memorization perfectly intact. The sensor and the actuator are different things.
I thought that was the best result of the two days. Before writing it up, I spent one hour searching the literature.
Constraining weight norm to eliminate this phenomenon: published in 2023.
The explicit predictive-variable-vs-causal-variable dissociation, with a closed-form timing law and experiments on the same tasks I'd used: already on arXiv.
Predicting the transition from internal mechanisms before validation accuracy moves: also already on arXiv, framed as an "interventional and predictive framework."
Independently rediscovering three published results is a decent sign your methodology works. It is not a paper.
That hour of searching was the highest-value hour of the entire session. It cost zero GPU time and prevented a submission that would have been desk-rejected.
What actually survived
Something did, and the shape of it is instructive. What survived is the part that wasn't about chasing the exciting phenomenon:
Why the trainless predictor works — the mechanism, confirmed across two architectures and two physically different scenes.
Where it stops working — and this is the part I've grown fondest of. On ordinary image classification the predictor doesn't just degrade, it inverts (correlation −1.00). We tried six different ways to patch it. All failed. The reason is structural: for natural images, the "clutter" you'd want to divide out is the signal the network learns from. You cannot separate them with any fixed measurement.
That last one is a negative result, and it's the most solid thing we produced. It draws a real line: trainless prediction of learnability is possible exactly when the discriminative signal already exists in a fixed representation. Inside that line, it works well. Outside it, no amount of cleverness helps.
The postscript that actually mattered
After the literature check I did the unglamorous thing: re-ran the two surviving headline results with five random seeds each instead of one or two. Sixty runs, overnight.
Both correlations went up — 0.884 → 0.932, and 0.82 → 0.850.
That's the tell. When an effect is real, cleaning up the noise makes it stronger. When it isn't, it evaporates — which is exactly what happened to the other seven. One configuration made the point perfectly: a single seed had reported 73% accuracy, and across five seeds it averaged 39%. That one lucky run had been quietly dragging the correlation down, and removing it improved the result.
Seven results died to more seeds. Two got better. Same treatment, opposite outcomes — which is the only reason to trust the two.
One last irony
The whole point of releasing this was to invite outside replication. Before publishing, I ran the replication kit the way a stranger would — from a clean directory, on a fresh machine path — and it broke three times in the first minute. Wrong package layout. A missing timestamp file. And the fatal one: the synthetic clock was spaced one second per frame, while the task samples memory at ages up to twenty-four hours, so every single run died on low >= high.
None of it had ever surfaced, because we always ran from the source tree where the imports happened to resolve. The kit built to let others check our work was itself unchecked.
The actual lesson
The instinct, when an agent can run experiments continuously, is to measure progress in results per hour. This session produced a lot of results per hour. Nearly all of them were wrong.
What made the remainder trustworthy wasn't the experiments. It was the controls, the pre-registration, and the willingness to run the one test most likely to destroy the thing you're excited about. The single most productive act across two days was a literature search that deleted half the output.
If you're doing this kind of work — with an agent or without one — the useful metric isn't how much you produced. It's how hard you tried to kill it, and what was still standing afterward.
Everything here is public — the code, the pre-registrations, the failures, and a one-command script so you can try to break it on your own machine: huggingface.co/resoajoe/loglens-learnability. Please do. A disagreement would be the most useful thing anyone could send me.