Age of Autoresearch

Andrei Filatov · 2026 · ~15 minutes

Chapter 1

A Lazy Evening

It was a regular weekday evening, and I was frankly too lazy to test hypotheses by hand.

Not lazy in the «tired, want to sleep» sense — a specific, engineering kind of lazy: I had a backlog of fifteen or so ideas that needed to be run, checked, filtered for garbage, repeated. Work I'd done dozens of times, every step memorized. And that evening I just didn't have it in me to sit down and click through the same thing by hand one more time.

For the past month and a half I'd been living without weekends, trying to become a nominal 10x engineer — doing way more without scaling up my own cognitive load. Sounds nice; in practice it meant I was stuffing agents into everything and hoping it would somehow speed itself up.

The first attempt was bold and dumb at the same time: I just fired off ten different tasks in parallel, agent on agent. The logic was simple — if one agent handles one task fine, ten agents will handle ten tasks at once, and I get my evening back. In practice, one of the ten almost always broke. And the moment it broke, all of my attention got sucked in. Effectively I spent more time cleaning up the mess than if I'd just done them one by one myself. The aftertaste was nasty: I'd supposedly split the work into parallel streams and ended up with plain parallel chaos.

The second attempt was smarter. I figured — okay, if an agent without an explicit plan produces nonsense, I'll give it a plan. I wrote out the agenda upfront: what to do in what order, which corner cases to watch, what counts as success. The agent behaved noticeably cleaner — fewer surprises, more cases covered. Better. But still kind of crappy — I was still sitting next to it cleaning up the loose ends, there were just fewer of them. It felt like I'd improved the supervision process rather than gotten rid of the need to supervise.

And so on that lazy evening, instead of trying for a third time to steer the agent better by hand, I did a lazy thing. I just described the hypothesis-generation pipeline in text — the way I usually come up with what to try next — and asked Gemini to evaluate the result of each attempt. Not to direct the process, not to decide the next step, but specifically to judge: here's a hypothesis, here's the result, is it good enough or not. I handed all of that to the agent and went to bed, because I was honestly too lazy to sit there and watch.

In the morning I opened the log and found that 40 hypotheses had been tested overnight.

Agent terminal: Pursuing goal (1d 1h 41m)

This is what it looks like from the outside: a line in a terminal and a counter that ticks while you sleep. The screenshot isn't mine — it's from sankalp's write-up, which is what the next chapter is about.

Not all forty were useful — some were outright garbage, some repeated things I'd already tried. But that didn't matter. All night I did nothing by hand — and ended up with more validated options than I'd have produced myself in a week with a laptop. And it became clear: the point wasn't to steer the agent better. The point was who evaluates the result.

It worked once, overnight, on one specific task. What was left was to understand what exactly had worked — and, oddly enough, what helps figure that out isn't my own experience, but the fact that someone else had stumbled onto almost the same thing a bit before me.

Chapter 2

It Already Had a Name

What happened that night wasn't an invention. It already had a name: autoresearch. And a couple of loud precedents before me.

First I stumbled on sankalp's post about QR decomposition. A classic problem: factor a matrix fast. He didn't touch the kernels by hand — he set Codex loose on the problem as an autonomous agent. It kept a beam of candidates, spawned sub-agents for profiling, math, and idea generation, and pruned the dumb branches as it went. The result — 232x speedup on batched QR and 12th place out of 183 participants, even though the author had no professional GPU experience.

Beam search diagram: candidate branches, merge points, best branch

sankalp's beam of candidates: branches live in parallel, the weak ones die off, the strong ones merge. Nobody picks the winner by hand — the criterion picks it. Diagram from sankalp's post.

And while reading it, it clicked: it's not about the agent and not about the prompt. The expert used to invest in the solution itself: features gathered by hand, heuristics invented out of their head, kernels written natively. Knowing how the problem is put together was the job. Now you barely need that knowledge directly.

You need to know how to build a loop that finds the solution itself: benchmark, oracle, stopping criterion. You don't write the kernel — you build the loop that searches over it. The same bitter lesson, except not about model architecture but about what the human does.

Two direct conclusions follow. First: you can physically do less work. That exact routine of hypothesis testing I was too lazy to do in the evening gets delegated — the agent grinds through it at night while I sleep. Second: the work gets done better. Not "the same thing, faster", but better — the oracle searches a wider space of solutions in the same time than a single person, and finds options I would simply never have reached on my own.

And sankalp isn't the only one. Karpathy built autoresearch: a 630-line script, an autonomous loop — the agent reads the training code, proposes a change, runs a five-minute run, measures the improvement, commits or rolls back, and goes again. Overnight — hundreds of experiments, not a single click. DeepMind's AlphaEvolve — the same idea at a bigger scale: Gemini plus an evolutionary framework plus automatic evaluators. They improved solutions for 20% of 50 open math problems and sped up Google's datacenters, chip design, and AI training.

Autoresearch progress chart: 83 experiments, 15 kept improvements

One night in Karpathy's repo: 83 experiments, 15 of them kept improvements. The grey dots are what the loop rejected on its own. Exactly the proportion of junk I saw in my own log that morning. Chart from karpathy/autoresearch.

Karpathy's README has a line that describes this shift more precisely than I managed to myself:

You're not touching any of the Python files like you normally would as a researcher. Instead, you are programming the program.md Markdown files that provide context to the AI agents and set up your autonomous research org.

The file the agent edits and the file the human edits are different files. All of the researcher's work moved into the second one.

Different names, different implementations, different scale. But one and the same pattern: expertise moves from "find the solution" to "build the system that finds the solution".

And this is still a frame, not an answer. To see where the frame's edges are, what helped me wasn't other people's cases but two of my own. One went smoothly — there's almost nothing to tell there. The second is the one where I had to argue with myself: I was sure I knew better.

Chapter 3

I Knew Better

For two weeks I was hand-building filters for a dataset.

The task looks simple — decide which samples to drop from training and which to keep. In practice it's a mix of engineering and curation: you read papers, look at the data, form a hypothesis, test it, throw it out, start over. I thought I understood this task. Armed with my ego and the knowledge of how it should be done, I spent two weeks pouring intuition and paper-reading into manual filters.

And then I did exactly what I did on that lazy evening — built a benchmark through Gemini as the oracle and pointed a VLM at it, zero-shot. Just to see whether it would beat my manual work. No heuristics, no hints, no involvement from me.

The first attempt was already better than two weeks of manual work. The second was better still.

It wasn't some cosmic gap. Not "the agent invented a filter I could never have come up with." The result was just objectively better, and the time it took — a couple of runs versus two weeks of my life. And that's when it genuinely stung — not because the method worked, but because I'd spent two weeks arguing with myself over a decision the automation brute-forced in a couple of attempts.

Before AlexNet, features were built by hand. Engineers came up with heuristics, teased out features, tuned pipelines — and that was expertise backed by years of work. Neural networks automated it: features started to be learned, and the work shifted to architecture and data. Now the researcher's expertise itself is being automated. You don't need to know the domain perfectly to make good decisions — it's enough to build an oracle that knows what counts as a good one, and let the agent loose. That's the bitter lesson on your own skin. Two weeks, beaten by a single night.

The second case went down much more calmly. Maybe the stakes were different — or maybe after the first one I stopped arguing with the oracle.

Chapter 4

No Silver Bullet

The second case was about speeding up training. No inner drama — just a task where frontier models are expensive and can be overkill, but if you point them at optimizing inference or training time, they save money and time. I left the model running for two days and got a 30% speedup. Pushing further got a lot harder, and there was no point spending more time on it. Smooth, mundane, no ego story.

But even on a smooth case — and even more so afterwards — pitfalls kept surfacing, the kind that mean you shouldn't treat the method as a silver bullet.

The first one — loss parity. I forgot to add a check to the oracle that the loss doesn't drift apart. The agent honestly reported "done, there's a speedup" — while in fact the loss was NaN everywhere. The model formally got faster but stopped learning. Once I added the loss parity check, the problem went away. A trivial thing that I still missed, because I trusted the agent too much to decide what counts as success.

The second one — an oracle that wasn't detailed enough. This surfaced later, on the task of removing text from images. I was testing different models but validating the result on the full image. Small artifacts — leftover letters, a half-erased character — I'd notice with my eyes, but the oracle wouldn't. It considered the text removed, and the solution it called best actually left garbage behind. As soon as I added a crop of the region where the text should have disappeared to the oracle — validating not the whole image but the fact of removal itself — the best solution turned out to be a completely different pipeline. The previous winner was simply exploiting the oracle's blindness.

One KernelBench-Mega session: speedup over the reference versus tokens spent

What a well-run session looks like — a breakdown of a single run on KernelBench-Mega. For the first 64% of the time there's no code at all: measuring the baseline, microbenchmarking barriers, deriving the roofline. The first working kernel appears at the 224k token mark. And near the end — the point I'm showing this for: "finer split-K regresses → measured, reverted". The hypothesis didn't work, the measurement caught it, rolled back.

The author of the benchmark described the same run in a single sentence:

It spent 64% of the session in silence timing the baseline, microbenchmarking grid barriers, deriving a ~29x bytes/token roofline. The one regression it tried (finer split-K) it measured and reverted instead of rationalizing.

Measured and reverted instead of rationalizing — that's what a working oracle is. My NaN happened precisely because there was nothing to measure with, and rationalizing was all the agent had left.

The general lesson is simple and unpleasant: building autoresearch is a delicate task where the time should go exactly into designing the oracle and the loop. The time doesn't go into steering the agent or hunting for a magic prompt — it goes into honestly and thoroughly describing what counts as a good solution. Treating this as an automatic silver bullet means sooner or later getting a NaN where your loss used to be.

And even when the oracle is built right and the loop runs smoothly, one variable remains that you can't design away — the model itself inside the loop. I ran this whole tuned process on different models, and it turned out that with the same oracle and the same task, the result depended not only on what I built, but also on who I entrusted it to.

Chapter 5

Not Raw Power, but Intuition

For the code speedup task I had three models: Fable 5, Claude Opus 4.8, and GPT-5.5. Same task, same oracle.

Fable 5 — I just described the task and it worked. The agent produced hypotheses that made sense, tested them, discarded the dumb ones. All I had to do was nod along at the sensible steps. Claude Opus 4.8 proposed a solution that hit OOM and kind of gave up right there — it didn't correct itself, didn't try another path. GPT-5.5 got there in the end, but I had to explicitly point it where to look, otherwise it just spun its wheels.

The difference isn't in the ability to execute steps. All three can read code, write code, run it, measure. The difference is domain intuition — how deeply the understanding of what even makes sense to try in this task is baked into the model. Fable 5 had that understanding. The other two didn't.

I compensate for the models' lack of intuition by stuffing context into the prompt — blogs, guides, other people's cases. But it's a limitation, and an annoying one: to know how to help the model, I have to already know the topic well myself. So the exact expertise I thought I'd automated away has to sit in my head anyway, so I can pull it out in chunks and feed it to the agent.

And it's not just my observation. The same pattern shows up in independent benchmarks. KernelBench-Mega is a benchmark for whole-block megakernels, where you have to fuse an entire model block into a single kernel instead of optimizing individual ops. Every previous model "won" on it only via a multi-kernel Triton pipeline — technically passes the test, but there's no honest fusion, it's a workaround. And only Fable 5, per the benchmark author's tweet, wrote the first real megakernel. A direct parallel to my pitfall from chapter four: a badly designed oracle allowed weaseling until someone did it honestly. AutoKaggle — the same pattern outside GPU: a multi-agent framework for Kaggle competitions, where the oracle is the competition itself and its tests, and across eight competitions it reached a 0.85 validation submission rate and 0.82 comprehensive score, at human level.

Elliot Arledge's tweet about the first real megakernel

The benchmark author lists who "won" before and by how much: Opus 4.8 — 14.4x, GLM-5.2 — 11.1x, GPT-5.5 — 4.3x, Sonnet 5 — 4.0x. All of them through a multi-kernel Triton pipeline that doesn't pass the authenticity gate. Full tweet.

So the best tool isn't the most powerful model, but the model with the right intuition for the specific task. And the more carefully the oracle is constructed, the clearer it becomes who has that intuition and who has to have it compensated with context.

From here this moves toward recursive loops — cycles where the agent improves not the model weights, but the system itself: the code, the pipeline, the artifact. Each improvement makes the next one easier. I hope tools will show up soon that you can just use. For now, every time I have to pick which loop fits which task, and it only works in spots.

So what does this say about my own role, if intuition — the only thing that's still mine — is becoming a commodity you can just stuff into a prompt?

Chapter 6

What Remains

To the question "will AI replace us" I now have a boring answer: for part of the work it already happened. Interns and juniors used to draw plots and check simple hypotheses — now an agent does all of that.

But that's about juniors. With me it's more complicated.

Three years ago I spent three days digging through the PyTorch Lightning source to figure out how checkpointing works. Three days — and I knew it at a level where I could explain it to anyone and fix anything. Now an agent does the same in an hour. Frees you from routine — sure. But it takes something in return. That sense of authorship you used to get from digging through the source by hand — "I understood this, I did this" — it's gone. In fact I did nothing, it was all the agents.

It's like moving from a senior role to a manager role. The story is exactly the same as in a regular career: you become a team lead — and you barely do anything with your own hands anymore. Before, you sat there and solved things yourself; now you watch others solve them and make sure they solve them in the right direction. The output is bigger than what you'd have done alone. But none of it is yours, made by your own hands. The position is less dopamine-rich: the old rush from solving something yourself won't be there.

And I thought I could stop there. You still need the manager, after all. Not because he's smarter than the people he manages, but because he knows where to point them — he has domain intuition. That was my conclusion in the previous chapter: the difference is not in power, it's in intuition. Where the model lacks intuition, I compensate by stuffing context into the prompt. Intuition was the last thing that was still mine. And then, over a couple of months, a few stories piled up that don't fit this picture.

The Jacobian Conjecture — a classical conjecture in algebra. If a polynomial map has a Jacobian that's a nonzero constant, the map is invertible. Recently — a counterexample in dimension three. And it wasn't found by a group of mathematicians. It was found by Fable 5, while I was watching the World Cup final.

Terence Tao, a Fields Medalist, wrote a post to digest it. He explains, retrospectively and geometrically, what the model found on its own. And here's what struck me: Tao sits down and counts how unlikely this is. A degree-seven polynomial. The Jacobian has a priori up to 1329 nonzero coefficients — against 360 degrees of freedom of a general polynomial map of the same degree. 1329 equations that must all vanish simultaneously, in a space of 360 variables. You don't find that by brute force — the chance of hitting the right point is zero. Which means Fable wasn't searching blindly. It saw the structure.

Paragraph from Terence Tao's post counting 1329 equations against 360 degrees of freedom

That very paragraph in Tao's post. The last sentence is the death sentence for brute force: "finding such a polynomial looks highly unlikely to be located by brute force".

At the end of the post there's a disclaimer that five years ago would have looked insane in a Fields Medalist's text:

I used an AI chatbot to discuss various aspects of this problem and to confirm several of the calculations made here.

Seeing where the solution lies, in a space where naive search is useless. That's exactly what I considered mine.

Back in university I was told there are different levels of knowing the material. First stage — you learn the terms: "Jacobian", "polynomial", "invertible map". You know they exist, but you don't understand them. Second — you know all the material: definitions, theorems, proofs. But you can't apply it — on the exam you fail a problem because it's slightly different from the template. And there's a third stage, which is what actually knowing the material means — intuition. Not just knowing that a fact exists, but seeing when to apply it, and recognizing the same structure in a new situation.

Models went through the same path. First, stochastic parrots — guessing the next word. Then they learned the material, they know the facts, but they break on anything slightly new. And now — the third stage. They see structure, they find solutions that weren't in the training data. A counterexample to a conjecture is not a reproduction of something the model saw, it's new mathematics. And there's no way to cheat here: a counterexample either works or it doesn't — in pure math there's no badly designed oracle you can game.

Okay, fine, suppose the model has intuition. But you still need the manager who points the way — that's what I was holding on to. The second story is exactly about this, and it's closer to me, because there you can see what the human did.

Mathematician Dmitry Rybin took the Dinitz–Garg–Goemans conjecture — a network flow problem, open since the nineties: can a fractional solution be made unsplittable without raising the cost and without overloading any edge by more than the size of the largest shipment. The counterexample was found by GPT-5.6 Pro: a graph with seven nodes, three shipments, fractional cost 58, while any unsplittable routing costs at least 60.

Counterexample to the Dinitz–Garg–Goemans conjecture: a seven-node graph with flows, costs and demands

The entire counterexample in full: seven nodes, one source s, three sinks with demands of 15, 10, and 15. Blue numbers are flow, red ones are edge cost. For thirty years nobody could either construct such a graph or prove it doesn't exist. Image from Rybin's tweet.

The interesting part here isn't the model, it's the prompts. There were four of them, under sixty words total: "construct a counterexample", "you should do a breakthrough", "it's enough of partial results, let's finish". Three runs in a row, an hour and a half each, the model kept returning partial results, and the human just asked it to keep going. There's zero domain intuition in those words. There's direction and a sense of when the result is good enough — which is exactly what I kept for myself as the manager. Turns out that job takes about a minute.

The same skill transfers to my work too. My personal labor — what even is it? A set of patterns I've built up over the years. "Seen this bug — I know where to look". "This architecture breaks like this — work around it like that". "This hypothesis won't work because this metric is lying". These are all patterns. And they can be learned. And a network that found a solution in a space of 1329 equations — it will learn my patterns too. It's only a question of price, and that price is cheaper every month than it was the month before.

And here's the strange thing: the fun hasn't gone anywhere. I still open the log in the morning and get sucked into whatever happened overnight, even though I did nothing with my own hands. Which means the fun was never about "I solved it myself" — it was about the problem moving forward. And now it moves a lot more: most of my model training I pull off alone, where before it took a team.

So I'm not quitting — I'm building the next loop. But to the question of where my value is now, I don't have an answer.

Sources