Short answer: autoresearch works on Kaggle, but not as-is. Karpathy's loop trusts one local metric, one GPU and one file. A live competition has a hidden test set, a noisy leaderboard, a daily submission limit and a field that moves every day. We ran an autoresearch-style loop on four live competitions — 136 recorded rounds, 94 of them launched with no chat open — and this is what had to change, and what it got us.
What autoresearch is
Andrej Karpathy's autoresearch is a deliberately small loop: an AI agent edits a training script, trains for five minutes on one GPU, and keeps the change only if validation loss improves; otherwise it reverts. At about a dozen experiments an hour, that is roughly a hundred overnight. The idea — keep or revert, against one honest number, all night — has since been copied into Claude Code skills, forks and research agents of every size.
Why Kaggle breaks the simple loop
- ▸The local metric is not the score. On Enveda CASMI 2026 our local MRR@25 fell while the public score rose, four submissions in a row. A loop that trusts local loss alone would have reverted the changes that helped.
- ▸The leaderboard is noisy. On the Gemma 4 Developer Agent the same file, byte for byte, scored 0.12 and then 0.17. A keep-or-revert rule needs a noise floor.
- ▸Submissions are scarce. Kaggle allows a few a day, so the loop cannot ask the leaderboard about every idea.
- ▸The field moves. On Squares in Squares our score improved almost daily while our rank fell from 174 to 401, because public notebooks kept getting better.
- ▸The compute is not yours. Code competitions run on Kaggle; others need Kaggle GPU quota, Colab or a rented GPU, each with its own limits.
What we changed
| Autoresearch | Kaggle Overnight (ResearchForge) | |
|---|---|---|
| What it changes | One training script | Any code a round needs, proposed by Claude Code, Cursor or your own model |
| Keep rule | Validation loss went down | Beats a frozen baseline and every earlier round; unscored rounds go to Kaggle |
| Where it runs | One local GPU, five minutes | Kaggle GPU, this machine, Colab, SSH or a cloud GPU, with a round time limit |
| Ideas | The agent’s own | arXiv, public notebooks checked hourly, your directions and its own results |
| Failures | Reverted | Kept with the reason, so the same idea is not retried |
| Unattended | A terminal left open | A background service; questions reach your phone |
| Submitting | Not applicable | Only with your Approve & submit, checked against the daily limit |
What it got us
| Competition | Rank | Top | Rounds | Standing |
|---|---|---|---|---|
| ARC-AGI-3 | 313 / 4,926 | 6.4% | 14 | 🥉 Bronze-medal range (public) |
| Squares in Squares | 401 / 2,086 | 19.2% | 46 | Playground, no medals; top 10% is 22.1528 |
| Enveda CASMI 2026 | 1,421 / 5,305 | 26.8% | 52 | 0.008 below the bronze line |
| Gemma 4 Developer Agent | 2,292 / 5,772 | 39.7% | 24 | Bronze line 0.20 |
Public leaderboard, 15 November 2026. Medals are decided on the private leaderboard, so these are standings, not medals. The full numbers are in our November standings.
Five lessons for any autoresearch loop
- ▸Measure the noise floor first. Run one thing twice before you trust a gain.
- ▸Check what your local metric holds out. It misled us on ARC-AGI-3 and on Enveda in October, and agreed with the leaderboard on Enveda in November.
- ▸Keep the failures. 47 of our 136 rounds failed or were rejected; each reason stopped a repeat.
- ▸A failed run is not a result. 16 Gemma rounds never produced a score because Colab sessions dropped; none of them judged an idea.
- ▸Keep a human on the irreversible step. The loop ran 94 rounds alone; it never submitted once without a click.
Other autoresearch-style tools
AIDE searches a tree of code solutions and is a common scaffold on MLE-bench, OpenAI's benchmark of 75 past Kaggle competitions run offline. The AI Scientist aims at the whole research workflow, up to a paper. Community autoresearch skills for Claude Code package Karpathy's loop for any measurable goal. Kaggle Overnight differs in where it works: live competitions, your own account and compute, and a record you can check rather than one final score.
Try it tonight
pip install researchforge, then /overnight <competition-url> in Claude Code or Cursor, or turn on auto-run and close the editor. The step-by-step guide is in the Kaggle Overnight docs; for your own repository, see the research loop.