← All posts
GuideEvidence from ARC-AGI-3, Enveda CASMI 2026, Squares in Squares and Gemma 4

Autoresearch for Kaggle: What 136 Overnight Rounds on Four Live Competitions Taught Us

Karpathy’s autoresearch loop on live Kaggle competitions: what had to change, a medal-range result, 136 recorded rounds and the lessons, failures included.

4
Live competitions
136
Rounds on record
Bronze range
Best standing
94
Launched unattended

Manas Joshi·November 15, 2026·9 min read

Short answer: autoresearch works on Kaggle, but not as-is. Karpathy's loop trusts one local metric, one GPU and one file. A live competition has a hidden test set, a noisy leaderboard, a daily submission limit and a field that moves every day. We ran an autoresearch-style loop on four live competitions — 136 recorded rounds, 94 of them launched with no chat open — and this is what had to change, and what it got us.

What autoresearch is

Andrej Karpathy's autoresearch is a deliberately small loop: an AI agent edits a training script, trains for five minutes on one GPU, and keeps the change only if validation loss improves; otherwise it reverts. At about a dozen experiments an hour, that is roughly a hundred overnight. The idea — keep or revert, against one honest number, all night — has since been copied into Claude Code skills, forks and research agents of every size.

Why Kaggle breaks the simple loop

  • ▸The local metric is not the score. On Enveda CASMI 2026 our local MRR@25 fell while the public score rose, four submissions in a row. A loop that trusts local loss alone would have reverted the changes that helped.
  • ▸The leaderboard is noisy. On the Gemma 4 Developer Agent the same file, byte for byte, scored 0.12 and then 0.17. A keep-or-revert rule needs a noise floor.
  • ▸Submissions are scarce. Kaggle allows a few a day, so the loop cannot ask the leaderboard about every idea.
  • ▸The field moves. On Squares in Squares our score improved almost daily while our rank fell from 174 to 401, because public notebooks kept getting better.
  • ▸The compute is not yours. Code competitions run on Kaggle; others need Kaggle GPU quota, Colab or a rented GPU, each with its own limits.

What we changed

AutoresearchKaggle Overnight (ResearchForge)
What it changesOne training scriptAny code a round needs, proposed by Claude Code, Cursor or your own model
Keep ruleValidation loss went downBeats a frozen baseline and every earlier round; unscored rounds go to Kaggle
Where it runsOne local GPU, five minutesKaggle GPU, this machine, Colab, SSH or a cloud GPU, with a round time limit
IdeasThe agent’s ownarXiv, public notebooks checked hourly, your directions and its own results
FailuresRevertedKept with the reason, so the same idea is not retried
UnattendedA terminal left openA background service; questions reach your phone
SubmittingNot applicableOnly with your Approve & submit, checked against the daily limit

What it got us

CompetitionRankTopRoundsStanding
ARC-AGI-3313 / 4,9266.4%14🥉 Bronze-medal range (public)
Squares in Squares401 / 2,08619.2%46Playground, no medals; top 10% is 22.1528
Enveda CASMI 20261,421 / 5,30526.8%520.008 below the bronze line
Gemma 4 Developer Agent2,292 / 5,77239.7%24Bronze line 0.20

Public leaderboard, 15 November 2026. Medals are decided on the private leaderboard, so these are standings, not medals. The full numbers are in our November standings.

📌
Honest attribution: our biggest jumps came from strong public work — a public notebook on ARC-AGI-3, a public pipeline on Enveda, public packings on Squares. The loop's job was to find it, check it, keep what held up and record what did not. That is still autoresearch; it is just honest about where the ideas came from.

Five lessons for any autoresearch loop

  • ▸Measure the noise floor first. Run one thing twice before you trust a gain.
  • ▸Check what your local metric holds out. It misled us on ARC-AGI-3 and on Enveda in October, and agreed with the leaderboard on Enveda in November.
  • ▸Keep the failures. 47 of our 136 rounds failed or were rejected; each reason stopped a repeat.
  • ▸A failed run is not a result. 16 Gemma rounds never produced a score because Colab sessions dropped; none of them judged an idea.
  • ▸Keep a human on the irreversible step. The loop ran 94 rounds alone; it never submitted once without a click.

Other autoresearch-style tools

AIDE searches a tree of code solutions and is a common scaffold on MLE-bench, OpenAI's benchmark of 75 past Kaggle competitions run offline. The AI Scientist aims at the whole research workflow, up to a paper. Community autoresearch skills for Claude Code package Karpathy's loop for any measurable goal. Kaggle Overnight differs in where it works: live competitions, your own account and compute, and a record you can check rather than one final score.

Try it tonight

pip install researchforge, then /overnight <competition-url> in Claude Code or Cursor, or turn on auto-run and close the editor. The step-by-step guide is in the Kaggle Overnight docs; for your own repository, see the research loop.

Try it on your next run

Every number in this post came from the ResearchForge record. Yours can too.