ARC-AGI-3
🥉 Bronze-medal rangeResearch loop · 14 experiments
Two ideas that scored higher on our offline benchmark did worse on the hidden games. Both were rejected on the evidence, not kept on a hunch.
Research + experiment loop
Autoresearch for your own repository. Let your coding agent explore overnight; ResearchForge keeps the evaluation deterministic: one frozen baseline, isolated experiments, hard constraints, repeated validation — and every rejected idea kept with its reason.
$ pip install researchforgeTwo ways in
Papers → landscape → hypotheses → a cited report.
$ researchforge research search$ researchforge research synthesize$ researchforge hypotheses generate$ researchforge report build# or, from Claude Code (no API key):$ /researchforge-start
Scan → research → contract → frozen baseline → unattended loop → validate → ship.
$ researchforge project create --mode improve_repository$ researchforge repo scan .$ researchforge research search && researchforge research synthesize$ researchforge report build$ researchforge contract generate && researchforge contract approve$ researchforge baseline run --n-runs 3$ researchforge autorun --target 0.85 --max-hours 8$ researchforge validate && researchforge ship pr# lost? researchforge status prints the next command
What each step does
arXiv search with deduplication and ranking; a landscape of research directions; hypotheses that each cite their evidence.
Metric, direction, hard constraints (latency, cost, accuracy) and protected paths, approved by you with a typed confirmation.
Measured in an isolated worktree before any change — or the mean of N runs, with the spread recorded — so every later number is comparable.
Each attempt runs in its own git worktree (venv or Docker) from the same baseline commit, through screening, a full benchmark and repeated validation.
Rounds of plan → run → record → re-synthesize over an experiment graph, building on earlier winners and merging independent ones. Stops on stall, target, rounds or hours.
The validated winner becomes a clean branch rebuilt from the baseline, with an engineering report and an optional pull request.
Compute by stage and by outcome, and token spend per project — measured, with unknown prices named rather than shown as zero.
autorun --kaggle tags how far each local metric can be trusted and what is worth Kaggle GPU time; --judgment asks you per experiment.
Leave it running
New · tested on the bundled exampleautorun stops when you close the terminal. auto-run hands the loop to a background monitor (launchd on macOS, a systemd timer on Linux, Task Scheduler on Windows) that checks every 15 minutes and starts a launch when everything allows it: the contract and baseline exist, nothing is paused, no plan or question waits for you, no launch is running, the driver is under its usage limit and today's rounds are under the cap.
approve, or a standing approval you type once. An unattended run can never approve a plan.researchforge ship. No branch, push or pull request happens on its own.$ researchforge auto-run settings --on --driver claude$ researchforge auto-run service$ researchforge auto-run status# plans wait for you: Approve plan in the dashboard, or$ researchforge experiment approve plan-001# the winner stays an experiment until you ship it$ researchforge ship pr
Who drives it
One driver per project, switchable any time. The same driver does the background sensors' AI work, and every round records which driver ran it.
Runs headless on your Claude subscription, so no API key is needed. Its limits are enforced by its allowed tools: only researchforge commands and edits inside the project.
Runs headless through the Cursor CLI. Headless Cursor needs --force, which allows any command, so its limits are instructions only — prefer it on a machine you trust.
Bring your own key: the terminal calls Anthropic, Gemini or OpenAI directly, or a local Ollama model — independent of any editor setup. Records the tokens each round used.
Sign a driver in once from your own terminal (researchforge kaggle driver-login claude) or on the dashboard's Keys & accounts page. Keys are never typed into a chat. Drivers in the docs →
It asks before it acts
Each project can have its own ntfy topic. When a plan waits for you, a high-priority alert arrives; you approve it on the dashboard under the 🔔, or in a terminal. The alert itself is plain text, so nothing is approved from a notification by accident.
plan-002 waits for your approval: … (3 experiment(s)). Approve plan on the dashboard, or researchforge experiment approve plan-002.
The alert text, as the repo auto-run sends it.
Alerts go only to a topic you set; with none, nothing is sent. Notifications in the docs →
One history
On Enveda CASMI 2026 the research loop ran in October and Kaggle Overnight took over in November. The dashboard shows them as one experiment tree: research-loop experiments dotted, overnight rounds solid, every edge from a recorded parent.

Real records
Public Kaggle leaderboard · 15 Nov 2026
Research loop · 14 experiments
Two ideas that scored higher on our offline benchmark did worse on the hidden games. Both were rejected on the evidence, not kept on a hunch.
Research loop LoRA screens, then Kaggle Overnight
Resubmitting identical code moved the public score by 0.05, so one leaderboard number is treated as noisy evidence, not a verdict.
Research loop in Oct, Kaggle Overnight in Nov · one history
In October the public score rose while local MRR@25 fell. In November the two finally moved together (local 0.889 → 0.915, public 0.166 → 0.213). A public pipeline advertised at 0.500 scored 0.479 for us — the record keeps both numbers.
Kaggle Overnight · 40 rounds run by auto-run
17 rounds improved the score; 16 were merges of public outputs, checked for overlap. Our own polishing, neighbour transfer and seed shapes ran 12 times for one gain of 0.0000015. The field moves daily, so a better score can still mean a lower rank.
Autoresearch is Andrej Karpathy’s small open-source loop in which an AI agent edits a training script, runs a five-minute experiment and keeps the change only if validation loss improves — about a hundred experiments overnight on one GPU. The name now covers any keep-or-revert research agent loop.
ResearchForge is an autoresearch-style loop built independently for real repositories and live Kaggle competitions. A contract fixes the metric and hard limits, the baseline is frozen first, each experiment runs in its own git worktree, finalists are re-validated, every rejected idea is kept with its reason, and the winner ships as a clean branch.
It covers the research half too: arXiv search, a landscape of directions and hypotheses that cite their evidence, then experiments that test them. Your coding agent — Claude Code, Cursor — or your own model does the thinking; ResearchForge decides what counts as evidence.
Yes. autorun runs round after round from a terminal, and auto-run hands the loop to a background service so it keeps going with the terminal closed. Each plan still waits for your approval unless you type a standing one, and it never ships on its own.