Research + experiment loop

Available now

From a question or a metric to a validated result.

Autoresearch for your own repository. Let your coding agent explore overnight; ResearchForge keeps the evaluation deterministic: one frozen baseline, isolated experiments, hard constraints, repeated validation — and every rejected idea kept with its reason.

$ pip install researchforge

Two ways in

Explore an idea, or improve a repository.

Explore a research idea

Papers → landscape → hypotheses → a cited report.

explore
$ researchforge research search
$ researchforge research synthesize
$ researchforge hypotheses generate
$ researchforge report build
 
# or, from Claude Code (no API key):
$ /researchforge-start

Improve a repository

Scan → research → contract → frozen baseline → unattended loop → validate → ship.

improve
$ researchforge project create --mode improve_repository
$ researchforge repo scan .
$ researchforge research search && researchforge research synthesize
$ researchforge report build
$ researchforge contract generate && researchforge contract approve
$ researchforge baseline run --n-runs 3
$ researchforge autorun --target 0.85 --max-hours 8
$ researchforge validate && researchforge ship pr
 
# lost? researchforge status prints the next command

What each step does

The scientific method, as a CLI.

Literature, grounded

Available now

arXiv search with deduplication and ranking; a landscape of research directions; hypotheses that each cite their evidence.

A contract before anything runs

Available now

Metric, direction, hard constraints (latency, cost, accuracy) and protected paths, approved by you with a typed confirmation.

Frozen baseline

Available now

Measured in an isolated worktree before any change — or the mean of N runs, with the spread recorded — so every later number is comparable.

Isolated experiments

Available now

Each attempt runs in its own git worktree (venv or Docker) from the same baseline commit, through screening, a full benchmark and repeated validation.

autorun

Available now

Rounds of plan → run → record → re-synthesize over an experiment graph, building on earlier winners and merging independent ones. Stops on stall, target, rounds or hours.

Ship with evidence

Available now

The validated winner becomes a clean branch rebuilt from the baseline, with an engineering report and an optional pull request.

What it cost

Available now

Compute by stage and by outcome, and token spend per project — measured, with unknown prices named rather than shown as zero.

Kaggle-aware mode

Available now

autorun --kaggle tags how far each local metric can be trusted and what is worth Kaggle GPU time; --judgment asks you per experiment.

Leave it running

New · tested on the bundled example

The loop keeps going with no terminal open.

autorun stops when you close the terminal. auto-run hands the loop to a background monitor (launchd on macOS, a systemd timer on Linux, Task Scheduler on Windows) that checks every 15 minutes and starts a launch when everything allows it: the contract and baseline exist, nothing is paused, no plan or question waits for you, no launch is running, the driver is under its usage limit and today's rounds are under the cap.

  • Approval stays yours. Each plan waits for Approve plan, where you type approve, or a standing approval you type once. An unattended run can never approve a plan.
  • It never ships. The winner stays an experiment until you run researchforge ship. No branch, push or pull request happens on its own.
  • Waits out usage limits. If the driver hits a limit, the monitor reads the reset time, tells you once and waits.
unattended
$ researchforge auto-run settings --on --driver claude
$ researchforge auto-run service
$ researchforge auto-run status
 
# plans wait for you: Approve plan in the dashboard, or
$ researchforge experiment approve plan-001
# the winner stays an experiment until you ship it
$ researchforge ship pr

Who drives it

Your coding agent, or your own key.

One driver per project, switchable any time. The same driver does the background sensors' AI work, and every round records which driver ran it.

Claude Code

Used in real runs

Runs headless on your Claude subscription, so no API key is needed. Its limits are enforced by its allowed tools: only researchforge commands and edits inside the project.

Cursor agent

Tested

Runs headless through the Cursor CLI. Headless Cursor needs --force, which allows any command, so its limits are instructions only — prefer it on a machine you trust.

BYOK

Tested

Bring your own key: the terminal calls Anthropic, Gemini or OpenAI directly, or a local Ollama model — independent of any editor setup. Records the tokens each round used.

Sign a driver in once from your own terminal (researchforge kaggle driver-login claude) or on the dashboard's Keys & accounts page. Keys are never typed into a chat. Drivers in the docs →

It asks before it acts

Decisions reach your phone. You make them on the dashboard.

Each project can have its own ntfy topic. When a plan waits for you, a high-priority alert arrives; you approve it on the dashboard under the 🔔, or in a terminal. The alert itself is plain text, so nothing is approved from a notification by accident.

plan-002 waits for your approval: … (3 experiment(s)). Approve plan on the dashboard, or researchforge experiment approve plan-002.

The alert text, as the repo auto-run sends it.

What else it sends

  • After each launch: what ran and the best result so far.
  • When it stops itself: a pause, a usage limit with its reset time, or a launch over its time limit.
  • Every morning: a daily summary, and new arXiv papers on the project's topic.

Alerts go only to a topic you set; with none, nothing is sent. Notifications in the docs →

One history

Both loops write to the same tree.

On Enveda CASMI 2026 the research loop ran in October and Kaggle Overnight took over in November. The dashboard shows them as one experiment tree: research-loop experiments dotted, overnight rounds solid, every edge from a recorded parent.

localhost:7842
One experiment tree holding research-loop experiments and Kaggle Overnight rounds in the ResearchForge dashboard

Real records

Four Kaggle competitions, every result on file.

Public Kaggle leaderboard · 15 Nov 2026

ARC-AGI-3

🥉 Bronze-medal range
313 / 4,926 · top 6.4%

Research loop · 14 experiments

Two ideas that scored higher on our offline benchmark did worse on the hidden games. Both were rejected on the evidence, not kept on a hunch.

Gemma 4 Agent

Bronze line 0.20
2,292 / 5,772 · top 39.7%

Research loop LoRA screens, then Kaggle Overnight

Resubmitting identical code moved the public score by 0.05, so one leaderboard number is treated as noisy evidence, not a verdict.

Enveda CASMI 2026

0.008 from bronze
1,421 / 5,305 · top 26.8%

Research loop in Oct, Kaggle Overnight in Nov · one history

In October the public score rose while local MRR@25 fell. In November the two finally moved together (local 0.889 → 0.915, public 0.166 → 0.213). A public pipeline advertised at 0.500 scored 0.479 for us — the record keeps both numbers.

Squares in Squares

Playground · no medals
401 / 2,086 · top 19.2%

Kaggle Overnight · 40 rounds run by auto-run

17 rounds improved the score; 16 were merges of public outputs, checked for overlap. Our own polishing, neighbour transfer and seed shapes ran 12 times for one gain of 0.0000015. The field moves daily, so a better score can still mean a lower rank.

Autoresearch questions

What is autoresearch?+

Autoresearch is Andrej Karpathy’s small open-source loop in which an AI agent edits a training script, runs a five-minute experiment and keeps the change only if validation loss improves — about a hundred experiments overnight on one GPU. The name now covers any keep-or-revert research agent loop.

How is ResearchForge different from Karpathy’s autoresearch?+

ResearchForge is an autoresearch-style loop built independently for real repositories and live Kaggle competitions. A contract fixes the metric and hard limits, the baseline is frozen first, each experiment runs in its own git worktree, finalists are re-validated, every rejected idea is kept with its reason, and the winner ships as a clean branch.

Is it an AI scientist?+

It covers the research half too: arXiv search, a landscape of directions and hypotheses that cite their evidence, then experiments that test them. Your coding agent — Claude Code, Cursor — or your own model does the thinking; ResearchForge decides what counts as evidence.

Can it run autoresearch overnight with nothing open?+

Yes. autorun runs round after round from a terminal, and auto-run hands the loop to a background service so it keeps going with the terminal closed. Each plan still waits for your approval unless you type a standing one, and it never ships on its own.