- Where: Kaggle, $850,000 total prize pool
- Format: interactive game environments the agent has never seen; scored on levels completed and action efficiency
- Runtime: notebook submissions on RTX 6000 (48 GB) GPUs, no internet, up to 9 hours
- Dates: submissions closed 2 Nov 2026 · papers due 8 Nov · results announced 4 Dec
- Field: 4,926 teams on the final public leaderboard
- More: all ARC-AGI-3 facts, dates and FAQ
The short version
We ran ResearchForge on ARC-AGI-3 from August to November. It recorded 18 hypotheses and 14 experiments; 10 were rejected with a written reason, and one — fine-tuning the model on its own winning games — was validated. The biggest single jump, though, did not come from our own ideas: it came on 1 October, when we rebuilt on a strong public notebook. We say that plainly because the point of keeping a record is that it tells you the truth about what worked.
Score timeline (public leaderboard)
| Date | Public score | What changed |
|---|---|---|
| 1–5 Aug | 0.08 | First agents: probe-based and graph-frontier exploration |
| 10 Aug | 1.21 | The “duck” agent harness + Qwen3.6-27B (FP8) served with vLLM — the base we built on |
| 18 Aug | 1.55 | Model upgrade Qwen3.6 → Qwen3.8-27B |
| 25 Aug | 1.94 | LoRA fine-tune (SFT / STaR) on 9 of the agent’s own winning games — validated |
| 3 Sep | 3.62 | Re-baseline on a faster NVFP4 build with multi-token prediction |
| 5–28 Sep | 2.08–3.86 | A/B rounds: reasoning steps, early stopping, strategy switching, repeat runs |
| 1 Oct | 21.06 | Rebuilt on a strong public notebook by dfranzen (public score 27.89) |
| 30 Oct | 32.51 | Our best public score, on our own variant of that pipeline |
| 2 Nov | 313th | Submissions closed: rank 313 of 4,926, inside the bronze-medal range |
The jump from 3.62 to 21.06 between September and October is not an experiment of ours paying off — it is the public notebook we built on. From there, our own variants took the best public score to 32.51.
What worked
- ▸A better base model. Moving from Qwen3.6 to Qwen3.8-27B lifted the score from 1.21 to 1.55.
- ▸Fine-tuning on the agent’s own wins. A small LoRA fine-tune (SFT / STaR) on just 9 of its winning games lifted 1.55 to 1.94 (+25%). This was the one experiment ResearchForge validated.
- ▸A faster quantized build. NVFP4 with multi-token prediction meant more actions inside the 9-hour limit.
- ▸Building on strong public work. The largest gain of the competition.
What we rejected, and why
Every one of these stayed in the record with its reason, so none of them got re-tried by accident:
| Idea | Why it was rejected |
|---|---|
| Executable world model + internal planning | Better offline (5 vs 4 levels) but 0.58 on the leaderboard vs 1.21 — the offline proxy failed |
| Cross-game memory | Positive offline, 1.35 on the leaderboard vs 1.94 without it |
| More fine-tuning data (9 → 22 games) | 1.20 vs 1.94 — naive data scaling hurt |
| Carry the world model across levels | 2–3 levels vs a baseline of 4, inside the noise band |
| Lower analyzer temperature (0.6 → 0.4) | Runs of 7 and 4 vs 4 — within run-to-run noise, not a reliable lever |
| Longer retained context | Same 4 levels as the baseline |
| Cross-game mechanic priors | No lift; the model likely already knows these mechanics |
| Centre-prior click tie-break | Worse than baseline offline |
| Graph-frontier exploration on the real 25 games | No change vs baseline |
The lesson: our offline benchmark lied, and so does a single submission
We screened ideas on an offline benchmark — levels completed across the 25 public games. It helped rule things out, but it did not predict the leaderboard: the world-model idea scored higher offline and far lower on the hidden games, and cross-game memory did the same. Offline runs also wandered between 2 and 4 levels with no change at all.
The leaderboard is noisy too. Our same “v5” configuration, submitted on different days in late October, scored anywhere from 21.75 to 32.51. That is why a public rank is a snapshot, not a verdict — and why ResearchForge treats every result as evidence with a confidence, rather than a pass/fail gate.
First published 11 August 2026; rewritten 4 November 2026 with the full run and updated 14 November with the medal lines. Final results follow on 4 December. Where our other runs stand: Kaggle standings, November 2026.