- Where: Google, on Kaggle (Featured), $75,000 prize pool, medals awarded
- Task: A coding agent on one Gemma 4 model that fixes bugs in real Python repositories
- Score: Share of tasks whose patch passes hidden tests, as in SWE-bench
- Hardware: 4 × NVIDIA L4, 32,768-token context, 4-bit Gemma 4 builds
- Dates: team merger 25 Nov 2026 · final deadline 2 Dec 2026
- Field: 5,691 teams on the public leaderboard (14 Nov); leader 0.29
- More: all Gemma 4 Developer Agent facts and FAQ
The short version
This is our least successful run so far, and the one that taught us most about noise. Four submissions from 25 October to 1 November went from 0.05 to 0.17, and the 0.17 was a deliberate control: the same file as a 0.12 submission, byte for byte. Since then ResearchForge has recorded 24 rounds; 20 failed, most of them on Colab compute rather than on the idea. A failed run is not a measurement, so none of them has earned a submission.
Score timeline (public leaderboard)
| Date | Public | What changed |
|---|---|---|
| 25 Oct | 0.05 | Baseline agent on Gemma 4 12B, code-graph tools removed |
| 27 Oct | 0.15 | Adapted from a public notebook reported at 0.22–0.24, on Gemma 4 31B |
| 29 Oct | 0.12 | The exact prompt of a public notebook reported as a stable 0.22 |
| 1 Nov | 0.17 | Control: the same file again — a 0.05 swing from noise alone |
A no-op that scored worse than the baseline
Early in November we screened a LoRA adapter initialised to zero, which by construction changes nothing. On the quick screen it scored 0.0 against a baseline of 0.3, and in a second run 0.2 against 0.4. Both were rejected, correctly, but the real lesson is that a quick screen of a few tasks cannot tell a change from no change on this competition. Two follow-up fine-tuning rounds failed to run at all.
Rounds since 7 November
Auto-run launched all 20 November rounds unattended, with Claude Code as the driver. Four produced a score: the best was 0.40 on a 10-task benchmark and 0.33 on 30 tasks. The other 16 failed for reasons the record kept: Colab could not assign the GPU we asked for, sessions exited or ran past the 30-minute limit, a downloaded template was saved under the wrong filename (our bug, fixed the next round), and twice a run finished without printing a score the loop could read.
Auto-run handled this the way it should. When a round would have failed the same way again, it stopped and asked a question in the dashboard instead of burning another run, and when a session timed out it collected the result later rather than starting over. What it cannot do is make an unreliable GPU session finish, so that is the next job, before any idea gets judged.
The lessons
- ▸Measure the noise floor before the first idea. 0.12 vs 0.17 on an identical file, and a no-op adapter scoring 0.0 vs 0.3, say that small gains here are invisible. More in leaderboard noise.
- ▸Reported public scores don’t transfer. Two public approaches reported at 0.22 scored 0.15 and 0.12 for us.
- ▸A failed run is not evidence. 16 of the 20 failures never produced a score, so they say nothing about the ideas behind them.
What’s next
Reliable compute first, then larger benchmarks so a gain can be told from noise, then the bronze line at 0.20 before 2 December. See all four runs in our November standings.