← All posts
GuideEvidence from Gemma 4 Developer Agent and ARC-AGI-3

Kaggle Leaderboard Noise: The Same Submission Scored 0.12, Then 0.17

We resubmitted a byte-identical Kaggle submission and the public score moved from 0.12 to 0.17. How to measure leaderboard noise before you trust a gain.

0.12 → 0.17
Identical file
2–4 levels
Offline repeats
Measure the floor
Rule

Manas Joshi·November 5, 2026·Updated November 14, 2026·6 min read

Short answer: on the Gemma 4 Developer Agent competition we resubmitted the exact same file, byte for byte, and the public score moved from 0.12 to 0.17. In competitions where the scored run is itself random, such as LLM agents with sampling and time limits, any gain smaller than that swing is not a gain until you have measured it twice.

Two kinds of leaderboard noise

  • ▸Noise in the scored run. In code and agent competitions Kaggle reruns your notebook or agent. Sampling, timeouts and tool failures change the result, so one file can score differently each time.
  • ▸Noise in the public split. Even a fixed CSV is scored on only part of the test set. Two close submissions can swap places on the private leaderboard, which is what a shake-up is.

What we measured

CompetitionSame thing, run twiceSpreadWhat it means
Gemma 4 Developer AgentIdentical submission file0.12 vs 0.17A 0.05 swing is within noise
ARC-AGI-3 (offline)Identical agent, repeated runs2 to 4 levelsChanges inside that band are ties
ARC-AGI-3 (offline)Temperature 0.6 → 0.47 and 4 vs 4Looked like a win; was noise
Enveda CASMI 2026A public pipeline, rerun by us0.500 reported vs 0.479A reported score is one draw too
Enveda CASMI 2026Two near-identical rounds0.213 vs 0.212A 0.001 step is a tie, not a loss

The Gemma resubmission was a deliberate control. Our previous submission had copied a public notebook’s prompt that its author reported as a stable 0.22, and it scored 0.12. Before changing anything, we resubmitted the same file to see how much of that gap was noise. It scored 0.17.

🗓️
Update, 14 Nov 2026. We are at rank 2,249 of 5,691 on Gemma 4 with that 0.17; the bronze-medal line is 0.20. Of the 24 rounds recorded since, 20 failed, most of them on Colab compute rather than on the idea. A run that never produced a score is not a measurement, so those count as evidence neither way. Full picture: Kaggle standings, November 2026.

How to measure your noise floor

  • ▸Spend one submission on a control. Resubmit an unchanged file early in the competition. The difference is your floor.
  • ▸Repeat offline runs. Run the same configuration several times with different seeds and record the spread, not just the mean.
  • ▸Treat anything inside the floor as a tie. Do not build the next idea on a winner you cannot tell from chance.
  • ▸Validate finalists. Before calling something better, run it and the baseline several more times. If the gap holds, keep it.

How ResearchForge handles it

ResearchForge screens each idea quickly, runs the survivors on the full benchmark, and can rerun finalists several times before a result is called validated. On ARC-AGI-3 that process validated one idea out of 14 and rejected the temperature change as noise. The same record also shows when your CV and leaderboard disagree, which is the other half of knowing what to trust.

Try it on your next run

Every number in this post came from the ResearchForge record. Yours can too.