Short answer: on the Gemma 4 Developer Agent competition we resubmitted the exact same file, byte for byte, and the public score moved from 0.12 to 0.17. In competitions where the scored run is itself random, such as LLM agents with sampling and time limits, any gain smaller than that swing is not a gain until you have measured it twice.
Two kinds of leaderboard noise
- ▸Noise in the scored run. In code and agent competitions Kaggle reruns your notebook or agent. Sampling, timeouts and tool failures change the result, so one file can score differently each time.
- ▸Noise in the public split. Even a fixed CSV is scored on only part of the test set. Two close submissions can swap places on the private leaderboard, which is what a shake-up is.
What we measured
| Competition | Same thing, run twice | Spread | What it means |
|---|---|---|---|
| Gemma 4 Developer Agent | Identical submission file | 0.12 vs 0.17 | A 0.05 swing is within noise |
| ARC-AGI-3 (offline) | Identical agent, repeated runs | 2 to 4 levels | Changes inside that band are ties |
| ARC-AGI-3 (offline) | Temperature 0.6 → 0.4 | 7 and 4 vs 4 | Looked like a win; was noise |
| Enveda CASMI 2026 | A public pipeline, rerun by us | 0.500 reported vs 0.479 | A reported score is one draw too |
| Enveda CASMI 2026 | Two near-identical rounds | 0.213 vs 0.212 | A 0.001 step is a tie, not a loss |
The Gemma resubmission was a deliberate control. Our previous submission had copied a public notebook’s prompt that its author reported as a stable 0.22, and it scored 0.12. Before changing anything, we resubmitted the same file to see how much of that gap was noise. It scored 0.17.
How to measure your noise floor
- ▸Spend one submission on a control. Resubmit an unchanged file early in the competition. The difference is your floor.
- ▸Repeat offline runs. Run the same configuration several times with different seeds and record the spread, not just the mean.
- ▸Treat anything inside the floor as a tie. Do not build the next idea on a winner you cannot tell from chance.
- ▸Validate finalists. Before calling something better, run it and the baseline several more times. If the gap holds, keep it.
How ResearchForge handles it
ResearchForge screens each idea quickly, runs the survivors on the full benchmark, and can rerun finalists several times before a result is called validated. On ARC-AGI-3 that process validated one idea out of 14 and rejected the temperature change as noise. The same record also shows when your CV and leaderboard disagree, which is the other half of knowing what to trust.