Short answer: trust your local cross-validation (CV) over the public leaderboard, but only once you know your CV measures the same thing the test set does. When the two disagree, the disagreement is information: it usually means your validation split does not look like the test data, or the public leaderboard is too small to separate two close submissions.
This guide uses a real case from our Enveda CASMI 2026 run, where the public score rose or held for four submissions in a row while our local metric fell each time, what we did about it, and what happened when the two started to agree.
Why CV and the public leaderboard disagree
- ▸Your validation does not match the test set. If the held-out data is easier, or drawn from a different distribution, CV rewards the wrong things.
- ▸The public split is small. The public leaderboard scores only part of the test set, so a small difference between two submissions can be chance.
- ▸You have overfit the leaderboard. Pick the best of many submissions on the public score and you are partly selecting for luck.
- ▸The scored run itself varies. In code and agent competitions, the same submission can score differently from run to run. See leaderboard noise.
A recorded case: the public score rose while CV fell
Enveda CASMI 2026 asks you to identify unknown molecules from mass spectra, scored by MRR@25. Our local benchmark held out spectra of compounds that were already in the training library, so it measured how well we recover known compounds. The test set includes many molecules the library has never seen. Here is every submission, in order:
| Date | Local MRR@25 | Public score | Change |
|---|---|---|---|
| 24 Oct | 0.8994 | 0.136 | Spectral library search only |
| 25 Oct | 0.8994 | 0.156 | Add a COCONUT exact-mass fallback for unknown molecules |
| 25 Oct | 0.8999 | 0.149 | Rank the fallback by a documentation prior |
| 29 Oct | 0.9032 | 0.158 | Reorder candidates with a second signal (from top public notebooks) |
| 30 Oct | 0.9000 | 0.160 | Mix fallback candidates in when the top match is weak (gate 0.5) |
| 31 Oct | 0.8978 | 0.164 | Gate raised to 0.7 |
| 31 Oct | 0.8915 | 0.166 | Gate raised to 1.0 |
| 31 Oct | 0.8906 | 0.166 | Gate removed |
| 1 Nov | 0.8649 | 0.164 | Fallback only, for ranks 2–25 |
From 30 October, each step that trusted the fallback for unknown molecules more lowered local MRR@25 while the public score rose or held, until the last step, which pushed it all the way and the public score fell back to 0.164. The local benchmark could not see the effect we were changing, because it contained no unknown molecules.
Update, 14 November: they finally moved together
In November the overnight loop kept going on the same competition, with a benchmark of its own, so compare numbers within each table rather than across them. This time the two mostly agreed: in five of seven steps, local MRR@25 and the public score moved the same way.
| Date | Local MRR@25 | Public score | Change |
|---|---|---|---|
| 11 Nov | 0.8947 | 0.166 | A more tolerant library similarity |
| 11 Nov | 0.9115 | 0.181 | A different library similarity |
| 11 Nov | 0.9124 | 0.175 | Its thresholds tuned — local up a hair, public down |
| 11 Nov | 0.9131 | 0.180 | Merged with an earlier reranking round |
| 13 Nov | 0.9141 | 0.191 | Spectrum clean-up before matching |
| 13 Nov | 0.9149 | 0.213 | A fragment-based score added to the ranking |
| 13 Nov | 0.9151 | 0.212 | That score tuned — a tie |
| 13 Nov | — | 0.479 | A public pipeline adopted whole; it prints no local score |
Two things stand out. The local metric is compressed: a local gain of 0.0008 came with +0.022 on the leaderboard (the fragment score), yet both disagreements were tuning steps that gained under 0.001 locally. So the direction of the local metric was worth following in November, but not its size. And the biggest jump, to 0.479, came from adopting a public pipeline whose author reported 0.500. It writes no local score, so the record marks it unscored and the leaderboard is the only evidence for it. That put us at rank 1,348 of 5,261, 0.008 below the bronze-medal line, with a month to go.
The same thing in an agent competition
On ARC-AGI-3 we screened ideas on 25 public games. An executable world model completed more levels offline (5 against 4) and scored 0.58 on the leaderboard against 1.21 without it. Cross-game memory also looked good offline and scored 1.35 against 1.94. Both were rejected on the leaderboard evidence.
What to do when they disagree
- ▸Check what your validation holds out. Split the way the test set is split: by molecule, by user, by time or by game, not by row.
- ▸Measure the noise floor first. Resubmit one file, or repeat one offline run, before you read anything into a small gain.
- ▸Track both numbers for every submission. One pair tells you nothing. A run of pairs moving in opposite directions, like the table above, tells you your CV is blind to something.
- ▸Choose final submissions on a CV you trust. If you cannot trust it yet, fix the validation before you pick.
- ▸Keep the rejected ideas. An idea that lost on a broken CV may win once the CV is fixed.
How ResearchForge keeps this visible
Every experiment is measured against a frozen baseline and kept or rejected with a written reason, and each submission note records the local score it was made with. That is why the table above could be rebuilt from the record: the disagreement was on file rather than in someone’s memory. Kaggle Overnight runs that loop on your competition while you sleep.