← All posts
GuideEvidence from Enveda CASMI 2026 and ARC-AGI-3

Kaggle CV vs Leaderboard: What to Trust When They Disagree

Your Kaggle CV and public leaderboard disagree: which do you trust? A recorded case where the public score rose 0.136 to 0.166 while local MRR@25 fell.

16
Submissions traced
CV ↓, public ↑
October
Both ↑ · 0.213
November

Manas Joshi·November 5, 2026·Updated November 14, 2026·7 min read

Short answer: trust your local cross-validation (CV) over the public leaderboard, but only once you know your CV measures the same thing the test set does. When the two disagree, the disagreement is information: it usually means your validation split does not look like the test data, or the public leaderboard is too small to separate two close submissions.

This guide uses a real case from our Enveda CASMI 2026 run, where the public score rose or held for four submissions in a row while our local metric fell each time, what we did about it, and what happened when the two started to agree.

Why CV and the public leaderboard disagree

  • ▸Your validation does not match the test set. If the held-out data is easier, or drawn from a different distribution, CV rewards the wrong things.
  • ▸The public split is small. The public leaderboard scores only part of the test set, so a small difference between two submissions can be chance.
  • ▸You have overfit the leaderboard. Pick the best of many submissions on the public score and you are partly selecting for luck.
  • ▸The scored run itself varies. In code and agent competitions, the same submission can score differently from run to run. See leaderboard noise.

A recorded case: the public score rose while CV fell

Enveda CASMI 2026 asks you to identify unknown molecules from mass spectra, scored by MRR@25. Our local benchmark held out spectra of compounds that were already in the training library, so it measured how well we recover known compounds. The test set includes many molecules the library has never seen. Here is every submission, in order:

DateLocal MRR@25Public scoreChange
24 Oct0.89940.136Spectral library search only
25 Oct0.89940.156Add a COCONUT exact-mass fallback for unknown molecules
25 Oct0.89990.149Rank the fallback by a documentation prior
29 Oct0.90320.158Reorder candidates with a second signal (from top public notebooks)
30 Oct0.90000.160Mix fallback candidates in when the top match is weak (gate 0.5)
31 Oct0.89780.164Gate raised to 0.7
31 Oct0.89150.166Gate raised to 1.0
31 Oct0.89060.166Gate removed
1 Nov0.86490.164Fallback only, for ranks 2–25

From 30 October, each step that trusted the fallback for unknown molecules more lowered local MRR@25 while the public score rose or held, until the last step, which pushed it all the way and the public score fell back to 0.164. The local benchmark could not see the effect we were changing, because it contained no unknown molecules.

📌
The lesson was not “ignore CV”. It was that this CV answered a different question from the leaderboard. The fix is a validation set that holds out whole molecules the library does not contain, so local and public scores measure the same thing.

Update, 14 November: they finally moved together

In November the overnight loop kept going on the same competition, with a benchmark of its own, so compare numbers within each table rather than across them. This time the two mostly agreed: in five of seven steps, local MRR@25 and the public score moved the same way.

DateLocal MRR@25Public scoreChange
11 Nov0.89470.166A more tolerant library similarity
11 Nov0.91150.181A different library similarity
11 Nov0.91240.175Its thresholds tuned — local up a hair, public down
11 Nov0.91310.180Merged with an earlier reranking round
13 Nov0.91410.191Spectrum clean-up before matching
13 Nov0.91490.213A fragment-based score added to the ranking
13 Nov0.91510.212That score tuned — a tie
13 Nov—0.479A public pipeline adopted whole; it prints no local score

Two things stand out. The local metric is compressed: a local gain of 0.0008 came with +0.022 on the leaderboard (the fragment score), yet both disagreements were tuning steps that gained under 0.001 locally. So the direction of the local metric was worth following in November, but not its size. And the biggest jump, to 0.479, came from adopting a public pipeline whose author reported 0.500. It writes no local score, so the record marks it unscored and the leaderboard is the only evidence for it. That put us at rank 1,348 of 5,261, 0.008 below the bronze-medal line, with a month to go.

The same thing in an agent competition

On ARC-AGI-3 we screened ideas on 25 public games. An executable world model completed more levels offline (5 against 4) and scored 0.58 on the leaderboard against 1.21 without it. Cross-game memory also looked good offline and scored 1.35 against 1.94. Both were rejected on the leaderboard evidence.

What to do when they disagree

  • ▸Check what your validation holds out. Split the way the test set is split: by molecule, by user, by time or by game, not by row.
  • ▸Measure the noise floor first. Resubmit one file, or repeat one offline run, before you read anything into a small gain.
  • ▸Track both numbers for every submission. One pair tells you nothing. A run of pairs moving in opposite directions, like the table above, tells you your CV is blind to something.
  • ▸Choose final submissions on a CV you trust. If you cannot trust it yet, fix the validation before you pick.
  • ▸Keep the rejected ideas. An idea that lost on a broken CV may win once the CV is fixed.

How ResearchForge keeps this visible

Every experiment is measured against a frozen baseline and kept or rejected with a written reason, and each submission note records the local score it was made with. That is why the table above could be rebuilt from the record: the disagreement was on file rather than in someone’s memory. Kaggle Overnight runs that loop on your competition while you sleep.

Try it on your next run

Every number in this post came from the ResearchForge record. Yours can too.