Difference between public and private benchmarks?

[justinborromeo]

I'm trying to replicate the brain-scoring of CORNet-S. I noticed that the brain-score leaderboard uses the MajajHong2015.IT-pls (V3) benchmark while the PyTorch candidate_models example uses dicarlo.MajajHong2015public.IT-pls. Are these the same?

[mschrimpf]

The website score on MajajHong2015 is run on a private (held-out) version of the data whereas the example uses a public version, so they are different. Scores on the two datasets should be related, but differ -- in my experience, the public version typically yields slightly higher scores whereas the private one seems more difficult.

We might release the currently private version at some point as we add other benchmarks with separate held-out data, but until then the best way to score models on those private data is to submit them through the website.