Right, I see that the docs also explain this. I would just imagine that the newer years might be harder (less likely that a model was trained on it).

In general, it is a point where I would love to examine different aggregations across the samples to see where the models fail and why. However, as it stands, EuroEval isn't really designed for this (not a critique, more of a wish).

I have had the thought of redoing EuroEval (NLG) using e.g.:

https://ai.pydantic.dev/evals/

To allow for more sample-specific examinations (it also makes a bunch of other things possible).

@KennethEnevoldsen Newer years turn out to have roughly similar performance, actually, which was a bit surprising to me. Probably because they're still rooted in the same learning material.

And your wish is very research oriented, where EuroEval is intended to follow the principle "do-one-thing-right": it provides robust evaluations and is opinionated, with little ability to tune parameters (such as metadata, as you're suggesting here). But for such things I'd suggest you simply fork the repo and create custom changes in the code there, to perform your analysis - otherwise the codebase would eventually get too bloated, I think.

For such custom analyses, the Pydantic eval framework could maybe be better suited?

@KennethEnevoldsen Newer years turn out to have roughly similar performance, actually, which was a bit surprising to me. Probably because they're still rooted in the same learning material.

Ahh that is nice to know

And your wish is very research oriented, where EuroEval is intended to follow the principle "do-one-thing-right" ...

Do you have a definition of that one thing?

I disagree that it is research-oriented; there are plenty of valid reasons to have sample-level information. Here are just a few:

• It makes development easier, e.g.:
  • Are there bad examples in the datasets. Could be examined by checking samples where all models perform poorly. Removing a sample can be done fully backward compatible
  • Adding new cases / datasets: Currently, this requires re-evaluating all models on the dataset. Having sample-level information could enable just evaluating the diff
  • examining new metrics
• understanding results, e.g.:
  • Understanding model differences: If model A is better than B at task 1, what does that really mean? Having a few examples of where they fail differently might help lead to a better model set
  • critical understanding of the benchmarks: Having examples also allows the user to see what the benchmark tests are, and allows them to be critical of the tests

That being said, I do think it would be a big change - it would require someone to work on it and probably agree that it doesn't fit well in the current structure of the repository - so I could def. see it as a separate project

I see the value of it, but I also see a lot of issues:

1. The core purpose is to provide an evaluation of models, not to use the evaluation during model development, which would very easily cause overfitting. The 'users' of the benchmark are primarily model users, rather than model developers;
2. Enabling a sample-wise comparison between models emphasises a focus on the individual datasets. Our focus is trying to measure the overall performance of the model, where the individual datasets (and especially individual samples) are not as important and detracts from the core aggregate evaluation;
3. Examining new metrics is a research question in my opinion;
4. Regarding the critical understanding of the benchmarks: we already have detailed dataset descriptions, and allow a --debug mode where you can see the individual samples and predictions for debugging purposes. I feel like that's a very high bar for transparency?
5. "Evaluating the diff" would mean that we had to store the sample-level information as well, which would cause the size of the stored results to explode. Currently it's already around 50 MB, only storing the basic aggregate scores, as we currently have 35,000 (filtered) evaluation results. Further, the actual evaluations are usually not the time intensive part when evaluating a single dataset across many models, it's simply downloading and loading the models.

I think we have some fundamental misunderstanding here - to be clear, I am not saying that you should change EuroEval, and I do think that sample-level metrics are better explored in a separate project.

1. I would argue that users would want this as well (developers can use the --debug after all)
2. But don't you agree that there are things that can be easily seen, but are hard to measure? Looking at 10 examples can often reveal much more about the differences or error modes
3. Yes, it would be a high bar
4. Yes, it would require that - it would likely take up multiple GB - the downloading/loading part is good to know

Sounds like the best argument is for me to code it up ;)