This is hard-coded in the rule. Would it make sense to expose this in the config (in particular given the expected rapid changes with new metrics. @cosunae in #144 has suggested a nice template for config 'sections', e.g.

runs:
...

experiment:
    dashboard:
        stratification: ...
    scorecard:
        regions:
            - all
            - mittelland
        scores: 
            U_10M: ["RMSE", "R2"]
            TOT_PREC: ["RMSE", "ETS_gt_0p0"]

showcases:
...

Also, I would suggest to only show ["RMSE", "R2"] by default. The other scores don't add much, or are even a bit dubious (such as correlation which cannot be generally converted to a skill score due to its support from -1 to 1). I know that Marco Arpagaus always wants to see STDE, but I think this is also based on a misunderstanding of what STDE is. STDE is only different from RMSE if we have 'systematic' domain-wide biases, which is usually not the case.

With #137 there are now additional scores available for categorical forecasts. ETS would be a candidate to include for TOT_PREC and wind in particular.

Hi Jonas, thanks for the suggestion. I included ETS in a recent commit, could you please have a look and let me know what you think?

Hi Alberto. I really like the way the new scorecard looks. Super cool.

I was wondering, if the referencing of the score shouldn't be made consistent with the way these are represented in the scores file (verif_aggregated.nc). As such, I would the variables section above expect to look something like this:

       variables=[
            "T_2M:RMSE,MAE",
            "TOT_PREC:RMSE,MAE,ETS_gt_0p0",
        ],

That way, we can specifically include / exclude scores (and thresholds) on a per variable basiss.

Hi thanks for the kind words!

This is actually already supported: that syntax in the variables param in report.smk gives

That works thanks to this function in report_scorecard:

def _parse_var_metrics(spec: str):
    """Parse a 'VAR:M1,M2,...' (or 'VAR' alone) item into (var, [metrics])."""
    if ":" in spec:
        var, metrics = spec.split(":", 1)
        return var.strip(), [m.strip() for m in metrics.split(",") if m.strip()]
    return spec.strip(), None  # None → use all scores for this variable

It splits each entry first on : and then splits the metrics on ,. So you can write just ETS to include all available thresholds, or list specific ones like ETS_gt_0p0 as in your example.

Currently we run this for all non-baseline runs against all baselines. It is configurable in the sense that you can copy-paste the config and remove runs / baselines that you don't want to be analysed. Is this enough?

Also here, I suggest to make this configurable

As mentioned previously, I would not include CORR as an eligible metric to add to the scorecard. Especially so, as R2 is virtually the same.

My comment from before about exposing the thresholds as well probably is better suited here. I.e. I would love to be able to specify what score AND what threshold to include on a per variable basis.

Actually this block is just a hardcoded fallback that only triggers when --variable is not passed at all (e.g., running the script manually). If the snakemake rule always provides explicit --variable arguments, this code path is never hit in the workflow.

Honestly, I'm not sure this fallback is that useful now... A cleaner alternative might be to drop it and instead show everything present in the data when no --variable is given (and same thing with the regions).