Is your feature request related to a problem? Please describe.

Current units (e.g. m/s, hPa) are not explicitly captured in the dataset building, or used across training and inference.
This body of works seeks to make Anemoi unit-aware.
The road map would be:

1. Capture units at dataset creation time.
2. Ability to flag unit differences when concatentating datasets with different units (e.g. mm and m for precipitation).
3. Ability to transform units to enable the unit-aware merging of datasets.
4. Ability in inference to detect unit inconsistencies in training and inference
5. Ability to transform units in inference to overcome the issues in 4.
6. Ability to prescribe the units to be written as output, e.g. to write different units in inference from those used in training.

The main driver for that change is that as ECMWF will migrate to use GRIB edition 2 throughout. With GRIB edition 2, parameters such as precipitations, will be encoded in mm instead of m, and cloud cover with have the range 0-100 instead of 0-1. This will be even more visible when ERA6 will be released, as it will adopted the new units. Anemoi will be affected as follows:

1. When creating a dataset using recipe that combines two or more datasets with the same variables encoded using different units.
2. When combining datasets at run time with open_dataset(dataset1, dataset2) for example.
3. When fine-tuning a model using data that have different units the one used in the original training
4. When running inference with input data that have different units than the ones used to train the model.

The PR ecmwf/anemoi-transform#258 of anemoi-transform add a units entry to the Variable class. It also implements routines to cross check the units of two datasets. In addition to checking the units, the code also checks periods over which the variables are constructed (e.g. for accumulated fields). Please not that if units are only known for one of the two variables being compared, the code will issue a warning instead of throwing an exception.

The PR ecmwf/anemoi-datasets#577 of anemoi-datasets implements the points 1 and 2 of the list above. Please note that the filters that are applied when combining datasets at build time or usage time have not been updated to update the variable units when relevant. This still need to be implemented inanemoi-transform.

Point 3 must be implemented in anemoi-training.

Point 4 must be implemented in anemoi-inference.

Currently any unit mismatch will trigger an exception (or issue a warning if the units are missing). We can decide if we want to consider automatic unit conversion.

We may also decide that the anemoi best practices be to always train models in SI units only, and convert input and output data a required.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Organisation

No response