Expected behaviour:
The graphics test workflow should pass when the plotted data is unchanged, regardless of formatting or cosmetic differences in the generated plots.
Actual behaviour:
test_graphics is failing when plots differ only in their formatting (e.g. fonts, spacing, margins, or styling), even though the underlying data and plotted values are identical. This can happen locally, through GH Actions, or both, and depends on matplotlib/freetype versions. Tests fail because they compare rendered plot images directly so formatting-only changes are treated as regressions. The false-positive test failures make it harder to identify genuine changes in plot output.
Points for Discussion
- Treat GH Actions output as truth, because the docker images are always the same.
- This specifically occurs when using the latest version of matplotlib (3.11). Pinning the version to 3.10.8 (like in GH Actions) prevents test failures.
Possible solutions:
- Force formatting for plots (at least in tests).
- Pin matplotlib version.
- Reupload new baseline images every new matplotlib release.
- Reinvent the wheel and find another library (not my preferred option).
Expected behaviour:
The graphics test workflow should pass when the plotted data is unchanged, regardless of formatting or cosmetic differences in the generated plots.
Actual behaviour:
test_graphicsis failing when plots differ only in their formatting (e.g. fonts, spacing, margins, or styling), even though the underlying data and plotted values are identical. This can happen locally, through GH Actions, or both, and depends on matplotlib/freetype versions. Tests fail because they compare rendered plot images directly so formatting-only changes are treated as regressions. The false-positive test failures make it harder to identify genuine changes in plot output.Points for Discussion
Possible solutions: