🦚  New Features

• Officially Support Windows: Test all functions in CI. Raise an OSError for a few functions that do not work yet in windows.
• Add sqlitelock.SQLiteLock and alternative to the filelock and portalocker libraries.
• Add datastructures.chunk_list

🐁  Minor Improvements

• Ad suppport for old EM27/SUN data in OPUS file reader
• Add test for datastructures.concat_lists
• Be more descriptive when GitHub API returns an error

🦚  New Features

• Add function code.download_github_release_asset
• Improve library import times (#28) when setting env var TUM_ESM_UTILS_EXPLICIT_IMPORTS=1 (see docs)
• Add function datastructures.concat_lists
• Add function system.get_physical_memory_usage
• Add function timing.timed_section
• Add function mathematics.divides_evenly (reason explained in the function docs)

🐜  Internal Changes

• Debug shell.get_commit_sha if git is not installed on the system
• Lower constraint for polars dependency to 1.16.0 because newer polars releases have memory leaks
• Debug opus.OpusHTTPInterface.some_macro_is_running for OPUS 8.9 (some error codes changed for OPUS 8.9)

🦚  New Features

• column.astronomy module: Convenience function for astronomical calculations
• column.ncep_profiles module: Loading GGG2020 atmospheric profiles into polars dataframes
• column.averaging_kernel module: Functions to store, load and apply a column averaging kernel.

Use the new column module by installing tum_esm_utils[modeling].

🦚  New Features

• opus.OpusHTTPInterface class
  Communicate with OPUS using its HTTP interface (see docs and API Reference)

🐜  Internal Changes

• Fix bug when only reading the first channel from OPUS files
• Refine pytests: add test groups, make tests run when .git folder is not present
• Add repository to Zenodo

🦚  New Features

• opus Module
  Read OPUS files. Credits to @FriedrichKlappenbach for decoding the OPUS file structure.

🐜  Internal Changes

• Switch from YAPF to RUFF formatting (easier to configure and better maintained)
• Rename polars dependency group to em27. The old name will be kept as an alias until the next breaking release (3.0.0)

🦚  New Features

• timing.ExponentialBackoff
  Add class to exponentially increase the waiting time after an error
  see in docs

• StricterBaseModel, StrictIPv4Adress
  More specialized Pydantic base classes
  see in docs

• timing.date_span_intersection, timing.datetime_span_intersection
  Functions to compute the overlap of two date/datetime periods
  see in docs

🦀  Dependency Updates

• Explicitly add Python 3.13 support (tests, PyPI package)

🦚  New Features

• em27.PROFFAST_MULTIPLIERS, em27. PROFFAST_UNITS, ...
  Add EM27/SUN constants for instrument colors, serial numbers, data units, etc.
  see in docs

• plotting.add_colorpatch_legend
  A function to add a "Colorpatch Legend" to a plot
  see in docs

• RandomLabelGenerator
  Generate random names - like Docker container names curious-einstein, wild-picard and so on
  see in docs

• tum_esm_utils.timing.parse_iso_8601_datetime
  Added because datetime.datetime.fromisoformat does not support all ISO 8601 timezone variants
  see in docs

• tum_esm_utils.files.list_directory
  Adds more options the os.listdir method - only include directories/files, filter by regex, ignore by Unix shell patterns
  see in docs

• tum_esm_utils.files.render_directory_tree
  Render a directory tree as a string
  see in docs

• tum_esm_utils.text.simplify_string_characters
  Simplify a string by replacing special characters with their ASCII counterparts and removing unwanted characters with. For example, simplify_string_characters("Héllo, wörld!") will return "hello-woerld".
  see in docs

• tum_esm_utils.text.replace_consecutive_characters
  Remove consecutive whitespaces or other characters with tum_esm_utils.text.replace_consecutive_characters
  see in docs

🦀  Dependency Updates

• Upgrade minimum Python version to 3.10. This follows SPEC 0 — Minimum Supported Dependencies recommendations.

Upgrade Numpy (v1 to v2) and Polars (v0 to v1).

Add Version validator (#19).

This release brings a significantly improved opus file validation based on Proffast 2.4. The old version could only handle a few reasons of file corruption. Furthermore the output did not list the files that the preprocessing could not even read.

Old validator:

detection_results = tum_esm_utils.em27.detect_corrupt_ifgs(
    "/path/to/a/folder/with/interferograms"
)
assert detection_results == {
    "md20220409s0e00a.0199": [
        "charfilter 'GFW' is missing",
        "charfilter 'GBW' is missing",
        "charfilter 'HFL' is missing",
        "charfilter 'LWN' is missing",
        "charfilter 'TSC' is missing",
    ]
}

New validator:

detection_results = tum_esm_utils.em27.detect_corrupt_opus_files(
    "/path/to/a/folder/with/interferograms"
)

detection_results == {
    'md20220409s0e00a.0199': [
        'Charfilter "DUR" is missing',
        'Charfilter "GBW" is missing',
        'Charfilter "GFW" is missing',
        'Charfilter "HFL" is missing',
        'Charfilter "LWN" is missing',
        'Charfilter "TSC" is missing',
        'Differing sizes of dual channel IFGs!',
        'IFG size too small!',
        'Inconsistent dualifg!',
        'Inconsistent parameter kind in OPUS file!'
    ],
    'comb_invparms_ma_SN061_210329-210329.csv': [
        'File not even readible by the parser'
    ],
    'comb_invparms_mc_SN115_220602-220602.csv': [
        'File not even readible by the parser'
    ],
    'md20220409s0e00a.0200': [
        'File not even readible by the parser'
    ]
}

The function tum_esm_utils.em27.detect_corrupt_ifgs has been renamed to tum_esm_utils.em27.detect_corrupt_opus_files but is of course still available under the old name too (until the next breaking release).

This breaking release is necessary to make the polars library an optional dependency. Until now, every codebase that wanted to use the tum-esm-utils library would have to use the same major release of polars.

The library now has two optional modules: em27 and plotting, for which the dependencies (polars and matplotlib are not automatically installed). Install them using:

pip install "tum-esm-utils[polars]"
pip install  "tum-esm-utils[plotting]"
pip install  "tum-esm-utils[polars,plotting]"

Due to this change, the base library is kept at ~12MB, including all (sub-)dependencies, and only if you want the plotting and EM27 extras, this grows to ~174MB.

🦞  Breaking changes

I included many breaking changes here, so there will not be a need for another breaking release soon.

• Module interferograms has been renamed em27 and made optional
• Removed the context and testing modules: functionality has been moved into timing and files
• Renamed module github to code
• Removed unused functionality and changed some return types (see #18 for complete list)
• Dropped Python3.8 support because Matplotlib also dropped it
• Changed from MIT to GPL license

🦚  New features

• Added new optional plotting module
• Refactored function to load Proffast2 output files
• Added function to code module to load files from GitLab