Skip to content

Gzip reading and compression abstraction#58

Merged
fsoubelet merged 15 commits into
masterfrom
gzip_compression
Sep 5, 2023
Merged

Gzip reading and compression abstraction#58
fsoubelet merged 15 commits into
masterfrom
gzip_compression

Conversation

@fsoubelet

@fsoubelet fsoubelet commented Sep 5, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

A proposed implementation for #57

  • Implements abstraction for the opener
  • Provides a ready-to-use (one import line) opener for gzipped files
  • Adds tests for with a gzipped file, those are a carbon copy of the tests for a regular binary file. May not be the best choice.

Tests ran fine locally but I ran into some issues building the documentation with the most recent version of numpy.

Out of scope but also included: the switch of CI to our reusable workflows, and the fix for the over-indexing in the documentation.

@fsoubelet fsoubelet self-assigned this Sep 5, 2023
@fsoubelet fsoubelet added Type: Documentation Improvements, updates and fixes to the documentation. Feature Request Type: CI/CD Improvements, updates and fixes to the continuous integration. labels Sep 5, 2023
JoschD
JoschD reviewed Sep 5, 2023
View reviewed changes
Comment thread sdds/reader.py Outdated
@fsoubelet fsoubelet changed the title [WIP] Gzip compression Gzip reading and compression abstraction Sep 5, 2023
@fsoubelet

fsoubelet commented Sep 5, 2023

Copy link
Copy Markdown
Member Author

The issue encountered here was that subscripting either contextlib.AbstractContextManager or collections.abc.Callable was not possible until PEP 585 and Python 3.9 so the code would raise on Python 3.8.

As a workaround, I have added a runtime check for the Python version and in case the user is running on Python 3.8 then OpenerType is simply collections.abc.Callable. There is no change of functionality.

As a side note, since our documentation workflow runs on Python 3.9 the online documentation holds the full typing information.

@fsoubelet fsoubelet requested a review from JoschD September 5, 2023 13:00
@fsoubelet fsoubelet merged commit ff390c1 into master Sep 5, 2023
@fsoubelet fsoubelet deleted the gzip_compression branch September 5, 2023 13:18
@fsoubelet fsoubelet mentioned this pull request Sep 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JoschD JoschD JoschD approved these changes

Assignees

@fsoubelet fsoubelet

Labels

Type: CI/CD Improvements, updates and fixes to the continuous integration. Type: Documentation Improvements, updates and fixes to the documentation.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fsoubelet @JoschD