Changelogs

• Introduce a new DatasetV2 class in a new polaris.experimental module.
• Explicitly rename the old dataset to DatasetV1, but create the Dataset alias for it.
• Extract a BaseDataset class that encompasses all functionality that is shared between V1 and V2.
• Restricted use of __getitem__ to only allow indexing a specific value or a row.
• Add a new Zarr manifest for upload integrity.
  • See Adding new Zarr manifest generation to DatasetV2 class #185

Checklist:

• Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
• Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
• Update the API documentation if a new function is added, or an existing one is deleted.
• Write concise and explanatory changelogs above.
• If possible, assign one of the following labels to the PR: feature, fix, chore, documentation or test (or ask a maintainer to do it for you).

Closes #132

The implementation has been relatively straight-forward given the PoC. There are two points that required some further thinking.

1. Naming and code organization: I wasn't sure how to name the new class, where to introduce it and whether to rename the old class. For clarity, I decided to rename the old class, but we do maintain the Dataset alias for it to not break backwards compatibility. I also introduced a new experimental module to house the V2 implementation.
2. Indexing: With Add a converter from PDB to Zarr to the DatasetFactory #171, we made it possible for the Zarr root to contain groups. This raises the question how to index those. For these cases, I decided to explicitly introduce a index array that needs to be part of that group.

On V1 and V2 compatibility

You can think of a Pandas DataFrame as a set of named NumPy arrays. For storage, the conversion from Pandas to a Zarr archive is therefore relatively straight-forward. Note that this is limited to the storage - It's not straight-forward to implement the Pandas API on top of a Zarr archive, but this was decided to be out of scope.

An example of this conversion was implemented in this test case. A toy example would look like:

    df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})

    root = zarr.open("path/to/arhive.zarr", "w")
    for col in df.columns:
        root.array(col, data=df[col].values)