Changelogs

This PR enables the use of a Hub-supplied storage JWT to access the R2 bucket. Compared to our current approach of signed URLs and fsspec-compliant implementation on top of the Hub's REST API, this approach does not require any requests to the Hub while the storage JWT is valid. This storage token is obtained through a standard OAuth exchange flow with the Hub. A client in possession of a valid, Hub-issued JWT (used to interact with the Hub's REST API) can exchange it for a valid storage JWT, enabling access for a scope (read or write) on one resource (specified as a URN, where datasets are currently the only resource supported.)

The final implementation is a context manager that handles getting a valid storage token when instantiated, and which exposes methods plus a Zarr store to support the current Dataset operations: getting/setting the root Parquet file, and getting/setting content in the optional Zarr extension.

The approach should be flexible enough to support the new flows in the upcoming XL Datasets.

Along the way, some adventures lead me to implement a Zarr store for S3. My initial approach to use Zarr's FSStore, on top of either fssspec's s3fs or PyArrow's S3FileSystem implementations encountered some snags:

• R2 has the specificity that, when using multipart upload, all parts (except for the last) must be of equal size. Something S3 does not require, and something s3fs does not do.
• PyArrow's implementation has been updated to fix the issue with R2, but being based on AWS' C++ SDK, does not offer the ability to set MD5 checksums on upload, or set metadata on the uploaded files. It's unfortunate that I discovered this after having built a full PyArrowFSStore, since a PyArrow filesystem is not fsspec-compliant, and that's what Zarr FSStore expects.

Ultimately, building a store implementation directly over Boto3 ended up the only answer that would satisfy our needs: equally-sized parts for multipart uploads, and integrity checks for the uploaded content.

Additional work

• This could do a better job of handling token expiry. There's probably a way to gracefully fetch a new storage token if it expires, provided the client's underlying token is still valid.
• The multipart upload could be the entrypoint for making a large upload retryable. It likewise could be made concurrent for some performance gains.

Closes #173

Checklist:

• Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
• Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
• Update the API documentation if a new function is added, or an existing one is deleted.
• Write concise and explanatory changelogs above.
• If possible, assign one of the following labels to the PR: feature, fix, chore, documentation or test (or ask a maintainer to do it for you).