Repository navigation
Tracking aiobotocore #528
Description
Activity
FYI I started to work on creating env files for testing here (#554). Could be a way to specify aiobotocore version in tests/ci.
Pinning aiobotocore to 1.4.2 depends on botocore 1.20, but up to date versions of boto3 need botocore >= 1.23.
Apologies if it's unrelated, but the error from #514 has shown up again in zarr-developers/zarr-python#914
I'm afraid you'll need to update your s3fs too. The required version of aiobotocore is now fixed, since that happened, but of course I can't fix the previous ones.
(If that was meant for me) @martindurant: isn't 2021.11.1 from zarr-developers/zarr-python#892 the latest?
@joshmoore
Yes - but now it works, right?I thought so, which was why I was worried when zarr-developers/zarr-python#914 started failing (last ~12 hours). I'll re-launch.
(No change)
Looks related to use of
fsspec[s3]. Updated on zarr-developers/zarr-python#914- added 2 commits that reference this issue
on Dec 21, 2021 It is now close impossible to use s3fs and boto3 in the same project. At least pip-tools doesn't know how to resolve a suitable boto3 version and we get 0.4.0 for s3fs, which hardcodes vulnerable urllib3 and all.
Reacted by JBurley and montagI want to echo @ztane that the
aiotbotocorepin and consequent incompatibility with any other package that requires (even vaguely modern)boto3** when managing viapip-toolsis a huge problem. Most critical ismoto[boto3]being incompatible, which essentially means projects cannot use s3fs and test their s3 protocols.I understand that packaging dependencies across projects is annoying for the maintainers of any individual github project, but I'd love to see this one get resolved.
Pip-tools improved their dependency resolver with backtracking 15 days ago and even that doesn't solve this.
** Because
boto3andaiobotocorecannot find mutually acceptablebotocore.Reacted by Josh Moore and montagI agree that installs not doing the right thing or not working at all, and requiring more precise definitions of package versions is a pain. However, it still seems like a lesser pain then relaxing the pin on our end, and having an upstream update break our code, as has happened before.
I would also not that the large fraction of the community that installs via conda/mamba, there is no problem.
Finally, testing with s3fs and moto is clearly possible, since we do exactly that in this repo :)
- added a commit that references this issue
on Aug 19, 2024
Starting with 2021.08.0, we have a hard pin on the aiobotocore version, after their removal of the top level reference to the session class broke our code.
For the future, how do we decide on the best pin? Do we need a set of rolling PRs updating the pin and seeing if tests pass?