Repository navigation
pip-compile doesn't support the new pip resolver #1190
Description
Activity
Well pip never considers the internal API stable right? :)
For my purposes I don’t actually need the new resolver logic except to deal with pypa/pip#5780. If you don’t think the new resolver internals are stable enough for pip-tools yet, I’m sure I can come up with some hack to work around this issue. :)
Reacted by Albert Tugushev and Maciej "RooTer" Urbański- added a commit that references this issue
on Sep 1, 2020 I'm hitting this error too.
The new resolver is going to be the default from middle or end of October if I understand correctly.
pypa/pip#8937
(This will be very exciting)It probably makes sense to ensure pip-tools works with both
--use-feature=2020-resolverand--use-deprecated-feature=legacy-resolverWe are aiming to release pip 20.3 about a week from now, on Wednesday or Thursday, Oct 28 or 29. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.
@atugushev @graingert How much do you already know about what needs to be done here in
pip-tools?This issue is critical as the new resolver is days away from becoming default. If you was one of those that added
use-feature = 2020-resolverto your~/.config/pip/pip.confin order to benefit from extra conflict resolution in newer resolver you are unable to use pip-compile at all.Reacted by Ricardo Decal and Eric ArellanoSo weirdly the 5.4 release seems to have fixed this for Python 3. But it still breaks for Python 2 (I know, I know). Can someone confirm this behavior, or is it just me?
I…uh…played a bit with the settings hence and forth and suddenly it started working and I can't break it anymore. 😳 It was 100% 5.4.0/20.3 before. I guess my computer is just haunted.
Reacted by Albert TugushevWell pip never considers the internal API stable right? :)
Can't we just use
resolvelibdirectly instead of the pip's vendored copy? This way pip-tools wouldn't need to rely on the internal pip's resolver wrapper but use it directly. Its API is quite simple (I'm in the progress of integrating resolvelib into ansible-galaxy CLI, for example) — using it would mean having a stable public API to rely on but the behavior may differ from what a given version of pip does, of course.1 remaining item
I believe you can work around that for now using
pip check?Well,.. I removed that line from that CI pipeline that was running pip-compile, but having to do this on multiple projects is no joy. We should still have to fix it. Sadly lots of people interested but nobody willing to do it.
Is anyone working on this? Thinking about taking a look but don't want to duplicate work.
This should actually be working now. I think this issue could be closed.
Which version of
pipare you seeing errors with? I usepip==21.0.1(new resolver enabled by default) with the latestpip-compileand things work correctly.okay well I have something that works with
pip installbut notpip compile. I'll see if I can boil it down to the base case."python 3.7.10 pip 21.0.1 pip-tools 6.0.1I have simplest example:
req.txt:asyncpg ~= 0.15.0 asyncpgsa >= 0.22.0pip-compile fails with:
Could not find a version that matches asyncpg>=0.22.0,~=0.15.0 (from -r req.txt (line 1)) Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.1, 0.6.3, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.20.0, 0.20.0, 0.20.1, 0.20.1, 0.21.0, 0.21.0, 0.22.0, 0.22.0 There are incompatible versions in the resolved dependencies: asyncpg~=0.15.0 (from -r req.txt (line 1)) asyncpg>=0.22.0 (from asyncpgsa==0.27.1->-r req.txt (line 2))but pip install succeed with:
... Successfully installed asyncpg-0.15.0 asyncpgsa-0.27.0 greenlet-1.0.0 sqlalchemy-1.4.6Tested on both pypi and github-master versions of pip-tools. Latest pip (since pip-tools required it).
If I change
req.txtas:asyncpg ~= 0.15.0 asyncpgsa >= 0.22.0, < 0.27.1Then I get pip-compile succeed with:
# # This file is autogenerated by pip-compile # To update, run: # # pip-compile req.txt # asyncpg==0.15.0 # via # -r req.txt # asyncpgsa asyncpgsa==0.27.0 # via -r req.txt greenlet==1.0.0 # via sqlalchemy sqlalchemy==1.4.6 # via asyncpgsaReacted by Hugo Ricateauconfirmed pip-compile can't handle,
boto3 botocore==1.20.20but pip finds valid solution:
boto3==1.17.20 botocore==1.20.20So sounds like no ones working on this?
Reacted by Niccolum, Robert Pollak, Hugo Ricateau, xiangqbj and Gonzalo PiottiHi, I'm a maintainer at https://github.com/pantsbuild/pants. We're working on a revamp of our 3rd-party support with lockfiles at the crux of the changes. We've been using pip-compile so far and are really happy with it so far, great UX!
This issue is a blocker for us to call the project complete. We're interested in taking on this issue, but are trying to scope it out: how much work would you anticipate this change will take?
Also, would anyone be available to help, like giving some tips or pair programming?
Finally, Pants still supports Python 2.7, so we'd ideally backport this feature to pip-tools 5, in addition to pip-tools 6. How feasible do you expect a backport like that would be?
Reacted by Nikita Melentev@Eric-Arellano there's a new lock file spec here too https://www.python.org/dev/peps/pep-0665/
Reacted by Eric ArellanoOhh yeah, it only took me to read the header to guess what this pep will propose: more TOML :( ... few pages later, I got my guess confirmed.
#1539 adds support for new resolver and fixes the issue:
Details
❯ pip-compile --resolver backtracking # # This file is autogenerated by pip-compile with python 3.8 # To update, run: # # pip-compile --resolver=backtracking # boto3==1.21.4 # via -r requirements.in botocore==1.24.5 # via # boto3 # s3transfer jmespath==0.10.0 # via # boto3 # botocore python-dateutil==2.8.2 # via botocore s3transfer==0.5.1 # via boto3 six==1.16.0 # via python-dateutil urllib3==1.26.8 # via botocoreReacted by Ben Spaulding, Tyler BoddySpargo, Gonzalo Piotti and Alex RestifoReacted by Eric Arellano, Tyler BoddySpargo and jake-sigtechReacted by Robert PollakThe resolver is released as part of pip-tools
v6.8.0🎉Reacted by Greg Roodt, Eric Arellano, Arthur Rio, Ben Spaulding, Juan Luis Cano Rodríguez, Matt Ritter and Thomas Grainger@atugushev I can't say how much this saved the day for me - thank you for getting this in there when you did!
Reacted by Greg Roodt, Awais Jibran and Gleb T.Reacted by Albert Tugushev and Awais Jibran
pip-compile doesn't support using the 2020 resolver.
Environment Versions
$ python -V: Python 3.7.8$ pip --version: pip 20.2$ pip-compile --version: pip-compile, version 5.3.0Steps to replicate
t.txt:pip-compile t.txt:Expected result
pip-compilecompletes successfully.Actual result
pip-compileerrors out.