Repository navigation
2020 resolver does not consider hashes in the constraints file #8792
Description
Activity
We completely overlooked this 🙂
Constraints went through a minor re-design in the new resolver, so there are a few decisions we need to make:
- Do we want to continue to support this usage? I’m inclined to say yes since this usage essentially adds further restrictions on what pip can choose when downloading distributions, which is not unlike version specifiers. Discussions in What does a URL constraint mean? #8253 seems like we are open to applying this kind of “adding further restrictions” semantic to constraints. (Note: we’ll still need to add an error message if we decide to not support this. The current implementation silently ignores hashes.)
- What should we do when hashes are specified in both constraints and requirements? There are three choices I can think of: requirements override constraints, union, and intersection. My intuition goes to intersection (i.e. the constraints restrict the list of hashes further) since that’s how version specifiers are merged. what does the current resolver do? What happens when a package appear multiple times in an install command with different list of hashes?
Interesting, I did not expect that hashes in the constraints file wouldn't be processed, so I was wrong about the cause of my issue. I've made my example simpler in a branch (https://github.com/jwhitlock/pip-resolver-demo/tree/just-constraints). I've added a note to my original bug, and plan to change the title, but I haven't re-written my original bug entirely.
My mental model of a constraints file is that it is the requirements of the requirements that I care about. For example, I use
Sphinxto generate documentation, but I don't particularly care aboutsphinxcontrib-qthelp, it just comes along for the ride. I've evangelized for this model with https://stackoverflow.com/questions/34645821/pip-constraints-files/36848206#36848206.This separation supports the way I approach package updates:
- When updating the project requirements:
- Update a few weeks after each release
- Read the changelog, and think about project changes before and after the update
- Run automated tests after updating
- Run some manual tests around the project functionality that uses that requirement
- Run the project in a staging environment overnight
- When updating the requirements of requirements (which I stick in constraints)
- Batch a bunch at once, otherwise only update for security issues
- Skip the changelog unless it is easy to find and skim
- Let continuous integration run automated tests
- Skip manual tests
- Briefly run on stage before going to production
The other nice feature of constraints is that, if a project requirement stopped requiring them, then they didn't get installed. Otherwise, it seems like I could solve my issue by changing
-c constraints.txtto-r constraints.txt.With that preamble, my answers are:
- Do we want to continue to support this usage? I hope so. When installing with hashes, the hash is as important as a version for the constraint. If someone is going to do something funky, I could imagine a new build for a widely used but somewhat boring requirement-of-requirements, and I'd want the hash to signal the shenanigans.
- What should we do when hashes are specified in both constraints and requirements? For my use case, an intersection (only hashes in all specs) would work. I'd go a step further, and say the hash list must be identical if present. This treats the version number + sorted hash list as the full package restriction.
I think other people's legit use would be a
requirements.txtfile with:Django>=3.0and a
constraints.txtwithDjango==3.1 \ --hash=sha256:1a63f5bb6ff4d7c42f62a519edc2adbb37f9b78068a5a862beff858b68e3dc8b \ --hash=sha256:2d390268a13c655c97e0e2ede9d117007996db692c1bb93eabebd4fb7ea7012b pytz==2020.1 \ --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \ --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048 sqlparse==0.3.1 \ --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \ --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548 asgiref==3.2.10 \ --hash=sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a \ --hash=sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aedso that
pip install -r requirements.txt -c constraints.txt --use-feature=2020-resolverworks and securely installs the listed packages.
- When updating the project requirements:
- changed the title
[-]2020 resolver fails to find hashes in constraint file from included requirement[/-][+]2020 resolver does not consider hashes in the constraints file[/+]on Aug 22, 2020 tl;dr; I feel that we should be cautious, but I don't have a strong objection.
I wouldn't describe the new resolver changes as a "minor redesign" - I think we did a fairly comprehensive overhaul and stripped constraints files down to being purely a way to specify global (version) limits for packages. That gave us a very specific behaviour, that translates well to how pip decides which version to install (don't even consider versions that don't match the limits in the constraint file).
I'm fine with adding extra functionality to constraint files that is in the same vein - removing certain candidates from consideration up front. As @uranusjr says, hashes can be viewed in this way, and so I feel that it's a reasonable extension. However, that logic does mean that IMO the only reasonable interpretation of the case when hashes are in both constraints and requirements is "intersection" - pip never considers anything that doesn't match the constraints, so if a requirement has a conflicting hash, pip will see nothing and hence will fail. I'd be unhappy with "requirements override" or "union" because they don't fit that model.
My mental model of a constraints file is that it is the requirements of the requirements that I care about.
That's not how I viewed them when I designed the new implementation. As I said above, the intended interpretation is that constraints limit what candidates pip sees. Most of the time, I don't expect there to be much difference in the viewpoints (which has positive and negative aspects - we won't conflict with people's expectations, which is good, but people may want to push constraints in a direction we didn't intend, which is less so...)
Reacted by Pradyun GedamSome additional context:
pip currently merges hash lists (i.e. a union operation) if a requirement is specified multiple times. So this
# This one points to the sdist. six==1.15.0 --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259 # This points to the wheel. six==1.15.0 --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5cedhas the same effect as
six==1.15.0 \ --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259 \ --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5cedThe ordering of hashes does not matter, nor whether the line is a constraint or requirement. So if we’re going to change the constraint file’s behaviour, we should probably also change the merge logic in requirements.txt as well. But the problem is, the merge is currently done during the parsing stage, and shared between both resolvers. This can be tricky to change 😥
Ah. Yes, that will be tricky. But IMO having a consistent model is the important thing (something pip has traditionally not been very good at doing 🙁) so we should either come up with a different model for constraints that makes union the "natural" interpretation¹, or bite the bullet and make the tricky change. Personally, I like the current model as it's very easy to explain.
¹ And, of course, review the existing constraints code to make sure it conforms to the changed model 🤷
Where does the merging occur?
My understanding was that we were handling the hash-checking-related rejections in Provider.find_matches (and, finally, in a loop in whatever the relevant method on
Factoryis).The merging is done in the
Hashesclass (which is onInstallRequirement). I traced the code and believe it’s possible to only change the new resolver implementation to do the intersection logic (and keep the legacy resolver as-is, performing union). PR coming shortly.Reacted by Pradyun GedamI traced the code and believe it’s possible to only change the new resolver implementation to do the intersection logic (and keep the legacy resolver as-is, performing union).
I did this too, and I was wondering whether I was missing something. :)
PR coming shortly.
\o/
6 remaining items
The 20.2.4 release didn't fix my issue. Should I open a new bug or re-open this one?
Consider this
requirements.txt:-c constraints.txt Django==3.1 \ --hash=sha256:1a63f5bb6ff4d7c42f62a519edc2adbb37f9b78068a5a862beff858b68e3dc8b \ --hash=sha256:2d390268a13c655c97e0e2ede9d117007996db692c1bb93eabebd4fb7ea7012band this
constraints.txt:pytz==2020.1 \ --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \ --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048 sqlparse==0.3.1 \ --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \ --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548 asgiref==3.2.10 \ --hash=sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a \ --hash=sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aedThis works:
pip install -r requirements.txtand this fails:
pip install -r requirements.txt --use-feature=2020-resolverwith:
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not: asgiref~=3.2.10 from https://files.pythonhosted.org/packages/d5/eb/64725b25f991010307fd18a9e0c1f0e6dff2f03622fc4bcbcdb2244f60d6/asgiref-3.2.10-py3-none-any.whl#sha256=9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed (from Django==3.1->-r requirements.txt (line 2))@jwhitlock I think the fix to the issue is surfacing another different bug (that was not triggered previously since pip fails before hitting it). It would be best to track it in another issue.
Reacted by Pradyun Gedam- locked as resolved and limited conversation to collaborators
on Oct 9, 2021
See:
Update 1:
The issue was not the projects multiple requirements file, but instead the combinations of using hashes and a constraints files. The current resolver uses hashes on a requirement in a constraints file, while the 2020 resolver ignores them, and fails to install because they do not have hashes. The description below describes the more complex version.
Update 2:
Merged in the changes, so the default branch of pip-resolver-demo describes the simpler version, and includes the
django.txt/django-versions.txtexample as well.What did you want to do?
Our requirements files include other files, as a way to only specify a requirement once for two different environments (development and building in ReadTheDocs.org):
default.txt:-c constraints.txt,-r docs.txt,-r shared.txtdocs.txt:-c constraints.txt,-r shared.txtshared.txt: Noneconstraints.txt: NoneAll of our requirements are specified with hashes, populated with hashin.
This works, without warnings, when installing with
pip install -r default.txtand pip 20.2.2.When installing with pip 20.2.2 or pip-20.3.dev0 (today's in-development version), this fails:
The
idna==2.10requirement is inconstraints.txt, with the sha256 hash.A similar error occurs when installing
docs.txt:Output
Here's
pip install -r default.txt --use-feature=2020-resolver:Here's
pip install -r docs.txt --use-feature=2020-resolver:For comparison, here's
pip install -r default.txt:Additional information
Full dependency tree: