Repository navigation
New resolver: “Nameless” constraint #8210
Description
Activity
One question - do we have any indication that anyone actually uses this functionality? It looks incredibly weird, and I don't see what the purpose of such a constraint is. (I can technically work it out, but why would anyone use it?)
I'd be reasonably comfortable taking the approach of (at least initially) dropping support for constructs like this in the new resolver, and seeing if anyone complains.
It would affect the rollout process and how we handle backward compatibility, though, because if someone does pop up saying they use this, we'd need a workaround for them while we work out how to add it. It also changes the project goal, as "all tests must pass" won't be quite right in that case.
@pradyunsg @brainwane Any thoughts? Is there any value in asking the user survey population "do you use constraints and if so how?" Or is that too small a sample to be worth it?
PS I'm not going to review the rest of the constraint-related issues until after the weekend, but I wanted to add my thoughts to this one to give people a chance to respond.
I vaguely remember being taught about mentioned constraints as a workaround to pip’s inability to merge extras (the feature described in #8211), but I assume many such usages are not needed anymore since the new resolver does not have the problem. As for direct URL in constraints, no idea at all.
This is definitely a good case to delibreately not include the feature because YAGNI and see what happens IMO.
100% on board for not implementing this, flagging this as a "this no longer works with the new resolver" line item somewhere, and seeing if anyone comes complaining about this during the beta. :)
Constraints were not a work-around for merging extras. They were added by Openstack because they want to make sure that the entirety of openstack is installable together, without having to either maintain 20 separate
requirements.txtfiles OR forcing the install of literally all of Openstack.See #2731 for a tiny bit more information. We would definitely want to reach out to the Openstack community to see if they're still using it.
It is my understanding as well. Sorry if my writing is causing misunderstandings, please feel free to add to/modify the issue.
We would definitely want to reach out to the Openstack community to see if they're still using it.
Openstack are actively testing the constraints implementation in the new resolver (@dhellmann contacted us to help out) so based on your comment, I'd suggest that we wait for feedback from them on whether their testing showed up any issues, and review these issues in the light of that feedback).
I wonder if these tests were mistakenly added from a "completeness" point of view ("because you can add any requirement in a constraints file, we need to decide what
foo[bar]means in a constraints file") rather than focusing on what was requested ("we should only allow version specifiers in a constraints file, so other types of requirement should be invalid or at least not supported")?As far as I know, OpenStack always uses package names, one rule with == or === and a version per constraint entry, and optionally an expression indicating which version of python needs the constraint (I can never remember the name of that feature, sorry). This allows the test jobs to "pin" to the same versions of a package across tests that have different combinations of applications installed.
The global constraints file for OpenStack is at https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt if you want to see an example.
Most projects also have a per-project constraints file using the lower bounds for their requirements, used for a separate unit test job and possibly an application-specific functional test job. We do not run the integration tests with the lower bounds constraints. For an example of one of these files, have a look at https://opendev.org/openstack/nova/src/branch/master/lower-constraints.txt There are going to be a lot of those, since every repo will have its own.
I have asked specifically about nameless constraints on the mailing list http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014790.html
/cc @prometheanfire
we currently do not allow nameless constraints or requirements. As I understand it, those are git/http based uris? we set
permit_urls=False@dhellmann @prometheanfire Could you aldo help verify whether there are currently projects depending on the extras behaviour, as described in #8211?
I'm a bit unsure about the precise meaning of 'nameless' and if this is coupled to the discussion around extras. However I will outline the way openstack-ansible uses constraints files to install a mixture of packages and source code just to make sure that this is captured.
Example is here http://paste.openstack.org/show/793386/ but short story is we use constraints like "git+https://opendev.org/openstack/glance@6e3ced8251cd6e273aa73f553a24fc475b219db5#egg=glance" and put URLs to remote constraints in constraints files all the time.
@jrosser So I'm not at all clear what such a constraint (a URL) would mean in practice. Constraints as I understand them, constrain a project by requiring a certain version or equivalent - so would your quoted example be saying that glance must be installed from precisely that URL - and hence that
pip install glance -c <that constraint file>would fail because you're not requesting the right URL?The constraint as we use it says "install this package, but take the source code from the URL at the SHA or branch specified". Here is an example I just did which installs ansible from our fork which has a bugfix applied:-
(test) ubuntu@bionic-test:~$ cat constraints.txt git+https://github.com/bbc/ansible@add-filters-os_router-os_subnet#egg=ansible (test) ubuntu@bionic-test:~$ pip install ansible --constraint constraints.txt Collecting ansible from git+https://github.com/bbc/ansible@add-filters-os_router-os_subnet#egg=ansible (from -c constraints.txt (line 1)) Cloning https://github.com/bbc/ansible (to add-filters-os_router-os_subnet) to /tmp/pip-build-yjrh1go0/ansible Collecting PyYAML (from ansible->-c constraints.txt (line 1)) Cache entry deserialization failed, entry ignored Cache entry deserialization failed, entry ignored Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB) 100% |████████████████████████████████| 276kB 2.3MB/s Collecting cryptography (from ansible->-c constraints.txt (line 1)) Cache entry deserialization failed, entry ignored Downloading https://files.pythonhosted.org/packages/58/95/f1282ca55649b60afcf617e1e2ca384a2a3e7a5cf91f724cf83c8fbe76a1/cryptography-2.9.2-cp35-abi3-manylinux1_x86_64.whl (2.7MB) 100% |████████████████████████████████| 2.7MB 367kB/s Collecting jinja2 (from ansible->-c constraints.txt (line 1)) Cache entry deserialization failed, entry ignored Cache entry deserialization failed, entry ignored <SNIP>So long as this all keeps working with the new resolver, it's all OK.
19 remaining items
pip install PKGsees the sdists and wheels on PyPI that relate to PKG. The finder in pip fetches data about those, and passes that information onto the resolver to decide which one to use. (The old resolver's logic is a lot more tangled, but conceptually that's how it works. The new resolver makes that separation cleanly).In particular, that invocation does not see https://github.com/OWNER/PKG/archive/master.zip.
A constraint file (as described in the docs, and as implemented in the new resolver) removes items from the list of available candidates, to constrain the options available to the resolver when deciding what to install.
The proposed behaviour for
pkg @"constraints" has an additional effect, in that it changes the list of candidates being considered, to add https://github.com/OWNER/PKG/archive/master.zip (and by implication, remove all the other options that the finder had identified).It's adding that extra candidate to the list that isn't a "constraint" as far as I can see, but rather an "extension" of the list.
Budget and planning constraints are another thing of course, that I totally understand. Although I guess that should be put aside while discussing the mid-long term desired feature set.
Agreed.
How about as an alternative, rather than (mis-)using constraints files for this, we had a separate feature
sources.txtthat allowed the user to specify where a package came from - one source per package, always a URL, so basically a file containing a list of direct URLs that would be used in place of the normal finder lookup via indexes etc?I'm sure this would immediately be subject to massive scope creep, as people decide they want to use it for all sorts of things where a custom index would be a better option, but maybe we could keep it under control 🙂
The part that I still don't get is why direct URLs would be considered a misuse of constraints. I mean when you view them as just another way to specify a version it all makes sense, no?
So a name=>url map such as
sources.txtwould be easily generated by extracting direct URLs from constraints? Assuming they are named of course, and naming them is an acceptable price to pay to use that with the new resolver.The part that I still don't get is why direct URLs would be considered a misuse of constraints. I mean when you view them as just another way to specify a version it all makes sense, no?
Sure. But if you see them as just a way to specify a version (by which I assume you mean version number), then there's no mechanism by which pip would know to consider that URL for installation, as it's not in the list of stuff the finder will locate. So a constraint of https://github.com/pypa/pip/archive/20.1.zip would be effectively identical to a constraint of
pip == 20.1, which I'm pretty sure is not what you intend...It's important here not to get misled by the current implemenation, which treats constraints as (very broadly) just ordinary requirements with a "do not install" flag set. That's not conceptually how they work, though, and in a very practical sense, it's not possible for the new resolver to implement them that way (I know, I tried...)
Assuming they are named of course, and naming them is an acceptable price to pay to use that with the new resolver.
Unnamed constraints are absolutely not allowed, IMO, as how would we know if they applied, if there's no name to match against what we're installing? And again, preparing a source dist just to say "so this is foo? Nope, we aren't interested in that" isn't reasonable. (It's less unreasonable in the current implementation, but again, don't get misled by that).
would be easily generated by extracting direct URLs from constraints?
Depends on your definition of "easy". :)
Needing to "backfeed" potential sources for a name in the new resolver's (correct) model of separating requirement vs installable thing is... definitely not easy in my book.
I'm happy to accept that there might be a need for a feature that says "override the finder to only return the following candidate(s) for this project" - but I don't think it should be part of the constraint mechanism, regardless of what historical accidents of implementation exist in the current constraint code.
+1
Part of the issue here is that "URLs in contraints" was not added intentionally, and has never undergone the "what can it be used for, how does that interact with other options we provide" etc discussions we have when adding functionality -- that's what's happening now. I don't think we should go unplanned functionality -> use cases, but rather use case -> functionality.
That said, it's already an undocumented thing you can do with pip, and Hyrum's Law applies here -- we'd need to do a deprecation cycle if we remove it, but (IMO) that's a separate conversation of what it means and whether that is a good solution for the problem it's solving.
How about as an alternative, rather than (mis-)using constraints files for this, we had a separate feature
sources.txtthat allowed the user to specify where a package came from - one source per package, always a URL, so basically a file containing a list of direct URLs that would be used in place of the normal finder lookup via indexes etc?I'm sure this would immediately be subject to massive scope creep, as people decide they want to use it for all sorts of things where a custom index would be a better option, but maybe we could keep it under control 🙂
I sort of like this. It meets the requirements I have, which could be described as "force installation from an unpublished location because you're running in a CI integration job that is trying to decide whether to publish the thing being tested". Regardless of whether that thing is eventually published to a custom index or PyPI, until the tests pass it isn't ready to be published at all.
Now, the unpublished thing may not have a predictable version. It's not likely to be tagged, so it will have some version like 10.2.1-55 or something, and I'm not even sure it's possible to put that into a constraint file. It's certainly inconvenient to have to update the constraint to something like ">10.2.1" every time a release is tagged. If you have a few repos, it's not a big deal. When you have 10s of dozens, manual processes don't work.
Selection doesn't have to be done via constraints, of course. If there's some other way to do it, that's fine. It could be the source list you describe, or it could even be a single local filesystem path so that if a package is found there then that version must be used. The CI job could pre-build wheels of everything it wants to install from source and drop them all into the same place. The key is that it's not just "a" source, we need some way to express that it is the only allowed source for a given package.
Reacted by Paul MooreIf that suggestion seems like it would suit people's requirements, let's thrash out the details in a new feature request PR. I'll try to get some time to write up the proposal, but it will certainly be a few days at least, and may be longer (I have quite a lot on my plate at the moment) so ping me if I forget, or someone else can start things going if they want.
@pfmoore I'm not convinced we really need to expose a new concept such as
source.txtto users.
I understand "link constraints" or "link overrides" have to be handled differently than version constraints in the resolver implementation, but for the users they can both be expressed in the same constraints files. I'd think that name=>link map can be built transparently for the user by filtering out link constraints at the very beginning of the resolution process.Unnamed constraints are absolutely not allowed
If everyone agrees on this point, I think we're good on this issue's goals. :)
Let's close this issue and move the conversation about URL requirements/sources.txt to a separate new issue?
Reacted by Tzu-ping Chung and Stéphane BidoulI'm gonna close this issue now -- please feel free to write a comment here, if you'd like to dispute with the conclusion here (unnamed constraints are not allowed).
- locked as resolved and limited conversation to collaborators
on Jun 24, 2020
Spawned from #8209.
According to
test_install_distribution_union_with_constraints, this is a valid constraint:This is quite difficult to deal with. We can store the constraint information in a separate mapping (with the URL as key), but I’m not sure how it can be fetched. Here’s how we get the relevant constraint to use:
I guess we can do something like:
but we probably can agree this looks super wrong. Some additional abstraction is needed here.