New Resolver: Rollout, Feedback Loops and Development Flow

I've been thinking a bit about #988 (duh!) -- specifically how to roll it out so as to minimize breakage and maximize the opportunity to get useful feedback from users.

Filing this issue now that I finally have both thumbs + time at hand to do so. Obviously, all of what follows is up for discussion. :)

My current plan for rolling out the new resolver is based on exposing the new resolver behind a flag. The flow would be to not document it initially and add big fat warnings on the use of the flag. Once it is less experimental and more beta-ish, we can start inviting users to play with the new resolver. This would involve CTAs to users for asking them to try it out and provide feedback. This information might also be printed when run with the flag.

In terms of feedback management, I am thinking of requesting for feedback on a different repository's issue tracker. The reasoning behind putting issues on a different issue tracker, is to minimize noise here + allow more focused discussions/investigation. I'd bubble up anything that's more than a "bug in the resolution" to the main issue tracker (this one).

In terms of transitioning, I think once there's enough confidence in the new resolution logic, we can look into how we want to handle the transition. Having put this behind a flag, we'll have 2 options -- directly switch over in a release or "stabilize" the new resolver and do a (maybe multi-release?) "transition period". I do think that we can do the transition planning later, when we have a better understanding of the exact trade-offs involved.

In terms of git/GitHub, this is probably the first "experimental" feature implementation within pip. FWIW, I'm planning to do experiments etc on my fork and regularly merging progress to pip's main repository itself (solely code, into pip._internal.resolution). I don't want to be noisy on the main repository but I do want to keep master in sync with work on this.

Note that I'm putting #5051 as a blocker for this work because of how painful dealing with build logic was when building the prototype.

I don't know how you have it planned out, but one comment is that I would encourage you to try to share code as much as possible between the new code and the current code, and refactor the current code as you're working to allow more sharing between the new and current code paths.

One reason is that if you're sharing more code, there will be less chance of breakage when you're toggling the new behavior off and on, because you'll be exercising that shared code in both states and you won't have as many potential differences in behavior to contend with.

This would involve CTAs to users for asking them to try it out and provide feedback

Our track record on getting advance feedback on new features has been pretty bad. We've tried beta releases, releasing new features with "opt out" flags that people can use if they hit issues, big publicity drives for breaking changes, and none of them seem to have worked.

My personal feeling is that "make it available and ask for feedback" is an interesting variation on what we've previously tried, but ultimately it won't make much difference. Too many people use the latest pip with default options in their automated build pipelines, and don't test before moving to a new pip version (we saw this with PEP 517).

I wonder - could we get a PSF grant to get resources to either do a big "real world" testing exercise for this feature, or (better still) develop a testing infrastructure for us? Such a project could include a call for projects to let us know their workflows and configurations, so that we can set up testing paths that ensure that new pip versions don't break them. Or even just use a grant to get someone experienced in the communications aspect of getting beta testers for new features to help us set up a better user testing programme?

In terms of git/GitHub, this is probably the first "experimental" feature implementation within pip

I'm not 100% sure what you mean by that. We've certainly had new features in the past that have been added while the "old way" was still present. We've not tended to leave them "off by default, enable to try them out", if that's what you mean, but that's mostly because we've never found any good way to get feedback (see above).

I spent ~60 minutes (re-re-re-re-re-)writing this one post, so now I will go take a look at places in New York! If you don't see an quick response from me, it's because I'll be in tourist mode.

I would encourage you to try to share code as much as possible between the new code and the current code, and refactor the current code as you're working to allow more sharing between the new and current code paths.

Definitely! This is 80% of why I'm putting #5051 ahead of this -- I intend to pay down a lot of the technical debt we've accumulated in our build logic so that it becomes easier to reuse (all of?) it. A bunch of the code will have to be 🔥 and I agree that the rest should definitely be reused as much as reasonable.

We've not tended to leave them "off by default, enable to try them out", if that's what you mean

Yep, indeed. I'm also hinting at development flow here -- IMO it would be okay to merge empty infrastructure (classes with a bunch of methods that are just raise NotImplementedError() that will get fleshed out in subsequent PRs) or one doesn't cover all the cases (half-baked implementation) into the master branch as long as that's only used behind the flag that is explicitly noted as "experimental/alpha".

re: feedback

I'm young, dumb and optimistic -- I want to make this rollout an opt-in, to get proactive feedback and act on it. By "proactive", I mean from folks who are willing to take out some extra time to try out alpha/beta functionality and inform us about how it is. I think if we make make enough noise and strategically target/reach out to people, we can get good "proactive" feedback from folks who have the time and energy to try out new functionality to help iron out the details/issues.

Looking at our recent "major" changes, I think most of the feedback we received was reactive -- from users realizing issues with their workflows when it broke, and then reaching out to inform us about it. A lot of them may not have had the time to help iron out the details of the new functionality, which causes a lot of friction. These also cost us a lot of our "churn budget" [1], which I don't want to spend more of, since Python Packaging doesn't really have much left anyway [2].

FWIW, I plan to borrow some ideas from the PyPI launch, like making blog posts at fairly visible locations (i.e. not my personal blog), possibly going on podcasts, well-timed actionable emails etc. I'm also looking for more prior art/avenues to communicate via. One of the (many!) things I learnt at PyCon, was that there are channels that we don't use, that will help spread information but won't seek out to check if we have any to spread.

To be clear, I'm not criticizing against the rollout approach we took for PEP 517, I think it's going well, especially given the fact that we're all volunteers. I'm trying to see what we can learn and actionable items to try to avoid the problems we had. Most of these items do involve more work from the maintainers and the main reason I am even spending all this time thinking about this, is because I view this as a fun learning exercise of how to do change management.

re: grants

Yep, I think we can definitely use a grant/more experienced person to help us figure out the communication, roll-outs and testing infrastucture. That does however need someone to do the grant-writing work and figuring out more concrete plans than I can make right now, since I don't have a more stable number of hours / week that I can guarantee.

FWIW, PSF has an ongoing contract to help figure out PyPA/Packaging-related communication with Changeset Consulting, so maybe we can leverage that?

I'm intentionally not @-mentioning people since this is fairly early in the planning state to add more people in the conversation.

Footnotes:

A really nice term that @ pganssle used that I'm definitely going to use.
This is why I've put Deprecate pip, pipX, and pipX.Y #3164 on the back burner, despite having an implementation of the "pip-cli" package proposed there and having reasonable consensus on how we want the rollout to look like.

I'm young, dumb and optimistic

:-) And I'm sometimes too old, weary and cynical. Let's go with your philosophy, it sounds much better :-)

Definitely! This is 80% of why I'm putting #5051 ahead of this -- I intend to pay down a lot of the technical debt we've accumulated in our build logic so that it becomes easier to reuse (all of?) it.

Great!

@pradyunsg

From IRC just now:

[sumanah] pradyunsg: is there anything we the pip & packaging community can do to help you get more work done faster on the resolver?
....
[pradyunsg] Actually, right now, inputs on #6536 would probably help me figure out how to approach the work / get feedback from people etc.
....
[sumanah] pradyunsg: re: New Resolver: Rollout, Feedback Loops and Development Flow #6536 -- the input you want is something like: is the feature flag approach a good idea? is it a good idea to get feedback via some mechanism other than the pip GitHub issues? is it a good idea to get a grant or similar to get realworld manual testing & robust testing infrastructure built, and/or proactive comms?
...
[pradyunsg] Yep -- whether the ideas I'm suggesting are good. Also any additional ideas/approaches/thoughts that might help the rollout + feedback be smoother would be awesome.

So:

Is the feature flag approach a good idea? Yes.

Is it a good idea to get feedback via some mechanism other than the pip GitHub issues? Yes. We should find automated ways to accept less structured bug reports from less expert users.

Would more robust testing infrastructure help? Yes, a lot, and this is someplace our sponsors might be able to help us out.

Could Changeset (me), under the existing contract with PSF to help with PyPA coordination/communications, help pip with proactive communications to get us more systematic realworld manual testing? Assuming that I have hours remaining in my contract by the time we want to start this rollout, yes.

is it a good idea to get a grant or similar to get more help with user experience, communications/publicity, and testing? Yes. The PSF grants would potentially be of interest, as would NLNet grants (for requests under 30,000 euros), potentially the Chan Zuckerberg essential open source software for science grant, and Mozilla's MOSS. The Packaging WG can be the applicant of record. If @pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod, I can start investigating those possibilities with the WG.

@pradyunsg

If @pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod,

It definitely sounds interesting to me :-)

@pradyunsg

@pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod

nods yeah that sounds interesting

@brainwane

Would more robust testing infrastructure help? Yes, a lot, and this is someplace our sponsors might be able to help us out.

@brainwane Also relevant here is https://github.com/pypa/integration-test. I think getting this set up, is another potential area for funding -- we should add this to https://wiki.python.org/psf/Fundable%20Packaging%20Improvements.

OK! I've started talking with the PSF and with the Chan Zuckerberg Initiative folks about applying for a CZI grant via the Packaging Working Group. I've added some details to the Fundable Packaging Improvements page about why the new pip resolver's important, and added the integration-test project to that list. And I've started gathering names of user experience experts who have the capacity to research our complicated all-on-the-command-line package distribution/installation toolchain, talk with users to understand their mental model of what's happening and what ought to happen, and advise maintainers.

If we get money via grants from MOSS, CZI, or NLNET, I think we'd get the money ... October at the earliest, probably. A grant directly from the PSF would be faster probably but "Our current focus is Python workshops, conferences (esp. for financial aid), and Python diversity/inclusivity efforts."

One consideration is that I know Brett & the folks over on the steering council are talking about investing in project management and looking into having some sort of paid resources for managing these projects (triage, project management, etc) and they are talking with the PSF directly. It may be worth reaching out and finding out what they are doing or thinking, since I heard some talks of long term sustainability and it'd be a good thing to be involved in those.

Feature flags are good, opt-ins are good. One thing you might consider is whether you could randomly prompt users to try out the resolver (like, very very very infrequently and only for an install at a time, i.e. not forcing them to turn it on permanently). Then you could indicate how the resolver was helpful (e.g. what did it do for them? what conflicts did it encounter and resolve?)

People coming from javascript or rust for example will also expect a lockfile of some kind, so that may be something to consider...

Sorry to jump in, glad to see this moving ahead!

My personal feeling is that "make it available and ask for feedback" is an interesting variation on what we've previously tried, but ultimately it won't make much difference. Too many people use the latest pip with default options in their automated build pipelines, and don't test before moving to a new pip version (we saw this with PEP 517).

As one of the people that got bit by some PEP 517 issues for this very reason, I'd actually love to see an opt-in way of testing things out. But I only know about this kinda stuff because i subscribed to all the python packaging news sources I could after the --no-use-pep517 flag issue. What I'm saying is that spreading this kind of news is hard and is probably why feedback is hard to get.

I think more people would be interested in this if the information could be disseminated better. Is that what the resources you are seeking would allow for?

To continue on what jriddy is saying, I also feel it'll be really hard to get people to test various feature flags if they have to know about them, make changes to their CI setup for each new flag, etc.

What would seem much more doable, however, is if there is only one feature flag to know about, to test "what's coming up next" in terms of changes that need to be tested. Then people and companies could setup their CI to run that also (without failing builds for errors). I'm thinking of something similar to Rust, where these kinds of changes bake in the "beta" channel of the toolchain, and it's easy to setup another CI channel to run things on the beta toolchain, and send errors to someone.

The key thing is, this setup needs to be learned about and done only once, instead of having to continuously learn about new individual feature flags, modify CI setups or test them manually.

At the end of the day, it's much better long-term for us to do change management for removing the legacy resolver, rather than to maintain it for any amount of time more than absolutely necessary.

Also note that the old resolver will never go away. Pip 20.3.3 will be available for download essentially forever. So if people must continue to use the old resolver, they can pin their version of pip. They just have to accept that they are using an unsupported version, and will benefit from no future improvements to pip. Obviously we don't want that to happen (there's a non-zero maintenance cost even for just having to close bug reports as "won't fix, not reproducible in a supported version of pip") but it's an option for the few users who need it.

@pfmoore

@pfmoore, would that it were that easy! I've already seen build failures because libraries themselves require pip version 20.3+. So even pinning pip to <=20.3 does not solve these issues.

@pradyunsg, thank you for the thoughtful response! If keeping the legacy resolver is untenable, is it possible for the new resolver to operate in a mode that doesn't fail the install? That is, something like an '--ignore-incompatibilies' flag. Yes: I know of a bunch of workarounds to install dependencies that are deemed "incompatible", but they aren't as nice as pip install -r requirements.txt.

@Tankanow

@Tankanow Question that will influence what is possible for pip maintainers going forward (given that, right now, I think we have about 0.2 people's time funded for pip maintenance): are you offering to comaintain this code, or offering funding, or offering to help gather funding, for further work? Thanks!

given that, right now, I think we have about 0.2 people's time funded for pip maintenance

If this 0.2 is supposed to be my time, that's not happened yet. We're definitely 100% volunteers at the moment.

@brainwane

@brainwane, thanks for pointing this out. I can't believe such an important part of the ecosystem is so underfunded. I applaud you and @pradyunsg and all of the others who've dedicated their time to this project.

I will ask my team about corporate contributions to the project. Re my own time and money, I'm happy to contribute one or both to the project when possible. How can I find out more about what is needed?

@pradyunsg

https://pip.pypa.io/en/latest/user_guide/#deprecation-timeline still says that 21.0 will include the removal of the legacy resolver; @pradyunsg could you please update that to "21.1 or 21.2"? That way I can respond to a tweet and point to that documentation.

Hi @Tankanow - I'm sorry for the delay. (I'm behind on correspondence.) https://github.com/psf/fundable-packaging-improvements/blob/master/FUNDABLES.md is the easiest place to look at what's needed in terms of corporate funding! And if you have some personal time to spend improving Python packaging tools, there are bugs in https://github.com/pypa/warehouse/ and https://github.com/pypa/virtualenv/issues that need fixing and are filed in issues. Thanks!

@Tankanow

If keeping the legacy resolver is untenable, is it possible for the new resolver to operate in a mode that doesn't fail the install? That is, something like an '--ignore-incompatibilies' flag.
Yes: I know of a bunch of workarounds to install dependencies that are deemed "incompatible", but they aren't as nice as pip install -r requirements.txt.

@Tankanow Providing “nice” ways to install broken dependencies, is essentially the same as removing incentive for the maintainers to fix their dependencies specifications. I don't think that would be a good move for the ecosystem, mid- and long-term.

@nbraud

@nbraud, this is NOT how dependencies work in the real world. There is no such thing as a broken dependency at the library level. The reason is simple: "compatibility" is a fallacy at the library level. Libraries don't use every line of code in their dependent libraries; they usually only use a few functions or classes. Only a library consumer knows which parts of the library they use.

For example

I use LibraryA and LibraryB.
LibraryA happens to have its own dependency on LibraryB.
LibraryB releases a new version that has some code I need.
I know that the code I use in LibraryA is not broken by the new LibraryB release (even if some other code of LibraryA is broken by the release, I don't care because I don't use that code)
LibraryA does not update it's transitive dependencies to the latest LibraryB
According to pip, there are no "compatible" versions of LibraryA and LibraryB ... but that's simply not true in my use case.

There is no way for library maintainers to know in advance all of the use cases of their libraries. In the end, the pip dependency resolver is a lot of maintenance for no good reason, because even if I fastidiously manage my dependencies to meet all of the transitive requirements, it's still no guarantee that all of the libraries will actually work together. Only a development team knows if their combination of libraries works in their context (runtime, use case, etc.) ... and they know that only by exercising their code (via tests and running in production).

That is, something like an '--ignore-incompatibilies' flag. Yes: I know of a bunch of workarounds to install dependencies that are deemed "incompatible", but they aren't as nice as pip install -r requirements.txt.

That's #8076, which is where I'd suggest taking the rest of this discussion.

@uranusjr

I'm behind on correspondence, and trying to catch up. In the course of closing some tabs, I came across several issues on GitHub, and tweets, that mention an issue/concern, in a Python-related project, involving pip's new resolver.

Anyone who is interested in helping a little bit: you can go there and comment to help them migrate to the new resolver.

Edit by @uranusjr: Use ordered list for easier reference.

I only read the first seven issues.

Does not seem related to pip at all? The reporter was trying to run python setup.py bdist_wheel.
Replied. (I think the issue is outdated.)
They seemed to have already found a solution but were unable to find a person to release it? Black still hasn’t had a release since, which is a quite serious issue on its own, but I don’t think we can help with that.
The issue report makes no sense to me. The requirements.txt is meant to be strict, and you shouldn’t use it (but pip install docker-py instead) to install. The reporter is the same as 2. and they got a similar reply there, so I think this is just a very confused user.
Replied.
The issue that seems to block them has been resolved, I’ve replied to see if they’re interested in progressing a fix.
It seems like they lost contact to the project maintainer and have forked the project to pypyodbc/pypyodbc, which fixed the issue. The old issue can’t be closed because they need to old maintainer to do that, so there’s nothing more we can do here.

Black maintainer chiming in:

Black still hasn’t had a release [...] but I don’t think we can help with that.

To clarify, please don't try to help us with this (releasing) cause you can't. There's some frustrating delays among the core team blocking the release. We got some rather inactive (yet important) core team members unfortunately (which is fine, just annoying that the core responsibilities haven't been managed well). Right now we are waiting for a bugfix from one of them to land.

We got plans to make releasing easier, enough to make it much more frequent, but progress has been painfully slow :/

Closing this out, since... uhm... we've released the resolver. 😅

Navigation Menu

Repository navigation

New Resolver: Rollout, Feedback Loops and Development Flow #6536

87 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

New Resolver: Rollout, Feedback Loops and Development Flow #6536

Description

Activity

cjerdonek commented on May 25, 2019

pfmoore commented on May 25, 2019

pradyunsg commented on May 26, 2019

pfmoore commented on May 26, 2019

cjerdonek commented on May 26, 2019

brainwane commented on Jun 13, 2019

pfmoore commented on Jun 13, 2019

pradyunsg commented on Jun 13, 2019

pradyunsg commented on Jun 19, 2019

brainwane commented on Jun 21, 2019

techalchemy commented on Jun 21, 2019

jriddy commented on Jun 27, 2019

chrish42 commented on Jul 5, 2019

87 remaining items

pfmoore commented on Jan 6, 2021

Tankanow commented on Jan 6, 2021

brainwane commented on Jan 6, 2021

pradyunsg commented on Jan 6, 2021

Tankanow commented on Jan 7, 2021

brainwane commented on Feb 5, 2021

nicoonoclaste commented on Feb 21, 2021

Tankanow commented on Feb 21, 2021

pradyunsg commented on Feb 21, 2021

brainwane commented on Apr 18, 2021

uranusjr commented on Apr 18, 2021

ichard26 commented on Apr 20, 2021

pradyunsg commented on Jan 28, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions