Deprecate --allow-unsafe option

@jcushman

I'm starting to think it might make sense to deprecate --allow-unsafe and make it the default behavior, but that would be as a separate issue/PR for discussion.

Originally posted by @jcushman in #806 (comment)

Additional context

pypa/pip#6459
https://stackoverflow.com/a/58864335/1529164

As a user, I'd love to see this! I find both the flag and the warning message printed without the flag confusing.

Add --no-allow-unsafe option to compile (#1265)

Out of curiosity, does anyone know the reason why these packages were considered unsafe in the first place?

I tried to track it down a while back to assure myself that --allow-unsafe is actually safe. Since I could not find anything I assumed the reason is that the result of installing from a requirements file could be different depending on the order that packages are installed and that, consequently, I would be safe if I installed setuptools and pip explicitly before the other packages. (I am aware that some other packages are also sensitive to order, such as opencv and opencv-contrib 😒)

I don't know, but atugushev asked the same question back in 2019 and tagged some people who might know, and no one offered any answers. So it seems like whatever the answer is, we're not super likely to find someone who remembers.

atugushev also noted in that thread that "unsafe packages are also used in pip-sync to ignore uninstall those packages," so maybe another guess could be that they were originally marked as "unsafe" packages because they were unsafe to uninstall rather than unsafe to pin, and that concept spread to other parts of the codebase accidentally.

Thanks! I never considered that pip-sync may be a factor. I just sort of assumed that it was copied from pip where setuptools, and later pip and wheel, have been excluded from freeze for time immemorial (the migration to git in 2008).

Would it be time to do the switch? --allow-unsafe has been deprecated for two years now and while I don't have any kind of numbers to back this up I think plenty of people have been using it just to silence the warning for much longer than that. I didn't even realise that it might be related to pip-sync since the most common use case of pip-tools for me has always been generating a "lockfile" to be fed for pip install -r in the CI. And I'm pretty sure I'm not alone with that one.

@atugushev

@atugushev I'm unclear of the scope of this issue... is it:

drive towards a decision of do/don't deprecate --allow-unsafe
the decision has already been made, execute the code / docs changes necessary to alert users of the impending removal
decision has been made and users have been warned, need to actually remove it in the next major release (too bad it didn't land in 7.0)

Anyway, curious what's the next action on this issue?

It's stated on deprecations section.

In future versions, the --allow-unsafe behavior will be enabled by default. Use --no-allow-unsafe to keep the old behavior. It is recommended to pass the --allow-unsafe now to adapt to the upcoming change.

We simply forgot to switch the default in 7.0. :/

I'd love to understand why these packages are considered (un)safe for pining...

One thing that comes to my mind: My application is really an application and not a library. I have a fully pinned set of dependencies, and my users should never use other versions (yes, we release updates if libraries have security fixes that are applicable to our usage of them). So the requirements.txt from pip-compile is used for my package's install_requires metadata.

Now, when someone who's installing that app creates a venv, then that venv contains pip and setuptools. Not sure which version is chosen, "latest" would be my expectation especially for pip. But now I pip install myapp, and unless I did a release very recently, it'll probably downgrade pip and setuptools - just because pip-compile now thinks it's a good idea to pin those as well.

AFAIK pip and setuptools are both VERY strict about backwards compatibility and I'd even consider them part of the Python stdlib even though they aren't really part of it. So why would I want to pin those...?

If there was at least an option to exclude those packages (either as "unsafe" or just ignoring them) without having to specify this on the command line all the time (e.g. a config file), I think the change of the default wouldn't feel so annoying.

setuptools is not particularly strict about backwards compatibility. I've been broken by it many times over. The community is moving away from setuptools having any sort of special-cased position in the ecosystem. There's maybe more of an argument for pip, since it's special-cased by stdlib via ensurepip / the last time I was broken by pip was 20.3. But pip does still make breaking changes, e.g. I know 24.1 will break some things for me (and if you see the changelog they make breaking changes every now and then).

IMO, pip-tools is definitely doing the right thing by changing the default here.

If explicitly specifying --unsafe-package setuptools is too verbose, maybe use a wrapper or an alias? An --unsafe-package-file option seems like a reasonable but separate feature request.