Skip to content

Black install via pip fails on embedded Python on Windows using latest pip version (20.3) #1847

Closed
Closed
Black install via pip fails on embedded Python on Windows using latest pip version (20.3)#1847
Labels
C: packagingInstallation and packaging of BlackT: bugSomething isn't working

Description

@robintw
robintw
opened on Nov 30, 2020

Describe the bug
A new version of pip has recently been released, which is a lot stricter in many ways. When installing the latest version of Black on a Windows machine using the embedded Python distribution, I get an error:

ERROR: Requested black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has different version in metadata: '0.0.0'

The package it is trying to download is the latest source release (20.8b1), and from what I can gather, pip finds that the version in the filename (20.8b1) conflicts with the version in the metadata inside the package - which it says is 0.0.0.

To Reproduce

  1. Install the embedded Python interpreter on Windows (https://docs.python.org/3/using/windows.html#the-embeddable-package)
  2. Install pip in the embedded Python interpreter (download https://bootstrap.pypa.io/get-pip.py and run it using the embedded Python interpreter's python.exe)
  3. Run python.exe -m pip install black --no-cache-dir
  4. You will see the following output:
c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Requested black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has different version in metadata: '0.0.0'

If the latest version which has a wheel release published to PyPI is specified instead, then it works fine:

c:\pepys-deploy\pepys-import>python -m pip install black==19.10b0 --no-cache-dir
Collecting black==19.10b0
  Downloading black-19.10b0-py36-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.1 MB/s
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black==19.10b0) (1.4.1)
Requirement already satisfied: toml>=0.9.4 in .\python\lib\site-packages (from black==19.10b0) (0.10.2)
Requirement already satisfied: regex in .\python\lib\site-packages (from black==19.10b0) (2020.11.13)
Requirement already satisfied: attrs>=18.1.0 in .\python\lib\site-packages (from black==19.10b0) (20.3.0)
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black==19.10b0) (1.4.4)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black==19.10b0) (0.8.1)
Requirement already satisfied: click>=6.5 in .\python\lib\site-packages (from black==19.10b0) (7.1.2)
Installing collected packages: black
  WARNING: The scripts black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-19.10b0

So, it seems to be something to do with the metadata checking before building the wheel which is going wrong.

The latest version of Black worked fine with the previous version of pip which didn't use the new strict metadata checker.

I can't seem to reproduce this in non-embedded Python, but it may be an issue there too. I'm aware that pip isn't officially supported for embedded Python on Windows, but it is widely used and all other packages seem to install fine:

Expected behavior
Expected to install successfully.

Environment (please complete the following information):

  • Version: 20.8b1
  • OS and Python version: Windows embedded Python 3.7

Does this bug also happen on master?
N/A as related to PyPI releases

Activity

  1. robintw
    added
    T: bugSomething isn't working
    on Nov 30, 2020
  2. robintw
    changed the title [-]Black install via pip fails on Windows using latest pip version (20.3)[/-] [+]Black install via pip fails on embedded Python on Windows using latest pip version (20.3)[/+] on Nov 30, 2020
  3. cooperlees
    self-assigned this
    on Nov 30, 2020
  4. cooperlees
    added
    help wantedExtra attention is needed
    C: packagingInstallation and packaging of Black
    stable
    on Nov 30, 2020

  5. cooperlees commented on Nov 30, 2020

    @cooperlees
    cooperlees
    on Nov 30, 2020
    Collaborator

    /me boots Windows VM to debug tonight or some time soon hopefully

  6. robintw commented on Nov 30, 2020

    @robintw
    robintw
    on Nov 30, 2020
    Author

    Just as an additional piece of information: the install works when when you tell pip to use it's legacy resolver which isn't as strict about metadata (see below). However, you can see that it is definitely confused about versions - it states Successfully installed black-0.0.0 at the end.

    c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir --use-deprecated=legacy-resolver
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in .\python\lib\site-packages (from black) (0.10.2)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black) (0.8.1)
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black) (1.4.1)
Requirement already satisfied: regex>=2020.1.8 in .\python\lib\site-packages (from black) (2020.11.13)
Requirement already satisfied: mypy-extensions>=0.4.3 in .\python\lib\site-packages (from black) (0.4.3)
Requirement already satisfied: click>=7.1.2 in .\python\lib\site-packages (from black) (7.1.2)
Requirement already satisfied: typing-extensions>=3.7.4 in .\python\lib\site-packages (from black) (3.7.4.3)
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-0.0.0-py3-none-any.whl size=124180 sha256=9d02c9cda7598e7caf1f73046db0bb8db2a1baf63bb54ac8c606c7ea8b143f82
  Stored in directory: C:\Users\ROBINW~1\AppData\Local\Temp\pip-ephem-wheel-cache-h9rra3jz\wheels\c5\85\79\f3af8daaf8037c0bf14beb3b7a1511a39b6e6902ca2aaf494e
Successfully built black
Installing collected packages: black
  WARNING: The scripts black-primer.exe, black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-0.0.0

    Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

  7. cooperlees commented on Nov 30, 2020

    @cooperlees
    cooperlees
    on Nov 30, 2020
    Collaborator

    Just as an additional piece of information: the install works when when you tell pip to use it's legacy resolver which isn't as strict about metadata (see below). However, you can see that it is definitely confused about versions - it states Successfully installed black-0.0.0 at the end.

    c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir --use-deprecated=legacy-resolver
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in .\python\lib\site-packages (from black) (0.10.2)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black) (0.8.1)
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black) (1.4.1)
Requirement already satisfied: regex>=2020.1.8 in .\python\lib\site-packages (from black) (2020.11.13)
Requirement already satisfied: mypy-extensions>=0.4.3 in .\python\lib\site-packages (from black) (0.4.3)
Requirement already satisfied: click>=7.1.2 in .\python\lib\site-packages (from black) (7.1.2)
Requirement already satisfied: typing-extensions>=3.7.4 in .\python\lib\site-packages (from black) (3.7.4.3)
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-0.0.0-py3-none-any.whl size=124180 sha256=9d02c9cda7598e7caf1f73046db0bb8db2a1baf63bb54ac8c606c7ea8b143f82
  Stored in directory: C:\Users\ROBINW~1\AppData\Local\Temp\pip-ephem-wheel-cache-h9rra3jz\wheels\c5\85\79\f3af8daaf8037c0bf14beb3b7a1511a39b6e6902ca2aaf494e
Successfully built black
Installing collected packages: black
  WARNING: The scripts black-primer.exe, black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-0.0.0

    Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

    Yup sure thing. I'll fix and push for a release before then. Thanks for raising.

  8. ichard26 commented on Nov 30, 2020

    @ichard26
    ichard26
    on Nov 30, 2020
    Collaborator

    I'll also look into this soon. I think this has something to do with setuptools-scm breaking as that's how the version is resolved in our source distributions. Why is it not working as expected? I don't know any answers but I'll do some digging.

  9. ichard26
    self-assigned this
    on Nov 30, 2020

  10. robintw commented on Nov 30, 2020

    @robintw
    robintw
    on Nov 30, 2020
    Author

    I imagine a partial solution would be to publish a wheel for the latest version - installing the latest version that had a wheel (19.x) worked fine for me. Obviously it'd be good to work out why it's actually happening too though!

  11. ichard26 commented on Nov 30, 2020

    @ichard26
    ichard26
    on Nov 30, 2020
    Collaborator

    We want to do more releases more often so I'm totally +1 for a new release. Need to check that the CHANGELOG is ready though. One thing, I can't do a release. I clearly have some power to push for a release, but no power to create and push one. That power lies with the other collaborators :-)

    @cooperlees, what do you think?

  12. pradyunsg commented on Nov 30, 2020

    @pradyunsg
    pradyunsg
    on Nov 30, 2020
    Contributor

    Since @cooperlees asked nicely...

    You seem to be hitting pypa/setuptools-scm#386. The main solution is listed in pypa/setuptools-scm#386 (comment). I'm not a 100% sure how you're generating release artifacts, (I don't see your release process documented -- it's usually a good idea to do so, to avoid mistakes!) but it does seem that the sdist doesn't have the metadata that setuptools_scm needs to properly version it, likely due to setuptools_scm not being installed/used in the build environment.

    Also, even though this is a pure-Python project, I strongly recommend uploading wheels to PyPI. Most users who use pip won't really download/uses the sdist if there's a wheel, so you'd also buy some time to investigate the underlying issue here by uploading a well-formed wheel. I'll leave it up to the maintainers to decide whether they want the 20.8 beta wheel built (checkout the 20.8 tag and build the wheel with pip wheel .?) or if they wanna put out a completely new release. :)

  13. RonnyPfannschmidt commented on Dec 5, 2020

    @RonnyPfannschmidt
    RonnyPfannschmidt
    on Dec 5, 2020

    at first glance i believe a part of the problem is that setup_requires is in setup.cfg but the recent enough setuptools version is not ensured
    a second part is that its observable that pip nukes the ability of setuptools_scm to fetch version data when it manages copying things to tmpdirs (and pip does not provide a way to get access to that metadata)

  14. 10 remaining items

  15. pradyunsg commented on Jan 18, 2021

    @pradyunsg
    pradyunsg
    on Jan 18, 2021
    Contributor

    FWIW, https://pypi.org/project/black/#data tells me that there's quite a few folks who can make the release. It might be worth reaching out to them as well.

  16. lucinvitae commented on Feb 26, 2021

    @lucinvitae
    lucinvitae
    on Feb 26, 2021

    I'm seeing a similar issue when pip-installing black within the python:3.8-slim docker image:

    Command ['/usr/local/bin/python', '-m', 'pip', 'install', '--no-deps', 'black==20.8b1'] errored with the following return code 1, and output: 
Looking in indexes: https://pypi.org/simple/
Collecting black==20.8b1
  Downloading black-20.8b1.tar.gz (1.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  WARNING: Requested black==20.8b1 from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea, but installing version 0.0.0
WARNING: Discarding https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea (from https://pypi.org/simple/black/) (requires-python:>=3.6). Requested black==20.8b1 from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has inconsistent version: filename has '20.8b1', but metadata has '0.0.0'
ERROR: Could not find a version that satisfies the requirement black==20.8b1
ERROR: No matching distribution found for black==20.8b1

    The issue has been occurring across our CI clusters for a month now, since mid-January.

    I don't think this issue is limited to Windows.

  17. lucinvitae commented on Feb 26, 2021

    @lucinvitae
    lucinvitae
    on Feb 26, 2021

    For folks looking to workaround this issue, downgrading has fixed the install issues in CI for our company:

    $ pip list | grep black
black                    19.10b0

  18. dciborow commented on Mar 10, 2021

    @dciborow
    dciborow
    on Mar 10, 2021

    Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

    This is no longer a working solution. Our build server went belly up. Could we get a binary release of black? That is how I am resolving for all my other packages...

  19. robintw commented on Mar 10, 2021

    @robintw
    robintw
    on Mar 10, 2021
    Author

    Pinging a few other people listed as maintainers on the PyPI page, in case they can help get a wheel release out, while we wait for a full fix. It seems that this is failing on multiple platforms, and causing issues for CI for a lot of people, and (I hope) should be a fairly simple fix to upload a binary wheel release.

    Is there anything you can do to help get a new release out as soon as possible: @ambv, @autophagy @carljm @willingc ?

  20. cooperlees commented on Mar 10, 2021

    @cooperlees
    cooperlees
    on Mar 10, 2021
    Collaborator

    I'd love to. I just don't have access. I've been personally trying to nudge @ambv ...

    Gonna be a rocky release tho. We have almost ~8 months of merges so people will need to be ready for that.

  21. JelleZijlstra commented on Mar 10, 2021

    @JelleZijlstra
    JelleZijlstra
    on Mar 10, 2021
    Collaborator

    @cooperlees I do have PyPI access, let's talk with the other maintainers to see if we can get a release out.

  22. ichard26 commented on Apr 25, 2021

    @ichard26
    ichard26
    on Apr 25, 2021
    Collaborator

    De-assigning myself since I do not have enough time to debug (also debugging is annoying since I dual boot Ubuntu and Windows but I only daily drive Ubuntu). Hopefully GH-2125 happens soon so this is resolved (or I guess worked around?).

  23. ichard26
    removed their assignment
    on Apr 25, 2021

  24. pradyunsg commented on Apr 25, 2021

    @pradyunsg
    pradyunsg
    on Apr 25, 2021
    Contributor

    FWIW, I just saw that this involves using an embedded interpreter. Using pip with an embedded Python interpreter is explicitly not supported, by Python or pip.

    From the link in OP:

    Using pip to manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates to users.

    I'd suggest closing this issue, since the specific usecase being asked for is not supported by the tooling being used.

  25. cooperlees commented on Apr 25, 2021

    @cooperlees
    cooperlees
    on Apr 25, 2021
    Collaborator

    Agree. The pip king has spoken.

  26. cooperlees
    closed this as completedon Apr 25, 2021
  27. ichard26
    removed
    help wantedExtra attention is needed
    stable
    on Apr 25, 2021
  28. ichard26
    unassigned
    cooperlees
    on Apr 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C: packagingInstallation and packaging of BlackT: bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions