Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

installing 'datrie' fails during compile (but pip succeeds) #7525

Open
samimia-swks opened this issue Sep 18, 2024 · 9 comments
Open

installing 'datrie' fails during compile (but pip succeeds) #7525

samimia-swks opened this issue Sep 18, 2024 · 9 comments
Labels
needs-mre Needs more information for reproduction question Asking for clarification or support

Comments

@samimia-swks
Copy link

uv 0.4.11 on Ubuntu 22.04

I can install datrie (a dependency of snakemake) by using pip in a pyenv managed python 3.9.6 or 3.12.3 with no issues. pypi is missing wheels for python >3.8 but the compile with gcc succeeds creating datrie-0.8.2-cp39-cp39-linux_x86_64.whl. See attached pip logs.

if I try the same with uv pip install datrie==0.8.2 --verbose I get error: command 'clang' failed: No such file or directory
So somehow uv seems to want to compile this with clang even though I have gcc and pip can use that. See attached uv logs.

If I do CC=$(which gcc) CXX=$(which g++) uv pip install it does use gcc but I get another error command '/tools/llvm/bin/llvm-ar' failed: No such file or directory further down the line.

The only thing that works is to install llvm and clang and then symlink llvm-ar from where it is installed to this seemingly hard-coded path '/tools/llvm/bin/llvm-ar'. This seems pretty hacky....

Now clearly this is mostly the fault of the datrie package, which is seemingly abandoned despite being a dependency of the very active and popular snakemake package, but my question is why does pip succeed where uv fails?

pip_datrie.log
uv_datrie.log
uv_datrie_gcc.log

@samimia-swks samimia-swks changed the title installing datrie fails during compile where pip succeeds installing 'datrie' fails during compile (but pip succeeds) Sep 18, 2024
@zanieb
Copy link
Member

zanieb commented Sep 19, 2024

Interestingly, when I use pip, it builds with clang not gcc. Are you using the same Python interpreter with uv pip? It looks like it's using 3.12 but the pip logs are using 3.9.

@zanieb
Copy link
Member

zanieb commented Sep 19, 2024

Can you reproduce this in CI or a container or something?

@samimia-swks
Copy link
Author

pip was using a pyenv installed python 3.9.6, which I think compiles python from source.
uv was using uv installed python (which I think are pre-compiled). my original logs were with python 3.12 but here they are with 3.9.6. it's the same thing.
uv_datrie.log

I have also attached logs for both pip and uv after installing clang and llvm (apt install clang llvm). it seems like pip still chooses to use gcc, but so does uv ?
pip_datrie_with_clang_installed.log
uv_datrie_with_clang_installed.log

I don't have a CI or container setup unfortunately. But my colleague reproduced this issue on his own Ubuntu machine, if that counts for something. pip works for both of us.

@zanieb
Copy link
Member

zanieb commented Sep 19, 2024

Are you still using the uv-managed Python? We build the Python distribution itself with clang so maybe that's the difference? You can pass --python <path> to make sure you're using the same pyenv interpreter to see if that makes a difference.

I don't have a CI or container setup unfortunately. But my colleague reproduced this issue on his own Ubuntu machine, if that counts for something.

We'll need some sort of minimal reproduction to investigate for real.

@zanieb zanieb added question Asking for clarification or support needs-mre Needs more information for reproduction labels Sep 19, 2024
@samimia-swks
Copy link
Author

Thanks for the quick response. See the logs below for an attempt at a minimal reproduction.
There are two python versions here:

  • one is installed by 'pyenv install 3.9.6' which compiles from source with gcc (~/.pyenv/versions/3.9.6/bin/python)

  • and another with 'uv python install 3.9.6' which as you said is a pre-compiled clang install (~/.local/share/uv/python/cpython-3.9.6-linux-x86_64-gnu/bin/python3 )

As you see in the log:

  • 'pip install' installs datrie successfully for the pyenv python with both CC=gcc and CC=clang.
  • 'uv pip install' fails with both CC=gcc and CC=clang.
  • 'uv pip install --python ~/.pyenv/versions/3.9.6/bin/python' succeeds with both CC=gcc and CC=clang.

In short, the problem appears related to the python installed by uv.

log.txt

@samimia-swks
Copy link
Author

samimia-swks commented Sep 20, 2024 via email

@zanieb
Copy link
Member

zanieb commented Sep 20, 2024

Hm. I thought a wrote a reply but it's not here :/

In brief, I think you're running into a quirk of the Python distribution (which is built with clang): https://github.com/indygreg/python-build-standalone/blob/main/docs/quirks.rst#references-to-build-time-paths

I tried to dig into why this specific path (/tools/llvm) is in the distribution but couldn't figure that out. Maybe because I was looking at an aarch64 build though.

@bluss
Copy link
Contributor

bluss commented Sep 20, 2024

Sysconfig contains this for a typical indygreg build (typical that I've looked at / linux):

AR = "/tools/llvm/bin/llvm-ar"

AR is one of the variables that sysconfigpatcher patches. It might be possible to try that tool and see if it helps the build (but no guarantees, it's empirical only.)

If you want to inspect those variables, here's the way to do it:

uv run python -m sysconfig

@zanieb
Copy link
Member

zanieb commented Sep 20, 2024

Thanks! I wonder if we can/should reset some of these at build time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-mre Needs more information for reproduction question Asking for clarification or support
Projects
None yet
Development

No branches or pull requests

3 participants