Cross-compile from sources for Yocto/Linux

I need to cross-compile scipy from source for Yocto/Linux. Even though I have touched issues

cross-compile for open-embedded or cross compiling fails or unable to install

I still see problems to cross-compile. Here scipy can't find openblas even though openblas has been added to system sysroot.

NOTE: Executing RunQueue Tasks
ERROR: scipy-1.0.0-r0 do_compile: python setup.py build execution failed.
ERROR: scipy-1.0.0-r0 do_compile: Function failed: do_compile (log file is located at /../build/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/scipy/1.0.0-r0/temp/log.do_compile.12169)
ERROR: Logfile of failure stored in: /../build/tmp/work/cortexa9hf-neon-poky-linux-gnueabi/scipy/1.0.0-r0/temp/log.do_compile.12169
Log data follows:
| DEBUG: Executing shell function do_compile
| ERROR: python setup.py build execution failed.
| ../build/tmp/sysroots/x86_64-linux/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
| warnings.warn(msg)
| lapack_opt_info:
| openblas_lapack_info:
| libraries openblas not found in ['../build/tmp/sysroots/x86_64-linux/usr/lib']
| NOT AVAILABLE

By cross-compiling with Yocto there several sysroots subdirectories (e.g. qemuarm, qemuarm-tcbootstrap, and x86_64-linux),

qemuarm-tcbootstrap directory is used as an intermediate directory
x86_64-linux directory is used o build host tools and libraries
qemuarm target machine directory for headers and libraries

I guess scipy is looking for a wrong directory for "openblas" (other dependencies )

libraries openblas not found in ['../build/tmp/sysroots/x86_64-linux/usr/lib']

How to fix it or set it correctly for cross compilation?

How to fix it or set it correctly for cross compilation?

I have no idea, sorry. In gh-8226 apparently someone had some success, but the official answer I think is that distutils does not properly support cross-compiling and that as NumPy/SciPy devs we accept patches where it make sense but have never tried to support this systematically.

Hi!

I managed with a (hoefully) properly cross compiled openblas to start cross compiling scipy. It fails because it cannot find a fortran compiler. Although $FC is properly set. I think, before writing many lines of code trying to guess the name of the fortran compiler, it would have been easier to just use $FC and let people set it according to their needs.
I am trying to read through those lines in numpy/distutils but I do not have much understanding of python. From my point of view, everything regarding fortran compiler can be replaced with env['FC'] and maybe a small fortran test program that uses f95 feature to test if the compiler works...

env['FC'] isn't exactly cross-platform friendly (Windows). But yeah, the distutils design is a mess due to historical reasons.

There seems to be a large number of reports on cross-compiling the last weeks, we never had those before. Is there a reason for that?

Hi,
I have no experience with how properly distribute python packages. But digging into scipy the hole day I get the impression, that distutils or how it is used is the root cause for not being able to cross compile. I can only suspect that a clean switch to setuptools might solve the problem?!

I cannot explain the increase in demands for cross compiling scipy, but if you want to have a product using scipy, using openembedded or buildroot is the only way to go in my humble opinion.

Unfortunately I need use a software component that again depends on scipy.

Funnily enough, I did manage to cross compile scipy with a couple of ugly hacks, and as of now it did not jump on me, but that does not mean, I completely works as expected.

that distutils or how it is used is the root cause for not being able to cross compile. I can only suspect that a clean switch to setuptools might solve the problem?!

Yes. It was never designed with cross-compiling in mind. And no, setuptools does almost nothing different, it is built on top of distutils and it inherits the whole build architecture. We're already using setuptools, but as a build tool at adds nothing of value.

There was a thread on a Yocto Project list in March. People have been attempting to build scipy with the openembedded build system on and off for several years. I just got asked about it again and will spend some cycles reviewing the progress to date. I do not see any fundamental reason why this shouldn't work without the right changes, the vast majority of python packages build fine for us.

@DasRoteSkelett

@DasRoteSkelett could you please share with us what those ugly hacks are? Or at least give us a few hints, in case someone else faces this problem in the future. I myself am in the same situation you were, trying to cross-compile scipy for openembedded and failing miserably :(

@Schyther

Hello @Schyther ,

please find in the tar a patch to numpy and the recipes that worked for me. Since I was cross compiling x86_64 on a x86_64, I cannot be 100% sure there is no host contamination.
IIRC, getting fortran to cross compile was also a hustle. If you also use openembedded / yocto, I can dig into what I did...

scipy-numpy-yocto-recipes.tar.gz

Regards,

Matthias

Thank you very much for your promptitude. I have applied the said patch and used the recipe you offered me. This time it sort of works, but fails somewhere near the end I might say prompting me that a certain static library is missing, even tho I can pretty clearly find it in both the native and target sysroots. I guess it might be happening because I am trying to cross-compile for python3, but I couldnt find a reason for that. Ill try to dig into the issue a little more, maybe I'm missing something and come back with an update :)

Hm, well, I remember that lapack, openblas and fortran needed some love as well.
Here are my recipes, I do not know if all is needed and up to date (was using rocko back then)
openplas-lapack-libgfortran.tar.gz

Regards

Im coming back with yet another update. The compilation was successful, but only after manually settings some of the env variables such as appending to LDSHARED and CFLAGS and LDFLAGS the location of npymath library (distutils was trying to search for the said library in the native sysroot, which contains x86 libraries and was failing with the motive that said library could not be found). Manually setting those flags solved the issue and the package compiled and installed just fine. Thank you again for your help ! :)

If you made the changes in a yocto recipe, it would be nice to share it. And thanks for the feedback!

@Schyther

@Schyther I'm trying to do the same thing with Python3 and Yocto. Could you share explicitly what you did to get SciPy to work? I'd appreciate any help. Thanks in advance.

@whitbennett

@whitbennett sorry for the delayed response. If you still need the recipe I will post as soon as possible, after I clean it up a little bit to make it more user friendly so to speak (some stuff I did there is pretty messy..). Also the compilation was successful but I cant promise that using the troublesome module will work as intended as I did not have enough time to test it.

@Schyther

@Schyther I was able to get lapack, libgfortran, and openblas to to compile, but I'm running into those same issues you brought up about being unable to find them when scipy tries to compile. Your help is most appreciated. I'll look into testing and posting the results here after I get it working.

@whitbennett

@whitbennett @DasRoteSkelett here is what I managed to bundle up
python3-scipy.zip

Hope this helps in any way. It's pretty much @DasRoteSkelett's recipe, with a few tweaks regarding the flags (as I have stated in a previous post)

@Schyther

@Schyther I cant find openblas-dev and I think it's holding me up. When I get to python3-scipy_1.1.0.bb:do_install, it fails on a error from numpy.distutils.
ImportError: cannot import name 'customized_ccompiler'
Any suggestions?

@whitbennett

@whitbennett be sure to add the following line to the openblas recipe in order to be able to use openblas-dev

FILES_${PN}-dev = "${includedir} ${libdir}/lib${PN}.so"

@whitbennett

I've been trying to get scipy compiled in a Yocto Rocko build as well. And got stuck at the same step as @whitbennett. After some searching I actually managed to fully compile scipy by adding lapack to DEPENDS and RDEPENDS, removing openblas-dev from DEPENDS and by using numpy version 1.14.5 (instead of 1.13.1). However when trying to run scipy on target I ran into #5800.

@whitbennett

I've been trying to get scipy compiled in a Yocto Rocko build as well. And got stuck at the same step as @whitbennett. After some searching I actually managed to fully compile scipy by adding lapack to DEPENDS and RDEPENDS, removing openblas-dev from DEPENDS and by using numpy version 1.14.5 (instead of 1.13.1). However when trying to run scipy on target I ran into #5800.

@Drazgor did you manage to solve your issue with #5800? I'm in same situation

I did not. Ended up removing scipy from the project since we were only using a few basic algorithms in the first place.

@Drazgor

@Drazgor I had ONLY_CBLAS=1 in my openblas recipe which seemed to cause that. I removed it and now everything is working.

Hi all, I was able to get Scipy to successfully compile, link, and package on Yocto Linux. I uploaded the recipes and patches to a repo here.

I realize the original post was about Yocto, but since the title of the issue is solely about cross compilation, I figured I'd share what I had to do using a different method.

I was able to get numpy/scipy cross-compiled for ARM, using crossenv. I needed to patch configure.ac so that -pthread was set correctly for the C++ compiler, see python/cpython#22525 for the patch + details. Our full cross-compilation docker image can be found here: https://github.com/robotpy/robotpy-cross-docker

Assuming one gets the cross-compilation environment setup properly, here's the steps I followed:

I use openblas, so setup ~/.numpy_site.cfg with the following contents so that it finds it:

[openblas]
libraries = openblas
library_dirs = /usr/local/arm-frc2020-linux-gnueabi/lib
include_dirs = /usr/local/arm-frc2020-linux-gnueabi/include
runtime_library_dirs = /usr/local/lib
extra_link_args = -lgfortran

Note: I also found that if I just copy the libraries and include files to the cross-venv (/build/venv/cross/lib and /build/venv/cross/include) then numpy would find OpenBLAS without needing to set up a ~/.numpy_site.cfg.

Then run these to install numpy

source /build/venv/bin/activate

# Numpy dependencies
build-pip install Cython==0.29.21
cross-expose Cython

# Cross compile numpy wheel
F90=arm-frc2020-linux-gnueabi-gfortran pip -v wheel --no-build-isolation numpy==1.19.2

# Install the compiled wheel in the cross env
cross-pip install numpy-1.19.2-cp39-cp39-linux_armv7l.whl

# Install numpy on the host also because numpy distutils requires it
# - The manylinux wheel works fine here for python 3.8, but since I moved
#   to Python 3.9 I actually built a host wheel in a separate step since
#   they don't distribute manylinux wheels for it yet
build-pip install numpy==1.19.2

Next, you have to build scipy. Numpy 1.18+ has an environment variable you can set to configure libraries (see https://numpy.org/devdocs/reference/distutils.html#numpy.distutils.misc_util.Configuration.add_npy_pkg_config), so we have to do that:

pkgcfg/npymath.ini:

[meta]
Name=npymath
Description=Portable, core math library implementing C99 standard
Version=0.1

[variables]
pkgname=numpy.core
pkgdir=/build/venv/cross/lib/python3.8/site-packages/numpy/core
prefix=${pkgdir}
libdir=${prefix}/lib
includedir=${prefix}/include

[default]
Libs=-L${libdir} -lnpymath
Cflags=-I${includedir}
Requires=mlib

And pkgcfg/mlib.ini:

[meta]
Name = mlib
Description = Math library used with this version of numpy
Version = 1.0

[default]
Libs=-lm
Cflags=

[msvc]
Libs=m.lib
Cflags=

Now we can build scipy:

NPY_PKG_CONFIG_PATH=/path/to/pkgcfg/ F90=arm-frc2020-linux-gnueabi-gfortran pip -v wheel --no-build-isolation scipy==1.5.3

And that's it! You have two numpy/scipy wheels, install them and they seem to work.

At some point in the near future I'll hopefully get all of this building via github actions, will end up in this repo (not published yet).

For those interested, I've started a conversation over on numpy's repo about the changes necessary to properly support cross-compilation using numpy's distutils. Once those changes have been made, there's some changes that will be needed in scipy's build, but that PR comes later.

Issue: ENH: move numpy.distutils to numpy_distutils toplevel package numpy/numpy#17620
PR: ENH: move numpy.{f2py, distutils} to toplevel package numpy/numpy#17632

xref gh-14812 for an overview of all cross-compilation issues and use-cases that still seem relevant. If anyone has a use case (platform or build system for example) that's not tracked there, it'd be great if you could comment there.

Now that we've switched to Meson, the cross-compilation situation is much more tractable. We have some very minimal docs on how to avoid running the Python interpreter (see here, and we have successful cross builds both with crossenv (conda-forge) and without it (Void Linux).

So there's nothing specifically left to do here - this issue is about the old setup.py-based build here, so I will close it. For further cross compilation improvements (which are certainly needed and desirable), see gh-14812.

Thanks all!

Navigation Menu

Repository navigation

Cross-compile from sources for Yocto/Linux #8571

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cross-compile from sources for Yocto/Linux #8571

Description

Activity

rgommers commented on Mar 17, 2018

DasRoteSkelett commented on Mar 22, 2018

rgommers commented on Mar 22, 2018

DasRoteSkelett commented on Mar 22, 2018

rgommers commented on Mar 22, 2018

balister commented on May 23, 2018

Schyther commented on Sep 19, 2018

DasRoteSkelett commented on Sep 20, 2018

Schyther commented on Sep 21, 2018

DasRoteSkelett commented on Sep 21, 2018

Schyther commented on Sep 24, 2018

DasRoteSkelett commented on Sep 24, 2018

whitbennett commented on Oct 16, 2018

Schyther commented on Oct 25, 2018

whitbennett commented on Oct 25, 2018

Schyther commented on Oct 26, 2018

whitbennett commented on Nov 2, 2018

Schyther commented on Nov 6, 2018

Drazgor commented on Jan 23, 2019

ttmms commented on Jan 3, 2020

Drazgor commented on Jan 3, 2020

ttmms commented on Jan 13, 2020

gpanders commented on Mar 2, 2020

virtuald commented on Oct 24, 2020

virtuald commented on Oct 25, 2020

rgommers commented on Oct 7, 2021

rgommers commented on Apr 24, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions