Skip to content

Optionally bootstrap missing Py dependencies #4445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nilsvu merged 5 commits into from
Dec 3, 2022
Merged

Optionally bootstrap missing Py dependencies #4445

nilsvu merged 5 commits into from
Dec 3, 2022

Conversation

@nilsvu
Copy link
Member

@nilsvu nilsvu commented Dec 1, 2022 •
edited
Loading

Proposed changes

When the optional BOOTSTRAP_PY_DEPS CMake flag is enabled, missing Python dependencies are installed to ${CMAKE_BINARY_DIR}/lib/python3.X/site-packages. This is an alternative if the user doesn't want to bother installing the dependencies themselves, e.g. on a cluster. If there's a problem with packages like h5py, numpy or scipy the user can/should still install them themselves to make sure they use the correct HDF5, BLAS, etc.

Enabling this on Wheeler, Ocean and Minerva. The other clusters have setup scripts that install the dependencies in the user directory anyway.

This also fixes #4124 on Wheeler.

Upgrade instructions

Code review checklist

  • The code is documented and the documentation renders correctly. Run
    make doc to generate the documentation locally into BUILD_DIR/docs/html.
    Then open index.html.
  • The code follows the stylistic and code quality guidelines listed in the
    code review guide.
  • The PR lists upgrade instructions and is labeled bugfix or
    new feature if appropriate.

Further comments

@nilsvu
Optionally bootstrap missing Py dependencies in build dir
7cf31b3 
When the optional `BOOTSTRAP_PY_DEPS` CMake flag is enabled, missing
Python dependencies are installed to the build directory.
This is an alternative if the user doesn't want to bother installing
the dependencies themselves, e.g. on a cluster. Packages like
h5py, numpy and scipy can/should still be installed by the user
to make sure they use the correct HDF5, BLAS, etc., but the new
`BOOTSTRAP_PY_DEPS` flag makes it easy to install pure-Python packages.
@nilsvu
Copy link
Member Author

nilsvu commented Dec 1, 2022

I tested this on Wheeler. @geoffrey4444 could you make sure this works on Ocean please? Just build and run unit tests.

@nilsvu nilsvu marked this pull request as ready for review December 1, 2022 20:51
@nilsdeppe
Copy link
Member

nilsdeppe commented Dec 2, 2022

Tests pass on Ocean, fail on wheeler:

Filters: "Unit.PointwiseFunctions.EquationsOfState.DarkEnergyFluid"
Traceback (most recent call last):
  File
  "/home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/core/__init__.py",
  line 23, in <module>
      from . import multiarray
        File
        "/home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/core/multiarray.py",
        line 10, in <module>
            from . import overrides
              File
              "/home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/core/overrides.py",
              line 6, in <module>
                  from numpy.core._multiarray_umath import (
                  ImportError:
                  /home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so:
                  undefined symbol: PyObject_SelfIter

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
  "/home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/__init__.py",
  line 140, in <module>
      from . import core
        File
  "/home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/core/__init__.py",
  line 49, in <module>
      raise ImportError(msg)
      ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from ""
    * The NumPy version is: "1.23.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was:
/home/nils/ds09/spectre/spectre_nils/build_clang/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so:
undefined symbol: PyObject_SelfIter

@nilsvu nilsvu force-pushed the bootstrap_py_deps branch from ad4c427 to 9a8e522 Compare December 2, 2022 22:53
nilsvu and others added 2 commits December 2, 2022 14:55
@nilsvu
Bootstrap missing Py deps on Wheeler, Ocean, Minerva
24749f9
@nilsvu
Require h5py >= 3.0.0
e46e189 
Older versions treat string encodings differently, leading
to compatibility issues.
@nilsvu nilsvu force-pushed the bootstrap_py_deps branch from 9a8e522 to 44ad7fe Compare December 2, 2022 22:56
@nilsvu
Copy link
Member Author

nilsvu commented Dec 2, 2022

Edited the wheeler env file to use the new miniconda module (thanks @nilsdeppe!). Could you try again please? Tests build and pass for me.

@nilsvu nilsvu force-pushed the bootstrap_py_deps branch from 306044c to a3fecd5 Compare December 2, 2022 23:54
nilsdeppe
nilsdeppe approved these changes Dec 3, 2022
View reviewed changes
Copy link
Member

@nilsdeppe nilsdeppe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've run the unit tests on both Wheeler and Ocean and they pass other than Unit.Visualization.Python.Render1D (Timeout)

@nilsvu nilsvu merged commit 6554239 into sxs-collaboration:develop Dec 3, 2022
@nilsvu nilsvu deleted the bootstrap_py_deps branch December 3, 2022 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@geoffrey4444 geoffrey4444 geoffrey4444 approved these changes

@nilsdeppe nilsdeppe nilsdeppe approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Unit.Options.Python.ExtractInputSourceYAMLFromH5 fails

3 participants

@nilsvu @nilsdeppe @geoffrey4444