Skip to content

Conversation

@tylerjereddy
Copy link
Contributor

@tylerjereddy tylerjereddy commented Apr 1, 2024

This PR contains a number of backports outlined below, and tentatively aims to position us to skip RC2 and just right to SciPy 1.13.0 "final" because of the urgency of supporting the ecosystem around NumPy 2.0.0 support/wheels. I'll add a checklist item below for hearing objections/last-minute changes related to this acceleration of pace. Locally, this branch passed full suite alongside NumPy 2.0.0rc1 and the full doc build passed locally as well.

Backports Included (so far):

  1. BUG: optimize: Fix constraint condition in inner loop of nnls #20168
  2. MAINT: sparse: add missing dict methods to DOK and tests #20175
  3. TST: linalg: fix complex sort in test_bad_geneig #20197
  4. BUG: Optimize: NewtonCG min crashes with xtol=0 #20299
  5. MAINT: bump pocketfft, MacOS patch #20313
  6. BUG: sparse: Restore random coordinate ordering to pre-1.12 results #20314
  7. BUG: signal: Fix scalar input issue of signal.lfilter #20318
  8. DOC: mention BSpline.insert_knot in the 1.13.0 release notes #20327
  9. BUG: linalg: raise an error in dnrm2(..., incx<0) #20341
  10. BUG: nelder-mead fix degenerate simplex #20345
  11. BLD: require pybind11 >=2.12.0 for numpy 2.0 compatibility #20347 (partial cherry-pick because pyproject.toml was already updated on rel branch)
  12. Do not segfault in svd(a) with VT.size > INT_MAX #20349
  13. BUG: optimize: Fix wrong condition to check invalid optimization state in optimize.scalar_search_wolfe2 #20350
  14. DOC: remove outdated NumPy imports note #20353
  15. ENH: Converting amos to std::complex #20359
  16. ENH: Rest of amos translation #20361 (only a direct pull-in of the license update from main, to mitigate risk)
  17. MAINT, BUG: bump OpenBLAS #20362
  18. BUG: interpolate: Fix wrong warning message if degree=-1 in interpolate.RBFInterpolator #20364
  19. MAINT: spatial: use cython_lapack in spatial/_qhull.pyx #20337

TODO:

  • check the diff over (I'm working a bit fast)
  • hear out any objections to the accelerated pace, and/or additional backports needed to make folks happy about it
  • try the full wheel builds on this branch before it goes in, to increase likelihood of smooth release process

ezander and others added 24 commits April 1, 2024 14:34
Co-Authored-By: Elmar Zander <elmar.zander@googlemail.com>
* add missing dict methods to DOK and tests

* turn off some dict dunders for dok_array

* tweak tests
* BUG: Optimize: NewtonCG min crashes with xtol=0

The minimize function crashes if `xtol` is set to `0`. This is
because the initial value of a loop variable is `2 * xtol`, which
is intended to ensure the loop runs at least once. However, if
`xtol = 0` then the loop never runs, a variable isn't instantiated
that is used outside of the loop, and an `UnboundLocalError` is
raised.

To fix this, the initial value of the loop variable is set to the
max float value.

* BUG: Optimize: UnboundLocalError for xtol=0 NCG

Corrected error from previous rework when switching to
`np.linalg.norm` that introduced the possibility of the unbound
variable error.

* TST: Optimize: NewtonCF Minimize when xtol=0

Regression test for scipygh-20214 which addressed an issue in the
Newton CG method which would raise an UnboundLocalVariable error
if `xtol` was set to zero. It also ensures the method finds the
solution in this case.

* MAINT: Add comment to explain initial value

Comment added to explain the purpose of setting `update_l1norm` to the max float value - this is to ensure the preceding while loop executes at least once.

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>

* TST: Test solution value as well as success

This is for the `xtol=0` test in the Newton CG minimize method.

* TST: Explicitly extract val from array

---------

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
* related to scipygh-20300, though awaits `conda-forge` testing
on old MacOS versions for actual confirmation of fix

* pulls in changes described at mreineck/pocketfft#11
and scipy/pocketfft#2

* I'm honestly not sure how much we gain by being conservative and
not just turning the code path off entirely
(mreineck/pocketfft@33ae5dc);
perhaps there are some performance advantages or something, but if
we have any more trouble with this we should obviously just do that
(and I suspect that's what NumPy should do for
numpy/numpy#25940)

[skip cirrus] [skip circle]
This was introduced in scipygh-20193; no longer needed with pybind11 2.12.0
linalg.svd with too large matrices may segfault if
max(m, n)*max(m, n) > int_max on large-memory machines (on smaller memory machines
it may fail with a MemoryError instead). The root cause is an integer overflow
in indexing 2D arrays, deep in the LAPACK code.
Thus, detect a possible error condition and bail out early.
Co-authored-by: Pearu Peterson <pearu.peterson@gmail.com>
* Converting amos to std::complex

* Define M_PI if not available (for windows)

* Bump test tolerance

---------

Co-authored-by: izaid <hi@irwinzaid.com>
Co-authored-by: Albert Steppi <albert.steppi@gmail.com>
* Fixes scipy#20294.

* Pull in new OpenBLAS binaries as discussed in the above issue,
to solve a bug that affects downstream consumers like `scikit-learn`.

* Some additional shims were needed: bumping `cibuildwheel` to
improve GitHub actions MacOS M1 runner support, and `openblas_support`
module needed an adjustment to process out an apparent syntax
error in the pkg-config file for openblas that we pull in from upstream.
…ate.RBFInterpolator` (scipy#20364)

* BUG: optimize: Fix wrong warning message if degree=-1 in interpolate.RBFInterpolator

* BUG: optimize: add a comment for test
* Prepare for SciPy `1.13.0` "final" release
instead of RC2 given urgency of supporting
NumPy 2 series.

* Remove note about SciPy `1.13.0` not being
released yet in the release notes, and update
the `1.13.0` release notes to reflect another
bump in the OpenBLAS version vendored in for PyPI
wheels.

* The usual version bumps removing `rc2` version
flags, and author/pr/issue list updates.
@tylerjereddy tylerjereddy added the maintenance Items related to regular maintenance tasks label Apr 1, 2024
@tylerjereddy tylerjereddy added this to the 1.13.0 milestone Apr 1, 2024
@github-actions github-actions bot added Documentation Issues related to the SciPy documentation. Also check https://github.com/scipy/scipy.org CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure C/C++ Items related to the internal C/C++ code base Cython Issues with the internal Cython code base Meson Items related to the introduction of Meson as the new build system for SciPy labels Apr 1, 2024
@ilayn
Copy link
Member

ilayn commented Apr 1, 2024

I agree with the looming urgency of the matter but I'd say please take your time so that we don't create a self-inflicting stress for you. No problem with skipping RC2 for me.

* attempt wheel build since regular
CI is passing (empty commit)

[wheel build]
@tylerjereddy
Copy link
Contributor Author

Regular CI was passing, full wheel build test started.

@tylerjereddy
Copy link
Contributor Author

All green with a few restarts. I'll let this sit overnight so there's a bit of time for comments or objections.

Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Tyler! I added a couple of very minor comments, but those seem optional to fix and don't even require retesting.

* `#19774 <https://github.com/scipy/scipy/issues/19774>`__: DOC: Detail what "concatenate" means in the context of \`spatial.transform.Rotation.concatenate\`
* `#19799 <https://github.com/scipy/scipy/issues/19799>`__: DOC: array types: update array validation guidance
* `#19813 <https://github.com/scipy/scipy/issues/19813>`__: BUG: typo in specfun.f?
* `#19819 <https://github.com/scipy/scipy/issues/19819>`__: BUG: In RPFInterpolator, wrong warning message if degree=-1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor, typo: RPFInterpolator -> RBFInterpolator (I fixed the issue title itself just now)

- The Modified Akima Interpolation has been added to
``interpolate.Akima1DInterpolator``, available via the new ``method``
argument.
- New method ``BSpline.insert_knot`` inserts a knot into a ``BSpline` instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor: missing a second backtick behind BSpline

reformat_pkg_file(target_path=target_path)


def reformat_pkg_file(target_path):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: if you're pushing another comment, perhaps add a comment that this is simple typo that is already fixed upstream in openblas#4594

@lucascolley lucascolley removed scipy.special scipy.spatial scipy.signal scipy.sparse scipy.linalg scipy.interpolate scipy.optimize scipy._lib C/C++ Items related to the internal C/C++ code base CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure Cython Issues with the internal Cython code base Meson Items related to the introduction of Meson as the new build system for SciPy labels Apr 2, 2024
* Fix a few typos and add a missing comment based
on code review.

[docs only]
@tylerjereddy
Copy link
Contributor Author

I addressed Ralf's comments and the docs build still passed after that.

I manually re-read the diff and didn't find anything accidentally pulled in. I did notice one new test that seems to use the "old" rather than newer/scoped NumPy random machinery, and one new error message code path with a typo in the message, but I don't think those merit additional attention for now.

@tylerjereddy tylerjereddy merged commit 15a69da into scipy:maintenance/1.13.x Apr 2, 2024
@tylerjereddy tylerjereddy deleted the treddy_prep_1_13_0_final branch April 2, 2024 18:02
@tylerjereddy
Copy link
Contributor Author

alright, starting release process, what could possibly go wrong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Documentation Issues related to the SciPy documentation. Also check https://github.com/scipy/scipy.org maintenance Items related to regular maintenance tasks

Projects

None yet

Development

Successfully merging this pull request may close these issues.