text:
BUG: Segfault in scipy/sparse/linalg/_eigen/arpack

Describe your issue.
I am installing scipy from source on an HPC on top of python 3.11.6. The code builds without errors, but scipy.test() segfaults at scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py. We have intel MKL on the HPC, so I built scipy with
CC=icx CXX=icpx FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp

Reproducing Code Example
python dev.py test -s sparse

Error message
ninja: Entering directory `/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build'
[4/4] Generating scipy/generate-version with a custom command
Build OK
💻  meson install -C build --only-changed
Installing, see meson-install.log...
Installation OK
SciPy from development installed path at: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages
Running tests for scipy version:1.13.0, installed at:/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0
configfile: pytest.ini
plugins: hypothesis-6.102.4
collected 11638 items / 61 deselected / 11577 selected

scipy/sparse/csgraph/tests/test_connected_components.py ........                                                                                [  0%]
scipy/sparse/csgraph/tests/test_conversions.py ...                                                                                [  0%]
scipy/sparse/csgraph/tests/test_flow.py ...........................................                                                                                [  0%]
scipy/sparse/csgraph/tests/test_graph_laplacian.py ......................................................................................................................... [  1%]
............................................................................................................................................................................ [  2%]
............................................................................................................................................................................ [  4%]
............................................................................................................................................................................ [  5%]
............................................................................................................................................................................ [  7%]
............................................................................................................................................................................ [  8%]
.................................................................................................................................................                            [ 10%]
scipy/sparse/csgraph/tests/test_matching.py ................................                                                                                [ 10%]
scipy/sparse/csgraph/tests/test_pydata_sparse.py ssssssssssssssssssssssssssssssssss                                                                                [ 10%]
scipy/sparse/csgraph/tests/test_reordering.py ...                                                                                [ 10%]
scipy/sparse/csgraph/tests/test_shortest_path.py ...................................                                                                                [ 11%]
scipy/sparse/csgraph/tests/test_spanning_tree.py .                                                                                [ 11%]
scipy/sparse/csgraph/tests/test_traversal.py ........                                                                                [ 11%]
scipy/sparse/linalg/_dsolve/tests/test_linsolve.py .s.s.s.s.ss..........s.....................                                                                               [ 11%]
scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py .Fatal Python error: Segmentation fault

Current thread 0x0000146b7118b740 (most recent call first):
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 883 in extract
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1357 in eigs
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1577 in eigsh
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 248 in eval_evec
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 425 in test_hermitian_modes
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/_lib/_testutils.py", line 106 in __call__
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 745 in scipy_tests
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 761 in run
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/action.py", line 461 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/task.py", line 473 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 176 in execute_task
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 220 in run_tasks
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 254 in run_all
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_run.py", line 265 in _execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 570 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 150 in parse_execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/api.py", line 50 in run_tasks
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 251 in run_doit_task
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pydevtool/cli.py", line 267 in callback
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 783 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1434 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1688 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/rich_click/rich_command.py", line 126 in main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1157 in __call__
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 1486 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy._lib._fpumode, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2 (total: 50)
Segmentation fault (core dumped)

SciPy/NumPy/Python version and system information
Python 3.11.6 (main, May 15 2024, 11:36:54) [GCC Intel(R) C++ gcc 11.3.1 mode] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.13.0 1.26.4 sys.version_info(major=3, minor=11, micro=6, releaselevel='final', serial=0)
/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
  warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "11.3.1",
      "commands": "icx"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.10",
      "commands": "cython"
    },
    "c++": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "11.3.1",
      "commands": "icpx"
    },
    "fortran": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "2023.1.0",
      "commands": "ifx"
    },
    "pythran": {
      "version": "0.14.0",
      "include directory": "../../../python/3.11.6/lib/python3.11/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "cross-compiled": true
  },
  "Build Dependencies": {
    "blas": {
      "name": "mkl-dynamic-lp64-iomp",
      "found": true,
      "version": "2023.1",
      "detection method": "pkgconfig",
      "include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
      "lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
      "openblas configuration": "unknown",
      "pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
    },
    "lapack": {
      "name": "mkl-dynamic-lp64-iomp",
      "found": true,
      "version": "2023.1",
      "detection method": "pkgconfig",
      "include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
      "lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
      "openblas configuration": "unknown",
      "pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.12.0",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/work2/noaa/co2/sbasu/packages/python/3.11.6/bin/python3.11",
    "version": "3.11"
  }
}

Thanks for the report @AgilentGCMS. That ARPACK code has had recurring issues with some BLAS functions (cdotc & co.) that are sensitive to Fortran compiler ABI conventions. MKL is built with the opposite convention as the default one (gfortran).
Maybe this particular crash was always an issue with MKL from OneAPI, or maybe it changed with https://github.com/scipy/scipy/pull/20232 (that was included in 1.13.0, which you're testing here). Do you know if this is a regression from 1.12.x or not for your setup?

I can reproduce this on Windows with scipy-1.13.1 and OneAPI MKL 2024.1.

Thanks @cgohlke for confirming. So it's a problem that's not platform-dependent.
I'll add a bit of context, since it may not be worth starting to debug here - the maintenance state of our ARPACK copy is very bad:

We're at ARPACK 3.3.0, from 2015 (last upgrade done in gh-5901). So it's about 9 years and 6 minor releases behind https://github.com/opencollab/arpack-ng. Open PRs to upgrade were never finished: gh-10375, gh-14786. I think a lot of issues may be hiding in ARPACK that have been solved upstream. The upgrade to 3.7.0 in the first PR did also uncover some regressions which made it harder to finish that PR.
The end goal is to get rid of ARPACK (and PROPACK) completely, as discussed in gh-18566.

It's also still possible that this is an MKL bug. We do test with Accelerate, which uses the same ABI conventions as MKL, and things work as expected there.

Seems like I am able to reproduce the segmentation fault with FC=ifx and C/C++ compilers from conda.
Command
FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp
Logs
SciPy from development installed path at: /home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages
Running tests for scipy version:1.15.0.dev0+1043.bbe993d, installed at:/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy
============================= test session starts ==============================
platform linux -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/czgdp18079/Quansight/scipy
configfile: pytest.ini
plugins: anyio-4.4.0, xdist-3.6.1, cov-5.0.0, hypothesis-6.104.2, timeout-2.3.1
collected 12224 items / 348 deselected / 11876 selected

scipy/sparse/csgraph/tests/test_connected_components.py ........         [  0%]
scipy/sparse/csgraph/tests/test_conversions.py ...                       [  0%]
scipy/sparse/csgraph/tests/test_flow.py ................................ [  0%]
...........                                                              [  0%]
scipy/sparse/csgraph/tests/test_graph_laplacian.py ..................... [  0%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  2%]
........................................................................ [  3%]
........................................................................ [  3%]
........................................................................ [  4%]
........................................................................ [  4%]
........................................................................ [  5%]
........................................................................ [  6%]
........................................................................ [  6%]
........................................................................ [  7%]
........................................................................ [  7%]
........................................................................ [  8%]
........................................................................ [  9%]
........................................................................ [  9%]
.........................                                                [  9%]
scipy/sparse/csgraph/tests/test_matching.py ............................ [ 10%]
....                                                                     [ 10%]
scipy/sparse/csgraph/tests/test_pydata_sparse.py sssssssssssssssssssssss [ 10%]
sssssssssssssssssssssssssssssssssss                                      [ 10%]
scipy/sparse/csgraph/tests/test_reordering.py ...                        [ 10%]
scipy/sparse/csgraph/tests/test_shortest_path.py ....................... [ 10%]
.................................................                        [ 11%]
scipy/sparse/csgraph/tests/test_spanning_tree.py .                       [ 11%]
scipy/sparse/csgraph/tests/test_traversal.py ........                    [ 11%]
scipy/sparse/linalg/_dsolve/tests/test_linsolve.py .s.s.s.s.ss.......... [ 11%]
s............................                                            [ 11%]
scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py .Fatal Python error: Segmentation fault

Current thread 0x00007efc3801b740 (most recent call first):
  File "/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 730 in iterate
  File "/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1357 in eigs
  File "/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1578 in eigsh
  File "/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 238 in eval_evec
  File "/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 421 in test_hermitian_modes
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/python.py", line 1632 in runtest
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy/_lib/_testutils.py", line 126 in __call__
  File "/home/czgdp18079/Quansight/scipy/dev.py", line 758 in scipy_tests
  File "/home/czgdp18079/Quansight/scipy/dev.py", line 774 in run
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/action.py", line 461 in execute
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/task.py", line 473 in execute
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/runner.py", line 176 in execute_task
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/runner.py", line 220 in run_tasks
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/runner.py", line 254 in run_all
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/cmd_run.py", line 265 in _execute
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/cmd_base.py", line 570 in execute
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/cmd_base.py", line 150 in parse_execute
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/doit/api.py", line 50 in run_tasks
  File "/home/czgdp18079/Quansight/scipy/dev.py", line 252 in run_doit_task
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pydevtool/cli.py", line 267 in callback
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/click/core.py", line 783 in invoke
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/click/core.py", line 1434 in invoke
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/click/core.py", line 1688 in invoke
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/rich_click/rich_command.py", line 152 in main
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/click/core.py", line 1157 in __call__
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/rich_click/rich_command.py", line 367 in __call__
  File "/home/czgdp18079/Quansight/scipy/dev.py", line 1657 in <module>

Extension modules: numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy._lib._fpumode, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2 (total: 49)

OK I bumped the ARPACK version in #21110. Maybe it can solve this issue too. No idea what the changes are as it still has issues unresolved upstream too.
Simultaneously, I started rewriting ARPACK work. The code is exactly what anyone would expect from a F77 monolith.

With your PR, the segmentation fault disappears from _arpack but comes in test_svds,
scipy/sparse/linalg/_eigen/tests/test_svds.py .......................... [ 14%]
................................................s.ss..s.ssss....s.ss..s. [ 15%]
ssss....s.ss..s.ssss....s.ss..s.ssss....s.ss..s.ssss....s.ss..s.ssss.... [ 16%]
............ssssssss.................................................... [ 16%]
............................................s.ss....s.ss....s.ss....s.ss [ 17%]
....s.ss....s.ss....s.ss....s.ss....s.ss....s.ss....s.ss....s.ss........ [ 17%]
........................................................................ [ 18%]
..............................s.Fatal Python error: Fatal Python error: Segmentation fault

Sigh... Language of scientific computing.

I think the problem is with ifx and not MKL/OpenBLAS. With gfortran, both MKL and OpenBLAS pass all the tests in sparse module. With ifx, both MKL and OpenBLAS give segmentation fault at the same point.

Could you run the test suite with -v option so that we know which test failed?

Yes. Now I will debug the tests. I wanted to see what all combinations work and what all fail. I have started with ifx and MKL combination.

Small update,
The problem is with the calling of wzdotc from complex16/zblasext.F in PROPACK.
When I inline zdotc from LAPACK (and rename it to pzdotc - nothing but a wrapper to wzdotc) in   complex16/zblasext.F, then segmentation fault goes away and all the tests in sparse module pass. Probably it's some ABI incompatibility between ifx and MKL/OpenBLAS.
See https://github.com/scipy/scipy/commit/3f61bc15474a1da27e7fe5e4be01a6322a380c86 and https://github.com/scipy/scipy/commit/ea11308ef3d6b0bb7939a6630fae77511c5688c5 in my branch of SciPy.

Now I merged the new version of ARPACK 3.9.1 which vendors its own zzdotc instead. So that should potentially solve the problem then.

Sure. I will try it in the morning.

If replacing the wzdotc wrapper by a vendored zzdotc function does fix the problem, then it'd also be good to know whether that is because we're using the dummy version of the wrapper dependency when we shouldn't (MKL should require the non-dummy version), or the wrapper code isn't actually doing what it's supposed to do.

The segmentation fault goes away but the following error is showing up with zzdotc, ifx and MKL combination. Let me check with gfortran.
----------------------------- Captured stdout call -----------------------------
 WARNING: Maximum dimension of Krylov subspace exceeded prior to convergence.
  Try increasing KMAX.
 neig =            0
=========================== short test summary info ============================
FAILED scipy/sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_PROPACK::test_svd_linop - numpy.linalg.LinAlgError: k=3 singular triplets did not converge within kma...
= 1 failed, 10204 passed, 1626 skipped, 348 deselected, 43 xfailed, 2 xpassed in 136.25s (0:02:16) =

Captured stdout is not important, that's expected. The PROPACK test can be adjusted too. Happy that you don't encounter any segfaults yet.

scipy/sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_PROPACK::test_svd_linop Fatal Python error: Segmentation fault gives segmentation faults with increased kmax.

After https://github.com/czgdp1807/PROPACK/commit/b59104e306f2164deca19f6a25f98c599d4c22e8 all the segmentation faults are fixed in python dev.py test -s sparse.

With ifx + MKL combination, no segmentation faults with https://github.com/czgdp1807/PROPACK/commit/b59104e306f2164deca19f6a25f98c599d4c22e8 in the whole test suite of SciPy but the following tests fail,
FAILED scipy/interpolate/tests/test_fitpack2.py::TestUnivariateSpline::test_out_of_range_regression - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints0-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints0-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints0-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints0-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints0-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints0-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints1-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints1-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints1-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints1-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints1-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints1-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-<lambda>-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-<lambda>-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-Bounds-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-Bounds-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints3-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints3-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints3-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints3-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints3-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints3-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints5-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints5-<lambda>-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints5-<lambda>-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints5-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints5-Bounds-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints5-Bounds-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints8-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints8-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints8-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints8-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints0-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints0-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints0-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints0-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints0-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints0-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints1-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints1-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints1-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints1-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints1-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints1-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-<lambda>-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-<lambda>-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-Bounds-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-Bounds-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints3-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints3-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints3-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints3-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints3-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints3-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints4-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints4-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints4-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints4-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints4-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints4-Bounds-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints5-<lambda>-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints5-<lambda>-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints5-<lambda>-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints5-Bounds-kwds0-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints5-Bounds-kwds1-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints5-Bounds-kwds2-SLSQP] - AssertionError:
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints8-<lambda>-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints8-<lambda>-kwds2-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints8-Bounds-kwds1-SLSQP] - assert np.False_
FAILED scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints8-Bounds-kwds2-SLSQP] - assert np.False_
= 81 failed, 50065 passed, 2775 skipped, 11739 deselected, 162 xfailed, 18 xpassed in 521.45s (0:08:41) =

ifx + MKL combination on Linux
SLSQP {'fun': <function setup_test_equal_bounds.<locals>.func at 0x7f263d199a80>, 'jac': False} <function setup_test_equal_bounds.<locals>.<lambda> at 0x7f2630fc6520> (None, None) None

True (1289.4752551349443, np.float64(1490.4970902896914)) (array([ 1.41371438,  2.        ,  0.87977266, -1.        ]), array([ 0.79393209,  2.        ,  0.74248698, -1.        ]))
Traceback (most recent call last):
  File "/home/czgdp18079/Quansight/scipy/../debug.py", line 188, in <module>
    test_equal_bounds(method, kwds, bound_type, constraints, callback)
  File "/home/czgdp18079/Quansight/scipy/../debug.py", line 147, in test_equal_bounds
    assert_allclose(res.fun, expected.fun, rtol=1.5e-6)
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1684, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/czgdp18079/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 885, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1.5e-06, atol=0

Mismatched elements: 1 / 1 (100%)
Max absolute difference among violations: 201.02183515
Max relative difference among violations: 0.13486899
 ACTUAL: array(1289.475255)
 DESIRED: array(1490.49709)

On my Apple Macbook Pro
SLSQP {'fun': <function setup_test_equal_bounds.<locals>.func at 0x104bcbb00>, 'jac': False} <function setup_test_equal_bounds.<locals>.<lambda> at 0x13a607b00> (None, None) None

True (1289.475255134854, 1289.4752550852413) (array([ 1.41371438,  2.        ,  0.87977267, -1.        ]), array([ 1.41371252,  2.        ,  0.87977062, -1.        ]))
4 4
(array([nan, nan]), array([nan, nan]))
(1289.475255134854, 1289.475255134854)
(array([1.41371438, 0.87977267]), array([1.41371438, 0.87977267]))

Python Code
import itertools
import platform
import numpy as np
from numpy.testing import (assert_allclose, assert_equal,
                           assert_almost_equal,
                           assert_no_warnings, assert_warns,
                           assert_array_less, suppress_warnings)
import pytest
from pytest import raises as assert_raises

from scipy import optimize
from scipy.optimize._minimize import Bounds, NonlinearConstraint
from scipy.optimize._minimize import (MINIMIZE_METHODS,
                                      MINIMIZE_METHODS_NEW_CB,
                                      MINIMIZE_SCALAR_METHODS)
from scipy.optimize._linprog import LINPROG_METHODS
from scipy.optimize._root import ROOT_METHODS
from scipy.optimize._root_scalar import ROOT_SCALAR_METHODS
from scipy.optimize._qap import QUADRATIC_ASSIGNMENT_METHODS
from scipy.optimize._differentiable_functions import ScalarFunction, FD_METHODS
from scipy.optimize._optimize import MemoizeJac, show_options, OptimizeResult
from scipy.optimize import rosen, rosen_der, rosen_hess

from scipy.sparse import (coo_matrix, csc_matrix, csr_matrix, coo_array,
                          csr_array, csc_array)

def setup_test_equal_bounds():

    np.random.seed(0)
    x0 = np.random.rand(4)
    lb = np.array([0, 2, -1, -1.0])
    ub = np.array([3, 2, 2, -1.0])
    i_eb = (lb == ub)

    def check_x(x, check_size=True, check_values=True):
        if check_size:
            assert x.size == 4
        if check_values:
            assert_allclose(x[i_eb], lb[i_eb])

    def func(x):
        check_x(x)
        return optimize.rosen(x)

    def grad(x):
        check_x(x)
        return optimize.rosen_der(x)

    def callback(x, *args):
        check_x(x)

    def constraint1(x):
        check_x(x, check_values=False)
        return x[0:1] - 1

    def jacobian1(x):
        check_x(x, check_values=False)
        dc = np.zeros_like(x)
        dc[0] = 1
        return dc

    def constraint2(x):
        check_x(x, check_values=False)
        return x[2:3] - 0.5

    def jacobian2(x):
        check_x(x, check_values=False)
        dc = np.zeros_like(x)
        dc[2] = 1
        return dc

    c1a = NonlinearConstraint(constraint1, -np.inf, 0)
    c1b = NonlinearConstraint(constraint1, -np.inf, 0, jacobian1)
    c2a = NonlinearConstraint(constraint2, -np.inf, 0)
    c2b = NonlinearConstraint(constraint2, -np.inf, 0, jacobian2)

    # test using the three methods that accept bounds, use derivatives, and
    # have some trouble when bounds fix variables
    methods = ('L-BFGS-B', 'SLSQP', 'TNC')

    # test w/out gradient, w/ gradient, and w/ combined objective/gradient
    kwds = ({"fun": func, "jac": False},
            {"fun": func, "jac": grad},
            {"fun": (lambda x: (func(x), grad(x))),
             "jac": True})

    # test with both old- and new-style bounds
    bound_types = (lambda lb, ub: list(zip(lb, ub)),
                   Bounds)

    # Test for many combinations of constraints w/ and w/out jacobian
    # Pairs in format: (test constraints, reference constraints)
    # (always use analytical jacobian in reference)
    constraints = ((None, None), ([], []),
                   (c1a, c1b), (c2b, c2b),
                   ([c1b], [c1b]), ([c2a], [c2b]),
                   ([c1a, c2a], [c1b, c2b]),
                   ([c1a, c2b], [c1b, c2b]),
                   ([c1b, c2b], [c1b, c2b]))

    # test with and without callback function
    callbacks = (None, callback)

    data = {"methods": methods, "kwds": kwds, "bound_types": bound_types,
            "constraints": constraints, "callbacks": callbacks,
            "lb": lb, "ub": ub, "x0": x0, "i_eb": i_eb}

    return data

eb_data = setup_test_equal_bounds()

def test_equal_bounds(method, kwds, bound_type, constraints, callback):
    """
    Tests that minimizers still work if (bounds.lb == bounds.ub).any()
    gh12502 - Divide by zero in Jacobian numerical differentiation when
    equality bounds constraints are used
    """
    # GH-15051; slightly more skips than necessary; hopefully fixed by GH-14882
    if (platform.machine() == 'aarch64' and method == "TNC"
            and kwds["jac"] is False and callback is not None):
        return 'Tolerance violation on aarch'

    lb, ub = eb_data["lb"], eb_data["ub"]
    x0, i_eb = eb_data["x0"], eb_data["i_eb"]

    test_constraints, reference_constraints = constraints
    if test_constraints and not method == 'SLSQP':
        return 'Only SLSQP supports nonlinear constraints'
    # reference constraints always have analytical jacobian
    # if test constraints are not the same, we'll need finite differences
    fd_needed = (test_constraints != reference_constraints)

    bounds = bound_type(lb, ub)  # old- or new-style

    kwds.update({"x0": x0, "method": method, "bounds": bounds,
                 "constraints": test_constraints, "callback": callback})
    res = optimize.minimize(**kwds)

    expected = optimize.minimize(optimize.rosen, x0, method=method,
                                 jac=optimize.rosen_der, bounds=bounds,
                                 constraints=reference_constraints)

    # compare the output of a solution with FD vs that of an analytic grad
    print(res.success, (res.fun, expected.fun), (res.x, expected.x))
    assert res.success
    assert_allclose(res.fun, expected.fun, rtol=1.5e-6)
    assert_allclose(res.x, expected.x, rtol=5e-4)

    if fd_needed or kwds['jac'] is False:
        expected.jac[i_eb] = np.nan
    print(res.jac.shape[0], 4)
    print((res.jac[i_eb], expected.jac[i_eb]))
    assert res.jac.shape[0] == 4
    assert_allclose(res.jac[i_eb], expected.jac[i_eb], rtol=1e-6)

    if not (kwds['jac'] or test_constraints or isinstance(bounds, Bounds)):
        # compare the output to an equivalent FD minimization that doesn't
        # need factorization
        def fun(x):
            new_x = np.array([np.nan, 2, np.nan, -1])
            new_x[[0, 2]] = x
            return optimize.rosen(new_x)

        fd_res = optimize.minimize(fun,
                                   x0[[0, 2]],
                                   method=method,
                                   bounds=bounds[::2])
        print((res.fun, fd_res.fun))
        print((res.x[[0, 2]], fd_res.x))
        assert_allclose(res.fun, fd_res.fun)
        # TODO this test should really be equivalent to factorized version
        # above, down to res.nfev. However, testing found that when TNC is
        # called with or without a callback the output is different. The two
        # should be the same! This indicates that the TNC callback may be
        # mutating something when it shouldn't.
        assert_allclose(res.x[[0, 2]], fd_res.x, rtol=2e-6)

i = 0
for method in eb_data["methods"]:
    for kwds in eb_data["kwds"]:
        for bound_type in eb_data["bound_types"]:
            for constraints in eb_data["constraints"]:
                for callback in eb_data["callbacks"]:
                    if i == 108:
                        print(method, kwds, bound_type, constraints, callback)
                        print()
                        test_equal_bounds(method, kwds, bound_type, constraints, callback)
                    i = i + 1

Actually, there is not very signficant problem with ifx . Its just that the code in slsqp_optmz.F is very rigid. Now I was debugging one example. There is this condition IF(wmax.LE.ZERO)                GOTO 280 (inside nnls subroutine) . Its intent is to make sure that all elements in w are non-positive. However, ifx compiled code results in wmax to be 1.6e-21. Not ZERO but very close to ZERO . So that causes chaotic computation because the above condition fails leading to further iterations which causes corruption of values (like correct values inside an array becoming arbitrarily large). If I change the above condition to, IF(ABS(wmax).LE.1e-20)                GOTO 280 then the code works. :D.

Do you have #21021 already on your branch which we recently merged? That should remove the need for the fortran compilation.

I merged main into my branch. The tests still fail. I think slsqp method still calls nnls written in scipy/optimize/slsqp/slsqp_optmz.f.

Ah yes my bad. Apologies i didn't push my slsqp to Scipy. I'm confusing myself finally reading too much fortran

In this, https://github.com/scipy/scipy/issues/20728#issuecomment-2223838292, I changed 1e-20 to 1e-19 in IF(ABS(wmax).LE.1e-20)                GOTO 280 and test_equal_bounds pass fully in optimize tests.
Seems like ifx has noisy behaviour with floating point computations. However, also floating point comparisons should always be done with some tolerances and not rigidly like, .EQ. or directly with ZERO.

With this https://github.com/scipy/scipy/pull/21173, things work with ifx + MKL on my Linux machine. I will address reviews and run the tests on CI, next week.

However, also floating point comparisons should always be done with some tolerances and not rigidly like, .EQ. or directly with ZERO

This is not always true, it needs some context, and there are always some problems that come very close to zero. Old foxes are not that careless about these algorithms doing such simple errors.
But here I think it won't matter since we are going to translate it anyways.
ifx and MKL  are probably too aggressive with FMA optimizations and we are seeing substantial deviations.

Old foxes are not that careless about these algorithms doing such simple errors.

True. May be as you said, ifx is doing FMA optimisations which is leading to these issues. I will try with ifort + MKL combination. ifort doesn't do optimisations by default, right? Is there any way we can silence optimisations with ifx?

I checked https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html. Seems like ifx optimisations can be disabled via, -O0 and -fpmodel=precise. Worth trying I think.

I must disappoint you that I really don't know much about fortran compilers. But indeed if you have capacity to try it out, that would be great.

Note that the floating-point precision flags for Intel compilers should already be set correctly, by this code: https://github.com/scipy/scipy/blob/8ccaf44b3644b9e3ad33afea7c6ef03780053fab/meson.build#L101-L129
Please verify that that's the case (the strict flag should appear in build/build.ninja). If so, the problem is a ifferent one.

Actually, FC="ifx -O0" and FC="ifx -O0 -fpmodel=precise doesn't make a difference. test_equal_bounds still fail because of the issues I mentioned in https://github.com/scipy/scipy/issues/20728#issuecomment-2225878433. So, I think the problem is with code generated by ifx.

@czgdp1807, the wmax is computed from the w values, which are computed by a dot-product. Dieter Kraft was using a manually unrolled version called ddot_sl.
Could you try replacing it with a different version?
C STEP TWO (COMPUTE DUAL VARIABLES)
C .....ENTRY LOOP A

C ( ... omitted ... )

      DO 120 iz=iz1,iz2
         j=INDEX(iz)
  120    w(j)=ddot_sl(m-nsetp,a(npp1,j),1,b(npp1),1)

C STEP THREE (TEST DUAL VARIABLES)

  130 wmax=ZERO
      DO 140 iz=iz1,iz2
      j=INDEX(iz)
         IF(w(j).LE.wmax)             GOTO 140
         wmax=w(j)
         izmax=iz
  140 CONTINUE

C .....EXIT LOOP A

      IF(wmax.LE.ZERO)                GOTO 280

Here is a test program with a few ways it can be done:
C test.f
C compile with ifx -O1 -fp-model=precise test.f ddot_sl.f -qmkl
      program test
      implicit none
      integer, parameter :: n = 100000000
      double precision a(n), b(n)

C Version in SLSQP (adapted by Dieter Kraft)
      external ddot_sl
      double precision ddot_sl

C BLAS (MKL) version
      external ddot
      double precision ddot

      call random_number(a)
      call random_number(b)

      print *, ddot_sl(n,a,1,b,1)
      print *, ddot(n,a,1,b,1)
      print *, dot_product(a,b)
      print *, sum(a*b)

      end

Sure @ivan-pi. Let me do it. I will update this comment if it works. gfortran works fine with the unrolled version. May be, ifx can give different results if I use yours. Let me see.

The problem is not with loop unrolling in ddot_sl. In fact, LAPACK also uses exactly same implementation for ddot. See, https://netlib.org/lapack/explore-html/d1/dcc/group__dot_ga2a42ecc597403b22ad786715c739196b.html. The problem is that ifx is not precise enough with DOUBLE PRECISION. I used REAL*16 inside ddot_sl (see the diff of slsqp_optimize.f in #21173) and I was able to remove IF(ABS(wmax).LE.1e-20)                GOTO 280. The only thing left is IF(rnorm.LT.ZERO) GOTO 50 in line 1094. Its safe and no tests fail with this (both ifx and gfortran work).

Why don't we just mark this SLSQP test as xfail and move on? The code will be rewritten in C or C++ anyway, so then the problem is solved.
You can determine in the test suite whether Intel Fortran was used to build from:
scipy.show_config(mode='dicts')['Compilers']['fortran']['name']

We now have a OneAPI CI job (all Intel compilers + MKL), so this should hopefully not regress again.

authors: AgilentGCMS cgohlke czgdp1807 ilayn ivan-pi rgommers


text:
MIMO package for Python control?

Does anyone know a python control package to analyse output for a MIMO system?
I tried lsim2 and it does not work for the following reason I guess ( the following lines are taken from source code of scipy.signal.lsim2:
tout, yout, xout = scipy.signal.lsim2(system, U=[signal1,signal2],T=T)
T = atleast_1d(T)
if len(T.shape) != 1:
raise ValueError("T must be a rank-1 array.")
if U is not None:
U = atleast_1d(U)
if len(U.shape) == 1:
U = U.reshape(-1, 1)
sU = U.shape
if sU[0] != len(T):
raise ValueError("U must have the same number of rows "
"as elements in T.")
If anyone knows how to fix it, please help!
Thanks

Check out the python-control package:

http://python-control.readthedocs.org/en/latest/index.html
https://github.com/python-control/python-control

Though I have not used it for this reason myself, the documentation mentions the ability to simulate MIMO forced responses: http://python-control.readthedocs.org/en/latest/control.html#time-domain-simulation
It's BSD licensed, as far as I am aware, so it could be nice to get some of it into scipy.signal...

source: gharchive/issue

repo: scipy/scipy

license_type: permissive

license_source: github-api


text:
Missing methods in multivariate_normal_frozen

Unlike stated in the documentation frozen multivariate gaussians do not have the same method, some of them are missing.
Either the documentation should be updated or the methods should be avaialble.
Reproducing code example:
from scipy.stats import multivariate_normal
model = multivariate_normal([1, 6], [[1, 0], [0, 1]])
print(model.cdf([0, 0]))

Error message:
AttributeError: 'multivariate_normal_frozen' object has no attribute 'cdf'

Scipy/Numpy/Python version information:
0.19.1 1.13.1 sys.version_info(major=3, minor=6, micro=2, releaselevel='final', serial=0)

Sorry, I was behind master. I thought this method would be the first thing to be implemented but it seems to be a very recent addition.

You reported using scipy 0.19.1.  The cdf (and logcdf) methods were added in scipy 1.0.0.  You'll have to upgrade to 1.0.0 to use cdf.

authors: GuillaumeLeclerc WarrenWeckesser


text:
'sum' in Rosenbrock function has an unclosed opening parenthesis

See here: https://github.com/scipy/scipy/blob/v0.15.1/scipy/optimize/optimize.py#L160

That was fixed some years back (v0.15.1 is from 2015).

authors: pv yurivict


text:
BUG: Fix Python crash by allocating larger array in LSQBivariateSpline

Reference issue
Closes gh-13356.
What does this implement/fix?
In scipy.interpolate.LSQBivariateSpline, which calls the Fortran subroutine surfit as dfitpack.surfit_lsq via F2PY, pad both of the knot arrays to the size demanded by the Fortran routine. This fixes the memory error when the sizes of the input knot arrays are different.
Additional information
Note: The call signature to dfitpack.surfit_lsq is changed as a result of this fix. The input parameters nx and ny are no longer optional. This will break client code that calls the F2PY-generated dfitpack.surfit_lsq() function directly.
An additional test is added that explicitly set the knot arrays to be unequal lengths. The input sizes in the test are also larger than the typical ones we've had so far.

Thanks @congma!

Note: The call signature to dfitpack.surfit_lsq is changed as a result of this fix. The input parameters nx and ny are no longer optional. This will break client code that calls the F2PY-generated dfitpack.surfit_lsq() function directly.

That's okay, that's definitely a private API.

Thanks @congma!

Note: The call signature to dfitpack.surfit_lsq is changed as a result of this fix. The input parameters nx and ny are no longer optional. This will break client code that calls the F2PY-generated dfitpack.surfit_lsq() function directly.

That's okay, that's definitely a private API.

Nice one @congma!
Merged thanks @congma @rgommers

Nice one @congma!
Merged thanks @congma @rgommers

You're welcome! Since this was essentially a hotfix, will it be backported to a bugfix release, or shall we wait till 1.7?

You're welcome! Since this was essentially a hotfix, will it be backported to a bugfix release, or shall we wait till 1.7?

I've added the backport-candidate label. Whether there will be a 1.6.1 and whether this one is going to be backported or not is up to Tyler. Typically we backport serious new regressions. This one is indeed annoying, however it's of a "been there forever" category, so backporting this one is not clear-cut.

I've added the backport-candidate label. Whether there will be a 1.6.1 and whether this one is going to be backported or not is up to Tyler. Typically we backport serious new regressions. This one is indeed annoying, however it's of a "been there forever" category, so backporting this one is not clear-cut.

Thanks for the explanation. I think in addition to being an annoyance, this one indeed has real-world consequences by silently overflowing the buffer. In fact, I encountered this bug when I was trying to use LSQBivariateSpline, in the way that it is intended to be used, as part of the day-to-day business. That no one had reported this seemed to me sheer (bad) luck, possibly compounded by the lack of test coverage. So my view is that, if it can fit into your development and release schedule, backporting would make SciPy more reliable and less crash-prone -- which to me sounds like a suitable case for backporting.

Thanks for the explanation. I think in addition to being an annoyance, this one indeed has real-world consequences by silently overflowing the buffer. In fact, I encountered this bug when I was trying to use LSQBivariateSpline, in the way that it is intended to be used, as part of the day-to-day business. That no one had reported this seemed to me sheer (bad) luck, possibly compounded by the lack of test coverage. So my view is that, if it can fit into your development and release schedule, backporting would make SciPy more reliable and less crash-prone -- which to me sounds like a suitable case for backporting.

source: gharchive/pull-request

authors: congma ev-br rgommers


text:
DOC: fix formatting of Methods in multivariate distributions

Reference issue

Not found.
What does this implement/fix?

Removed unnecessary backquotes, which were not working as codeblock.
Additional information

Similar commit is https://github.com/scipy/scipy/commit/5cdc56aaf43ed6e2f097ead93063b5a26b098c7a

您的邮件我已收到，我会加班加点的看完您珍贵的来信，尽快给您回复的！谢谢啦！陈鑫


text:
Added class ExactKrylovJacobian

Hi,
This PR addresses issue #15367
I've been using this code for a while now in a personal project of mine.
I use it as a parametrized substitution of scipy.optimize.nonlin.KrylovJacobian. It showed very good convergence behavior near a solution, where massive cancellation is avoided.
My usage case looks something like this:
if (Use_exact_Jacobian):

    FGrad = lambda x,dx : Compute_action_hess_mul(x,dx,callfun)
    jacobian = ExactKrylovJacobian(exactgrad=FGrad,**jac_options)

else:
    jacobian = scipy.optimize.nonlin.KrylovJacobian(**jac_options)

x0 = np.copy(best_sol.x)
opt_result = scipy.optimize.nonlin.nonlin_solve(F=F,x0=x0,jacobian=jacobian,verbose=disp_scipy_opt,maxiter=maxiter,f_tol=gradtol,line_search=line_search,callback=best_sol.update,raise_exception=False)

I have little experience contribution to big open source project, so don't hesitate to tell me if more info is required etc...
In particular, I have little understanding of the inner workings of scipy.optimize, so I am not sure how to best propagate these changes at the user level.
Cheers,

Thank you @gabrielfougeron for the PR. Provided @andyfaff confirms this is wanted, it would need some tests.
On my side, it looks very simple and I feel it would be better suited in some doc. I will let specialists weight in.

Indeed it's nothing extravagant in terms of code, but in order to achieve this, you really need to get deep into the inner workings of scipy.optimize. As a layman, it was actually quite hard to achieve, and the optimize series of calls really felt like a deep rabbithole.
As mentioned in the issue though, the functionality is already present for scipy.optimize.minimize via the hessp argument. So I can't believe I would be the only one to need/want this kind of addition.

@tupui, I'm not familiar/expert with nonlin, going to need another expert here.

@zhaog6 you've been improving Krylov methods, maybe you have an opinion on whether this is worth adding?

Thanks @gabrielfougeron for your contribution. What does the keyword Exact in ExactKrylovJacobian mean? (Generally, it means to use like automatic differentiation (AD-like) to exactly find Jacobian/gradient of function, but it's not common).
The matvec in KrylovJacobian implemented Jacobian-Free Newton-Krylov, and the finite difference form for approximating Jacobian is popular, which is enough in most cases. With your issue, I guess you might need a better approximation of Jacobian.
Have you implemented Compute_action_hess_mul by high-order difference or other approximate methods in your personal project? Could you provide the specific form of f mentioned in the issue #15367?

The keyword maybe could have been better chosen indeed. In this case, it is meant at giving the user the opportunity to provide their own hand coded jacobian matrix vector product. In my case, I compute it with no approximation. The soundness of such an approach of course depends on the problem at hand, but it made sense in my case.

In this case, it is meant at giving the user the opportunity to provide their own hand coded jacobian matrix vector product.

I see, that's what I mean. Whether it is exact Jacobian or design high-order approximation for Jacobian (by evaluating matrix-vector products), a matvec interface that can be customized by users is needed. It would be nice to include related references and test cases for this interface.

authors: andyfaff gabrielfougeron rgommers tupui zhaog6


text:
DOC: update pydata-sphinx theme

This is a provision for the next release of the PyData Sphinx Theme which should happen in the following days (RC2 already up).
This will be needed if the doc is build after 0.10 is released otherwise there will be no search bar. The new search bar is a way better experience. Overall the new version provides a lot of improvements.

@tupui tried to resolve merge conflicts. Hope I did it correctly. (Oops, probably shouldn't have skipped most CI. )

Thanks Matt for having a look and catching the lint issue 👍

The new theme version is out, but I guess this is blocked until the build time issues reported in https://github.com/pydata/pydata-sphinx-theme/issues/889 are resolved?

@rgommers yes this is blocked until the build time issues are resolved. Unless we are fine with either removing the templates (not sure) or doubling build time (probably not). I've quickly pinned the version in #16922 so that it would not break or CI and not block them to release either.

Thanks. Yes indeed, doubling build time would be really bad, it is already quite painful. Not sure about the templates either, needs trying.

Still an issue with 0.12. I will remove the templates so we can have a look at the output here. Besides the build time, the new versions are great improvements and it's sad we are missing out.

And again an issue with 0.13.

Based on past discussion this seems unlikely to be solved for 1.11, but I'd be very happy to be proven wrong! :)

I am afraid you are correct... 😢

Still not there yet with the 0.14.1 release, but there seems to be an improvement. Now the build step finished and we are stuck at the upload.

It looks like they're closer to a root cause for the slowdown issue in https://github.com/pydata/pydata-sphinx-theme/issues/889, but it seems unlikely that will be fully solved before we branch in ~2 weeks, though a fix for 1.13.0 does seem like it might happen.
I suppose I could leave this to the last minute, but currently looks likely that I'd bump the milestone again.

I'm bumping the milestone after re-reading the "upstream" status--sounds like there's a modest chance we may be able to consider moving forward in a few months with 1.13.0 perhaps.

Great news from the team's team! 🚀  Once https://github.com/pydata/pydata-sphinx-theme/pull/1609 is in, we might be able to update. I will keep an eye and re-work this PR once they cut a new release.

Alright, we are golden!
Thank you @drammock and  @choldgraf for tackling this 💪🎉
I will wait for the next official release and we can deploy this.
@tylerjereddy if the next point release can wait a bit, would be great to have this in.

I've seen, yay 😃 I will update the PR after work.

Checking out the page in the artefacts of the build_docs job, this still seems to have some issues:

The header looks pretty objectively worse than the current main page IMO:

While it's more of a minor stylistic nit, I kinda prefer the slightly warmer colour scheme we have now, and (if the search bar is permanently moved from left to top-right) the horizontal balance is off to me - the page is leaning left to my tastes.
Back to the new page, the following "Back to top" annoyingly does not fade away:

It disappears when scrolling down slightly, but that's unintuitive IMO.

I will see if we have some custom CSS that would go against the template (I will check NumPy's config too).
The long term goal was to progressively remove as much customization as possible. The theme is supposed to have better defaults (more inclusive colours, better UI, etc.)

The long term goal was to progressively remove as much customization as possible.

Absolutely in favour of that and don't want to stand in the way. Though ideally we should try not to "regress" too much in terms of UI/UX while doing so.

In my local testing when fixing the theme I had tweaked your conf.py to put search back in the left sidebar. I'd recommend doing that here, and having a longer discussion / separate PR about which upstream layout and style changes you want to override and which you want to keep.
I think I also had tweaked the top navbar ... I'll be back at my desk later today and can tell you then how exactly I changed the conf.
-------- Original Message --------
On 1/4/24 06:01, h-vetinari  wrote:

The long term goal was to progressively remove as much customization as possible.

Absolutely in favour of that and don't want to stand in the way. Though ideally we should try not to "regress" too much in terms of UI/UX while doing so.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

What I need to really fix though is the number of levels for the API part.
We have a crazy "logic" with hidden tree (I tried once to refactor that madness but did not see it through) and we need the custom html for that. I had deactivated this bit during the debug to see if that fixed the build time issue.

Thanks @drammock 🙌 feel free to make a PR on my fork on this branch. Might be easier than sharing bits of code there.

I did the updates. Seems like we have the same state as on develop with the new version 😃
There is still an issue with the sidebar TOC for the API reference page, but that's a known existing issue that we think is linked to how the website is hosted on CircleCI. Locally the TOC is there, style (not stacked) can be discussed later.
@drammock let us know if you have some quick fixes for 1. the navbar, 2. the back to top button and 3. TOC (though as I said, I think there is no change besides the style)

@tupui I just sent a PR into this branch with a couple conf.py tweaks. However, for me locally the API pages don't just have a poorly formatted left sidebar TOC, they have the wrong TOC (the one from the "development" page). I see that here too: https://output.circle-artifacts.com/output/job/056b1ded-d18d-4e94-bdcb-cba697e7bc45/artifacts/0/html/reference/index.html

Thanks @drammock, I just merged it.
Yes for the formatting this is what I mean and what we see at the moment on the dev doc:

Locally for me though, I have the correct content. We are hosting differently the doc for dev and releases and we have assumed it was an issue linked to how files are being distributed.

Ready from my side and I think we should merge ASAP. There are no new bugs, we finally can update and benefit from all the work that went through over the year. We can always work out extra CSS later on if wanted.

@tupui locally I've just fixed the left-sidebar not-in-a-list problem.  I'll open another PR to this branch.  I still can't grok why it's showing the wrong TOC though.

The PR build is now doing something different: no left-sidebar TOC at all on the API pages: https://output.circle-artifacts.com/output/job/ddf5b596-d4b4-4ead-993a-697c7199f96d/artifacts/0/html/reference/index.html
I don't know all the details about how your stable vs dev docs are hosted, but @tupui seems to think it's OK if those sidebars don't look right? Not really my call to make. I'm happy to keep looking into this but can't promise I'll get to it today.

I just built locally and get the expected behaviour. So yes, I still think it's linked to the deployment. And I mean, even this new wrong behaviour would be better than the current wrong behaviour 😅

@drammock in an attempt to resolve the issue, I had made an attempt at re working the sidebar with #18604. But I did not get it to work. Maybe you would spot something I missed.

@tupui I checked the build of your documentation and I think you should consider moving the version swithcer back next to the logo of scipy. I checked on small screen and the navbar_end placeholder gets crowded, putting the titles on 2 lines:
On the other hand you have lot of room left next to your logo:

@12rambau happy to do that if that fixes the issue 😃 Do you want to make a PR on the branch or shall I do it?

as my screenshot shows I'm still on my company computer I should not be working on opensource 😄 so maybe for the sake of time it's better if you do it

Done 😉 It's slightly better. There is a small transition, but that's a nitpick at that point.

Another nit but on small screens a burger button appears on the right which opens a menu with nothing on

Hey @tupui, did you want me to try to get this into 1.12.0 or 1.13.0, based on how likely you think it is for problems to arise? 1.12.0 is almost at RC2 (waiting on a Pythran release I think); 1.13.0 might be closer to early February (to support NumPy 2.0.0).

Hey @tupui, did you want me to try to get this into 1.12.0 or 1.13.0, based on how likely you think it is for problems to arise? 1.12.0 is almost at RC2 (waiting on a Pythran release I think); 1.13.0 might be closer to early February (to support NumPy 2.0.0).

I would have tried 1.12.x if that's not so much trouble now. But your call. At that point we can wait a few more months if that's easier.
I would just like that we try to get this in quick. This way more people see the new style and people can improve/tweak the CSS if wanted.

Another nit but on small screens a burger button appears on the right which opens a menu with nothing on

Yes this is to be done on the theme side. i.e. not for now. (IIRC I actually reported that at some point when I tested previous versions 😅)

The PR build is now doing something different: no left-sidebar TOC at all on the API pages

I think that this could arguably block inclusion in 1.12.0 if a fix isn't found in time. The rest of the issues that have been raised seem minor though and should be fine to save for later IMO.

Please have a look at the current dev doc. It is actually worse. It shows the TOC from another section.
As I said above, we don't get the issue with a local build. We had a look with @melissawm a while back and could not get why this was the case. The positive thing we have seen though is that the once we release, all is good.
This could be explain as the deployment is different. Released docs are hosted on a server we administrate while dev and CI are both hosted on CircleCI and GitHub pages which both serve static files.

I found the issue for the TOC https://github.com/pydata/pydata-sphinx-theme/issues/854
@drammock this seems to still be an issue.

Please have a look at the current dev doc. It is actually worse. It shows the TOC from another section.

Got you, sorry I was looking at the stable docs where it's okay. In that case, I don't see any blockers for this in 1.12 here! Of course, there could be other problems we haven't found yet, but I think the 'TOC from another section' is enough of a bug to justify the backport. And as you've said, this will unblock further improvements.

I found the issue for the TOC pydata/pydata-sphinx-theme#854
@drammock this seems to still be an issue.

That issue is about the right sidebar (the in-page TOC), I thought that was working ok here?

@drammock this is the issue: https://github.com/scipy/scipy/pull/16660#issuecomment-1879277743

for your landing page, could you add html_theme.sidebar_secondary.remove: true to your index.rst page metadata (very top of the page)
This is what we are using for our landing page and it work like a charm.

@h-vetinari I made the update 😃
@drammock there is still an issue. I said earlier that locally the sidebar was ok for the reference pages, but I made a mistake and had 0.14.x when building. With 0.15.1, it's indeed empty. I played around and I can get the TOC back if I remove the hidden markup (see last commit). The output looks good, only issue is that the build time explodes again (see the CI before the last commit).
Hence now I pushed the version with the empty TOC as it's the only one that builds in a reasonable time.
I personally would be for going with this because it's still an improvement over the current state of the dev doc deployment. The drawback being that now to deploy releases, we would need to edit the reference/index.rst to remove :hidden:. (Unless someone finds another fix)
@tylerjereddy your call 😃 We can also modify dev.py to add a doc release. It would just do this small edit.

@tupui I think it's the same issue as https://github.com/pydata/pydata-sphinx-theme/issues/1625 and should hopefully be fixed by https://github.com/pydata/pydata-sphinx-theme/pull/1632. I'm planning to make some thorough tests on monday but if you can test that PR branch with scipy build that would be helpful

(also my best guess of what was wrong:  internally we had a includehidden=False default. When calculating the sidebar partial TOCTree it was thus ignoring any hidden TOCs, so for the API page the TOCTree got pruned down to only the non-hidden TOC --- AKA the development/contrib page one)

@h-vetinari I made the update 😃

Thank you, I appreciate it! Matplotlib is not 100% consistent; once you use >=3.5 and once >3.5. I don't really care either way, JFYI.
Also, thanks a lot for your help @drammock & @12rambau! :pray:

@drammock I just tested and it does bring back the TOC on the reference page, but at the cost of a very long build time again.

Pythran is out so I may try to get RC2 out tomorrow. If this is to be backported, is someone confident enough to squash-merge and say that it is "somewhat safe?"

Pythran is out so I may try to get RC2 out tomorrow. If this is to be backported, is someone confident enough to squash-merge and say that it is "somewhat safe?"

As an outsider I don't think I really get a vote here, but personally I'd prefer to wait. Weird things keep happening and I'd like to figure out why.

Understood, I'll leave the backport label but won't place extra strain here to get it done right away.

I've just opened another PR into @tupui's branch, thinking I had finally solved things. Then I build the docs locally one more time just to make sure, and discovered I was wrong, and I think now I might know why.  It appears that the API pages can sometimes end up with the wrong left-sidebar TOC when building docs in parallel.  I'm not yet sure how deterministic this is, so I'm doing a few more test builds to see if I can narrow it down.

Awesome thanks @drammock

So I'm pretty much convinced now that the "wrong sidebar on API pages" is 100% a problem of serial-vs-parallel sphinx build. When I build with python dev.py doc -j1 it is always correct, when I build with -j8 it is always wrong. So the short-term solution is to mark pydata-sphinx-theme as parallel_write_safe=False :(  until such time as I or someone else can figure out how to fix it.
On the plus side, with https://github.com/pydata/pydata-sphinx-theme/pull/1642 I now have the local SciPy doc build down to between 6 and 7 minutes with -j1. In addition, I think that once the theme is marked as parallel-write-unsafe, y'all could still use parallel jobs during read which should still yield a non-trivial speed up over single threaded.
I'll post back again once https://github.com/pydata/pydata-sphinx-theme/pull/1642 is merged, and once we've updated the parallel_write_safe flags for the theme.

Thanks @drammock for the investigation. 6-7 mins is definitely ok, is this with no-cache? That would be a good surprise as with the old version this is what I was getting but in parallel with -j8.

Thanks @drammock for the investigation. 6-7 mins is definitely ok, is this with no-cache? That would be a good surprise as with the old version this is what I was getting but in parallel.

yes, this is with --no-cache :)

$ python dev.py doc clean  # just to make sure
$ time python dev.py doc --no-cache -j8
...
WARNING: the pydata_sphinx_theme extension is not safe for parallel writing
WARNING: doing serial write
...
real    3m49.691s
user    10m5.980s
sys     2m42.991s
$ python dev.py doc clean  # just to make sure
$ time python dev.py doc --no-cache
...
real    8m1.070s
user    8m15.306s
sys     2m22.436s

(not sure why it's 8 min this time instead of 6-7 I saw before, but still an improvement!)

Nice! The scaling was either way not so linear 😅

What's the status/plan here? Incidentally, I don't see any CI results anymore... 🤔

We're just waiting for https://github.com/pydata/pydata-sphinx-theme/pull/1642 I believe.

We're just waiting for pydata/pydata-sphinx-theme#1642 I believe.

I'll self-merge that this Friday unless one of the other maintainers asks for more time to review. Additional steps:

merge https://github.com/tupui/scipy/pull/34 into this PR
push out a point-release of the theme
update the theme min-version constraint in your pyproject.toml / requirements.txt / CI configs / etc

@tupui we had a false start; the GH release of 0.15.2 didn't make it to PyPI due to a new upstream bug in sphinx causing the tests to fail (which prevented our publish action from succeeding).  But that's all fixed now, and 0.15.2 is now on PyPI. Can you manually re-trigger CIs here (or push an empty commit)?

Thanks @drammock, just re-launched 😃🤞

13 mins! And it looks like the warnings are safe.

Just for fun, I clicked on the "Explain this error" with AI 🤣 Looks like Sphinx is still too hard for a LLM

@drammock I will have to test locally but the sidebar is still off I am afraid. The build time is good though 😃 Was there another PR you wanted to make on this branch?

I will have to test locally but online, the sidebar is still off I am afraid.

It's sidebar and top bar, it seems. I.e. clicking on the "API reference" tab on the top still shows "Development" tab being highlighted (matching the content of the left sidebar).

Yes, same as current deployment.

Argh, ok. I'll try to look again tomorrow.

[x] make sure that pydata-sphinx-theme 0.15.2 is actually getting installed into CircleCI environment (no "show installed versions" step in CircleCI, but the rendered doc says 0.15.2 in the footer, so we can be fairly sure)
[ ] check if sphinx version matters.  CircleCI logs say it was built with Sphinx 5.3.0; my local tests from before were using Sphinx 7.2.6.
[ ] ...
[ ] profit

@drammock if building locally leads to all sidebars etc being ok. Then I think we are good. We can only know for sure once we deploy for real but it might just be again the same issue we have right now with our current dev doc.

@drammock if building locally leads to all sidebars etc being ok. Then I think we are good. We can only know for sure once we deploy for real but it might just be again the same issue we have right now with our current dev doc.

in that case I guess to get the CircleCI build to "pass" you need to somehow make it OK/expected to see
WARNING: the pydata_sphinx_theme extension is not safe for parallel writing
WARNING: doing serial write

... or else stop running CircleCI with -j2

@drammock thank you. For now let's try with removing -j2 because IIUC the warning is a log and would need to write a filter on an event handler.
Otherwise I had the chance to run this locally and it seems all good.

@h-vetinari it would be helpful if you had time to confirm this too. And if everything is fine, I would suggest to move this forward.

@h-vetinari it would be helpful if you had time to confirm this too.

So far everything I've done was check the docs as rendered by CI (though IMO this is also the closest we're going to get in advance to what the actual deployment will look like), and with the latest change, the rendered docs look fine AFAICT.
Given the behaviour observed by @drammock, it seems pretty clear that the -j2 is the culprit, given the lack of thread-safety of PST?

And if everything is fine (and CI comes back green), I would suggest to move this forward.

Fine by me to move forward with this and merge, though given the current 42 commits[^1], I think we should either squash, or clean up the git history.
[^1]: the answer to life, the universe and everything!

@h-vetinari I am fine with a squash 👍
(In general I am a partisan of squash merge either way 😉)

Squash is fine with me, FWIW. I'll keep looking into why it wasn't enough to just mark the theme as parallel write unsafe... But that shouldn't hold up things here.

Thank you all for your help and dedication! This is great that we can finally move forward and be in sync with the theme.
@tylerjereddy would it be possible for you to re-publish the doc for 1.12 with this?

@h-vetinari Please feel free to merge. For the message I don't think we need much more than saying we updated the theme. No need to talk about all the commits in the squash. The only important thing to me is to keep the co-author mentions.

authors: 12rambau drammock h-vetinari j-bowhay lucascolley mdhaber melissawm rgommers tupui tylerjereddy


text:
DOC: replace only first paragraph of rng documentation

Reference issue
gh-21823
What does this implement/fix?
Currently, the _transition_to_rng decorator replaces the entirety of a function's rng decumentation with boilerplate, but sometimes additional information is desired. This PR changes the logic so that only the first paragraph is replaced; anything after the first blank line is preserved.
Additional information
For example, in gh-21823, we'd like for this text to be included in the rng description:

Specify rng for repeatable minimizations. The random numbers generated only affect the visiting distribution function and new coordinates generation.

but currently it is omitted. Compare against docs rendered with this PR.

Looks like it renders correctly. Code looks ok.

authors: andyfaff mdhaber


text:
TST: stats: skip a test to avoid a spurious log(0) warning

As a follow-up to gh-5409, suppress two spurious warnings from leaking implementation details.
Specifically, these two distributions have a=-inf, then a-1 = -inf too; then logcdf(-inf, arg) calls into the private _logcdf, which is a generic log(_cdf), and _cdf(-inf) returns zero. This does not happen for distributions with a finite value of a or specifically defined _logpdf method. Here it's just easier to filter out the warnings IMO.

This just removes the test.  If a user makes a similar call, they still get the warning, correct?

Yes. Which was there before this PR:
>>> dlaplace.logcdf(dlaplace.a, 1)
/usr/lib/python2.7/dist-packages/scipy/stats/distributions.py:4972: RuntimeWarning: divide by zero encountered in log
  return log(self._cdf(x, *args))
-inf
>>> import scipy
>>> scipy.__version__
'0.9.0'

I added these two cases to gh-5202: https://github.com/scipy/scipy/issues/5202#issuecomment-150814044

This fixes test errors, so in it goes. Thanks @ev-br, @WarrenWeckesser

authors: WarrenWeckesser ev-br rgommers


text:
MAINT: drop Python 2.7 and 3.4

Updates all build files, docs, CI config files. Also adds # cython: language_level=3 to get rid of the many warnings from Cython.
Should also solve the issues with building wheels (gh-9574), but let's check that after merge.

@tylerjereddy Just a reminder that scipy-wheels master branch will also need an update after this goes in.

LGTM.

@rgommers a practical question, why do we bump the NumPy requirement all the way up to 1.13+? It looks like there are 3.6 wheels for 1.12 on PyPi (but only up to 3.5 for NumPy 1.11).

If I remember correctly that is the earliest version of NumPy that has the wheels for all platforms available.

I see win 32/64, linux 686/x86_64, and OSX for Python 3.5 and 3.6 for NumPy 1.12.0 in the pip link I posted above. Do we need others or do these not work properly...?

Matplotlib is numpy 1.11+ currently as that was the oldest numpy that supported py3.6 iirc.
For Matplotlib we plan to drop py3.5 support in mpl3.1 (early next year), based on 36 months of support for a given python minor version (see https://matplotlib.org/devel/min_dep_policy.html).  We should try to aligen these policies across the ecosystem if possible.

Python 3.7 requires NumPy >= 1.13.1, and it's nicer to have the same minimum numpy version for each python version. This was in my proposal on the list.

Should we also remove the from __future__ ... clauses? Feels quite awkward though 😃

Given the rise of user-space python I am less concerned about staying compatible with the version of python shipped by the stable linux distributions.  As they are stable, I do not expect them to package new versions of the scipy stack for their main line (they tend to stay with what ever was out when they released).  If users want to user newer versions of our stuff they should be using some sort of user-space enviroment management (as sudo pip install will eventually end in tears) so it does not seem too big of a lift to use one that also manages python versions.
I like the time-based policy because it is not subjective (don't have to define a 'major' linux distribution, don't have to decide when 'enough' of them have moved, etc) and it easy to compute (well, as easy as estimating a release date and doing date-math ever is ;) ).
Changing the numpy support policy to "the oldest that supports the newest python supported" is interesting.

Should we also remove the from __future__ ... clauses? Feels quite awkward though

We decided not to do any cleanups yet, because 1.2.x is an LTS release and we want to keep the work for backports as low as possible. So I'd prefer to defer all cleanups for now rather than weigh up for each one if it would hinder backports.

Given the rise of user-space python I am less concerned about staying compatible with the version of python shipped by the stable linux distributions.

I sympathize with the argument, especially now that conda has become a good option. But I know a number of people disagree, and they may have a point - installing Python from source is not something that many people want to do, and not everyone will want to use conda. So this argument is unlikely to be accepted.
I like this discussion and aiming at standardizing this as much as possible. Will move it to a separate issue to not lose track of this.

Changing the numpy support policy to "the oldest that supports the newest python supported" is interesting.

We can't always do this, because we're trying to keep at least 4 supported numpy versions, to help users in their upgrade paths. So say we'll need numpy 1.17.0 for Python 3.8, then we'll go back to requiring different numpy versions for different Python versions.

Two approving reviews and no unaddressed comments, so let's get this in. Thanks all.

Thanks @rgommers

Thanks @charris. Yes for master both should be 1.13.3. For 1.2.x there should not be any changes compared to 1.1.1

Looks like SciPy built against earlier NumPy versions (1.13.1) but tested with later versions (1.15.4) emits several thousands of PendingDeprecationWarnings. Not sure why that is. Fixed in scipy-wheels by specifying 1.13.3 so the first version bigger than 1.13.1 (cached?) isn't pulled in for the testing, but the problem is probably still there. Should be tested locally just to see.

@rgommers What Mac version? Currently scipy-wheels is using 10.6, should it be 10.9?

yes 10.9 sounds right

@rgommers OK, I'll put that in scipy-wheels master. What about the v1.2.x? Scipy 1.2.0 came out with the old OpenBLAS library, and it also has an svd bug, #9620. Let's bring @tylerjereddy in also. I'll make a PR for that and leave the decision up to others.

Would it be worth adding SciPy to Python 3 Statement (as NumPy was)?

I opened an issue for that a while ago: https://github.com/python3statement/python3statement.github.io/issues/158

Thanks for the update.

authors: charris ev-br ilayn jakirkham larsoner rgommers tacaswell


text:
User enabled gears, phase 1

Allows for versioned gears to be stored, without disturbing existing endpoints too much.
Closes #645, which has the details of which endpoints changed.
@ryansanford This has two implications on the infra:

Gears should not be registered if they already exist
The database update script has a one-time one-second per existing gear document, for irritating reasons that aren't worth getting into

Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

@kofalt
Can you make sure to update/add relevant RAML docs? This link should display current snapshot from github on this branch: https://scitran.github.io/core/api/v1/index.html?branch=gear-versioning-p1

@kofalt
Is it intentional these 2 GETs now both return the whole gear doc? I'm testing with postman against the latest commit.
/api/gears?fields=all
/api/gears

Yes, I'm surprised you caught that :slightly_smiling_face:
I removed that field because Dan never ended up using it, and it only adds implementation burden with this new setup. I'll update the docs.

authors: kofalt ryansanford

repo: scitran/core

license: MIT


text:
New feature: los_intersect

This function calculates the latitude and longitude where a line-of-sight, defined by azimuth and tilt (off-nadir) angles from a starting point, intersects with the Earth ellipsoid. It also returns the slant range from the starting point to the intersection point. It has similar functionality to the lookatSpheroid routine in the Matlab mapping toolbox but the algorithm is based on https://gist.github.com/stephenHartzell/00e812ceef6901f30290203945e2cf8b#file-los_intersection-py by @StephenHartzell.
BTW, I think there may be a small inaccuracy at high latitudes in the enu2uvw function.

Thanks @rpavlick , would be interested to hear more about possible enu2uvw issue

Never mind, it was actually the other library I was testing it against that had the problem. Thanks for putting together pymap3d, it is very useful.

authors: rpavlick scivision


text:
Updated for latest version of PowerPlatform, with Platform Library support

Update that upgrades some dependencies, and allows a new feature for the PowerPlatform, allowing the usage of platform libraries for React and FluentUI, reducing production bundle sizing.   React has been moved to a dev-dependency and downgraded to utilise the same version as the PowerPlatform.
See: https://learn.microsoft.com/en-us/power-apps/developer/component-framework/react-controls-platform-libraries

Updated Mocks.
Updated StandardControl
Updated README and code examples

repo: scottdurow/pcf-react


text:
Adding new functions to libraries

This new functions should be used to in general scripts

Thanks. glad you were able to make use of this

Was very useful!
I hope soon be able to send you a SmartZone implementation pull request, which uses this new functions

sounds good. The customer I used this with wasn't using smartzone.  When I
first did this I was hoping I could use ansible but the integration from
netmiko and related libraries wasn't in as good a shape as it is now so I
rolled my own to facilitate a migration.   I figured I'd put it out there
for people to use and modify at will.
On Mon, Mar 6, 2017 at 6:45 PM, Italo Santos notifications@github.com
wrote:

Was very useful!
I hope soon be able to send you a SmartZone implementation pull request,
which uses this new functions
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/scottharney/python-mdszoning/pull/1#issuecomment-284583630,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF_xSJzNd3ye1zRu00-kPj9QccOEjHp-ks5rjKiNgaJpZM4MU43d
.

--
Scott Harney
scotth@scottharney.com
504-298-8649 cell
https://www.scottharney.com

authors: italux scottharney

repo: scottharney/python-mdszoning

license_source: bigquery