[SciPy-dev] numpy/scipy some remarks

[Pearu Peterson]

On Wed, 18 Jan 2006, Arnd Baecker wrote:

> Hi,
>
> I just successfully installed a recent numpy/scipy using MKL as math
> library on Itanium2.
> I have few minor comments, which might cause confusion
> to new-comers (and me as well;-) to the numpy/scipy combo:
>
> 1.) What does the overwriting stuff mean?
>
>    In [1]: import scipy
>    In [2]: scipy.pkgload()
> Overwriting lib=<module 'scipy.lib' from
> '/home/baecker/python2/scipy_icc_lintst_n_mkl/lib/python2.4/site-packages/scipy/lib/__init__.pyc'>
> from
> /home/baecker/python2/scipy_icc_lintst_n_mkl/lib/python2.4/site-packages/scipy/lib/__init__.pyc
> (was <module 'numpy.lib' from
> '/home/baecker/python2/scipy_icc_lintst_n_mkl/lib/python2.4/site-packages/numpy/lib/__init__.pyc'>
> from
> /home/baecker/python2/scipy_icc_lintst_n_mkl/lib/python2.4/site-packages/numpy/lib/__init__.pyc)

It means exactly what is written. You can disable these messages by 
defining env variable:

   export SCIPY_IMPORT_VERBOSE=-1

Basically, it says which numpy symbols are overwritten by scipy symbols in 
scipy namespace. Only scipy developers should watch out these messages.

> 2.) Because I did not use ATLAS, I get the following warnings
>    during testing (scipy.test(10): Ran 1038 tests in 21.509:
>
> ****************************************************************
> WARNING: clapack module is empty
> -----------
> See scipy/INSTALL.txt for troubleshooting.
> Notes:
> * If atlas library is not found by numpy/distutils/system_info.py,
>  then scipy uses flapack instead of clapack.
> ****************************************************************
>
> ****************************************************************
> WARNING: cblas module is empty
> -----------
> See scipy/INSTALL.txt for troubleshooting.
> Notes:
> * If atlas library is not found by numpy/distutils/system_info.py,
>  then scipy uses fblas instead of cblas.
> ****************************************************************
>
>
> I am not sure, if it is clear (even after reading  scipy/INSTALL.txt),
> what one has to do with this warning.
> Should one in addition also install atlas for the clapack/cblas
> part?
> (Personally I think that for MKL one should just go for the fortran
> wrappers, and don't think about the clapack/cblas stuff.
> Isn't this only of importance, if the arrays have the wrong
> orderering in memory to avoid a copy?)

These messages are old and should be fixes. I agree that they can be 
optimized when using MKL.

> 3.) There was one of those statistical failures again:
>
> ======================================================================
> FAIL: check_normal (scipy.stats.morestats.test_morestats.test_anderson)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File
> "/home/baecker/python2/scipy_icc_lintst_n_mkl/lib/python2.4/site-packages/scipy/stats/tests/test_morestats.py",
> line 47, in check_normal
>    assert_array_less(A, crit[-2:])
>  File
> "/home/baecker/python2//scipy_icc_lintst_n_mkl/lib/python2.4/site-packages/numpy/testing/utils.py",
> line 207, in assert_array_less
>    assert cond,\
> AssertionError:
> Arrays are not less-ordered (mismatch 50.0%):
>        Array 1: 0.86583024233101469
>        Array 2: [ 0.858               1.0210000000000001]
>
>
>  ((Of course, the more often one builds, the more events of this
>   type will happen ;-))
>
> Would it make sense to run these statistical tests eg. up to 5 times,
> and only if they fail all the time, to issue a FAILURE?

+1

> 4.) help(scipy)
>
>    Contents
>    --------
>
>      numpy name space
>
>   [[ looks quite empty here -
>      is any more contents (eg. a short example?) planned ]]

Yes, documentation patches are welcome.

>    Available subpackages
>    ---------------------
>    stats       --- Statistical Functions [*]
>    sparse      --- Sparse matrix [*]
>    lib         --- Python wrappers to external libraries [*]
>    linalg      --- Linear algebra routines [*]
>    signal      --- Signal Processing Tools [*]
>    misc        --- Various utilities that don't have another home.
>    interpolate --- Interpolation Tools [*]
>    optimize    --- Optimization Tools [*]
>    cluster     --- Vector Quantization / Kmeans [*]
>    fftpack     --- Discrete Fourier Transform algorithms [*]
>    io          --- Data input and output [*]
>    integrate   --- Integration routines [*]
>    lib.lapack  --- Wrappers to LAPACK library [*]
>    special     --- Special Functions [*]
>    lib.blas    --- Wrappers to BLAS library [*]
>      [*] - using a package requires explicit import (see pkgload)
>
>   [[ couldn't one get rid of all these [*]? They don't look nice.
>      Only `misc` is different 	 ]]

:) Suggestions how to denote packages that require explicit import more 
nicely are  welcome.

>    Global symbols from subpackages
>    -------------------------------
>    stats   --> find_repeats
>    misc    --> info, factorial, factorial2, factorialk, comb, who, lena,
>                central_diff_weights, derivative, pade, source
>    fftpack --> fft, fftn, fft2, ifft, ifft2, ifftn, fftshift, ifftshift,
>                fftfreq
>
>    Utility tools
>    -------------
>
>      test        --- Run scipy unittests
>      pkgload     --- Load scipy packages
>      show_config --- Show scipy build configuration
>      show_numpy_config --- Show numpy build configuration
>
>   [[ this is directly mapped to `numpy`s show - is it really needed here
>     as well? ]]
>
>      __version__ --- Scipy version string
>      __numpy_version__ --- Numpy version string

The tools are different though. Compare scipy.test() with numpy.test() for 
instance. The same holds for other symbols. So, the answer is yes, they 
are needed.

>   [[ same with this one, why not just have it in numpy?]]
>    Environment variables
>    ---------------------
>
>      SCIPY_IMPORT_VERBOSE --- pkgload verbose flag, default is 0.

See previous answer.

Pearu