[Numpy-discussion] Help building numpy

Bruce Southey bsouthey@gmail....
Fri Oct 8 11:35:57 CDT 2010


  On 10/08/2010 10:58 AM, Ian Goodfellow wrote:
> Here's the output on my atlas library:
> file -L /usr/local/atlas/lib/libatlas.so
> /usr/local/atlas/lib/libatlas.so: ELF 64-bit LSB shared object, 
> x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> It looks mostly the same as yours except it says "not stripped" while 
> yours says "stripped." Do you know what that means?
>
> Here's my timing code (but the difference between installations is 
> pretty obvious even without a timer, 8 full seconds on my machine 
> versus .2 seconds on a slower machine at work):
>
> import numpy as N
> import time
> rng = N.random.RandomState([0,1,2])
> A = rng.randn(1000,1000)
> t1 = time.time(); x = N.dot(A,A); t2 = time.time()
> print t2-t1
>
>
>
>
> On Fri, Oct 8, 2010 at 11:49 AM, Bruce Southey <bsouthey@gmail.com 
> <mailto:bsouthey@gmail.com>> wrote:
>
>     On 10/08/2010 10:01 AM, Ian Goodfellow wrote:
>>     I'm using 64-bit Ubuntu 10.04. I originally tried building
>>     without site.cfg and got the same result. After that I removed
>>     the installation and the numpy/build directory and tried again
>>     with site.cfg.
>>
>>     Here's the otuput of my show_config(). Does this mean it's
>>     actually using atlas? I ran 'make time' in the atlas directory
>>     and got higher % of clock speed numbers than the reference so I
>>     would be surprised if my atlas build is this slow.
>>
>>     atlas_threads_info:
>>         libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
>>         library_dirs = ['/usr/local/atlas/lib']
>>         define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>>         language = f77
>>         include_dirs = ['/usr/include']
>>     blas_opt_info:
>>         libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
>>         library_dirs = ['/usr/local/atlas/lib']
>>         define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>>         language = c
>>         include_dirs = ['/usr/include']
>>     atlas_blas_threads_info:
>>         libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
>>         library_dirs = ['/usr/local/atlas/lib']
>>         define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>>         language = c
>>         include_dirs = ['/usr/include']
>>     lapack_opt_info:
>>         libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
>>         library_dirs = ['/usr/local/atlas/lib']
>>         define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>>         language = f77
>>         include_dirs = ['/usr/include']
>>     lapack_mkl_info:
>>       NOT AVAILABLE
>>     blas_mkl_info:
>>       NOT AVAILABLE
>>     mkl_info:
>>       NOT AVAILABLE
>>
>>
>>     On Fri, Oct 8, 2010 at 10:53 AM, Bruce Southey
>>     <bsouthey@gmail.com <mailto:bsouthey@gmail.com>> wrote:
>>
>>         On 10/08/2010 09:06 AM, Benjamin Root wrote:
>>>         On Fri, Oct 8, 2010 at 8:47 AM, Ian Goodfellow
>>>         <goodfellow.ian@gmail.com <mailto:goodfellow.ian@gmail.com>>
>>>         wrote:
>>>
>>>             Can anyone explain how to get numpy to recognize atlas?
>>>             I have atlas built and installed, and I put what I
>>>             thought should be sufficient, based on numpy's
>>>             INSTALL.TXT, to make numpy use atlas in my site.cfg then
>>>             ran python setup.py install. The resulting build is
>>>             incredibly slow. Multiplying a 1,000 x 1,000 matrix
>>>             takes 8 seconds. The default packages take .8 . On a
>>>             computer at work with a similar but slower cpu to mine,
>>>             it only takes .2, so I know I should be able to get a
>>>             lot more speed out of numpy.
>>>
>>>             This is my site.cfg:
>>>
>>>             [atlas]
>>>             atlas_libs = lapack, f77blas, cblas, atlas
>>>
>>>             [DEFAULT]
>>>             library_dirs = /usr/local/atlas/lib
>>>             include_dir = /usr/local/atlas/include
>>>
>>>             Thanks in advance,
>>>             Ian
>>>
>>>
>>>         Ian, it would be very helpful if you could post the output
>>>         from the build of numpy.  That log can tell us whether it
>>>         found the libraries or not when building.
>>>
>>>         Ben Root
>>>
>>>
>>>         _______________________________________________
>>>         NumPy-Discussion mailing list
>>>         NumPy-Discussion@scipy.org
>>>         <mailto:NumPy-Discussion@scipy.org>
>>>         http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>         I do not use a 'site.cfg' even though I build my own numpy
>>         but use the standard Fedora (13) packages for atlas. So try
>>         it without defining it - remove the installation and build
>>         directories first!
>>
>>         Can you please provide the operating system (including 32bit
>>         or 64 bit), Python version and numpy version?
>>
>>         What is the output of the numpy function 'show_config()'?
>>         I added mine below from a 64-bit linux system so you can see
>>         that numpy is looking in the '/usr/lib64' directory.
>>
>>         Bruce
>>
>>         >>> import numpy as np
>>         >>> np.show_config()
>>         atlas_threads_info:
>>             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>             library_dirs = ['/usr/lib64/atlas']
>>             define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>             language = f77
>>             include_dirs = ['/usr/include']
>>         blas_opt_info:
>>             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>             library_dirs = ['/usr/lib64/atlas']
>>             define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>             language = c
>>             include_dirs = ['/usr/include']
>>         atlas_blas_threads_info:
>>             libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>             library_dirs = ['/usr/lib64/atlas']
>>             define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>             language = c
>>             include_dirs = ['/usr/include']
>>         lapack_opt_info:
>>             libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>             library_dirs = ['/usr/lib64/atlas']
>>             define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>             language = f77
>>             include_dirs = ['/usr/include']
>>         lapack_mkl_info:
>>           NOT AVAILABLE
>>         blas_mkl_info:
>>           NOT AVAILABLE
>>         mkl_info:
>>           NOT AVAILABLE
>>         >>>
>>
>>
>>
>>         _______________________________________________
>>         NumPy-Discussion mailing list
>>         NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org>
>>         http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>>
>>     _______________________________________________
>>     NumPy-Discussion mailing list
>>     NumPy-Discussion@scipy.org  <mailto:NumPy-Discussion@scipy.org>
>>     http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>     It appears to finding something but I do not know enough about
>     that aspect.
>
>     I presume that you installed your own version of atlas since you
>     are using /usr/local. So you need to be careful of the flags used
>     especially that it is 64-bit and uses the appropriate SSE
>     instructions for your processor. For example, on my system:
>
>     $ file -L /usr/lib64/atlas/libatlas.so
>     /usr/lib64/atlas/libatlas.so: ELF 64-bit LSB shared object,
>     x86-64, version 1 (SYSV), dynamically linked, stripped
>
>     Can you please provide the code you are using for timing for the
>     'slow' system?
>
>     By the way, there was this thread:
>     'HOWTO build NumPy without external math libs'
>     http://mail.scipy.org/pipermail/numpy-discussion/2010-July/051513.html
>
>     BLAS=None LAPACK=None ATLAS=None python setup.py build
>
>
>     Bruce
>
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org>
>     http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Don't use time!
Use timeit.

I suspect your build is using numerical libraries that have not been 
build correctly for your system. But I don't know anything about that.

What are the flags used to build lapack and atlas?

I told the lazy way of using timeit but may help you identify the slowness.
To capture the output I use two files for stdout and stderr:

$ python setup.py build > b.log 2>e.log
$ python setup.py install # done as root

1) Using numpy built as:
'BLAS=None LAPACK=None ATLAS=None python setup.py build'

$ python -m timeit -s "import numpy as N;  rng = 
N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000); x = N.dot(A,A)"
100000000 loops, best of 3: 0.012 usec per loop

2) Using numpy built just without Atlas:
'ATLAS=None python setup.py build'
$ python -m timeit -s "import numpy as N;  rng = 
N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000); x = N.dot(A,A)"
100000000 loops, best of 3: 0.012 usec per loop

3) Using Atlas without dot:
$ python -m timeit -s "import numpy as N; rng = 
N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000)"
100000000 loops, best of 3: 0.012 usec per loop

4) Using Atlas with dot:
$ python -m timeit -s "import numpy as N; rng = 
N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000); x = N.dot(A,A)"
100000000 loops, best of 3: 0.0131 usec per loop



Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20101008/33dac12f/attachment-0001.html 


More information about the NumPy-Discussion mailing list