[SciPy-user] Bundling numpy/scipy with applications

Johann Cohen-Tanugi cohen@slac.stanford....
Tue Nov 13 01:12:04 CST 2007


David Cournapeau wrote:
> Johann Cohen-Tanugi wrote:
>   
>> David Cournapeau wrote:
>>     
>>> Michael Hearne wrote:
>>>   
>>>       
>>>> I'm creating a Python application that is internal to my organization, 
>>>> but will be installed on both Linux and Mac OS X machines.  The 
>>>> application depends heavily on a number of "non-pure" modules (those 
>>>> that have C/C++/FORTRAN components), like numpy, scipy, gdal, etc.
>>>>
>>>> What is the most "pythonic" way to bundle these types of modules 
>>>> inside my application?
>>>>
>>>> I've investigated dist_utils and setuptools, and I don't see an easy 
>>>> way with those tools to include build instructions for packages built 
>>>> with autconf tools.
>>>>
>>>> Is my only recourse to write a custom install script that calls the 
>>>> "configure;make;make install" from a shell?
>>>>
>>>>     
>>>>         
>>> You have two problems: building and packaging. For packaging, autoconf 
>>> will not help you much outside providing a coherence between packages, I 
>>> think; distutils/setuptools have packaging tools, but I don't know how 
>>> good they are.
>>>
>>> On Linux, the packaging system depends on the distribution: since it is 
>>> only internal, the problem is severely simplified, since you are likely 
>>> to target only one format (rpm, deb, etc...). On Mac OS X, you have also 
>>> tools to package .pkg and .mpkg (which are a set of .pkg), available 
>>> freely on apple dev website; I have not used them much, but they look 
>>> simple and easy to use for simple command line packages.
>>>
>>> You could hack something in distutils to call configure and so on, but I 
>>> don't think it will be pleasant. Building packages using 
>>> distutils/setuptools is easy, and do not need much configuration. So I 
>>> think it is more natural to use a top level tool calling 
>>> distutils/setuptools than to use distutils as the top level tool. As a 
>>> pythonic way, you may consider build tools, such as scons or waf, both 
>>> written in python.
>>>
>>> But frankly, I would not bother too much about pythonic way: since some 
>>> tools will be shell based anyway (autotools, packaging tools), and you 
>>> don't have to care about windows, using make and shell may actually be 
>>> easier.
>>>
>>> For the build part, you may take a look at my garnumpy package: it is 
>>> essentially a set of rules for Gnu Makefiles, and it can build numpy + 
>>> scipy, with a configurable set of dependencies (ATLAS, NETLIB 
>>> BLAS/LAPACK, fftw, etc....). It can build both distutils and 
>>> autotools-based packages.
>>>
>>> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/garnumpy/garnumpy-0.4.tbz2
>>>
>>> I used it sucessfully on linux and cygwin, so I would say it should work 
>>> on Mac OS X without too much trouble. The only thing which will be 
>>> likely to be a pain is fat binaries (Universal). I use it to build a 
>>> totally self contained numpy/scipy installation, which is a first step 
>>> toward packaging. If you think it can be useful to you, don't hesitate 
>>> to ask questions; there is also a bzr archive if you want to have access 
>>> to the dev history of the tool
>>>
>>> cheers,
>>>
>>> David
>>> _______________________________________________
>>> SciPy-user mailing list
>>> SciPy-user@scipy.org
>>> http://projects.scipy.org/mailman/listinfo/scipy-user
>>>   
>>>       
>> Hi David,
>> I am having a hell of a time with lapack/atlas. 
>>     
> That's because they are far from trivial to build correctly. As always 
> with build problems, it is never complicated, but problems are often 
> difficult to track down. To make things worse, some linux distribution 
> (including fedora and suse) did include bogus packages for blas and 
> lapack. That's why I did this garnumpy thing in the first place: quickly 
> build blas/lapack correctly, so that I can test things more easily.
>   
>> I already posted the 
>> issue I currently have with scipy after following the building doc in 
>> scipy web site (see 
>> http://projects.scipy.org/pipermail/scipy-user/2007-November/014506.html). 
>> Besides, octave build seems to also have serious issues with my 
>> lapack/atlas build.
>>     
> Which distribution are you using ?
>   
>> So I immediately downloaded youor package, but I seem to have a problem 
>> with the numpy download :
>> make[3]: Leaving directory 
>> `/data1/sources/python/garnumpy-0.4/platform/numpy'
>> # Change default path when looking for libs to fake dir,
>> # so we can set everything by env variables
>> cd work/main.d/numpy-1.0.3.1 && 
>> PYTHONPATH=/home/cohen/garnumpyinstall/lib/python2.5/site-packages:/home/cohen/garnumpyinstall/lib/python2.5/site-packages/gtk-2.0 
>> /usr/bin/python \
>>         setup.py config_fc --fcompiler=gnu config
>> Traceback (most recent call last):
>>   File "setup.py", line 90, in <module>
>>     setup_package()
>>   File "setup.py", line 60, in setup_package
>>     from numpy.distutils.core import setup
>>   File 
>> "/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/__init__.py", 
>> line 39, in <module>
>>     import core
>>   File 
>> "/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/__init__.py", 
>> line 8, in <module>
>>     import numerictypes as nt
>>   File 
>> "/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/numerictypes.py", 
>> line 83, in <module>
>>     from numpy.core.multiarray import typeinfo, ndarray, array, empty, dtype
>> ImportError: No module named multiarray
>> make[2]: *** [configure-custom] Error 1
>> make[2]: Leaving directory 
>> `/data1/sources/python/garnumpy-0.4/platform/numpy'
>> make[1]: *** [../../platform/numpy/cookies/main.d/install] Error 2
>> make[1]: Leaving directory 
>> `/data1/sources/python/garnumpy-0.4/platform/scipy'
>> make: *** [imgdep-main] Error 2
>>
>>     
> This is really strange, I have never seen this error. This is what I 
> would do:
>     - remove garnumpyinstall directory
>     - go into bootstrap/lapack, and do make install
>     - go into platform/numpy, do make install
> At this point, you should have an installed numpy in garnumpyinstall: 
> test it (import numpy; numpy.test(level = 9999, verbosity = 9999). If 
> this works, then go into platform/scipy. The errors should be clearer.
>   
do make install in platform/numpy fails exactly in the same way :
[cohen@localhost numpy]$ make install
[===== NOW BUILDING:    numpy-1.0.3.1   =====]
        [fetch] complete for numpy.
        [checksum] complete for numpy.
        [extract] complete for numpy.
        [patch] complete for numpy.
# Change default path when looking for libs to fake dir,
# so we can set everything by env variables
cd work/main.d/numpy-1.0.3.1 && 
PYTHONPATH=/home/cohen/garnumpyinstall/lib/python2.5/site-packages:/home/cohen/garnumpyinstall/lib/python2.5/site-packages/gtk-2.0 
/usr/bin/python \
        setup.py config_fc --fcompiler=gnu config
Traceback (most recent call last):
  File "setup.py", line 90, in <module>
    setup_package()
  File "setup.py", line 60, in setup_package
    from numpy.distutils.core import setup
  File 
"/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/__init__.py", 
line 39, in <module>
    import core
  File 
"/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/__init__.py", 
line 8, in <module>
    import numerictypes as nt
  File 
"/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/numerictypes.py", 
line 83, in <module>
    from numpy.core.multiarray import typeinfo, ndarray, array, empty, dtype
ImportError: No module named multiarray

>> Any idea? Your package is anyway a great idea. I would actually love to 
>> see it work, of course, but also allow for the possibility to point to 
>> already exisiting source directories for numpy and scipy (for instance 
>> from a previous svn checkout). I did not read the doc of your package so 
>> it might actually be there....
>>     
> You cannot use svn, because this prevents patching from working, and 
> also there is a checksum when downloading: this is a major limitation 
> (but not so major if you realize that the only thing you really want to 
> use from svn are numpy and scipy; you can reuse blas/lapack/fftw/atlas, 
> and that's my main use when testing on different platforms).
>
> You can reuse downloaded archives, though:
>
> make garchive
>
> it will download all the tarballs (you cannot choose a subset, 
> unfortunately) and put them into a directory which will be reused 
> automatically afterwards. This means that even if you do make clean 
> anywhere in the source tree, you won't need to download over and over 
> the same things.
>
> cheers,
>
> David
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user@scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>   



More information about the SciPy-user mailing list