[SciPy-user] Bundling numpy/scipy with applications

Johann Cohen-Tanugi cohen@slac.stanford....
Mon Nov 12 23:58:47 CST 2007


David Cournapeau wrote:
> Michael Hearne wrote:
>   
>> I'm creating a Python application that is internal to my organization, 
>> but will be installed on both Linux and Mac OS X machines.  The 
>> application depends heavily on a number of "non-pure" modules (those 
>> that have C/C++/FORTRAN components), like numpy, scipy, gdal, etc.
>>
>> What is the most "pythonic" way to bundle these types of modules 
>> inside my application?
>>
>> I've investigated dist_utils and setuptools, and I don't see an easy 
>> way with those tools to include build instructions for packages built 
>> with autconf tools.
>>
>> Is my only recourse to write a custom install script that calls the 
>> "configure;make;make install" from a shell?
>>
>>     
> You have two problems: building and packaging. For packaging, autoconf 
> will not help you much outside providing a coherence between packages, I 
> think; distutils/setuptools have packaging tools, but I don't know how 
> good they are.
>
> On Linux, the packaging system depends on the distribution: since it is 
> only internal, the problem is severely simplified, since you are likely 
> to target only one format (rpm, deb, etc...). On Mac OS X, you have also 
> tools to package .pkg and .mpkg (which are a set of .pkg), available 
> freely on apple dev website; I have not used them much, but they look 
> simple and easy to use for simple command line packages.
>
> You could hack something in distutils to call configure and so on, but I 
> don't think it will be pleasant. Building packages using 
> distutils/setuptools is easy, and do not need much configuration. So I 
> think it is more natural to use a top level tool calling 
> distutils/setuptools than to use distutils as the top level tool. As a 
> pythonic way, you may consider build tools, such as scons or waf, both 
> written in python.
>
> But frankly, I would not bother too much about pythonic way: since some 
> tools will be shell based anyway (autotools, packaging tools), and you 
> don't have to care about windows, using make and shell may actually be 
> easier.
>
> For the build part, you may take a look at my garnumpy package: it is 
> essentially a set of rules for Gnu Makefiles, and it can build numpy + 
> scipy, with a configurable set of dependencies (ATLAS, NETLIB 
> BLAS/LAPACK, fftw, etc....). It can build both distutils and 
> autotools-based packages.
>
> http://www.ar.media.kyoto-u.ac.jp/members/david/archives/garnumpy/garnumpy-0.4.tbz2
>
> I used it sucessfully on linux and cygwin, so I would say it should work 
> on Mac OS X without too much trouble. The only thing which will be 
> likely to be a pain is fat binaries (Universal). I use it to build a 
> totally self contained numpy/scipy installation, which is a first step 
> toward packaging. If you think it can be useful to you, don't hesitate 
> to ask questions; there is also a bzr archive if you want to have access 
> to the dev history of the tool
>
> cheers,
>
> David
> _______________________________________________
> SciPy-user mailing list
> SciPy-user@scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-user
>   
Hi David,
I am having a hell of a time with lapack/atlas. I already posted the 
issue I currently have with scipy after following the building doc in 
scipy web site (see 
http://projects.scipy.org/pipermail/scipy-user/2007-November/014506.html). 
Besides, octave build seems to also have serious issues with my 
lapack/atlas build.

So I immediately downloaded youor package, but I seem to have a problem 
with the numpy download :
make[3]: Leaving directory 
`/data1/sources/python/garnumpy-0.4/platform/numpy'
# Change default path when looking for libs to fake dir,
# so we can set everything by env variables
cd work/main.d/numpy-1.0.3.1 && 
PYTHONPATH=/home/cohen/garnumpyinstall/lib/python2.5/site-packages:/home/cohen/garnumpyinstall/lib/python2.5/site-packages/gtk-2.0 
/usr/bin/python \
        setup.py config_fc --fcompiler=gnu config
Traceback (most recent call last):
  File "setup.py", line 90, in <module>
    setup_package()
  File "setup.py", line 60, in setup_package
    from numpy.distutils.core import setup
  File 
"/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/__init__.py", 
line 39, in <module>
    import core
  File 
"/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/__init__.py", 
line 8, in <module>
    import numerictypes as nt
  File 
"/data1/sources/python/garnumpy-0.4/platform/numpy/work/main.d/numpy-1.0.3.1/numpy/core/numerictypes.py", 
line 83, in <module>
    from numpy.core.multiarray import typeinfo, ndarray, array, empty, dtype
ImportError: No module named multiarray
make[2]: *** [configure-custom] Error 1
make[2]: Leaving directory 
`/data1/sources/python/garnumpy-0.4/platform/numpy'
make[1]: *** [../../platform/numpy/cookies/main.d/install] Error 2
make[1]: Leaving directory 
`/data1/sources/python/garnumpy-0.4/platform/scipy'
make: *** [imgdep-main] Error 2

Any idea? Your package is anyway a great idea. I would actually love to 
see it work, of course, but also allow for the possibility to point to 
already exisiting source directories for numpy and scipy (for instance 
from a previous svn checkout). I did not read the doc of your package so 
it might actually be there....

anyway, thanks in advance for your help.
Johann


More information about the SciPy-user mailing list