[SciPy-user] Bundling numpy/scipy with applications
Tue Nov 13 01:12:04 CST 2007
David Cournapeau wrote:
> Johann Cohen-Tanugi wrote:
>> David Cournapeau wrote:
>>> Michael Hearne wrote:
>>>> I'm creating a Python application that is internal to my organization,
>>>> but will be installed on both Linux and Mac OS X machines. The
>>>> application depends heavily on a number of "non-pure" modules (those
>>>> that have C/C++/FORTRAN components), like numpy, scipy, gdal, etc.
>>>> What is the most "pythonic" way to bundle these types of modules
>>>> inside my application?
>>>> I've investigated dist_utils and setuptools, and I don't see an easy
>>>> way with those tools to include build instructions for packages built
>>>> with autconf tools.
>>>> Is my only recourse to write a custom install script that calls the
>>>> "configure;make;make install" from a shell?
>>> You have two problems: building and packaging. For packaging, autoconf
>>> will not help you much outside providing a coherence between packages, I
>>> think; distutils/setuptools have packaging tools, but I don't know how
>>> good they are.
>>> On Linux, the packaging system depends on the distribution: since it is
>>> only internal, the problem is severely simplified, since you are likely
>>> to target only one format (rpm, deb, etc...). On Mac OS X, you have also
>>> tools to package .pkg and .mpkg (which are a set of .pkg), available
>>> freely on apple dev website; I have not used them much, but they look
>>> simple and easy to use for simple command line packages.
>>> You could hack something in distutils to call configure and so on, but I
>>> don't think it will be pleasant. Building packages using
>>> distutils/setuptools is easy, and do not need much configuration. So I
>>> think it is more natural to use a top level tool calling
>>> distutils/setuptools than to use distutils as the top level tool. As a
>>> pythonic way, you may consider build tools, such as scons or waf, both
>>> written in python.
>>> But frankly, I would not bother too much about pythonic way: since some
>>> tools will be shell based anyway (autotools, packaging tools), and you
>>> don't have to care about windows, using make and shell may actually be
>>> For the build part, you may take a look at my garnumpy package: it is
>>> essentially a set of rules for Gnu Makefiles, and it can build numpy +
>>> scipy, with a configurable set of dependencies (ATLAS, NETLIB
>>> BLAS/LAPACK, fftw, etc....). It can build both distutils and
>>> autotools-based packages.
>>> I used it sucessfully on linux and cygwin, so I would say it should work
>>> on Mac OS X without too much trouble. The only thing which will be
>>> likely to be a pain is fat binaries (Universal). I use it to build a
>>> totally self contained numpy/scipy installation, which is a first step
>>> toward packaging. If you think it can be useful to you, don't hesitate
>>> to ask questions; there is also a bzr archive if you want to have access
>>> to the dev history of the tool
>>> SciPy-user mailing list
>> Hi David,
>> I am having a hell of a time with lapack/atlas.
> That's because they are far from trivial to build correctly. As always
> with build problems, it is never complicated, but problems are often
> difficult to track down. To make things worse, some linux distribution
> (including fedora and suse) did include bogus packages for blas and
> lapack. That's why I did this garnumpy thing in the first place: quickly
> build blas/lapack correctly, so that I can test things more easily.
>> I already posted the
>> issue I currently have with scipy after following the building doc in
>> scipy web site (see
>> Besides, octave build seems to also have serious issues with my
>> lapack/atlas build.
> Which distribution are you using ?
>> So I immediately downloaded youor package, but I seem to have a problem
>> with the numpy download :
>> make: Leaving directory
>> # Change default path when looking for libs to fake dir,
>> # so we can set everything by env variables
>> cd work/main.d/numpy-188.8.131.52 &&
>> /usr/bin/python \
>> setup.py config_fc --fcompiler=gnu config
>> Traceback (most recent call last):
>> File "setup.py", line 90, in <module>
>> File "setup.py", line 60, in setup_package
>> from numpy.distutils.core import setup
>> line 39, in <module>
>> import core
>> line 8, in <module>
>> import numerictypes as nt
>> line 83, in <module>
>> from numpy.core.multiarray import typeinfo, ndarray, array, empty, dtype
>> ImportError: No module named multiarray
>> make: *** [configure-custom] Error 1
>> make: Leaving directory
>> make: *** [../../platform/numpy/cookies/main.d/install] Error 2
>> make: Leaving directory
>> make: *** [imgdep-main] Error 2
> This is really strange, I have never seen this error. This is what I
> would do:
> - remove garnumpyinstall directory
> - go into bootstrap/lapack, and do make install
> - go into platform/numpy, do make install
> At this point, you should have an installed numpy in garnumpyinstall:
> test it (import numpy; numpy.test(level = 9999, verbosity = 9999). If
> this works, then go into platform/scipy. The errors should be clearer.
do make install in platform/numpy fails exactly in the same way :
[cohen@localhost numpy]$ make install
[===== NOW BUILDING: numpy-184.108.40.206 =====]
[fetch] complete for numpy.
[checksum] complete for numpy.
[extract] complete for numpy.
[patch] complete for numpy.
# Change default path when looking for libs to fake dir,
# so we can set everything by env variables
cd work/main.d/numpy-220.127.116.11 &&
setup.py config_fc --fcompiler=gnu config
Traceback (most recent call last):
File "setup.py", line 90, in <module>
File "setup.py", line 60, in setup_package
from numpy.distutils.core import setup
line 39, in <module>
line 8, in <module>
import numerictypes as nt
line 83, in <module>
from numpy.core.multiarray import typeinfo, ndarray, array, empty, dtype
ImportError: No module named multiarray
>> Any idea? Your package is anyway a great idea. I would actually love to
>> see it work, of course, but also allow for the possibility to point to
>> already exisiting source directories for numpy and scipy (for instance
>> from a previous svn checkout). I did not read the doc of your package so
>> it might actually be there....
> You cannot use svn, because this prevents patching from working, and
> also there is a checksum when downloading: this is a major limitation
> (but not so major if you realize that the only thing you really want to
> use from svn are numpy and scipy; you can reuse blas/lapack/fftw/atlas,
> and that's my main use when testing on different platforms).
> You can reuse downloaded archives, though:
> make garchive
> it will download all the tarballs (you cannot choose a subset,
> unfortunately) and put them into a directory which will be reused
> automatically afterwards. This means that even if you do make clean
> anywhere in the source tree, you won't need to download over and over
> the same things.
> SciPy-user mailing list
More information about the SciPy-user