[Numpy-discussion] include file location and .eggs

Andrew Straw strawman at astraw.com
Mon Jan 23 09:09:01 CST 2006


konrad.hinsen at laposte.net wrote:

> On Jan 19, 2006, at 18:31, Andrew Straw wrote:


>> = Why can't eggs use traditional include locations? =
>> Multiple version support. Using eggs, it is possible to have multiple
>> versions of the same distribution, such as numpy, installed within a
>> single Python installation. (I use "distribution" to mean  collection of
>
>
> How is multiple version support in eggs supposed to work in the case 
> of packages that contain C modules?

Eggs will allow you to do it, but you'll have to specify the exact
dependencies in the setup.py file.

>
> Let me give a concrete example: Numeric, ScientificPython, and MMTK. 
> Numeric provides a C API through its header files. ScientificPython 
> and MMTK both contain C modules that depend on that C API. 
> ScientificPython also provides a C API for two of its modules, which 
> in turn MMTK depends on.

Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5.
You'd make sure the call to setup() (which is now imported from
setuptools, not distutils.core) included the following:

setup( name='Scientific',
    version='2.4.2',
    ext_modules=stuff_that_depends_on_Numeric_23_5,
    install_requires='Numeric==23.5')

Clearly, you could do this for all versions of Numeric and Scientific
you want to run together. Obviously, it could be done in a more
automated way (by querying what version of Numeric is being used to
build). To have two versions of Scientific 2.4.2 installed, however,
they would have to have different names, so perhaps '2.4.2_Numeric23.5'
would work well for the example above. Now, you can just do your imports
as normal, which will result in the the highest version number
available. To specify versions, use 'import pkg_resources;
pkg_resources.require('Numeric==23.5'); import Numeric'.

An exception is raised if you do 'import Numeric' and it imports
Numeric==24.2 and then you later try and import Scientific with only
version 2.4.2_Numeric23.5 available. (At least, if an exception isn't
raised, I'm 99% sure this would be considered a bug and would be fixed.)
So, the import order is important -- it's probably best to import
Scientific first in this example so that the correct version of Numeric
can be selected behind the scenes without requiring you to write any
pkg_resources.require() calls.

N.B. For setuptools to get a package's (e.g. Numeric's) version number,
Numeric must be built with setuptools -- not necessarily as an egg, but
at least with the egg metadata available. If you don't build as an egg,
though, you can't have more than one version installed. FYI, to build a
conventional (non-egg) with setuptools so that it includes this metadata
from an unmodified package that doesn't itself use setuptools, use the
following
python -c "import setuptools; execfile('setup.py')" install
--single-version-externally-managed

>
> Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my 
> machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then 
> need four sets of the binary modules of Scientific:
>
> 1) 2.4.2 for Numeric 23.5
> 2) 2.4.2 for Numeric 24.2
> 3) 2.5.1 for Numeric 23.5
> 4) 2.5.1 for Numeric 24.2

See above.

>
> I would also need four sets of the binary modules of MMTK, assuming 
> that I am happy with just one version of MMTK.

The above scales, so I don't think any new info is needed here.

>
> Unless some mechanism (which would have to be part of egg 
> construction or egg installation) makes sure that the correct 
> binaries are used for the particular combination of versions that I 
> wish to use, I will end up having lots of mysterious crashes 
> resulting from incompatible binary versions.

That's exactly the point -- eggs do ensure the correct binary version is
used. To come full circle to the question of header file location, this
is exactly why eggs need to have multiple versions of the header files
available, each version located within their respective egg. And this is
also why keeping the header files in
/usr/local/include/python2.4/Numeric/arrayobject.h is incompatible with
one of the best features of eggs.

>
> If the egg mechanism does take care of such dependencies, then it 
> could easily (and thus should) take care of selecting the right 
> combination of header files. If it doesn't, then it should not be 
> used at all for packages containing C or Pyrex modules, and packages 
> like numpy should better not accomodate to eggs because in the long 
> run that will only cause support headaches.

So how, about the converse? :) If setuptools does take care of
dependencies (and it does), should packages like numpy definitely
accommodate eggs? I certainly think so! :)

Andrew




More information about the Numpy-discussion mailing list