[SciPy-dev] Re: [Numpy-discussion] Release of SciPy Core 0.4 (Beta)

Robert Kern rkern at ucsd.edu
Sun Oct 2 21:26:46 CDT 2005

Travis Oliphant wrote:
> Robert Kern wrote:
>> No one should have to totally rewrite their setup.py scripts, just add
>> two lines:
>>  from scipy.distutils.misc_util import get_scipy_include_dirs
>>  myext = Extension('myext', ...
>>                    include_dirs=get_scipy_include_dirs())
> What I'm saying is that the default setup command needs to include the
> directory where the headers are installed as well.
> Yes, this is only two lines, but I think it will cause issues for
> current packages that already use Numeric.   They will have to re-write
> their setup.py scripts before they can use scipy_core.

I think that overstates the issue. They don't have to rewrite their
setup.py scripts from top to bottom. They just add to it. There are no
other complications. Odds are that most setup.py scripts try to import
Numeric to check for its existence; people will likely have to do some
touch-ups to them regardless.

> How standard is it to put headers all over the place.  It seems much
> more natural to have one or two include root trees where all headers go.
> I also think the default place to install headers should be where Python
> installs them. This should only change if the user cannot write there
> for whatever reason.
> On my system, for example, it seems silly to install to
> /usr/lib/python2.4/site-packages/scipy/base/include/scipy  
> when because I can write to site-packages I can write to /usr/lib, I
> could also write to
> /usr/include/python2.4/scipy

I also don't see the harm in it.

> If the user can't write to the default include directory, then just
> place the headers into another "user-default" directory -- either
> specified by the user or some higher-level place:
> If scipy is installed here:
> /some/foo/directory/scipy
> then the headers should be here:
> /some/foo/directory/include/scipy
> and not where they currently are (clear down in the guts of scipy).
> Preferably, I suppose you would just install to
> $PREFIX/include/pythonX.X/scipy
> where the user can fix the PREFIX.

$PREFIX isn't reliable. Some setups don't have the UNIX-like
$prefix/lib, $prefix/include, etc. structure. For instance, non-root
users on Macs would install packages to
~/Library/Python/2.4/site-packages/. There's no place for non-root users
to install include files.

Then those users will have to modify every scipy-using package's
setup.py to point to the nonstandard include directory. I would much
prefer there to be one standard way to get the include directory for
scipy. That way, it could be done in the upstream package's setup.py
*once* and it will work unchanged on *every* system.

There was never a standard secondary directory for when you couldn't
install to the Python include directory. People had to find a place to
put the headers and manually modify the setup.py scripts to look there
for headers, too.

Putting the headers in the package also allows scipy_core to be packaged
as a PythonEgg, which does not bundle things installed by install_header
(nor can it be modified do so reliably). Packaging the headers in the
package itself induces a very small one-time cost and resolves problems
that we've had for years.

> I'm really concerned about making it easy for people to be able to build
> their packages against the new scipy_core system easily.   This new
> include system seems strange to me.  Are their other packages out there
> that offer a C-API.  How do they package their include file?

AFAICT, we're the only community (Numeric, numarray, Konrad's
Scientific) that installs header files into Python include directory.
numarray has a way to inquire from the package where its headers were
installed just like I'm suggesting, although they install headers to the
Python directory anyways. That mechanism fails when numarray is bundled
as an Egg since it is expecting to locate the headers in a place
external to the Egg.

wxPython installs headers with the wxWidgets headers (also breaking Eggs).

The egenix tools (mxBeeBase, mxStack, etc.) install headers into the
package itself.

Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

More information about the Scipy-dev mailing list