[SciPy-dev] Scipy for Numarray
jmiller at stsci.edu
Wed Sep 22 16:16:12 CDT 2004
On Wed, 2004-09-22 at 13:42, Pearu Peterson wrote:
> On Wed, 22 Sep 2004, Charles Harris wrote:
> > Hi All,
> > I've been thinking about getting started on a version of scipy for
> > numarray. It's going to be a big job, so I have questions to put to the
> > community.
> > 1. How should it be set up in cvs? There is currently a scipy module to
> > be checked out, should we have a numarray_scipy module at the same
> > level?
> > 2. Where should it install? To avoid name conflicts, I think it might be
> > reasonable to install it, or its components, in the numarray packages
> > directory.
> > 3. There is stuff like Pycrust and ipython that is not directly related
> > to scipy, should it be taken over also, or perhaps broken out into a
> > scipy addons module.
> > 4. There is a lot of infrastructure for the installation and testing of
> > scipy. I am not very familiar with that, so how does it need to be
> > modified to install a numarray version of scipy.
> > 5. How can we keep both versions updated?
> Let me not give direct answers to your all specific questions but give
> comments on the issue that hopefully will answer the questions.
> First, there have been some off-list discussion about numarray integration
> with scipy among developers of both projects and at the moment the plan
> seems to be to provide scipy installation that has two versions of
> extension modules that are built against Numeric and numarray,
> respectively. When importing scipy then only one version of extension
> modules will be imported based on the state of some external flag,
> e.g. environment variable (imho, using env. variable is reasonable under
> unix system but how this sounds for Windows users? Alternative ideas?).
For matplotlib we primarily use a setting in the .rc file but also
support a command line switch of --Numeric or --numarray. Matplotlib
also fails over to whichever array package is actually present.
This selection is done through a module called "matplotlib.numerix"
which provides the same basic facilities for either numarray or Numeric.
> Second, we would like to avoid creating numarray branch of scipy if
> possible. Otherwise, merging the Numeric and numarray branches in future
> might turn out to be quite a painful task.
> On the implementation of the plan:
> 1) An extension module, say foo, will be built into three files:
> _foo_na.so - foo built against numarray
> _foo_np.so - foo built against Numeric
> foo.py - Python module that imports one of the above modules
> This task should be doable on the top level of the scipy sources
> (in setup.py and scipy_distutils) without touching scipy subpackages
> at all. And f2py already supports numarray.
Top level support is a very cool idea. I just implemented this approach
"manually" for matplotlib (only two extensions were needed) and so far
it seems to work fine.
> 2) All scipy subpackages should explicitely import and use
> scipy_base functions instead of Numeric. Currently there are number of
> scipy subpackages that need to be fixed in this respect. The task should
> be trivial to do.
> 3) The most tricky part of getting numarray+scipy to work seems to
> be building scipy_base against numarray as scipy_base extension modules
> are very much Numeric specific.
Looking at this today, I see two problems for numarray (there may be
others) in scipy_base:
1. fastumathmodule.c uses the Numeric ufunc C-API which numarray doesn't
support (yet anyway). IMHO, this breaks down into:
a. Ordinary numarray ufuncs are probably good enough because
anyone that really wants fastumath is not going to use numarray
for other performance reasons anyway.
b. There are some new features in fastumath that need to be
supported. These latter features could be added directly to
2. Other scipy_base modules import Numeric and its addon modules
directly, similar to your (2). Instead, they should import from a TBD
More information about the Scipy-dev