[SciPy-dev] Re: scipy.special for numarray advice

pearu at cens.ioc.ee pearu at cens.ioc.ee
Tue Mar 8 16:56:24 CST 2005


Hi Todd,

PS: I am sending this thread also to scipy-dev list, I hope you don't
mind.

I just commited `build_ext --backends` support to scipy_distutils CVS.
At the moment I have tested it only on scipy_distutils/tests/f2py_ext
example. For example, try

  cd scipy_distutils/tests/f2py_ext
  python setup.py build_src build_ext --backends=numarray,numeric build
  python tests/test_fib2.py    # default array backend is numarray
                               # as it appears first in --backends list.
  python tests/test_fib2.py --numarray
  python tests/test_fib2.py --numeric

after you have installed updated scipy_distutils.

Note that scipy_distutils/command/build_src.py does not use
scipy_base/numerix.py array backend detection mechanism as it does
not work quite right. For example, it should remove --numarray or
--numeric from sys.argv once the array backend is established.
Also, to keep --backends feature general, instead of detecting
NUMERIX env. variable, the uppercased backend name env. variable is
checked instead. For example, the last two commands above would be
equivalent to

  NUMARRAY=1 python tests/test_fib2.py
  NUMERIC=1 python tests/test_fib2.py

I am not still sure whether this would be the way to choose the array
backend package. Other suggestions are welcome.

Let me know if there are any trouble with using --backends feature while
building scipy modules for different array backends.

Regards,
Pearu

On Fri, 4 Mar 2005, Todd Miller wrote:

> Fantastic!  This sounds just right.  I think your idea about the
> longhand version of --backends=numeric,numarray and the resulting
> non-intrusive extensibility is a good one,  both from the point of view
> of clarity and future growth.
> 
> Regards,
> Todd
> 
> On Thu, 2005-03-03 at 20:21, pearu at cens.ioc.ee wrote:
> > Hi Todd,
> > 
> > Just a quick note on what I started to work on. I have introduced
> > --backends option to build_ext command that takes a comma separated list
> > of the following words: nc, na, (and in future, n3 for Numeric3).  
> > 
> > For example, if --backends=nc,na is used and some setup.py file defines
> > Extension('foo',..) then scipy_distutils replaces this with two Extension
> > instances, Extension('_nc.foo',..) and Extension('_na.foo',..), and
> > creates a foo.py file that will import either _nc.foo or _na.foo.
> > 
> > This approach has an advantage that there is no need to modify existing
> > setup.py files as well as f2py2e. In addition, the order of words in
> > --backends may define the default backend and one can build extension
> > modules for arbitrary number of array backends. Of cource, the sources of
> > extension modules must be aware of -DNUMERIC, -DNUMARRAY, (and in future,
> > -DNUMERIC3) cpp macros. 
> > 
> > I hope that I can finish --backends support by the end of this week. It is
> > almost working on my local tree, I only have to figure out the proper
> > content for the foo.py file and make sure that all extension sources are
> > compiled to different locations for different backends (this should avoid
> > the need to introduce all these nc_* and na_* files).
> > 
> > [Words nc,na,n3 could be replaced with numeric,numarray,numeric3,
> > respectively, if that would be clearer. Acctualy, the corresponding upper
> > cased words could then define cpp macros and in principle one could
> > introduce new array backends without the need to modify
> > scipy_distutils.]
> > 
> > The above approach assumes that only one array backend is used in one
> > python session. It would be possible to choose the proper backend also in
> > runtime based on the types of arguments but I'd leave this feature for
> > future because it's a non-trivial task. However, note that implementing
> > such an extension would require only figuring out the proper contents for
> > the foo.py file.
> > 
> > Regards,
> > Pearu
> > 
> > On Thu, 3 Mar 2005, Todd Miller wrote:
> > 
> > > We're still working at STScI on porting SciPy to numarray.   Obviously
> > > if Numeric3 succeeds, all this is moot,  but in the interim we're
> > > proceeding.
> > > 
> > > We're starting by porting scipy.special to numarray so I've been
> > > thinking about how to build scipy extensions for numarray.
> > > My initial thoughts were to duplicate what we did for matplotlib and
> > > scipy_base so that scipy could be built to simultaneously support either
> > > Numeric or numarray;  this might potentially double the size of the
> > > binary but makes distribution and switching back and forth easier.  The
> > > other option is to use the current scheme used with f2py:  build for
> > > numarray or Numeric but not both.
> > > 
> > > My current hand implemented porting scheme is roughly like this:
> > > 
> > > 1.  Re-do the modules for extensions using the Numeric ufunc API and use
> > > the numarray N-ary ufunc API.   This is fairly simple table driven
> > > Python code now that numarray has N-ary ufunc support.  This always has
> > > to be hand done because the Ufunc APIs are different.
> > > 
> > > 2. Re-compile f2py .pyf files for both numarray and Numeric by renaming
> > > and copying to na_*.pyf and nc_*.pyf.  Internally,  the .pyf module
> > > names get pre-pended with na_ or nc_ but that's it.  This could be
> > > completely automated in f2py or the distutils.  I've been thinking about
> > > a "numarray certified" switch which tells f2py to build for both. 
> > > 
> > > 3. For both ufunc API and f2py modules,  create Python proxy modules
> > > named after the current Numeric module.  In the proxy,  do something
> > > like:
> > > 
> > > if scipy_base.numerix.which[0] == "numeric":
> > > 	from nc_specfun import *
> > > else:
> > > 	from na_specfun import *
> > > 
> > > where nc_specfun is a Numeric version of the extension,  and na_specfun
> > > is the numarray version.  The Python proxies have a minor disadvantage
> > > that they do not effectively override the existing .so modules;  those
> > > have to be removed because they have "priority" over the same named .py
> > > at import time.
> > > 
> > > 4. Modify setup_special.py to build both versions of the extensions, one
> > > with -DNUMARRAY=1,  generate numarray wrapper code, etc.  
> > > 
> > > One obstacle is that fortranobject.o gets built and cached for either
> > > Numeric or numarray so that it's difficult to build both kinds of
> > > extensions with a single setup.  Nevertheless,  with some shennanigans
> > > it's already easily possible to build scipy.special for both.  I think
> > > one hack to make this work easier (rather than figuring out where
> > > fortranobject.o is) is to have f2py temp copy fortranobject.c to
> > > na_fortranobject.c and nc_fortranobject.c.
> > > 
> > > So,  I'm writing to see if you have any top-of-the-head comments or
> > > suggestions about this.  I understand if you have limited time or
> > > interest at this point but thought I should still ask.
> > > 
> > > Do you have an opinion on the "single binary for both array packages vs.
> > > build for either/or" choice?
> > > 
> > > Do you have any suggestions on how to get the distutils or f2py to build
> > > extensions for both Numeric and numarray?
> > > 
> > > Regards,
> > > Todd
> > > 
> > > 
> > > 
> -- 
> 
> 




More information about the Scipy-dev mailing list