[SciPy-dev] Some concerns on Scipy development

pearu at scipy.org pearu at scipy.org
Tue Mar 26 05:14:17 CST 2002

Hi Travis,

On Tue, 26 Mar 2002, Travis Oliphant wrote:

> On Tue, 26 Mar 2002, Pearu Peterson wrote:
> >
> > 1) It is generally a good style of programming if a module imports only
> > thouse resources that it actually uses. This helps, for example, in
> > localizing and fixing bugs in a particular module level, which will be
> > much easier when doing it in a global scipy level.
> > Testing routines should be particullary careful not importing
> > irrelevant modules, escpecially, if these are also under testing.
> >
> The point with SciPy is to bring together a lot of different modules in
> one umbrella to make it easier to use.

I agree.

> My vision is that SciPy provides a basic set of functionality that lives
> under the SciPy namespace (this functionality encompasses at least
> Numeric and MLab and other basic functions).


> Modules (e.g. scipy.module_name) could in principle be distributed
> separately, but most likely will be written to depend on basic (and
> extended) scipy  functionality.

The "but" part will probably the killer for the first part.
My point is to avoid this from the very beginning when introducing a new
module. Later the dependency on basic scipy may become necessity, but 
may be not: I believe that the later may be the case especially for
wrappers to external libraries.

> > For a particular example, many modules import MLab or scipy just to
> > access only the resources in Numeric. I would remove all such imports in
> > favour of importing Numeric directly.
> But, I see SciPy subsuming Numeric's function calls.  In other words, a
> person shouldn't have to remember whether a certain expression they use
> regularly comes (originally) from Numeric or SciPy.

I agree with that as an end-user when importing scipy.
But as a developer, I am looking the code and it will not be clear where
this or that function comes from. And I am sure that also users would
appreciate if it would be clearer. I presume that SciPy will
dependent on Numeric for a long time.

> No package in SciPy should use RandomArray, or LinearAlgebra, or FFT
> (ideally) as these are all supported in SciPy.

I agree with that.

> > 2) There has been wished (see numpy-discussion list) that scipy wouldn't
> > be such a big monolitic environment. Instead, it should be possible to
> > install Scipy subpackages as standalone packages if that is possible in
> > principle. It has an advantage that other projects could use parts of
> > Scipy codes without actually building the whole scipy.
> It is hoped that this will be possible.  That is the point of "levels."
> The basic package will likely have to be installed in order to get other
> modules.  I just don't think it's worth the effort to satisfy one
> dissonant user who will probably never like SciPy no matter what we do.

For example, linalg2 needed only scipy_distutils (and that comes with
f2py) and nothing from scipy basic level, to be fully functional and
useful. Until it imported scipy...

> > > Or is this the Scipy policy to force installing/using either the whole
> > scipy package or nothing from it? I think that then some experts will not
> > use scipy at all (also I find it less stimulating to contribute to
> > scipy).
> No, but, the user should have to install some "base package."

Which currently means the whole scipy. Sure, one can remove unnecessay
modules from scipy.__init__, but then I would not call the resulting
installation as a `scipy' package, though it will share the name.
By installing scipy.module_name as a standalone module I mean that 
one makes a change to directory module_name and executes there
  ./setup_module_name.py install
and module_name will be installed as module_name, not scipy.module_name. 

Also, testing scipy.module_name becomes difficult if module_name imports
scipy: one has to install scipy before it is possible to run tests in
scipy/module_name directory. Such a waste of time... not to speak
about difficulty to repair possible bugs...

> > 3) The idea of different levels in SciPy is not working if the lowest
> > level modules import everything. It would be less confusing to drop the
> > idea of leveling from start -- I am not proposing to acctually go for it.
> > Instead, as I see it, Scipy should be rather thin interface between an
> > user and modules that it provides, with some additional (high
> > level) convinience functions (that may import "everything"). Lower level
> > modules should not depend each other unless there is a good reason for
> > that.
> First, lower levels don't depend on functions in higher levels, as far as
> I know (If they did the code wouldn't import), so I'm not sure where that
> statement comes from.

Sure. But the lower levels import the higher levels (that is included 
in import scipy statement). To me strictly only vice-versa makes sense.

> I'm not really sure how your overall idea is different from what we are
> doing,  other than the fact that I think Numeric is subsumed by SciPy
> (there are  future compatibility issues for this choise as well with
> Numeric in a state of flux).
> The level idea is just a loose association to try and capture
> different dependencies.  It solves the problem of importing things in
> the right order in the __init__ file.  It also helps to see which packages
> can be "installed separately."

I appreciate the idea. I believe that the roots of problems of importing
things in a possibly wrong order come from the fact that submodules try to
import everything -- I have several times been faced with this issue in
different projects. I have worked out a possible way to avoid these
problems with an effect that all submodules have everything and yet
they import very little. I don't want to suggest this approach to SciPy
right now though. But to get an idea, see


> > I wonder how other Scipy developers feel about these issues?
> > Do you think that my concerns are somehow relevant or not?
> > Is it acceptable trying gradually to turn lower level modules to be
> > independent from each other if possible?
> If you have particular suggestions please state them.  I'm always willing
> to consider re-factoring code.

In particular, I am currently interested in linalg2 and developing it
without the need to rebuild the whole scipy too often, already
building linalg2 alone takes some precious time. Note that you don't need
to bother with that provided that if you don't mind if I can change some
things in your commits to linalg2.


More information about the Scipy-dev mailing list