[SciPy-dev] Package organization

Pearu Peterson pearu at scipy.org
Mon Oct 17 03:34:29 CDT 2005



On Sun, 16 Oct 2005, Robert Kern wrote:

> Pearu Peterson wrote:
>>
>>> Perhaps you could explain to me how we benefit putting all of the raw
>>> wrappers into scipy.lib.*. I don't really see any.
>>
>> The scipy namespace will be cleaner, all packages that scipy will contain,
>> are more or less in the same level. Things that are in scipy.lib would be
>> raw wrappers with pythonic interfaces to various libraries that packages
>> in scipy level can use. Packages in scipy.lib would depend only on
>> scipy_core, not on each other.
>
> I would only point out that most of the time, it's those Python
> interfaces (what would go into sundials.py) that we want to expose to
> the user. When SUNDIALS gets wrapped by f2py, we then write a bit of
> Python to make the interface convenient and robust. That's pretty much
> it. The functions we write in that layer are what we want users to be
> using. I don't want that functionality hidden away in
> scipy.lib.sundials; I want it in scipy.sundials.

If odepack/vode will be under scipy.ode and quadpack under scipy.integrate 
then couldn't sundials be under scipy.dae? It's more informative for those
not knowing about sundials but need to solve dae's.

That's also related to my preference of putting wrappers under scipy.lib
and let higher lever packages such as scipy.ode, scipy.integrate, etc
to import needed tools from scipy.lib. After all, higher level packages 
may use different library wrappers for the same task, just using different 
numerical methods that may be more suitable for a particular problem.

>>> If I were starting scipy all over again, I would also make scipy.ode for
>>> ODEPACK and VODE and leave scipy.integrate for QUADPACK and the other
>>> "integrals of functions" routines. I'm not sure it's worth doing the
>>> latter at this point.
>>
>> I think in scipy 0.1 or so we had all packages, both wrapper packages as
>> well as toolkits kind of packages, in scipy namespace. I think it was a
>> mess. Moving wrapper packages to scipy.lib would reduce this mess a bit.
>
> That's where we differ, I think. I don't see the distinction between
> wrapper packages and toolkits. Most of the time, the wrapper is the
> toolkit. It's just that we tended to throw together a bunch of wrappers
> into one module and called it a toolkit. If we put all of the wrappers
> into scipy.lib.*, we're going to end up with several scipy.* packages
> that are just aliases to the scipy.lib.* packages.

I guess we have different experiences. For example, I found that it's 
easier to maintain blas/lapack wrappers under scipy.lib rather than under 
scipy.linalg. The building time for these wrappers is relatively long so 
that developing/debugging process of these packages is easier if the 
modularity is higher. On the other hand, scipy.fftpack contains wrappers 
to Fortran fftpack, fftw, and djbfft and in this case it would not be 
practical to separate wrappers according to underlying libraries as it 
was possible to unify the interface to various FFT programs.

>> I think that odepack/vode as well as quadpack should be separated
>> from scipy.integrate. My orginal idea was to move the wrappers to
>> scipy.lib and scipy.integrate would just use them from there.
>> The main advantage is that when some other scipy package might use, say,
>> odepack, but not quadpack, then does not need to install the whole
>> scipy.integrate. With this separation, the scipy will be more modular.
>
> We're shooting for the same goal. I just think we should split them into
> scipy.ode (ODEPACK and VODE) and scipy.integrate (QUADPACK and
> quadrature.py) and be done with it. You get the same level of modularity.

See above about scipy.dae suggestion.

> Splitting the wrappers into separate scipy.lib.* modules doesn't assist
> modularity unless the scipy.* packages that depend on them are also
> appropriately split up as well. At that point, most of our scipy.*
> packages are just aliases to scipy.lib.* packages.
>
> But like I said, my technical objection is now null, so I'm mostly
> arguing from aesthetics. de gustibus non est disputandum, and all that.

I think we should think further in future in this respect. If scipy would 
not grow considerably then I completely would agree with you. But as I see 
it, the current scipy contains a rather small set of tools compared to 
what it could contain. I just have a premonition that perhaps some over 
modularity today will pay off in future.

I have learned from the hard way to prefer the modularity already from the 
maintainence point of view. When tracking down bugs, the modularity gives 
great advantages. And considering how many scipy packages currently lack 
even basic unittests, there are lots of bugs to be discovered yet.

And I don't think that some internal modularity of scipy would confuse 
end users, they would still use only higher level scipy packages that 
live directly under scipy.

I would sum up the implications of this thread as follows. We should view 
each scipy package separately. Some of them can be completely standalone 
(which is always a good property of a package), they contain both toolkits 
and wrappers to domain specific libraries. But there are also scipy 
packages that would use the same library wrappers and so such libraries 
should be made directly available for all such scipy packages. And by 
directly I mean that one scipy package should not import another scipy 
package in order to access some particular wrapper library.

So, as a first instance, when developing a scipy package, library wrappers 
can be placed under this pacticular scipy package. But once the library 
wrappers could be used by another scipy package, there should be an easy 
way to move library wrappers under scipy.lib.

Would such a compromise be acceptable?

Pearu




More information about the Scipy-dev mailing list