[SciPy-dev] Package organization

[Pearu Peterson]

On Fri, 14 Oct 2005, Travis Oliphant wrote:

> First.   I like what Fernando is trying to do with the scipy.kits
> concept.   The most valuable thing scipy could provide is a mechanism
> for auto-indexing your scipy-related packages for a help browser.  I
> don't think packages would need to live under scipy to accomplish this
> (but it would make it easier).   There are other mechanism like the
> environment variable concept that could also collect package
> information.  In fact, I wonder if scipy could detect that some module
> imported it and add that to some internal index (or only add it if the
> module defined some __scipy__  name or something...

That's a difficult problem and I think with some nasty hacks it could be 
solved, e.g. overwriting builtin __import__ function by adding to it some
callback mechanism.

> Second.  Regarding the debate over the .lib directory.   I could accept
> the fact that an extremely flat packaging structure has advantages.
> After all MATLAB "gets away" with an enormously flat system.  I also see
> the dependency problem with a hierarchial approach.  I think Robert
> raises a valid point, that a nested hierarchy is largely just a
> documentation/indexing issue.

Then we have been talking about different things. I agree that too deep 
hierarchy can cause problems but I have never throught about it from the 
documentation point of view. And I never meant that scipy packages should 
be organized with a deep directory structure, the depth 2 or 3 is more 
than enough.

> It  could be handled using some standard
> form of keywords/GAMS classification numbers in the sub-packages
> themselves.
>
> For example in optimize.py
>
> __gamskeys__ = {'fmin': 'G1a1a'}
>
> Then scipy could provide an indexing system that would pick all of these
> names up and produce a GAMS hierarchy of code available.

That's a nice idea.

> I think one of the purpose of the .lib directory is to extract those
> elements of the current full scipy that cause interdependencies among
> the modules to a 'library' directory that holds the actual dependencies.
>  It could still serve that purpose.    If we just start moving things
> to the library on a case-by-case basis, it might help us decide what
> should really go there.
>
> While I don't really have enough data to form a strong opinion, I like
> the idea of a flat structure under scipy with perhaps a few
> sub-directories like lib and sandbox.

I think that putting certain scipy packages under subdirectories lib, 
sandbox, and maybe kits, is all that I would wish.

I have designed scipy.distutils to handle scipy package directories 
with any depth. But I hope that scipy.distutils design did not give this 
impression that we would need more than these subdirectories in scipy.

Pearu