[SciPy-dev] Package organization

[Perry Greenfield]

I've finally had a chance to read over the messages on this topic and 
thought I would make a few comments. It's certainly possible that I 
misunderstand some of the points already made so I'll summarize some 
(not all) of the issues raised, at least those I have an interest in, 
as well as asking for clarification for those issues that I didn't see 
clearly spelled out.

1) flatter is better than deeper. I'm inclined to agree with this. The 
objection is that flat will lead to many name collisions within scipy. 
I guess this depends on the next issue.

2) are all the items within the scipy namespace controlled? Some of the 
discussions suggested that mechanisms be provided to extend the search 
path for scipy libraries and such. I guess the question I have is why 
such libraries should be loaded into the scipy namespace. Nothing 
prevents them from having dependencies on scipy, but do they actually 
need to be loaded into its namespace? I can't think of any reason why. 
If not, then that means the scipy project is the master of the 
namespace and should detect when someone tries to add a duplicate name.

3) This doesn't mean that naming problems won't arise. Using the 
example given regarding astropy, I'm not sure that an astropy wcs 
package should just go right into the scipy namespace. That acronym 
might be used by other specialties in entirely different ways. In this 
case there is something to be said for qualifying the name somehow. 
That might be with the name itself, i.e., astrowcs. or that domain 
specific knowledge does justify some nesting. Perhaps we should 
consider a policy where mathematical algorithms are treated as general. 
After all, they may be used in many specialties of science and 
engineering. So it is important not to group them by kind of algorithm, 
or by those that might use it. On the other hand, there are many 
domain-specific areas where it is unlikely that the package or module 
will be used in another. I doubt that many other areas (but in fact, a 
few do) will use the FITS format. A library like that could reasonably 
be confined to its own namespace (like astrolib). I don't know if there 
is any simple approach to this. It seems to me some judgment is 
required. One could argue that something like astronomical-specific 
libraries shouldn't fall under scipy at all but in its own package.

4) So why not start flat with algorithms, and leave domains to define 
their own name space as a starting point? If it is necessary to move 
things into scipy because they have usefulness over more than one 
domain, then do that when the need arises. Being conservative on this 
can probably put off name collision issue for a good while.

4a) But as pointed out, there are occasions where one wants a generic 
module or package (e.g., fft) for which there may be many 
implementations. With the flat approach, the generic name should be 
reserved for some sort of redirection mechanism to use a more specific 
implementation.

5) One of the points raised is that lib can be used to indicate items 
for which many other packages have dependencies. Perhaps. I suspect 
that things won't be so clear. Maybe the right thing to do is just to 
provide tools to give packages an easy way of ensuring that all needed 
dependent scipy elements are already installed and informing the user 
which are missing (ideally, automatically installing them as needed, 
but I don't know if we need to be that ambitious at the start). Perhaps 
a sandbox is useful to the extent it doesn't force a final name choice.

6) Then again, I could be entirely confused about the real issues :-)

Perry