[SciPy-dev] MCMC, Kalman Filtering, AI for SciPy?
eric at enthought.com
Thu Sep 30 02:40:59 CDT 2004
Robert Kern wrote:
> Charles Harris wrote:
>> I agree that search and indexing are the best ways to find stuff, but
>> I am mostly concerned as to where to commit stuff. Clustering, where
>> does that go?
> scipy.cluster I would imagine. ;-)
>> Lattice methods, where do they go? How about useful data structures
>> or combinatorics? So on and so forth. I think the upper level GAMS
>> categories cover sufficient range that most things can be put into a
>> directory without embarrassment. As to the detailed breakdown in the
>> GAMS sub-classifications, I am not so sure.
> To make the discussion a bit more concrete, here is an example
> directory structure corresponding to the top-level GAMS
> classifications. The names are all my own, so feel free to pretend
> they are something more to your liking.
> <snip hierarchy>
> Now that I see it, it is somewhat appealing. I would probably want to
> break up some of those into two or more top-level groups. I definitely
> don't want to see too many subpackages under each of the top-level
> groups ("Flat is better than nested.").
Here is where the current SciPy modules would likely get lumped in the
linalg/ linalg, sparse
optimization/ optimize, ga
graphics/ xplt, gplt, plt
other/ cow, cluster ??, signal ??
(Cluster and signal didn't fit anywhere obvious to me)
The naming conventions are often quite similar. The SciPy names are
generally shorter which is nice for typing. Where SciPy has multiple
packages [(linalg, sparse), (optimize, ga), etc.], it is likely a good
idea. Like you, I don't want to see a deep nesting in the package
Looking at this, I don't see any real reason to reorganize top level
package names. Are any of them that bad or misleading? On the other
hand, I do think we should reorganize the functions within them some to
fix the places where they are organized based on "build" convenience
instead of actual function. This will probably necessitate the addition
of new top level groups and maybe the pruning of one of the current
ones. I've made a Wiki page to keep suggestions that people have:
If you update the page, you might also post to python-dev so that people
know to go check on the Wiki (that is so painful...). We can obviously
also just discuss it here and then transfer to the Wiki later. [side
note: this using a wiki and a mailing list for communication is also a
> Fernando, could you give an example or two where you would want to
> replicate a function across sub-packages? I'm wary of doing so as
> there is already the enormous amount of replication with respect to,
> at least, the base Numeric functions. Try scipy.special.<tab> in
> IPython. I realize what you're proposing doesn't even come close to
> that, but I'd like an example in any case.
I don't like the replication idea very well. I think things should live
in one place. Otherwise people will wonder if two functions that are
actually the same have different purposes, implementation, etc.
> And since we are talking about re-organization, is there anything we
> can do about the problem I just mentioned? It wreaks havoc with not
> only tab-completion but also automatic documentation generation .
> Is it practical to be careful about what we import into __init__.py?
> By which I mean not doing "from foo import *" in __init__.py where
> foo.py does "from scipy_base import *". On the other hand, explicitly
> listing all of the names in special is gonna be a major pain and
> fragile to boot.
I used to love "from xxx import *" and swore it was the right way to
handle Numeric, etc. since I few "array" and friends as builtin
functions... I guess I've been hanging out with to many computer
scientists lately. Or perhaps it is the few times where I have wondered
"where is function xxx [which is broken] coming from?" and struggled
through a large codebase to track it down. We've had a nasty bug or two
where one import * unexpectedly clobbered some functions from a previous
import *. In any case, it is a (seldom broken) policy to never use
import * in our code bases at Enthought. It is probably a good idea to
apply this same policy to SciPy. Doing so would partially solve the
problem you discuss.
The 2nd thought we have had on this is to put a "filter" tag within the
doc-string of a module or package to specify a set of functions that
tab-complete tools should ignore. You could say "filter: sin, cos, ..."
to remove a set of names or "filter: Numeric-all" (or something like
that) to exclude all the functions in a module/package from the
tab-complete list for a module. IPython, PyCrust, Idle, and others
could standardize on a format so that they all could benefit.
>  http://www.scipy.org/documentation/apidocs/scipy/scipy.special.html
More information about the Scipy-dev