[Numpy-discussion] package vs module

Pearu Peterson pearu at cens.ioc.ee
Tue Jan 7 02:22:03 CST 2003


On Mon, 6 Jan 2003, Perry Greenfield wrote:

> The main disadvantages I see so far are:
> 
> 1) One will either have to change import statements in old code
>    to match the new style (a pain, but generally changing imports
>    is not terribly difficult since they are easy to identify) or
>    explicitly add the path to each 3rd party module to Python
>    Path (or some equivalent).
> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can
>    require having people install FFT in the Standard Library 
>    directory structure after Python is installed. Rather, we would
>    probably have numarray look for extension modules in a standard
>    named site-packages directory (or site-numarray?) or otherwise
>    check a numarraypath environmental variable so that
>    import numarray.FFT works properly. Perhaps others have ideas
>    about how to best handle this.
> 
> Any other issues being overlooked?

There is one, though not so critical at this point but I will raise
it anyway. In summary, I am +1 for making numarray a package.

The issue is releated to import time and memory usage: more extension
modules in a package increase both of them, even if users have no
indention to use these modules. On slower machines this may cause
inconvinieces, especially in applications that call Python multiple times
for short tasks containing numarray operation.

Let me repeat, currently this is not a problem neither with Numeric
(because it never imports its extension modules) or numarray until
numarray will contain a number of extension modules that
presumably are not small.

For a realistic example of this issue consider Scipy (as a sort of upper
bound what numarray may become one day). Scipy contains a linalg module
that is an (almost complete) wrapper to ATLAS/BLAS/LAPACK libraries and
therefore importing the corresponding extension modules can be both time
and memory consuming.  For example, importing scipy to Python may take 2-5
seconds on PII 400MHz, mainly because of loading the linalg extension
modules. This time may be annoying for small but frequent tasks.

I wish Python import mechanism would be a bit smarter or lazier in
loading extension modules that are never used...

Pearu





More information about the Numpy-discussion mailing list