[Numpy-discussion] "import numpy" is slow

David Cournapeau cournape@gmail....
Fri Aug 1 22:45:56 CDT 2008


On Sat, Aug 2, 2008 at 5:33 AM, Ondrej Certik <ondrej@certik.cz> wrote:

> But I am still unhappy about it, I'd like if the package could import
> much faster, because it adds up, when you need to import 7 packages
> like that, it's suddenly 1s and that's just too much.

Too much for what ? We need more information on the kind of things
people who complaing about numpy startup cost are doing. I suggested
lazy import a few weeks ago when this discussion started (with the
example of bzr instead of hg), but I am less convinced that it would
be that useful, because numpy is fundamentally different than bzr/hg.
As robert said, it would bring some complexity, and in an area where
python is already "fishy".

When you import numpy, you expect some core things to be available,
and they are the ones who take the most time. In bzr/hg, you use a
*program*, and you can relatively easily change the API because not
many people use it. But numpy is essentially an API, not a tool, so we
don't have this freedom. Also, it means it is relatively easy for
bzr/hg developers to control lazy import ,because they are the users,
and users of bzr/hg don't deal with python directly. If our own lazy
import has some bugs, it will impact many people who will not be able
to trace it.

The main advantage I see with lazy imports is that it avoids someone
else from breaking the speed-up work by re-importing globally a costly
package.

> But of course everything within the constrains that Robert has
> outlined. From the theoretical point of view, I don't understand why
> python cannot just import numpy (or any other package) immediatelly,
> and only at the moment the user actually access something, to import
> it in real.

I guess because it would be complex to do everywhere while keeping all
the semantics of python import. Also, like everything "lazy", it means
it is more complicated to follow what's happening. Your examples show
that it would be complex to do.

As I see it, there are some things in numpy we could do a bit
differently to cut significantly import times (a few ten ), without
changing much. Let's try that first.

> Mercurial uses a lazy import module, that does exactly
> this. Maybe that's an option?

Note that mercurial is under the GPL :)

cheers,

David


More information about the Numpy-discussion mailing list