[SciPy-Dev] lazy imports feedback request

Ralf Gommers ralf.gommers@googlemail....
Fri Sep 30 03:21:22 CDT 2011

On Thu, Sep 29, 2011 at 7:18 AM, Paul Ivanov <pivanov314@gmail.com> wrote:

> Friends and uncles,
> I'm seeking feedback about some lazy import functionality I'm trying to
> implement in nitime. In short, "foo = LazyImport('foo')" returns something
> that
> will act as foo if you did just "import foo" - but only perform the import
> when
> foo is actually used, not when it is defined.
> I'll outline the highlights in this email, but you can see the full
> four-leaf
> clover pull request discussion here:
> https://github.com/nipy/nitime/pull/88
> Here's the code, with an explanatory docstring
> class LazyImport(object):
>    """
>    This class takes the module name as a parameter, and acts as a proxy for
>    that module, importing it only when the module is used, but effectively
>    acting as the module in every other way (including inside IPython with
>    respect to introspection and tab completion) with the *exception* of
>    reload().
>    >>> mlab = LazyImport('matplotlib.mlab')
>    No import happens on the above line, until we do something like call an
>    mlab method or try to do tab completion or introspection on mlab
> in IPython.
>    >>> mlab
>    <module 'matplotlib.mlab' will be lazily loaded>
>    Now the LazyImport will do an actual import, and call the dist function
> of
>    the imported module.
>    >>> mlab.dist(1969,2011)
>    42.0
>    >>> mlab
>    <module 'matplotlib.mlab' from '.../site-packages/matplotlib/mlab.pyc'>
>    """
>    def __init__(self, modname):
>        self.__lazyname__= modname
>    def __getattribute__(self,x):
>        # This method will be called only once
>        name = object.__getattribute__(self,'__lazyname__')
>        module =__import__(name, fromlist=name.split('.'))
>        # Now that we've done the import, cutout the middleman
>        class LoadedLazyImport(object):
>            __getattribute__ = module.__getattribute__
>            __repr__ = module.__repr__
>        object.__setattr__(self,'__class__', LoadedLazyImport)
>        return module.__getattribute__(x)
>    def __repr__(self):
>        return "<module '%s' will be lazily loaded>" %\
>                object.__getattribute__(self,'__lazyname__')
> The win is that when you get a faster import and smaller memory footprint
> by
> not loading those modules which your code path ends up not using.
> The functionality of lazyimports.LazyImport is generic enough to allow the
> lazily imported module to act as the module in almost every way (tab
> completion, introspection for docstrings and sources) except reloading is
> not supported.
> Although I have not spent a lot of time trying to make reloading work, I
> think
> it is not a big deal because here in the intent is to lazily load external
> packages (matplotlib, scipy, and numpy's nosetools) which are not meant to
> be
> developed simultaneously with the project wishing to load them in a lazy
> manner.
> A difference with another strategy, such as one Keith Goodman highlighted
> in a
> thread announcing Bottlneck 0.4.1 on [SciPy-User] back in March which
> looks like this:
> #
> http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhead
> email = None
> def parse_email():
>   global email
>   if email is None:
>       import email
> is that LazyImport keeps you from having to check if a given module
> has been lazily
> loaded in every place that it is used.
> Fernando Perez suggested I contact this list because he recalls similar
> functionality being implemented in scipy at some point, and then removed
> because it was problematic.

Until recently this was indeed in scipy, using the PackageLoader in
numpy/_import_tools.py. You will likely run into strange issues like
http://projects.scipy.org/scipy/ticket/1501 at some point, so you have to
have a very good use-case to justify this kind of lazy loading. For scipy
that wasn't really the case.

Plus the code you have to write to get the functionality you want is quite


> thanks a lot in advance for any feedback. I'm cc-ing my primary
> address in hope that your mail reader will do the same.
> best,
> --
> Paul Ivanov
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-dev/attachments/20110930/93fbbf3c/attachment.html 

More information about the SciPy-Dev mailing list