[SciPy-Dev] lazy imports feedback request

Ralf Gommers ralf.gommers@googlemail....
Fri Sep 30 03:21:22 CDT 2011


On Thu, Sep 29, 2011 at 7:18 AM, Paul Ivanov <pivanov314@gmail.com> wrote:

> Friends and uncles,
>
> I'm seeking feedback about some lazy import functionality I'm trying to
> implement in nitime. In short, "foo = LazyImport('foo')" returns something
> that
> will act as foo if you did just "import foo" - but only perform the import
> when
> foo is actually used, not when it is defined.
>
> I'll outline the highlights in this email, but you can see the full
> four-leaf
> clover pull request discussion here:
> https://github.com/nipy/nitime/pull/88
>
> Here's the code, with an explanatory docstring
>
> class LazyImport(object):
>    """
>    This class takes the module name as a parameter, and acts as a proxy for
>    that module, importing it only when the module is used, but effectively
>    acting as the module in every other way (including inside IPython with
>    respect to introspection and tab completion) with the *exception* of
>    reload().
>
>    >>> mlab = LazyImport('matplotlib.mlab')
>
>    No import happens on the above line, until we do something like call an
>    mlab method or try to do tab completion or introspection on mlab
> in IPython.
>
>    >>> mlab
>    <module 'matplotlib.mlab' will be lazily loaded>
>
>    Now the LazyImport will do an actual import, and call the dist function
> of
>    the imported module.
>
>    >>> mlab.dist(1969,2011)
>    42.0
>    >>> mlab
>    <module 'matplotlib.mlab' from '.../site-packages/matplotlib/mlab.pyc'>
>    """
>    def __init__(self, modname):
>        self.__lazyname__= modname
>    def __getattribute__(self,x):
>        # This method will be called only once
>        name = object.__getattribute__(self,'__lazyname__')
>        module =__import__(name, fromlist=name.split('.'))
>        # Now that we've done the import, cutout the middleman
>        class LoadedLazyImport(object):
>            __getattribute__ = module.__getattribute__
>            __repr__ = module.__repr__
>        object.__setattr__(self,'__class__', LoadedLazyImport)
>        return module.__getattribute__(x)
>    def __repr__(self):
>        return "<module '%s' will be lazily loaded>" %\
>                object.__getattribute__(self,'__lazyname__')
>
> The win is that when you get a faster import and smaller memory footprint
> by
> not loading those modules which your code path ends up not using.
>
> The functionality of lazyimports.LazyImport is generic enough to allow the
> lazily imported module to act as the module in almost every way (tab
> completion, introspection for docstrings and sources) except reloading is
> not supported.
>
> Although I have not spent a lot of time trying to make reloading work, I
> think
> it is not a big deal because here in the intent is to lazily load external
> packages (matplotlib, scipy, and numpy's nosetools) which are not meant to
> be
> developed simultaneously with the project wishing to load them in a lazy
> manner.
>
> A difference with another strategy, such as one Keith Goodman highlighted
> in a
> thread announcing Bottlneck 0.4.1 on [SciPy-User] back in March which
> looks like this:
>
> #
> http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhead
> email = None
> def parse_email():
>   global email
>   if email is None:
>       import email
>
> is that LazyImport keeps you from having to check if a given module
> has been lazily
> loaded in every place that it is used.
>
> Fernando Perez suggested I contact this list because he recalls similar
> functionality being implemented in scipy at some point, and then removed
> because it was problematic.
>

Until recently this was indeed in scipy, using the PackageLoader in
numpy/_import_tools.py. You will likely run into strange issues like
http://projects.scipy.org/scipy/ticket/1501 at some point, so you have to
have a very good use-case to justify this kind of lazy loading. For scipy
that wasn't really the case.

Plus the code you have to write to get the functionality you want is quite
ugly.....

Cheers,
Ralf


> thanks a lot in advance for any feedback. I'm cc-ing my primary
> address in hope that your mail reader will do the same.
>
> best,
> --
> Paul Ivanov
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-dev/attachments/20110930/93fbbf3c/attachment.html 


More information about the SciPy-Dev mailing list