[IPython-dev] Musings: syntax for high-level expression of parallel (and other) execution control

Gael Varoquaux gael.varoquaux@normalesup....
Tue Sep 8 07:26:07 CDT 2009


On Sat, Sep 05, 2009 at 11:36:50PM -0700, Fernando Perez wrote:
> though it does have the same misconception that just about every other
> document about decorators has, namely

> """a function, that takes a function as input, and ***returns a
> function*** """

> The part between ** above is not correct,  and this is a subtle but
> critical point here.  

Actually, I'd like to jump in here. This is slightly off-topic, but I
believe of interest to this mailing list. I have recently been revisiting
my decoration code, to fight a common mistake I had been doing, and it
was partly due to the heavy use of a simplified pattern for decorating
that is underlying in the quote above.

The pattern
------------

________________________________________________________________________________
def with_print(func):
    """ Decorate a function to print its arguments.
    """
    def my_func(*args, **kwargs):
	print args, kwargs
	return func(*args, **kwargs)

    return my_func

@with_print
def f(x):
    print 'f called'

________________________________________________________________________________

The nice thing about this pattern is that is it quite easy to type, and
to read.


Why it is harmful
------------------

The decorated function is actually the function 'my_func', with a
reference to the orginal function 'func', a part of the scope of the
decorator 'with_print', and thus in the closure of the with_print
function.

The problem is that we have a closure here. Thus we have a variable that
are hard to get to (the undecorated function), and the decorated function
is not pickable (which is more and more important to me, e.g. for
parallel computing).


Solutions
-----------

Avoiding the closure
.....................


Use objects as a scope, rather than a closure:
________________________________________________________________________________
class WithPrint(object):
    
    def __init__(self, func):
        self.func = func
    
    def __call__(self, *args, **kwargs):
        print args, kwargs
        return self.func(*args, **kwargs)
________________________________________________________________________________


This solution is not enough: the following code won't pickle:
________________________________________________________________________________
@WithPrint
def g(x):
    print 'g called'
________________________________________________________________________________


The reason this won't pickle is that we have a name collision: the code
above expands to:
________________________________________________________________________________
def g(x):
    print 'g called'

g = WithPrint(g)
________________________________________________________________________________

and trying to pickle raises the following PicklingError:

Can't pickle <function g at 0x6ed2a8>: it's not the same object as __main__.g

If we do:
________________________________________________________________________________
def g(x):
    print 'g called'

h = WithPrint(g)
________________________________________________________________________________

we can pickle h, hurray!

Using functools.wraps
......................

However, Python comes with the answer in the standard libary
functools.wraps does the name unmangling.

Thus the following code produces a pickleable f:
________________________________________________________________________________
from functools import wraps

def with_print(func):
    """ Decorate a function to print its arguments.
    """
    @wraps(func)
    def my_func(*args, **kwargs):
        print args, kwargs
        return func(*args, **kwargs)

    return my_func

@with_print
def f(x):
    print 'f called'
________________________________________________________________________________


The pickling works simply because using functools.wraps resets the
.func_name attribute of f to have a well-defined import path. Thus
pickling works, simply by storing the import path, as all pickling of
functions.

Notice that there is only a one-line difference with the original code!

I actually tend to use a combination of both solution (an object, using
functools.wraps), to keep a reference on the undecorated functions.


Take home messages
-------------------

    - Decorators can be more clever than you think, and my not return
      objects as simple as you think
    - Think about pickling, or you'll get bitten at some point

  and most important:

    - Use functools.wraps

Sorry for going off-topic,

Gaël

-------------- next part --------------
A non-text attachment was scrubbed...
Name: tmp.py
Type: text/x-python
Size: 1323 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/ipython-dev/attachments/20090908/6a3f6f06/attachment.py 


More information about the IPython-dev mailing list