[IPython-user] can't pickle interactively defined function

[Robert Kern]

Robin wrote:
> Hi,
> 
> After having some trouble with the processing module in ipython I
> think the problem is related to functions defined interactively in
> ipython (or through an %edit) being unpickle-able. Since this has
> nothing to do really with the processing module I thought I'd start a
> new thread. The simplest way to demonstrate the problem is below:
> 
> In [1]: from pooltest import f
> In [2]: f??
> Type:           function
> Base Class:     <type 'function'>
> String Form:    <function f at 0x2532370>
> Namespace:      Interactive
> File:           /Users/robince/bzr/handy/pooltest.py
> Definition:     f(x)
> Source:
> def f(x):
>     return x*x
> In [3]: from cPickle import dumps
> In [4]: dumps(f)
> Out[4]: 'cpooltest\nf\np1\n.'
> In [5]: def f(x):
>    ...:     return x*x
>    ...:
> In [6]: f??
> Type:           function
> Base Class:     <type 'function'>
> String Form:    <function f at 0x25323b0>
> Namespace:      Interactive
> File:           /Users/robince/bzr/handy/<ipython console>
> Definition:     f(x)
> Docstring [source file open failed]:
>     <no docstring>
> 
> In [7]: dumps(f)
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> 
> /Users/robince/bzr/handy/<ipython console> in <module>()
> 
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy_reg.pyc
> in _reduce_ex(self, proto)
>      67     else:
>      68         if base is self.__class__:
> ---> 69             raise TypeError, "can't pickle %s objects" % base.__name__
>      70         state = base(self)
>      71     args = (self.__class__, base, state)
> 
> TypeError: can't pickle function objects
> 
> Is this a bug?

It's certainly undesirable. BTW, to replicate the exact code path from 
multiprocessing, use dumps(f, 2). Using the pure Python pickle module helped me 
track it down a bit more:

In [17]: pickle.dumps(f, 2)
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)

/Users/rkern/today/<ipython console> in <module>()

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.pyc in 
dumps(obj, protocol)
    1364 def dumps(obj, protocol=None):
    1365     file = StringIO()
-> 1366     Pickler(file, protocol).dump(obj)
    1367     return file.getvalue()
    1368

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.pyc in 
dump(self, obj)
     222         if self.proto >= 2:
     223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
     225         self.write(STOP)
     226

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.pyc in 
save(self, obj)
     284         f = self.dispatch.get(t)
     285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
     287             return
     288

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.pyc in 
save_global(self, obj, name, pack)
     746             raise PicklingError(
     747                 "Can't pickle %r: it's not found as %s.%s" %
--> 748                 (obj, module, name))
     749         else:
     750             if klass is not obj:

PicklingError: Can't pickle <function f at 0xaeb9f0>: it's not found as __main__.f

In [18]: import __main__

In [19]: __main__.
__main__.__IP              __main__.__getattribute__  __main__.__reduce__
__main__.__builtins__      __main__.__hash__          __main__.__reduce_ex__
__main__.__class__         __main__.__init__          __main__.__repr__
__main__.__delattr__       __main__.__module__        __main__.__setattr__
__main__.__dict__          __main__.__name__          __main__.__str__
__main__.__doc__           __main__.__new__           __main__.__weakref__
__main__.__file__          __main__.__nonzero__


We're not putting things defined in the namespace in the __main__ module. I'm 
not sure what the best approach should be, here. There might be multiple 
namespaces floating around. Presumably, they shouldn't all be linked to __main__.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco