[IPython-User] ipython parallelism returning user-defined types

Robert Kern robert.kern@gmail....
Wed Nov 10 13:20:01 CST 2010


On 11/10/10 11:33 AM, Brian Granger wrote:
> John,
>
> On Tue, Nov 9, 2010 at 2:18 AM, John Reid <j.reid@mail.cryst.bbk.ac.uk
> <mailto:j.reid@mail.cryst.bbk.ac.uk>> wrote:
>
>     Hi,
>
>     I'm having some problems returning named tuples from a multi-engine
>     client. I do something like:
>
>     from IPython.kernel import client
>     mec = client.MultiEngineClient()
>
>     @mec.parallel()
>     def parallel_get_starts(arg):
>     <do some work>
>          return <some named tuples>
>
>     results = parallel_get_starts(<some list>)
>
>
>     but I always get errors like:
>     TypeError: __new__() takes exactly 8 arguments (2 given)
>
>
> Hmm, that is an odd error that I have never seen.  Even the fact that it is a
> TypeError is odd to me.  A couple of things to look at:
>
> * Make sure that named tuples can be pickled and unpickled.

namedtuple objects can be pickled, but you have to be particularly careful. 
Creating a namedtuple type at the prompt and then trying to send that over won't 
work because the type won't be available on the other side. IPython's FakeModule 
will cause more complications. Creating a namedtuple type inside of a function 
is even worse since the pickle cannot address it by name. If you create the 
namedtuple type in a real module that is available to both interpreters, it will 
probably work.

> * See if you can come up with a super simple example that displays this error
> (isolate it).
> * See if you can figure out where the TypeError is coming from.

It is possible that there is a name collision.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco



More information about the IPython-User mailing list