[IPython-User] ipython parallelism returning user-defined types

Brian Granger ellisonbg@gmail....
Wed Nov 10 15:39:24 CST 2010


On Wed, Nov 10, 2010 at 11:20 AM, Robert Kern <robert.kern@gmail.com> wrote:

> On 11/10/10 11:33 AM, Brian Granger wrote:
> > John,
> >
> > On Tue, Nov 9, 2010 at 2:18 AM, John Reid <j.reid@mail.cryst.bbk.ac.uk
> > <mailto:j.reid@mail.cryst.bbk.ac.uk>> wrote:
> >
> >     Hi,
> >
> >     I'm having some problems returning named tuples from a multi-engine
> >     client. I do something like:
> >
> >     from IPython.kernel import client
> >     mec = client.MultiEngineClient()
> >
> >     @mec.parallel()
> >     def parallel_get_starts(arg):
> >     <do some work>
> >          return <some named tuples>
> >
> >     results = parallel_get_starts(<some list>)
> >
> >
> >     but I always get errors like:
> >     TypeError: __new__() takes exactly 8 arguments (2 given)
> >
> >
> > Hmm, that is an odd error that I have never seen.  Even the fact that it
> is a
> > TypeError is odd to me.  A couple of things to look at:
> >
> > * Make sure that named tuples can be pickled and unpickled.
>
> namedtuple objects can be pickled, but you have to be particularly careful.
> Creating a namedtuple type at the prompt and then trying to send that over
> won't
> work because the type won't be available on the other side. IPython's
> FakeModule
> will cause more complications. Creating a namedtuple type inside of a
> function
> is even worse since the pickle cannot address it by name. If you create the
> namedtuple type in a real module that is available to both interpreters, it
> will
> probably work.
>
>
Robert makes good points.  You do have to make sure types are in modules
when using IPython.  I would put namedtuple in a module, and then make sure
you import that module in both the client and engine namespaces.


> > * See if you can come up with a super simple example that displays this
> error
> > (isolate it).
> > * See if you can figure out where the TypeError is coming from.
>
> It is possible that there is a name collision.
>
>
Yes, that sounds plausible.  Putting namedtuple in a module will also help
any name collisions.

Brian


> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma
>  that is made terrible by our own mad attempt to interpret it as though it
> had
>  an underlying truth."
>   -- Umberto Eco
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20101110/19bfbb7e/attachment.html 


More information about the IPython-User mailing list