[IPython-User] Parallel: problem with user-defined class in apply()

MinRK benjaminrk@gmail....
Thu Mar 21 16:13:19 CDT 2013


This is an issue with the way instances are pickled - they only get a
reference to their class (in this case __main__.MyModel),
so this class must be defined on the engines prior to the task running.
 Now, the obvious answer is to `push(MyModel=MyModel)` to your engines,
prior to calling apply, but if you are using 0.13, you will find that this
does not work (classes are pushable in master).

The gist:

- In 0.13.1 you must define the classes locally *and* remotely
- In master, this is easier, and you can just push classes like you would
with a function
- There's a PR that eliminates even the two-step process with the @require
decorator

All of these in action:

http://nbviewer.ipython.org/5216756

-MinRK


On Thu, Mar 21, 2013 at 11:13 AM, Jacob Vanderplas <
jakevdp@cs.washington.edu> wrote:

> Hi,
> I'm having an issue trying to execute a function in parallel which
> references a class object.  Here's a minimal snippet:
>
> #------------------------------------------------
> from IPython.parallel import Client
> client = Client()
> lbv = client.load_balanced_view()
>
> # the real model object is much more involved...
> class MyModel(object):
>     def maximize(self, r):
>         return max(r, 4)
>
> def evaluate(r, model):
>     return model.maximize(r)
>
> model = MyModel()
> print lbv.apply(evaluate, 2, model).get()
> #-----------------------------------------------
>
> when I run this, I get the following error:
>
> IPython.parallel.error.RemoteError: AttributeError('DummyMod' object has
> no attribute 'MyModel')
>
> I've found that I can get around this by moving the definition of MyModel
> to a separate .py file, and importing it from there rather than defining it
> in-place.  Is this generally the case in IPython parallel that objects
> defined within the script cannot be used in parallel execution?  Is there a
> way to get around this without using separate files?
>
> Thanks,
>    Jake
>
>
>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130321/0f9c3b09/attachment.html 


More information about the IPython-User mailing list