[IPython-User] parallel computations on a global variable

Moritz Beber moritz.beber@gmail....
Tue Feb 12 17:23:19 CST 2013


On 12 Feb 2013 19:42, "MinRK" <benjaminrk@gmail.com> wrote:
> On Mon, Feb 11, 2013 at 9:33 AM, Moritz Emanuel Beber <
moritz.beber@gmail.com> wrote:
>> Dear all,
>> I'm trying to parallalize the computation of all pairwise inner products
of the column vectors of a matrix. Unfortunately, I have some issues:
>> Since I want to avoid passing the large matrix many times, I pushed the
matrix to the engines, making it available to the namespace there. I can
access that matrix as a global just fine using a function defined in the
notebook, doing the same in a module fails... (Please have a look at the
two attached files.)
>> When I use an IPython.parallel.Reference to the global matrix and pass
that to the function, that works in the external module. Of course, this
means I have to pass a large list containing the reference to the map
> This is the most common confusion when working with IPython.parallel.
> The gist: interactively defined functions and objects are in the
`__main__` module, so `globals()` in those scopes points to the same
> When you move something to a module, you change what `globals()` points
to.  You can use the @interactive decorator (in IPython.parallel)
> to make functions defined in modules behave as if they were interactively
defined (i.e. globals will be __main__ rather than the defining module).

Awesome, thank you for the explanation. I figured it must be a namespace
issue since the variable wasn't in the globals but I didn't know about the

> So basically, in your code all you need to add is:
> from IPython.parallel import require, interactive
> @interactive
> @require("numpy")
> def parallel_inner(i, j):
>     ...
> Also note that to decorate with require, you need the `@`, which is
missing in your code.

Oops :P

> More detail here and here.

A great reminder to start checking SO for IPython questions more often.

>> Insights and advice is appreciated,
>> Moritz
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130213/70506b78/attachment.html 

More information about the IPython-User mailing list