[IPython-User] parallel computations on a global variable

Moritz Beber moritz.beber@gmail....
Tue Feb 12 17:23:19 CST 2013


Hi,

On 12 Feb 2013 19:42, "MinRK" <benjaminrk@gmail.com> wrote:
>
>
>
> On Mon, Feb 11, 2013 at 9:33 AM, Moritz Emanuel Beber <
moritz.beber@gmail.com> wrote:
>>
>> Dear all,
>>
>> I'm trying to parallalize the computation of all pairwise inner products
of the column vectors of a matrix. Unfortunately, I have some issues:
>>
>> Since I want to avoid passing the large matrix many times, I pushed the
matrix to the engines, making it available to the namespace there. I can
access that matrix as a global just fine using a function defined in the
notebook, doing the same in a module fails... (Please have a look at the
two attached files.)
>>
>> When I use an IPython.parallel.Reference to the global matrix and pass
that to the function, that works in the external module. Of course, this
means I have to pass a large list containing the reference to the map
function.
>
>
> This is the most common confusion when working with IPython.parallel.
>
> The gist: interactively defined functions and objects are in the
`__main__` module, so `globals()` in those scopes points to the same
namespace.
> When you move something to a module, you change what `globals()` points
to.  You can use the @interactive decorator (in IPython.parallel)
> to make functions defined in modules behave as if they were interactively
defined (i.e. globals will be __main__ rather than the defining module).

Awesome, thank you for the explanation. I figured it must be a namespace
issue since the variable wasn't in the globals but I didn't know about the
solution.

>
> So basically, in your code all you need to add is:
>
> from IPython.parallel import require, interactive
>
> @interactive
> @require("numpy")
> def parallel_inner(i, j):
>     ...
>
> Also note that to decorate with require, you need the `@`, which is
missing in your code.

Oops :P

>
> More detail here and here.
>

A great reminder to start checking SO for IPython questions more often.

>
>>
>>
>> Insights and advice is appreciated,
>> Moritz
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>

Cheers,
Moritz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130213/70506b78/attachment.html 


More information about the IPython-User mailing list