[IPython-User] parallel computations on a global variable

MinRK benjaminrk@gmail....
Tue Feb 12 12:42:25 CST 2013


On Mon, Feb 11, 2013 at 9:33 AM, Moritz Emanuel Beber <
moritz.beber@gmail.com> wrote:

> Dear all,
>
> I'm trying to parallalize the computation of all pairwise inner products
> of the column vectors of a matrix. Unfortunately, I have some issues:
>
> Since I want to avoid passing the large matrix many times, I pushed the
> matrix to the engines, making it available to the namespace there. I can
> access that matrix as a global just fine using a function defined in the
> notebook, doing the same in a module fails... (Please have a look at the
> two attached files.)
>
> When I use an IPython.parallel.Reference to the global matrix and pass
> that to the function, that works in the external module. Of course, this
> means I have to pass a large list containing the reference to the map
> function.
>

This is the most common confusion when working with IPython.parallel.

The gist: interactively defined functions and objects are in the `__main__`
module, so `globals()` in those scopes points to the same namespace.
When you move something to a module, you change what `globals()` points to.
 You can use the @interactive decorator (in IPython.parallel)
to make functions defined in modules behave as if they were interactively
defined (i.e. globals will be __main__ rather than the defining module).

So basically, in your code all you need to add is:

from IPython.parallel import require, interactive

@interactive
@require("numpy")
def parallel_inner(i, j):
    ...

Also note that to decorate with require, you need the `@`, which is missing
in your code.

More detail here<http://stackoverflow.com/questions/10857250/python-name-space-issues-with-ipython-parallel>
and
here<http://stackoverflow.com/questions/12304847/ipython-parallel-computing-namespace-issues>
.



>
> Insights and advice is appreciated,
> Moritz
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130212/bb4dc433/attachment.html 


More information about the IPython-User mailing list