[IPython-User] Parallel question: Sending data directly between engines

Matthew Rocklin mrocklin@gmail....
Wed Jan 25 15:57:34 CST 2012


I see in PR 1295 you're pickling all transmitted objects. Is there a
cleaner/faster way of doing this for arrays?
Using cPickle on a 1000 by 1000 array of float32s takes about a second on
my machine.

On Wed, Jan 25, 2012 at 3:34 PM, Matthew Rocklin <mrocklin@gmail.com> wrote:

> I'm going to rehijack this thread back to the simple peer-peer case.
>
> I was playing with the interengine example with numpy arrays and running
> into issues. It looks like its sending the array's data but none of the
> object data.
>
> In [1]: import numpy as np
>
> In [2]: run interengine.py
>
> In [3]: A, B = rc[0], rc[1]
>
> In [4]: A['data'] = np.ones((2,2))
>
> In [5]: send(rc, 0, 1, 'data')
>
> In [6]: A['data']
> Out[6]:
> array([[ 1.,  1.],
>        [ 1.,  1.]])
>
> In [7]: B['data']
> Out[7]:
> ['\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?']
>
> Is there a clean and fast way to make B's data variable a numpy array? I
> suspect this is handled already in the high-level interface. I wasn't able
> to locate this bit of code. Is there somewhere in particular I should be
> looking for this kind of thing?
>
> Best,
> -Matt
>
> On Thu, Jan 19, 2012 at 7:38 PM, Fernando Perez <fperez.net@gmail.com>wrote:
>
>> On Thu, Jan 19, 2012 at 4:50 PM, MinRK <benjaminrk@gmail.com> wrote:
>> > simple binary-tree engine interconnect example:
>> > https://github.com/ipython/ipython/pull/1295
>>
>> Thanks!  Olivier, if you have any feedback on this one, we'd appreciate
>> it.
>>
>> Cheers,
>>
>> f
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120125/186dd96b/attachment.html 


More information about the IPython-User mailing list