[IPython-User] Parallel question: Sending data directly between engines
Wed Jan 25 16:26:24 CST 2012
See this pyzmq example for non-copying sends/recvs of numpy arrays:
You will have to write your own serialize/deserialize functions,
depending on your data structures, but that example shows that simple
numpy arrays are trivial.
On Wed, Jan 25, 2012 at 14:04, Olivier Grisel <email@example.com> wrote:
> 2012/1/25 Matthew Rocklin <firstname.lastname@example.org>:
>> I see in PR 1295 you're pickling all transmitted objects. Is there a
>> cleaner/faster way of doing this for arrays?
>> Using cPickle on a 1000 by 1000 array of float32s takes about a second on my
> If cPickle is the bottleneck, then maybe stealing this code might be useful:
> This implementation is file centric but it might be possible to adapt
> to zmq sockets or in-memory buffers.
> http://twitter.com/ogrisel - http://github.com/ogrisel
> IPython-User mailing list
More information about the IPython-User