[IPython-User] Parallel question: Sending data directly between engines
Wed Jan 25 16:04:29 CST 2012
2012/1/25 Matthew Rocklin <firstname.lastname@example.org>:
> I see in PR 1295 you're pickling all transmitted objects. Is there a
> cleaner/faster way of doing this for arrays?
> Using cPickle on a 1000 by 1000 array of float32s takes about a second on my
If cPickle is the bottleneck, then maybe stealing this code might be useful:
This implementation is file centric but it might be possible to adapt
to zmq sockets or in-memory buffers.
http://twitter.com/ogrisel - http://github.com/ogrisel
More information about the IPython-User