[IPython-User] Parallel question: Sending data directly between engines

MinRK benjaminrk@gmail....
Wed Jan 25 16:26:24 CST 2012


See this pyzmq example for non-copying sends/recvs of numpy arrays:
https://github.com/zeromq/pyzmq/blob/master/examples/serialization/serialsocket.py#L33

You will have to write your own serialize/deserialize functions,
depending on your data structures, but that example shows that simple
numpy arrays are trivial.

-MinRK

On Wed, Jan 25, 2012 at 14:04, Olivier Grisel <olivier.grisel@ensta.org> wrote:
> 2012/1/25 Matthew Rocklin <mrocklin@gmail.com>:
>> I see in PR 1295 you're pickling all transmitted objects. Is there a
>> cleaner/faster way of doing this for arrays?
>> Using cPickle on a 1000 by 1000 array of float32s takes about a second on my
>> machine.
>
> If cPickle is the bottleneck, then maybe stealing this code might be useful:
>
>  https://github.com/joblib/joblib/blob/master/joblib/numpy_pickle.py
>
> This implementation is file centric but it might be possible to adapt
> to zmq sockets or in-memory buffers.
>
> --
> Olivier
> http://twitter.com/ogrisel - http://github.com/ogrisel
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user


More information about the IPython-User mailing list