[IPython-User] Parallel question: Sending data directly between engines

Olivier Grisel olivier.grisel@ensta....
Wed Jan 25 16:04:29 CST 2012


2012/1/25 Matthew Rocklin <mrocklin@gmail.com>:
> I see in PR 1295 you're pickling all transmitted objects. Is there a
> cleaner/faster way of doing this for arrays?
> Using cPickle on a 1000 by 1000 array of float32s takes about a second on my
> machine.

If cPickle is the bottleneck, then maybe stealing this code might be useful:

  https://github.com/joblib/joblib/blob/master/joblib/numpy_pickle.py

This implementation is file centric but it might be possible to adapt
to zmq sockets or in-memory buffers.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel


More information about the IPython-User mailing list