[IPython-User] Parallel question: Sending data directly between engines

Olivier Grisel olivier.grisel@ensta....
Wed Jan 25 16:35:59 CST 2012


2012/1/25 MinRK <benjaminrk@gmail.com>:
> See this pyzmq example for non-copying sends/recvs of numpy arrays:
> https://github.com/zeromq/pyzmq/blob/master/examples/serialization/serialsocket.py#L33
>
> You will have to write your own serialize/deserialize functions,
> depending on your data structures, but that example shows that simple
> numpy arrays are trivial.

It might thus be possible to write a non-memory-copying ZMQ-aware Pickler
implementation that streams the arrays efficiently yet
transparently and only copy the python object structure boilerplate as
a prefix in the packed message. That might require a bit of work to
get right though.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel


More information about the IPython-User mailing list