[IPython-User] Parallel question: Sending data directly between engines

[Olivier Grisel]

2012/1/25 MinRK <benjaminrk@gmail.com>:
> See this pyzmq example for non-copying sends/recvs of numpy arrays:
> https://github.com/zeromq/pyzmq/blob/master/examples/serialization/serialsocket.py#L33
>
> You will have to write your own serialize/deserialize functions,
> depending on your data structures, but that example shows that simple
> numpy arrays are trivial.

It might thus be possible to write a non-memory-copying ZMQ-aware Pickler
implementation that streams the arrays efficiently yet
transparently and only copy the python object structure boilerplate as
a prefix in the packed message. That might require a bit of work to
get right though.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel