[IPython-User] Parallel question: Sending data directly between engines

Olivier Grisel olivier.grisel@ensta....
Sun Jan 8 13:36:28 CST 2012


2012/1/8 Fernando Perez <fperez.net@gmail.com>:
> On Sun, Jan 8, 2012 at 11:12 AM, Olivier Grisel
> <olivier.grisel@ensta.org> wrote:
>> I don't know as I am not familiar with the implementations of MPI
>> runtimes nor the inner workings of vowpal wabbit and its Hadoop
>> AllReduce integration. I would guess there is some kind of external
>> monitoring process that can detect those failures and dynamically
>> rewire the connected nodes to a new engines or to one another.
>
> OK, thanks.  This is a very useful discussion, thanks for taking the
> time to educate me on the matter!

No problem, it's a pleasure to have a place to discuss this kind of
topics. I really like the pyzmq API and I am convinced that
IPython.parallel + the scipy ecosystem is a very promising framework
to build upon for medium to large scale data analytics. I am really
looking forward to the pycon sprint to dive deeper in the code.

Unfortunately I already have a bunch of WIP pull-requests on
scikit-learn that I would like to finish before diving into the
IPython.parallel code base and find ways to leverage or extend it to
address the machine learning use case I have in mind. I just wanted to
give some hints on what I have in mind to make the sprint and other
potential collaborations more productive.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel


More information about the IPython-User mailing list