[IPython-User] IPython on EC2 with hundreds of cores

Fernando Perez fperez.net@gmail....
Tue Sep 25 15:37:32 CDT 2012


Hi Jon,

On Tue, Sep 25, 2012 at 12:31 AM, Jon Olav Vik <jonovik@gmail.com> wrote:
> Yes. See below for how I have modified my configuration. It seems to
> me that the default timeouts are suitable for a single computer, but
> need to be increased to several minutes for the case when hundreds
> of engines connect over a network that may be under high load. I
> have also slowed down heartbeating, hoping that this would reduce
> traffic. (Haven't got around to writing a proper benchmark to optimize the
> various settings, sorry.)

You're one of the people (at least that we know of) who has pushed the
furthest with ipython.parallel in large environments, and your posts
to the list are a continued source of invaluable wisdom.  We're very
grateful for that already, but I was wondering if you'd perhaps be
willing to contribute a section to the documentation, something along
the lines of "using ipython parallel in large clusters and other
high-performance scenarios" or somesuch.

Given that this would be docs-only, you could even submit the pull
request by directly editing the reST files directly on the github web
interface, so you don't even have to do a classic fork/clone/push
cycle.

I understand you're probably very busy as it is, but this is just a
thought for a rainy afternoon or a day when you want to engage in a
bit of 'productive procrastination' away from your main projects :)

Regards,

f


More information about the IPython-User mailing list