[IPython-User] IPython on EC2 with hundreds of cores

Robert Nishihara robertnishihara@gmail....
Mon Sep 24 18:26:34 CDT 2012


Does anyone have experience running IPython on Amazon's EC2 service with
hundreds of cores? I am seeing a problem that only occurs when I try to
start a cluster with a large number of engines (over 500).

Using starcluster with the ipcluster plugin, and following the directions
here <http://star.mit.edu/cluster/docs/0.93.3/plugins/ipython.html>, I am
able to start a cluster with slightly over 300 engines. Everything works
fine.

Specifically, suppose I request 10 of the cc2.8xlarge instance nodes. Then
starcluster (with the ipcluster plugin) successfully starts a cluster with
323 engines. I can connect to them with the command

    rc = parallel.Client(packer='pickle')

However, if I try the same thing with 17 or so nodes, the cluster still
starts fine, but the command

    rc = parallel.Client(packer='pickle')

times out. I think the engine registration may be timing out. Has anyone
seen a similar problem?

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120924/0105f5df/attachment.html 


More information about the IPython-User mailing list