[IPython-User] max engines?
Mon Aug 9 14:28:50 CDT 2010
There are a couple of things to consider with our current design:
1. Eventually you will hit a file descriptor limit per process. You can
usually change this by messing with the ulimits.
2 . The performance. The current twisted based design doesn't scale very
well, but the basic idea is that the more engines you connect, the more
memory the controller will consume and the more latency you will see. Once
the lantecies get to be as long as the tasks the engines are performing, you
will stop observing speedups.
We are currently working on a new zeromq based architecture that will
dramatically improve the scalability of the parallel stuff. We are not sure
how far it will go, but i am hopeful that the new stuff will get us up to
On Fri, Aug 6, 2010 at 2:52 PM, Darren Govoni <email@example.com> wrote:
> What's the practical maximum number of engines a controller can
> support? 10? 50? 100?
As long as you are not sending lots of data between the engines and client,
I think 64-128 is doable with the current design. Best to just give it a
> IPython-User mailing list
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-User