[IPython-User] [IPython-user] max engines?

Fernando Perez fperez.net@gmail....
Sun Sep 26 19:21:47 CDT 2010


Hi Brett,

On Tue, Aug 31, 2010 at 10:20 PM, dragm <brett.bryan@csiro.au> wrote:
>
> Hi
>
> I'm running into limits on the number of engines at around 256 running on a
> linux cluster with PBS. Could you please expand on a couple of points you
> raised below Brian. First, what are the file descriptor limits and how would
> I change them using ulimits. Please provide more detailed instructions on
> this. Second, what are the timelines and ETA on the zeromq architecture? I'm
> sure it will be eagerly awaited!

for some reason a batch of old emails was just 'burped' recently by
the scipy.org server, so we hadn't seen your message.

For file descriptors, you can try

ulimit -n <number>

but it's possible that your system has this number hardcoded and that
you can't change it.  These references might help:

http://tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/x4733.html
http://serverfault.com/questions/48717/practical-maximum-open-file-descriptors-ulimit-n-for-a-high-volume-system

As for the zmq work: in very raw, experimental form, we hope to merge
most of it in the next couple of weeks.  Much of what had to go into
pyzmq itself has already been merged, and the parts for IPython should
come in soon.

There will be more work to do to integrate the 'outer layers' of entry
point scripts that will take a couple of months.  And given the fact
that, as Brian mentioned, we don't have a security system for zmq like
we had for twisted, the most likely scenario will be to keep the
twisted system around (they can coexist) until we get either enough
time or help from others to implement authentication and encription
for the new code.  Help is very much welcome!

Regards,

f


More information about the IPython-User mailing list