[IPython-User] Parallel IPython -- too many open files

[Erik Petigura]

Dear IPython,

I've been running code on 8 engines via:

   ipcluster start --n=8

The code ran for a while, but then it died with:

   ERROR: Internal Python error in the inspect module.
   Below is the traceback from this internal error.
   Traceback (most recent call last):
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/IPython/core/ultratb.py", line 750, in structured_traceback
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/IPython/core/ultratb.py", line 232, in _fixed_getinnerframes
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 1041, in getinnerframes
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 1001, in getframeinfo
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 456, in getsourcefile
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 485, in getmodule
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 469, in getabsfile
     File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/posixpath.py", line 347, in abspath
OSError: [Errno 24] Too many open files

I increased the limits on the number of open files from 256 to:

   [537][/Users/petigura]$ launchctl limit maxfiles
        maxfiles    1000000        1000000   

My code ran longer but died.  Is IPython opening a bunch of files when it manages its engines?

Thanks in advance for any help. 

Erik