[IPython-User] Parallel IPython -- too many open files

Erik Petigura eptune@gmail....
Sun Jan 8 01:27:04 CST 2012

Hi, Min.

On Jan 7, 2012, at 4:51 PM, Min RK wrote:

> It would help a great deal to see your actual code, or better yet a minimal test case to cause the issue.

Here's the code that uses the IPython.parallel module.  This typically doesn't crash.  When I loop through it ~80 times, I get the too many files error:

   def fitcandW(t,f,dL,par=False):
       n = len(dL)
       if par:
           from IPython.parallel import Client
           rc = Client()
           lview = rc.load_balanced_view()
           resL = lview.map(fitcand , n*[t] , n*[f] , dL, n*[False] ,block=True)
           resL = map(fitcand , n*[t] , n*[f] , dL)
       return resL

> Is it your client process that is running out of FDs?
> Are you perchance creating clients repeatedly?  If at all possible, continue to use the same client for the duration of the process.  Creating a client opens several socket connections to speak to the cluster, and if they are not properly cleaned up, then your fd count will obviously rise.  Deleting a client *should* close all its sockets, but that assumes you have actually cleared all references to it (not always an easy task, especially in an interactive IPython session).  Client.close() will explicitly close all of its sockets.

I think this is the issue.  I added 


and the code ran to completion.



> -MinRK
> On Jan 7, 2012, at 16:00, Erik Petigura <eptune@gmail.com> wrote:
>> Dear IPython,
>> I've been running code on 8 engines via:
>>  ipcluster start --n=8
>> The code ran for a while, but then it died with:
>>  ERROR: Internal Python error in the inspect module.
>>  Below is the traceback from this internal error.
>>  Traceback (most recent call last):
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/IPython/core/ultratb.py", line 750, in structured_traceback
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/site-packages/IPython/core/ultratb.py", line 232, in _fixed_getinnerframes
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 1041, in getinnerframes
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 1001, in getframeinfo
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 456, in getsourcefile
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 485, in getmodule
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/inspect.py", line 469, in getabsfile
>>    File "/Library/Frameworks/EPD64.framework/Versions/7.1/lib/python2.7/posixpath.py", line 347, in abspath
>> OSError: [Errno 24] Too many open files
>> I increased the limits on the number of open files from 256 to:
>>  [537][/Users/petigura]$ launchctl limit maxfiles
>>       maxfiles    1000000        1000000   
>> My code ran longer but died.  Is IPython opening a bunch of files when it manages its engines?
>> Thanks in advance for any help. 
>> Erik
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user

More information about the IPython-User mailing list