[IPython-user] Cannot start ipcluster

Gökhan Sever gokhansever@gmail....
Sun Oct 18 14:08:01 CDT 2009


On Sun, Oct 18, 2009 at 12:27 PM, Brian Granger <ellisonbg.net@gmail.com>wrote:

> Also, once you fix the import/name error, can you send me your script.
> That way I can see if there are any performance issues we can address.
>
> Cheers,
>
> Brian



This is a slightly modified version of the original script that I used for
timings. I just defined a generic function to find the sea files and extract
paths so that I can pass into the main processing function to be mapped.
Also attaching to file for easy access. Will test with MultiEngineClient
approach as well.

#!/usr/bin/env python

"""
Execute postprocessing_saudi script in parallel using IPython's parallel
computing features. Make sure "ipcluster local n -2" has run in a separate
shell window before running this script
"""

# Gokhan Sever <gokhansever@gmail.com>
# University of North Dakota - Dept. of Atmospheric Sciences
# Written : October 18, 2009 with help from Brain Granger

from IPython.kernel.client import TaskClient
from subprocess import call
import os


def find_sea_files():

    file_list, path_list = [], []
    init = os.getcwd()

    for root, dirs, files in os.walk('.'):
        dirs.sort()
        for file in files:
            if file.endswith('.sea'):
                file_list.append(file)
                os.chdir(root)
                path_list.append(os.getcwd())
                os.chdir(init)

    return file_list, path_list


def process_all(path, file):
    import os
    from subprocess import call
    os.chdir(path)
    call(['postprocessing_saudi', file])


if __name__ == '__main__':
    tc = TaskClient()
    files, paths = find_sea_files()
    tc.map(process_all, paths, files)



>
>
> On Sun, Oct 18, 2009 at 10:26 AM, Brian Granger <ellisonbg.net@gmail.com>wrote:
>
>>
>> from IPython.kernel.client import TaskClient
>>> from subprocess import call
>>>
>>> def process(file):
>>>     call(['postprocessing_saudi', file])
>>>
>>>
>> This function (process) will be pickled and sent to the engines.  But, the
>> engines don't have call imported yet.
>> Two ways of accomplishing this:
>>
>> def process(file):
>>     from subprocess import call
>>     call(['postprocessing_saudi', file])
>>
>> But, this has a small amount of overhead by importing call each time.
>>
>> To import it once, do:
>>
>> from IPython.kernel.client import MultiEngineClient
>> mec = MultiEngineClient()
>> mec.execute('from subprocessing import call')
>>
>> before you submit the tasks.  The MultiEngineClient interface doesn't do
>> load balancing, just does things on
>> the engines you specify (the default is all).
>>
>> Give that a shot...
>>
>> Brian
>>
>>
>>> files =
>>> ['/home/gsever/Desktop/test/20090317_131342/PostProcessing/09_03_17_13_13_42.sea',
>>>
>>>  '/home/gsever/Desktop/test/20090318_075533/PostProcessing/09_03_18_07_55_33.sea',
>>>
>>>  '/home/gsever/Desktop/test/20090319_110816/PostProcessing/09_03_19_11_08_16.sea',
>>>
>>>  '/home/gsever/Desktop/test/20090320_064651/PostProcessing/09_03_20_06_46_51.sea']
>>>
>>> if __name__ == '__main__':
>>>     tc = TaskClient()
>>>     tc.map(process, files)
>>>
>>>
>>> Should I return something from the process function and why it is
>>> complaining about "NameError: global name 'call' is not defined" I am
>>> already importing the call on top.
>>>
>>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>>  Doing "ps -aux | grep ip" yields
>>>>>>>
>>>>>>> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
>>>>>>> /usr/bin/python /usr/bin/ipcluster local -n 4
>>>>>>> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
>>>>>>> /usr/bin/python /usr/bin/ipcontroller
>>>>>>> --logfile=/home/gsever/.ipython/log/ipcontroller
>>>>>>> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>>
>>>>>>> and system monitor shows this tasks as sleeping. See the screenshot :
>>>>>>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>>>>>>
>>>>>>>
>>>>>> It could show up as sleeping and still be working.
>>>>>>
>>>>>
>>>>> They might be working, but they are not letting me give them extra work
>>>>> :)
>>>>>
>>>>>
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>> I hope this is not another Fedora 11 related bug.
>>>>>>>
>>>>>>>
>>>>>>>>  Cheeers,
>>>>>>>>
>>>>>>>> Brian
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead
>>>>>>>>>   import sha
>>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>>
>>>>>>>>> I[2]: mec = MultiEngineClient()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>>> ConnectionRefusedError                    Traceback (most recent
>>>>>>>>> call last)
>>>>>>>>>
>>>>>>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>>>>>>
>>>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc
>>>>>>>>> in get_multiengine_client(furl_or_file)
>>>>>>>>>      67     """
>>>>>>>>>      68     client =
>>>>>>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>>>>>>> ---> 69         furl_or_file)
>>>>>>>>>      70     return client.adapt_to_blocking_client()
>>>>>>>>>      71
>>>>>>>>>
>>>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>>>>>>> in blockingCallFromThread(f, *a, **kw)
>>>>>>>>>      70         @raise: any error raised during the callback chain.
>>>>>>>>>      71         """
>>>>>>>>> ---> 72         return
>>>>>>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>>>      73
>>>>>>>>>      74 else:
>>>>>>>>>
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>>>>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>>>     112     result = queue.get()
>>>>>>>>>     113     if isinstance(result, failure.Failure):
>>>>>>>>> --> 114         result.raiseException()
>>>>>>>>>     115     return result
>>>>>>>>>     116
>>>>>>>>>
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>>>>>>> in raiseException(self)
>>>>>>>>>     324         information if available.
>>>>>>>>>     325         """
>>>>>>>>> --> 326         raise self.type, self.value, self.tb
>>>>>>>>>     327
>>>>>>>>>     328
>>>>>>>>>
>>>>>>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>>>>>>> Connection refused.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I had installed Twisted and other parallel computing requirements
>>>>>>>>> following the instructions that were listed on the documentation pages.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> mec.get_ids()
>>>>>>>>>>
>>>>>>>>>> It may all be working...all you see are deprecation warnings
>>>>>>>>>> related to Python 2.6.
>>>>>>>>>>
>>>>>>>>>> One more word.  The parallel computing stuff is not working in
>>>>>>>>>> trunk right now (I am fixing it),
>>>>>>>>>> so please stick with 0.10.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was bitten by the trunk a few times. For over a month I am using
>>>>>>>>> the downgraded (0.10) version :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Brian
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <
>>>>>>>>>> gokhansever@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I want to experiment IPython's parallel computing functionality.
>>>>>>>>>>> This far I couldn't progress much because ipcluster instantiation stalls
>>>>>>>>>>> giving the following messages without dropping me into the main IPython
>>>>>>>>>>> shell.
>>>>>>>>>>>
>>>>>>>>>>> My intention is parallelise a small Python script that calls an
>>>>>>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>>>>>>> and calls the main external script via subprocess module's call function.
>>>>>>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>>>>>>> process.
>>>>>>>>>>>
>>>>>>>>>>> From the IPython parallel computing documentation, it seems like
>>>>>>>>>>> what I want is doable in IPython. However I need some advice whether my
>>>>>>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>>>>>>> warning messages.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead
>>>>>>>>>>>   import sha
>>>>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>>>>>>> pid=11066
>>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>>>>>>> starting...
>>>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11067
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11068
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11069
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11070
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067,
>>>>>>>>>>> 11068, 11069, 11070]
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Here is my system info:
>>>>>>>>>>>
>>>>>>>>>>> ================================================================================
>>>>>>>>>>> Platform     :
>>>>>>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>>>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>>>>>>>> IPython      : 0.10
>>>>>>>>>>> NumPy      : 1.4.0.dev
>>>>>>>>>>>
>>>>>>>>>>> ================================================================================
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Gökhan
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> IPython-user mailing list
>>>>>>>>>>> IPython-user@scipy.org
>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Gökhan
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gökhan
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gökhan
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Gökhan
>>>
>>
>>
>


-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/4805457b/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: proall_parallel.py
Type: application/octet-stream
Size: 1114 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/4805457b/attachment-0001.obj 


More information about the IPython-user mailing list