[IPython-user] Cannot start ipcluster
Gökhan Sever
gokhansever@gmail....
Sun Oct 18 14:08:01 CDT 2009
On Sun, Oct 18, 2009 at 12:27 PM, Brian Granger <ellisonbg.net@gmail.com>wrote:
> Also, once you fix the import/name error, can you send me your script.
> That way I can see if there are any performance issues we can address.
>
> Cheers,
>
> Brian
This is a slightly modified version of the original script that I used for
timings. I just defined a generic function to find the sea files and extract
paths so that I can pass into the main processing function to be mapped.
Also attaching to file for easy access. Will test with MultiEngineClient
approach as well.
#!/usr/bin/env python
"""
Execute postprocessing_saudi script in parallel using IPython's parallel
computing features. Make sure "ipcluster local n -2" has run in a separate
shell window before running this script
"""
# Gokhan Sever <gokhansever@gmail.com>
# University of North Dakota - Dept. of Atmospheric Sciences
# Written : October 18, 2009 with help from Brain Granger
from IPython.kernel.client import TaskClient
from subprocess import call
import os
def find_sea_files():
file_list, path_list = [], []
init = os.getcwd()
for root, dirs, files in os.walk('.'):
dirs.sort()
for file in files:
if file.endswith('.sea'):
file_list.append(file)
os.chdir(root)
path_list.append(os.getcwd())
os.chdir(init)
return file_list, path_list
def process_all(path, file):
import os
from subprocess import call
os.chdir(path)
call(['postprocessing_saudi', file])
if __name__ == '__main__':
tc = TaskClient()
files, paths = find_sea_files()
tc.map(process_all, paths, files)
>
>
> On Sun, Oct 18, 2009 at 10:26 AM, Brian Granger <ellisonbg.net@gmail.com>wrote:
>
>>
>> from IPython.kernel.client import TaskClient
>>> from subprocess import call
>>>
>>> def process(file):
>>> call(['postprocessing_saudi', file])
>>>
>>>
>> This function (process) will be pickled and sent to the engines. But, the
>> engines don't have call imported yet.
>> Two ways of accomplishing this:
>>
>> def process(file):
>> from subprocess import call
>> call(['postprocessing_saudi', file])
>>
>> But, this has a small amount of overhead by importing call each time.
>>
>> To import it once, do:
>>
>> from IPython.kernel.client import MultiEngineClient
>> mec = MultiEngineClient()
>> mec.execute('from subprocessing import call')
>>
>> before you submit the tasks. The MultiEngineClient interface doesn't do
>> load balancing, just does things on
>> the engines you specify (the default is all).
>>
>> Give that a shot...
>>
>> Brian
>>
>>
>>> files =
>>> ['/home/gsever/Desktop/test/20090317_131342/PostProcessing/09_03_17_13_13_42.sea',
>>>
>>> '/home/gsever/Desktop/test/20090318_075533/PostProcessing/09_03_18_07_55_33.sea',
>>>
>>> '/home/gsever/Desktop/test/20090319_110816/PostProcessing/09_03_19_11_08_16.sea',
>>>
>>> '/home/gsever/Desktop/test/20090320_064651/PostProcessing/09_03_20_06_46_51.sea']
>>>
>>> if __name__ == '__main__':
>>> tc = TaskClient()
>>> tc.map(process, files)
>>>
>>>
>>> Should I return something from the process function and why it is
>>> complaining about "NameError: global name 'call' is not defined" I am
>>> already importing the call on top.
>>>
>>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>> Doing "ps -aux | grep ip" yields
>>>>>>>
>>>>>>> gsever 15467 0.7 0.3 20700 14264 pts/2 S+ 00:02 0:00
>>>>>>> /usr/bin/python /usr/bin/ipcluster local -n 4
>>>>>>> gsever 15468 1.1 0.5 40424 23544 pts/2 S+ 00:02 0:00
>>>>>>> /usr/bin/python /usr/bin/ipcontroller
>>>>>>> --logfile=/home/gsever/.ipython/log/ipcontroller
>>>>>>> gsever 15469 1.1 0.4 36980 20332 pts/2 S+ 00:02 0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>> gsever 15470 1.1 0.5 37396 20668 pts/2 S+ 00:02 0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>> gsever 15471 0.9 0.4 37064 20392 pts/2 S+ 00:02 0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>> gsever 15472 1.2 0.4 36976 20328 pts/2 S+ 00:02 0:00
>>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>>
>>>>>>> and system monitor shows this tasks as sleeping. See the screenshot :
>>>>>>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>>>>>>
>>>>>>>
>>>>>> It could show up as sleeping and still be working.
>>>>>>
>>>>>
>>>>> They might be working, but they are not letting me give them extra work
>>>>> :)
>>>>>
>>>>>
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>> I hope this is not another Fedora 11 related bug.
>>>>>>>
>>>>>>>
>>>>>>>> Cheeers,
>>>>>>>>
>>>>>>>> Brian
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead
>>>>>>>>> import sha
>>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>>
>>>>>>>>> I[2]: mec = MultiEngineClient()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>>> ConnectionRefusedError Traceback (most recent
>>>>>>>>> call last)
>>>>>>>>>
>>>>>>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>>>>>>
>>>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc
>>>>>>>>> in get_multiengine_client(furl_or_file)
>>>>>>>>> 67 """
>>>>>>>>> 68 client =
>>>>>>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>>>>>>> ---> 69 furl_or_file)
>>>>>>>>> 70 return client.adapt_to_blocking_client()
>>>>>>>>> 71
>>>>>>>>>
>>>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>>>>>>> in blockingCallFromThread(f, *a, **kw)
>>>>>>>>> 70 @raise: any error raised during the callback chain.
>>>>>>>>> 71 """
>>>>>>>>> ---> 72 return
>>>>>>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>>> 73
>>>>>>>>> 74 else:
>>>>>>>>>
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>>>>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>>> 112 result = queue.get()
>>>>>>>>> 113 if isinstance(result, failure.Failure):
>>>>>>>>> --> 114 result.raiseException()
>>>>>>>>> 115 return result
>>>>>>>>> 116
>>>>>>>>>
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>>>>>>> in raiseException(self)
>>>>>>>>> 324 information if available.
>>>>>>>>> 325 """
>>>>>>>>> --> 326 raise self.type, self.value, self.tb
>>>>>>>>> 327
>>>>>>>>> 328
>>>>>>>>>
>>>>>>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>>>>>>> Connection refused.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I had installed Twisted and other parallel computing requirements
>>>>>>>>> following the instructions that were listed on the documentation pages.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> mec.get_ids()
>>>>>>>>>>
>>>>>>>>>> It may all be working...all you see are deprecation warnings
>>>>>>>>>> related to Python 2.6.
>>>>>>>>>>
>>>>>>>>>> One more word. The parallel computing stuff is not working in
>>>>>>>>>> trunk right now (I am fixing it),
>>>>>>>>>> so please stick with 0.10.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was bitten by the trunk a few times. For over a month I am using
>>>>>>>>> the downgraded (0.10) version :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Brian
>>>>>>>>>>
>>>>>>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <
>>>>>>>>>> gokhansever@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I want to experiment IPython's parallel computing functionality.
>>>>>>>>>>> This far I couldn't progress much because ipcluster instantiation stalls
>>>>>>>>>>> giving the following messages without dropping me into the main IPython
>>>>>>>>>>> shell.
>>>>>>>>>>>
>>>>>>>>>>> My intention is parallelise a small Python script that calls an
>>>>>>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>>>>>>> and calls the main external script via subprocess module's call function.
>>>>>>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>>>>>>> process.
>>>>>>>>>>>
>>>>>>>>>>> From the IPython parallel computing documentation, it seems like
>>>>>>>>>>> what I want is doable in IPython. However I need some advice whether my
>>>>>>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>>>>>>> warning messages.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [gsever@ccn Desktop]$ ipcluster local -n 4
>>>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead
>>>>>>>>>>> import sha
>>>>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>>>>>>> pid=11066
>>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>>>>>>> starting...
>>>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11067
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11068
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11069
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>>> pid=11070
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067,
>>>>>>>>>>> 11068, 11069, 11070]
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>>> instead\n import sha\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Here is my system info:
>>>>>>>>>>>
>>>>>>>>>>> ================================================================================
>>>>>>>>>>> Platform :
>>>>>>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>>>>>>> Python : ('CPython', 'tags/r26', '66714')
>>>>>>>>>>> IPython : 0.10
>>>>>>>>>>> NumPy : 1.4.0.dev
>>>>>>>>>>>
>>>>>>>>>>> ================================================================================
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Gökhan
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> IPython-user mailing list
>>>>>>>>>>> IPython-user@scipy.org
>>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Gökhan
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gökhan
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gökhan
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Gökhan
>>>
>>
>>
>
--
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/4805457b/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: proall_parallel.py
Type: application/octet-stream
Size: 1114 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/4805457b/attachment-0001.obj
More information about the IPython-user
mailing list