[IPython-user] Cannot start ipcluster

Brian Granger ellisonbg.net@gmail....
Sun Oct 18 12:27:18 CDT 2009


Also, once you fix the import/name error, can you send me your script.  That
way I can see if there are any performance issues we can address.

Cheers,

Brian

On Sun, Oct 18, 2009 at 10:26 AM, Brian Granger <ellisonbg.net@gmail.com>wrote:

>
> from IPython.kernel.client import TaskClient
>> from subprocess import call
>>
>> def process(file):
>>     call(['postprocessing_saudi', file])
>>
>>
> This function (process) will be pickled and sent to the engines.  But, the
> engines don't have call imported yet.
> Two ways of accomplishing this:
>
> def process(file):
>     from subprocess import call
>     call(['postprocessing_saudi', file])
>
> But, this has a small amount of overhead by importing call each time.
>
> To import it once, do:
>
> from IPython.kernel.client import MultiEngineClient
> mec = MultiEngineClient()
> mec.execute('from subprocessing import call')
>
> before you submit the tasks.  The MultiEngineClient interface doesn't do
> load balancing, just does things on
> the engines you specify (the default is all).
>
> Give that a shot...
>
> Brian
>
>
>> files =
>> ['/home/gsever/Desktop/test/20090317_131342/PostProcessing/09_03_17_13_13_42.sea',
>>
>>  '/home/gsever/Desktop/test/20090318_075533/PostProcessing/09_03_18_07_55_33.sea',
>>
>>  '/home/gsever/Desktop/test/20090319_110816/PostProcessing/09_03_19_11_08_16.sea',
>>
>>  '/home/gsever/Desktop/test/20090320_064651/PostProcessing/09_03_20_06_46_51.sea']
>>
>> if __name__ == '__main__':
>>     tc = TaskClient()
>>     tc.map(process, files)
>>
>>
>> Should I return something from the process function and why it is
>> complaining about "NameError: global name 'call' is not defined" I am
>> already importing the call on top.
>>
>>
>>
>>>
>>>>
>>>>>
>>>>>
>>>>>>  Doing "ps -aux | grep ip" yields
>>>>>>
>>>>>> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
>>>>>> /usr/bin/python /usr/bin/ipcluster local -n 4
>>>>>> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
>>>>>> /usr/bin/python /usr/bin/ipcontroller
>>>>>> --logfile=/home/gsever/.ipython/log/ipcontroller
>>>>>> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
>>>>>> /usr/bin/python /usr/bin/ipengine
>>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>>
>>>>>> and system monitor shows this tasks as sleeping. See the screenshot :
>>>>>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>>>>>
>>>>>>
>>>>> It could show up as sleeping and still be working.
>>>>>
>>>>
>>>> They might be working, but they are not letting me give them extra work
>>>> :)
>>>>
>>>>
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>> I hope this is not another Fedora 11 related bug.
>>>>>>
>>>>>>
>>>>>>>  Cheeers,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead
>>>>>>>>   import sha
>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>
>>>>>>>> I[2]: mec = MultiEngineClient()
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>> ConnectionRefusedError                    Traceback (most recent
>>>>>>>> call last)
>>>>>>>>
>>>>>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>>>>>
>>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc
>>>>>>>> in get_multiengine_client(furl_or_file)
>>>>>>>>      67     """
>>>>>>>>      68     client =
>>>>>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>>>>>> ---> 69         furl_or_file)
>>>>>>>>      70     return client.adapt_to_blocking_client()
>>>>>>>>      71
>>>>>>>>
>>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>>>>>> in blockingCallFromThread(f, *a, **kw)
>>>>>>>>      70         @raise: any error raised during the callback chain.
>>>>>>>>      71         """
>>>>>>>> ---> 72         return
>>>>>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>>      73
>>>>>>>>      74 else:
>>>>>>>>
>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>>>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>>     112     result = queue.get()
>>>>>>>>     113     if isinstance(result, failure.Failure):
>>>>>>>> --> 114         result.raiseException()
>>>>>>>>     115     return result
>>>>>>>>     116
>>>>>>>>
>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>>>>>> in raiseException(self)
>>>>>>>>     324         information if available.
>>>>>>>>     325         """
>>>>>>>> --> 326         raise self.type, self.value, self.tb
>>>>>>>>     327
>>>>>>>>     328
>>>>>>>>
>>>>>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>>>>>> Connection refused.
>>>>>>>>
>>>>>>>>
>>>>>>>> I had installed Twisted and other parallel computing requirements
>>>>>>>> following the instructions that were listed on the documentation pages.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> mec.get_ids()
>>>>>>>>>
>>>>>>>>> It may all be working...all you see are deprecation warnings
>>>>>>>>> related to Python 2.6.
>>>>>>>>>
>>>>>>>>> One more word.  The parallel computing stuff is not working in
>>>>>>>>> trunk right now (I am fixing it),
>>>>>>>>> so please stick with 0.10.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I was bitten by the trunk a few times. For over a month I am using
>>>>>>>> the downgraded (0.10) version :)
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Brian
>>>>>>>>>
>>>>>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <
>>>>>>>>> gokhansever@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I want to experiment IPython's parallel computing functionality.
>>>>>>>>>> This far I couldn't progress much because ipcluster instantiation stalls
>>>>>>>>>> giving the following messages without dropping me into the main IPython
>>>>>>>>>> shell.
>>>>>>>>>>
>>>>>>>>>> My intention is parallelise a small Python script that calls an
>>>>>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>>>>>> and calls the main external script via subprocess module's call function.
>>>>>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>>>>>> process.
>>>>>>>>>>
>>>>>>>>>> From the IPython parallel computing documentation, it seems like
>>>>>>>>>> what I want is doable in IPython. However I need some advice whether my
>>>>>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>>>>>> warning messages.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>> instead
>>>>>>>>>>   import sha
>>>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>>>>>> pid=11066
>>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>>>>>> starting...
>>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>> pid=11067
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>> pid=11068
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>> pid=11069
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>>> pid=11070
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067,
>>>>>>>>>> 11068, 11069, 11070]
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>>> instead\n  import sha\n'
>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Here is my system info:
>>>>>>>>>>
>>>>>>>>>> ================================================================================
>>>>>>>>>> Platform     :
>>>>>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>>>>>>> IPython      : 0.10
>>>>>>>>>> NumPy      : 1.4.0.dev
>>>>>>>>>>
>>>>>>>>>> ================================================================================
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Gökhan
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> IPython-user mailing list
>>>>>>>>>> IPython-user@scipy.org
>>>>>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gökhan
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gökhan
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Gökhan
>>>>
>>>
>>>
>>
>>
>> --
>> Gökhan
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/c2e40ae9/attachment-0001.html 


More information about the IPython-user mailing list