[IPython-user] Cannot start ipcluster

Brian Granger ellisonbg.net@gmail....
Sun Oct 18 12:26:17 CDT 2009


> from IPython.kernel.client import TaskClient
> from subprocess import call
>
> def process(file):
>     call(['postprocessing_saudi', file])
>
>
This function (process) will be pickled and sent to the engines.  But, the
engines don't have call imported yet.
Two ways of accomplishing this:

def process(file):
    from subprocess import call
    call(['postprocessing_saudi', file])

But, this has a small amount of overhead by importing call each time.

To import it once, do:

from IPython.kernel.client import MultiEngineClient
mec = MultiEngineClient()
mec.execute('from subprocessing import call')

before you submit the tasks.  The MultiEngineClient interface doesn't do
load balancing, just does things on
the engines you specify (the default is all).

Give that a shot...

Brian


> files =
> ['/home/gsever/Desktop/test/20090317_131342/PostProcessing/09_03_17_13_13_42.sea',
>
>  '/home/gsever/Desktop/test/20090318_075533/PostProcessing/09_03_18_07_55_33.sea',
>
>  '/home/gsever/Desktop/test/20090319_110816/PostProcessing/09_03_19_11_08_16.sea',
>
>  '/home/gsever/Desktop/test/20090320_064651/PostProcessing/09_03_20_06_46_51.sea']
>
> if __name__ == '__main__':
>     tc = TaskClient()
>     tc.map(process, files)
>
>
> Should I return something from the process function and why it is
> complaining about "NameError: global name 'call' is not defined" I am
> already importing the call on top.
>
>
>
>>
>>>
>>>>
>>>>
>>>>>  Doing "ps -aux | grep ip" yields
>>>>>
>>>>> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
>>>>> /usr/bin/python /usr/bin/ipcluster local -n 4
>>>>> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
>>>>> /usr/bin/python /usr/bin/ipcontroller
>>>>> --logfile=/home/gsever/.ipython/log/ipcontroller
>>>>> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
>>>>> /usr/bin/python /usr/bin/ipengine
>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
>>>>> /usr/bin/python /usr/bin/ipengine
>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
>>>>> /usr/bin/python /usr/bin/ipengine
>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
>>>>> /usr/bin/python /usr/bin/ipengine
>>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>>
>>>>> and system monitor shows this tasks as sleeping. See the screenshot :
>>>>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>>>>
>>>>>
>>>> It could show up as sleeping and still be working.
>>>>
>>>
>>> They might be working, but they are not letting me give them extra work
>>> :)
>>>
>>>
>>>>
>>>> Brian
>>>>
>>>>
>>>>> I hope this is not another Fedora 11 related bug.
>>>>>
>>>>>
>>>>>>  Cheeers,
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead
>>>>>>>   import sha
>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>
>>>>>>> I[2]: mec = MultiEngineClient()
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> ConnectionRefusedError                    Traceback (most recent call
>>>>>>> last)
>>>>>>>
>>>>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>>>>
>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc in
>>>>>>> get_multiengine_client(furl_or_file)
>>>>>>>      67     """
>>>>>>>      68     client =
>>>>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>>>>> ---> 69         furl_or_file)
>>>>>>>      70     return client.adapt_to_blocking_client()
>>>>>>>      71
>>>>>>>
>>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>>>>> in blockingCallFromThread(f, *a, **kw)
>>>>>>>      70         @raise: any error raised during the callback chain.
>>>>>>>      71         """
>>>>>>> ---> 72         return
>>>>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>      73
>>>>>>>      74 else:
>>>>>>>
>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>>     112     result = queue.get()
>>>>>>>     113     if isinstance(result, failure.Failure):
>>>>>>> --> 114         result.raiseException()
>>>>>>>     115     return result
>>>>>>>     116
>>>>>>>
>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>>>>> in raiseException(self)
>>>>>>>     324         information if available.
>>>>>>>     325         """
>>>>>>> --> 326         raise self.type, self.value, self.tb
>>>>>>>     327
>>>>>>>     328
>>>>>>>
>>>>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>>>>> Connection refused.
>>>>>>>
>>>>>>>
>>>>>>> I had installed Twisted and other parallel computing requirements
>>>>>>> following the instructions that were listed on the documentation pages.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> mec.get_ids()
>>>>>>>>
>>>>>>>> It may all be working...all you see are deprecation warnings related
>>>>>>>> to Python 2.6.
>>>>>>>>
>>>>>>>> One more word.  The parallel computing stuff is not working in trunk
>>>>>>>> right now (I am fixing it),
>>>>>>>> so please stick with 0.10.
>>>>>>>>
>>>>>>>
>>>>>>> I was bitten by the trunk a few times. For over a month I am using
>>>>>>> the downgraded (0.10) version :)
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Brian
>>>>>>>>
>>>>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <
>>>>>>>> gokhansever@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I want to experiment IPython's parallel computing functionality.
>>>>>>>>> This far I couldn't progress much because ipcluster instantiation stalls
>>>>>>>>> giving the following messages without dropping me into the main IPython
>>>>>>>>> shell.
>>>>>>>>>
>>>>>>>>> My intention is parallelise a small Python script that calls an
>>>>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>>>>> and calls the main external script via subprocess module's call function.
>>>>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>>>>> process.
>>>>>>>>>
>>>>>>>>> From the IPython parallel computing documentation, it seems like
>>>>>>>>> what I want is doable in IPython. However I need some advice whether my
>>>>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>>>>> warning messages.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead
>>>>>>>>>   import sha
>>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>>>>> pid=11066
>>>>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>>>>> starting...
>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead\n  import sha\n'
>>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>> pid=11067
>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>> pid=11068
>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>> pid=11069
>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>>> pid=11070
>>>>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067,
>>>>>>>>> 11068, 11069, 11070]
>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead\n  import sha\n'
>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead\n  import sha\n'
>>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead\n  import sha\n'
>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>>> instead\n  import sha\n'
>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here is my system info:
>>>>>>>>>
>>>>>>>>> ================================================================================
>>>>>>>>> Platform     :
>>>>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>>>>>> IPython      : 0.10
>>>>>>>>> NumPy      : 1.4.0.dev
>>>>>>>>>
>>>>>>>>> ================================================================================
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Gökhan
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> IPython-user mailing list
>>>>>>>>> IPython-user@scipy.org
>>>>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gökhan
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gökhan
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Gökhan
>>>
>>
>>
>
>
> --
> Gökhan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/5fa42e9c/attachment-0001.html 


More information about the IPython-user mailing list