[IPython-user] Cannot start ipcluster

Gökhan Sever gokhansever@gmail....
Sun Oct 18 02:15:44 CDT 2009


On Sun, Oct 18, 2009 at 1:19 AM, Brian Granger <ellisonbg.net@gmail.com>wrote:

>
> This is my first question to you :) It just stops after issuing all those
>> Deprecation warnings. I can't drop back into Bash either let alone Python or
>> IPython. It doesn't give any error message even. I insert a few enters but
>> this is no-avail either.
>>
>
>
> Ahh, I think I understand the confusion.  ipcluster starts a server and
> *doesn't return* (it is iterating in a select/poll loop).  You have to start
> ipcluster in one terminal window and then open another terminal window and
> run ipython.  By using IPython.kernel.client.MultiEngine or TaskClient you
> then connect to the server that is running and run you job.
>
> Can you give that a shot?
>
> Brian
>
>

Still no success :(

Here is what I do:

from IPython.kernel.client import TaskClient
from subprocess import call

def process(file):
    call(['postprocessing_saudi', file])

files =
['/home/gsever/Desktop/test/20090317_131342/PostProcessing/09_03_17_13_13_42.sea',
 '/home/gsever/Desktop/test/20090318_075533/PostProcessing/09_03_18_07_55_33.sea',
 '/home/gsever/Desktop/test/20090319_110816/PostProcessing/09_03_19_11_08_16.sea',
 '/home/gsever/Desktop/test/20090320_064651/PostProcessing/09_03_20_06_46_51.sea']

if __name__ == '__main__':
    tc = TaskClient()
    tc.map(process, files)


Should I return something from the process function and why it is
complaining about "NameError: global name 'call' is not defined" I am
already importing the call on top.



>
>>
>>>
>>>
>>>>  Doing "ps -aux | grep ip" yields
>>>>
>>>> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
>>>> /usr/bin/python /usr/bin/ipcluster local -n 4
>>>> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
>>>> /usr/bin/python /usr/bin/ipcontroller
>>>> --logfile=/home/gsever/.ipython/log/ipcontroller
>>>> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
>>>> /usr/bin/python /usr/bin/ipengine
>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
>>>> /usr/bin/python /usr/bin/ipengine
>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
>>>> /usr/bin/python /usr/bin/ipengine
>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
>>>> /usr/bin/python /usr/bin/ipengine
>>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>>
>>>> and system monitor shows this tasks as sleeping. See the screenshot :
>>>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>>>
>>>>
>>> It could show up as sleeping and still be working.
>>>
>>
>> They might be working, but they are not letting me give them extra work :)
>>
>>
>>>
>>> Brian
>>>
>>>
>>>> I hope this is not another Fedora 11 related bug.
>>>>
>>>>
>>>>>  Cheeers,
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>>>
>>>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead
>>>>>>   import sha
>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>
>>>>>> I[2]: mec = MultiEngineClient()
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>> ConnectionRefusedError                    Traceback (most recent call
>>>>>> last)
>>>>>>
>>>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>>>
>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc in
>>>>>> get_multiengine_client(furl_or_file)
>>>>>>      67     """
>>>>>>      68     client =
>>>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>>>> ---> 69         furl_or_file)
>>>>>>      70     return client.adapt_to_blocking_client()
>>>>>>      71
>>>>>>
>>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>>>> in blockingCallFromThread(f, *a, **kw)
>>>>>>      70         @raise: any error raised during the callback chain.
>>>>>>      71         """
>>>>>> ---> 72         return
>>>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>      73
>>>>>>      74 else:
>>>>>>
>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>>>     112     result = queue.get()
>>>>>>     113     if isinstance(result, failure.Failure):
>>>>>> --> 114         result.raiseException()
>>>>>>     115     return result
>>>>>>     116
>>>>>>
>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>>>> in raiseException(self)
>>>>>>     324         information if available.
>>>>>>     325         """
>>>>>> --> 326         raise self.type, self.value, self.tb
>>>>>>     327
>>>>>>     328
>>>>>>
>>>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>>>> Connection refused.
>>>>>>
>>>>>>
>>>>>> I had installed Twisted and other parallel computing requirements
>>>>>> following the instructions that were listed on the documentation pages.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> mec.get_ids()
>>>>>>>
>>>>>>> It may all be working...all you see are deprecation warnings related
>>>>>>> to Python 2.6.
>>>>>>>
>>>>>>> One more word.  The parallel computing stuff is not working in trunk
>>>>>>> right now (I am fixing it),
>>>>>>> so please stick with 0.10.
>>>>>>>
>>>>>>
>>>>>> I was bitten by the trunk a few times. For over a month I am using the
>>>>>> downgraded (0.10) version :)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <gokhansever@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I want to experiment IPython's parallel computing functionality.
>>>>>>>> This far I couldn't progress much because ipcluster instantiation stalls
>>>>>>>> giving the following messages without dropping me into the main IPython
>>>>>>>> shell.
>>>>>>>>
>>>>>>>> My intention is parallelise a small Python script that calls an
>>>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>>>> and calls the main external script via subprocess module's call function.
>>>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>>>> process.
>>>>>>>>
>>>>>>>> From the IPython parallel computing documentation, it seems like
>>>>>>>> what I want is doable in IPython. However I need some advice whether my
>>>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>>>> warning messages.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead
>>>>>>>>   import sha
>>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>>>> pid=11066
>>>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>>>> starting...
>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead\n  import sha\n'
>>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>> pid=11067
>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>> pid=11068
>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>> pid=11069
>>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>>> pid=11070
>>>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067,
>>>>>>>> 11068, 11069, 11070]
>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead\n  import sha\n'
>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead\n  import sha\n'
>>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead\n  import sha\n'
>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>>> instead\n  import sha\n'
>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is my system info:
>>>>>>>>
>>>>>>>> ================================================================================
>>>>>>>> Platform     :
>>>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>>>>> IPython      : 0.10
>>>>>>>> NumPy      : 1.4.0.dev
>>>>>>>>
>>>>>>>> ================================================================================
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gökhan
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> IPython-user mailing list
>>>>>>>> IPython-user@scipy.org
>>>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gökhan
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Gökhan
>>>>
>>>
>>>
>>
>>
>> --
>> Gökhan
>>
>
>


-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/e8d4cce8/attachment-0001.html 


More information about the IPython-user mailing list