[IPython-user] Cannot start ipcluster

Brian Granger ellisonbg.net@gmail....
Sun Oct 18 01:19:37 CDT 2009


> This is my first question to you :) It just stops after issuing all those
> Deprecation warnings. I can't drop back into Bash either let alone Python or
> IPython. It doesn't give any error message even. I insert a few enters but
> this is no-avail either.
>


Ahh, I think I understand the confusion.  ipcluster starts a server and
*doesn't return* (it is iterating in a select/poll loop).  You have to start
ipcluster in one terminal window and then open another terminal window and
run ipython.  By using IPython.kernel.client.MultiEngine or TaskClient you
then connect to the server that is running and run you job.

Can you give that a shot?

Brian


>
>
>>
>>
>>>  Doing "ps -aux | grep ip" yields
>>>
>>> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
>>> /usr/bin/python /usr/bin/ipcluster local -n 4
>>> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
>>> /usr/bin/python /usr/bin/ipcontroller
>>> --logfile=/home/gsever/.ipython/log/ipcontroller
>>> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
>>> /usr/bin/python /usr/bin/ipengine
>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
>>> /usr/bin/python /usr/bin/ipengine
>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
>>> /usr/bin/python /usr/bin/ipengine
>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
>>> /usr/bin/python /usr/bin/ipengine
>>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>>
>>> and system monitor shows this tasks as sleeping. See the screenshot :
>>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>>
>>>
>> It could show up as sleeping and still be working.
>>
>
> They might be working, but they are not letting me give them extra work :)
>
>
>>
>> Brian
>>
>>
>>> I hope this is not another Fedora 11 related bug.
>>>
>>>
>>>>  Cheeers,
>>>>
>>>> Brian
>>>>
>>>>
>>>>>
>>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead
>>>>>   import sha
>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated
>>>>>
>>>>> I[2]: mec = MultiEngineClient()
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> ConnectionRefusedError                    Traceback (most recent call
>>>>> last)
>>>>>
>>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>>
>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc in
>>>>> get_multiengine_client(furl_or_file)
>>>>>      67     """
>>>>>      68     client =
>>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>>> ---> 69         furl_or_file)
>>>>>      70     return client.adapt_to_blocking_client()
>>>>>      71
>>>>>
>>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>>> in blockingCallFromThread(f, *a, **kw)
>>>>>      70         @raise: any error raised during the callback chain.
>>>>>      71         """
>>>>> ---> 72         return
>>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>>      73
>>>>>      74 else:
>>>>>
>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>>     112     result = queue.get()
>>>>>     113     if isinstance(result, failure.Failure):
>>>>> --> 114         result.raiseException()
>>>>>     115     return result
>>>>>     116
>>>>>
>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>>> in raiseException(self)
>>>>>     324         information if available.
>>>>>     325         """
>>>>> --> 326         raise self.type, self.value, self.tb
>>>>>     327
>>>>>     328
>>>>>
>>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>>> Connection refused.
>>>>>
>>>>>
>>>>> I had installed Twisted and other parallel computing requirements
>>>>> following the instructions that were listed on the documentation pages.
>>>>>
>>>>>
>>>>>
>>>>>> mec.get_ids()
>>>>>>
>>>>>> It may all be working...all you see are deprecation warnings related
>>>>>> to Python 2.6.
>>>>>>
>>>>>> One more word.  The parallel computing stuff is not working in trunk
>>>>>> right now (I am fixing it),
>>>>>> so please stick with 0.10.
>>>>>>
>>>>>
>>>>> I was bitten by the trunk a few times. For over a month I am using the
>>>>> downgraded (0.10) version :)
>>>>>
>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <gokhansever@gmail.com>wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I want to experiment IPython's parallel computing functionality. This
>>>>>>> far I couldn't progress much because ipcluster instantiation stalls giving
>>>>>>> the following messages without dropping me into the main IPython shell.
>>>>>>>
>>>>>>> My intention is parallelise a small Python script that calls an
>>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>>> and calls the main external script via subprocess module's call function.
>>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>>> process.
>>>>>>>
>>>>>>> From the IPython parallel computing documentation, it seems like what
>>>>>>> I want is doable in IPython. However I need some advice whether my
>>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>>> warning messages.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead
>>>>>>>   import sha
>>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>>> pid=11066
>>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>>> starting...
>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead\n  import sha\n'
>>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>> pid=11067
>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>> pid=11068
>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>> pid=11069
>>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>>> pid=11070
>>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067,
>>>>>>> 11068, 11069, 11070]
>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead\n  import sha\n'
>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead\n  import sha\n'
>>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead\n  import sha\n'
>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>>> instead\n  import sha\n'
>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>>
>>>>>>>
>>>>>>> Here is my system info:
>>>>>>>
>>>>>>> ================================================================================
>>>>>>> Platform     :
>>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>>>> IPython      : 0.10
>>>>>>> NumPy      : 1.4.0.dev
>>>>>>>
>>>>>>> ================================================================================
>>>>>>>
>>>>>>> --
>>>>>>> Gökhan
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> IPython-user mailing list
>>>>>>> IPython-user@scipy.org
>>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gökhan
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Gökhan
>>>
>>
>>
>
>
> --
> Gökhan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091017/bd0f53c9/attachment-0001.html 


More information about the IPython-user mailing list