[IPython-user] Cannot start ipcluster

Brian Granger ellisonbg.net@gmail....
Sun Oct 18 00:45:35 CDT 2009


> I can't drop into ipython in this case :(
>
>
Why not?  Can you drop into python?


> Doing "ps -aux | grep ip" yields
>
> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
> /usr/bin/python /usr/bin/ipcluster local -n 4
> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
> /usr/bin/python /usr/bin/ipcontroller
> --logfile=/home/gsever/.ipython/log/ipcontroller
> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
> /usr/bin/python /usr/bin/ipengine
> --logfile=/home/gsever/.ipython/log/ipengine15468-
> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
> /usr/bin/python /usr/bin/ipengine
> --logfile=/home/gsever/.ipython/log/ipengine15468-
> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
> /usr/bin/python /usr/bin/ipengine
> --logfile=/home/gsever/.ipython/log/ipengine15468-
> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
> /usr/bin/python /usr/bin/ipengine
> --logfile=/home/gsever/.ipython/log/ipengine15468-
>
> and system monitor shows this tasks as sleeping. See the screenshot :
> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>
>
It could show up as sleeping and still be working.

Brian


> I hope this is not another Fedora 11 related bug.
>
>
>> Cheeers,
>>
>> Brian
>>
>>
>>>
>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>> instead
>>>   import sha
>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>> DeprecationWarning: the sets module is deprecated
>>>
>>> I[2]: mec = MultiEngineClient()
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> ConnectionRefusedError                    Traceback (most recent call
>>> last)
>>>
>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>
>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc in
>>> get_multiengine_client(furl_or_file)
>>>      67     """
>>>      68     client =
>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>> ---> 69         furl_or_file)
>>>      70     return client.adapt_to_blocking_client()
>>>      71
>>>
>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>> in blockingCallFromThread(f, *a, **kw)
>>>      70         @raise: any error raised during the callback chain.
>>>      71         """
>>> ---> 72         return
>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>      73
>>>      74 else:
>>>
>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>     112     result = queue.get()
>>>     113     if isinstance(result, failure.Failure):
>>> --> 114         result.raiseException()
>>>     115     return result
>>>     116
>>>
>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>> in raiseException(self)
>>>     324         information if available.
>>>     325         """
>>> --> 326         raise self.type, self.value, self.tb
>>>     327
>>>     328
>>>
>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>> Connection refused.
>>>
>>>
>>> I had installed Twisted and other parallel computing requirements
>>> following the instructions that were listed on the documentation pages.
>>>
>>>
>>>
>>>> mec.get_ids()
>>>>
>>>> It may all be working...all you see are deprecation warnings related to
>>>> Python 2.6.
>>>>
>>>> One more word.  The parallel computing stuff is not working in trunk
>>>> right now (I am fixing it),
>>>> so please stick with 0.10.
>>>>
>>>
>>> I was bitten by the trunk a few times. For over a month I am using the
>>> downgraded (0.10) version :)
>>>
>>>
>>>>
>>>> Cheers,
>>>>
>>>> Brian
>>>>
>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <gokhansever@gmail.com>wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I want to experiment IPython's parallel computing functionality. This
>>>>> far I couldn't progress much because ipcluster instantiation stalls giving
>>>>> the following messages without dropping me into the main IPython shell.
>>>>>
>>>>> My intention is parallelise a small Python script that calls an
>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>> and each folder is independent from each other. My script visits each folder
>>>>> and calls the main external script via subprocess module's call function.
>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>> really want is to put externally called scripts into separate threads, so
>>>>> that I don't need to wait the previous job to be done during the processing
>>>>> process.
>>>>>
>>>>> From the IPython parallel computing documentation, it seems like what I
>>>>> want is doable in IPython. However I need some advice whether my
>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>> warning messages.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead
>>>>>   import sha
>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated
>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>> pid=11066
>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>> starting...
>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead\n  import sha\n'
>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>> pid=11067
>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>> pid=11068
>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>> pid=11069
>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>> pid=11070
>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067, 11068,
>>>>> 11069, 11070]
>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead\n  import sha\n'
>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead\n  import sha\n'
>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead\n  import sha\n'
>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>> instead\n  import sha\n'
>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>
>>>>>
>>>>> Here is my system info:
>>>>>
>>>>> ================================================================================
>>>>> Platform     :
>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>> IPython      : 0.10
>>>>> NumPy      : 1.4.0.dev
>>>>>
>>>>> ================================================================================
>>>>>
>>>>> --
>>>>> Gökhan
>>>>>
>>>>> _______________________________________________
>>>>> IPython-user mailing list
>>>>> IPython-user@scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Gökhan
>>>
>>
>>
>
>
> --
> Gökhan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091017/41869774/attachment-0001.html 


More information about the IPython-user mailing list