[IPython-user] Cannot start ipcluster

Gökhan Sever gokhansever@gmail....
Sun Oct 18 01:07:57 CDT 2009


On Sun, Oct 18, 2009 at 12:45 AM, Brian Granger <ellisonbg.net@gmail.com>wrote:

>
> I can't drop into ipython in this case :(
>>
>>
> Why not?  Can you drop into python?
>

This is my first question to you :) It just stops after issuing all those
Deprecation warnings. I can't drop back into Bash either let alone Python or
IPython. It doesn't give any error message even. I insert a few enters but
this is no-avail either.


>
>
>> Doing "ps -aux | grep ip" yields
>>
>> gsever   15467  0.7  0.3  20700 14264 pts/2    S+   00:02   0:00
>> /usr/bin/python /usr/bin/ipcluster local -n 4
>> gsever   15468  1.1  0.5  40424 23544 pts/2    S+   00:02   0:00
>> /usr/bin/python /usr/bin/ipcontroller
>> --logfile=/home/gsever/.ipython/log/ipcontroller
>> gsever   15469  1.1  0.4  36980 20332 pts/2    S+   00:02   0:00
>> /usr/bin/python /usr/bin/ipengine
>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>> gsever   15470  1.1  0.5  37396 20668 pts/2    S+   00:02   0:00
>> /usr/bin/python /usr/bin/ipengine
>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>> gsever   15471  0.9  0.4  37064 20392 pts/2    S+   00:02   0:00
>> /usr/bin/python /usr/bin/ipengine
>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>> gsever   15472  1.2  0.4  36976 20328 pts/2    S+   00:02   0:00
>> /usr/bin/python /usr/bin/ipengine
>> --logfile=/home/gsever/.ipython/log/ipengine15468-
>>
>> and system monitor shows this tasks as sleeping. See the screenshot :
>> http://img148.imageshack.us/img148/6240/screenshotsystemmonitor.png
>>
>>
> It could show up as sleeping and still be working.
>

They might be working, but they are not letting me give them extra work :)


>
> Brian
>
>
>> I hope this is not another Fedora 11 related bug.
>>
>>
>>>  Cheeers,
>>>
>>> Brian
>>>
>>>
>>>>
>>>> I[1]: from IPython.kernel.client import MultiEngineClient
>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>> instead
>>>>   import sha
>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>> DeprecationWarning: the sets module is deprecated
>>>>
>>>> I[2]: mec = MultiEngineClient()
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> ConnectionRefusedError                    Traceback (most recent call
>>>> last)
>>>>
>>>> /home/gsever/Desktop/<ipython console> in <module>()
>>>>
>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc in
>>>> get_multiengine_client(furl_or_file)
>>>>      67     """
>>>>      68     client =
>>>> blockingCallFromThread(_client_tub.get_multiengine_client,
>>>> ---> 69         furl_or_file)
>>>>      70     return client.adapt_to_blocking_client()
>>>>      71
>>>>
>>>> /home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc
>>>> in blockingCallFromThread(f, *a, **kw)
>>>>      70         @raise: any error raised during the callback chain.
>>>>      71         """
>>>> ---> 72         return
>>>> twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
>>>>      73
>>>>      74 else:
>>>>
>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
>>>> in blockingCallFromThread(reactor, f, *a, **kw)
>>>>     112     result = queue.get()
>>>>     113     if isinstance(result, failure.Failure):
>>>> --> 114         result.raiseException()
>>>>     115     return result
>>>>     116
>>>>
>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
>>>> in raiseException(self)
>>>>     324         information if available.
>>>>     325         """
>>>> --> 326         raise self.type, self.value, self.tb
>>>>     327
>>>>     328
>>>>
>>>> ConnectionRefusedError: Connection was refused by other side: 111:
>>>> Connection refused.
>>>>
>>>>
>>>> I had installed Twisted and other parallel computing requirements
>>>> following the instructions that were listed on the documentation pages.
>>>>
>>>>
>>>>
>>>>> mec.get_ids()
>>>>>
>>>>> It may all be working...all you see are deprecation warnings related to
>>>>> Python 2.6.
>>>>>
>>>>> One more word.  The parallel computing stuff is not working in trunk
>>>>> right now (I am fixing it),
>>>>> so please stick with 0.10.
>>>>>
>>>>
>>>> I was bitten by the trunk a few times. For over a month I am using the
>>>> downgraded (0.10) version :)
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Brian
>>>>>
>>>>> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <gokhansever@gmail.com>wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I want to experiment IPython's parallel computing functionality. This
>>>>>> far I couldn't progress much because ipcluster instantiation stalls giving
>>>>>> the following messages without dropping me into the main IPython shell.
>>>>>>
>>>>>> My intention is parallelise a small Python script that calls an
>>>>>> external set of scripts that process the dataset I have in-hand. It is not a
>>>>>> huge computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it
>>>>>> takes about 1.5 hours to process the whole dataset. Looking at the system
>>>>>> monitor I see that the workload is not equally distributed in between CPUs
>>>>>> (one of them usually much lazier than the other.) I am sure parallezing the
>>>>>> code run would boost the processing speed. In my dataset I have 17 folders
>>>>>> and each folder is independent from each other. My script visits each folder
>>>>>> and calls the main external script via subprocess module's call function.
>>>>>> Processing starts with the first folder, and doesn't work on the next folder
>>>>>> unless the processing finishes with the previous folder. Basically, what I
>>>>>> really want is to put externally called scripts into separate threads, so
>>>>>> that I don't need to wait the previous job to be done during the processing
>>>>>> process.
>>>>>>
>>>>>> From the IPython parallel computing documentation, it seems like what
>>>>>> I want is doable in IPython. However I need some advice whether my
>>>>>> understanding is correct in this aspect. Also for the solution of the below
>>>>>> warning messages.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> [gsever@ccn Desktop]$  ipcluster local -n 4
>>>>>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead
>>>>>>   import sha
>>>>>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated
>>>>>> 2009-10-17 18:59:37-0500 [-] Log opened.
>>>>>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>>>>>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>>>>>> pid=11066
>>>>>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish
>>>>>> starting...
>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead\n  import sha\n'
>>>>>> 2009-10-17 18:59:38-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>> 2009-10-17 18:59:39-0500 [-] Controller started
>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>> pid=11067
>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>> pid=11068
>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>> pid=11069
>>>>>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>>>>>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>>>>>> pid=11070
>>>>>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067, 11068,
>>>>>> 11069, 11070]
>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead\n  import sha\n'
>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead\n  import sha\n'
>>>>>> 2009-10-17 18:59:39-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead\n  import sha\n'
>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>>>>>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>>>>>> instead\n  import sha\n'
>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>> 2009-10-17 18:59:40-0500 [-]
>>>>>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>>>>>> DeprecationWarning: the sets module is deprecated\n'
>>>>>>
>>>>>>
>>>>>> Here is my system info:
>>>>>>
>>>>>> ================================================================================
>>>>>> Platform     :
>>>>>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>>>>>> Python       : ('CPython', 'tags/r26', '66714')
>>>>>> IPython      : 0.10
>>>>>> NumPy      : 1.4.0.dev
>>>>>>
>>>>>> ================================================================================
>>>>>>
>>>>>> --
>>>>>> Gökhan
>>>>>>
>>>>>> _______________________________________________
>>>>>> IPython-user mailing list
>>>>>> IPython-user@scipy.org
>>>>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Gökhan
>>>>
>>>
>>>
>>
>>
>> --
>> Gökhan
>>
>
>


-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/723f3ef7/attachment-0001.html 


More information about the IPython-user mailing list