[IPython-user] Cannot start ipcluster

Gökhan Sever gokhansever@gmail....
Sat Oct 17 23:10:37 CDT 2009


On Sat, Oct 17, 2009 at 10:57 PM, Brian Granger <ellisonbg.net@gmail.com>wrote:

> I am not yet on Python 2.6 so there may be some issues there...
>
> questions:
>
> * What version of IPython are you using?  I am guessing 0.10?
>

I am using 0.10.


> * Have you tried to connect to the cluster:
>
> from IPython.kernel.client import MultiEngineClient
> mec = MultiEngineClient()
>

Without ipcluster local -n 4, when I just launch an ipython shell and
execute these two lines, I get an error:


I[1]: from IPython.kernel.client import MultiEngineClient
/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead
  import sha
/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated

I[2]: mec = MultiEngineClient()

---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)

/home/gsever/Desktop/<ipython console> in <module>()

/home/gsever/Desktop/python-repo/ipython/IPython/kernel/client.pyc in
get_multiengine_client(furl_or_file)
     67     """
     68     client =
blockingCallFromThread(_client_tub.get_multiengine_client,
---> 69         furl_or_file)
     70     return client.adapt_to_blocking_client()
     71

/home/gsever/Desktop/python-repo/ipython/IPython/kernel/twistedutil.pyc in
blockingCallFromThread(f, *a, **kw)
     70         @raise: any error raised during the callback chain.
     71         """
---> 72         return
twisted.internet.threads.blockingCallFromThread(reactor, f, *a, **kw)
     73
     74 else:

/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/internet/threads.pyc
in blockingCallFromThread(reactor, f, *a, **kw)
    112     result = queue.get()
    113     if isinstance(result, failure.Failure):
--> 114         result.raiseException()
    115     return result
    116

/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/failure.pyc
in raiseException(self)
    324         information if available.
    325         """
--> 326         raise self.type, self.value, self.tb
    327
    328

ConnectionRefusedError: Connection was refused by other side: 111:
Connection refused.


I had installed Twisted and other parallel computing requirements following
the instructions that were listed on the documentation pages.



> mec.get_ids()
>
> It may all be working...all you see are deprecation warnings related to
> Python 2.6.
>
> One more word.  The parallel computing stuff is not working in trunk right
> now (I am fixing it),
> so please stick with 0.10.
>

I was bitten by the trunk a few times. For over a month I am using the
downgraded (0.10) version :)


>
> Cheers,
>
> Brian
>
> On Sat, Oct 17, 2009 at 5:41 PM, Gökhan Sever <gokhansever@gmail.com>wrote:
>
>> Hello,
>>
>> I want to experiment IPython's parallel computing functionality. This far
>> I couldn't progress much because ipcluster instantiation stalls giving the
>> following messages without dropping me into the main IPython shell.
>>
>> My intention is parallelise a small Python script that calls an external
>> set of scripts that process the dataset I have in-hand. It is not a huge
>> computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it takes
>> about 1.5 hours to process the whole dataset. Looking at the system monitor
>> I see that the workload is not equally distributed in between CPUs (one of
>> them usually much lazier than the other.) I am sure parallezing the code run
>> would boost the processing speed. In my dataset I have 17 folders and each
>> folder is independent from each other. My script visits each folder and
>> calls the main external script via subprocess module's call function.
>> Processing starts with the first folder, and doesn't work on the next folder
>> unless the processing finishes with the previous folder. Basically, what I
>> really want is to put externally called scripts into separate threads, so
>> that I don't need to wait the previous job to be done during the processing
>> process.
>>
>> From the IPython parallel computing documentation, it seems like what I
>> want is doable in IPython. However I need some advice whether my
>> understanding is correct in this aspect. Also for the solution of the below
>> warning messages.
>>
>> Thanks.
>>
>>
>> [gsever@ccn Desktop]$  ipcluster local -n 4
>> /usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>> instead
>>   import sha
>> /usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>> DeprecationWarning: the sets module is deprecated
>> 2009-10-17 18:59:37-0500 [-] Log opened.
>> 2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
>> '--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
>> pid=11066
>> 2009-10-17 18:59:37-0500 [-] Waiting for controller to finish starting...
>> 2009-10-17 18:59:38-0500 [-]
>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>> instead\n  import sha\n'
>> 2009-10-17 18:59:38-0500 [-]
>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>> DeprecationWarning: the sets module is deprecated\n'
>> 2009-10-17 18:59:39-0500 [-] Controller started
>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>> pid=11067
>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>> pid=11068
>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>> pid=11069
>> 2009-10-17 18:59:39-0500 [-] Process ['ipengine',
>> '--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
>> pid=11070
>> 2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067, 11068,
>> 11069, 11070]
>> 2009-10-17 18:59:39-0500 [-]
>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>> instead\n  import sha\n'
>> 2009-10-17 18:59:39-0500 [-]
>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>> instead\n  import sha\n'
>> 2009-10-17 18:59:39-0500 [-]
>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>> DeprecationWarning: the sets module is deprecated\n'
>> 2009-10-17 18:59:40-0500 [-]
>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>> DeprecationWarning: the sets module is deprecated\n'
>> 2009-10-17 18:59:40-0500 [-]
>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>> instead\n  import sha\n'
>> 2009-10-17 18:59:40-0500 [-]
>> '/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
>> DeprecationWarning: the sha module is deprecated; use the hashlib module
>> instead\n  import sha\n'
>> 2009-10-17 18:59:40-0500 [-]
>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>> DeprecationWarning: the sets module is deprecated\n'
>> 2009-10-17 18:59:40-0500 [-]
>> '/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
>> DeprecationWarning: the sets module is deprecated\n'
>>
>>
>> Here is my system info:
>>
>> ================================================================================
>> Platform     :
>> Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
>> Python       : ('CPython', 'tags/r26', '66714')
>> IPython      : 0.10
>> NumPy      : 1.4.0.dev
>>
>> ================================================================================
>>
>> --
>> Gökhan
>>
>> _______________________________________________
>> IPython-user mailing list
>> IPython-user@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>


-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091017/21601535/attachment.html 


More information about the IPython-user mailing list