[IPython-user] Cannot start ipcluster

Gökhan Sever gokhansever@gmail....
Sat Oct 17 19:41:10 CDT 2009


Hello,

I want to experiment IPython's parallel computing functionality. This far I
couldn't progress much because ipcluster instantiation stalls giving the
following messages without dropping me into the main IPython shell.

My intention is parallelise a small Python script that calls an external set
of scripts that process the dataset I have in-hand. It is not a huge
computing power demanding task but in my Intel 2.5Ghz Dual Core 2 it takes
about 1.5 hours to process the whole dataset. Looking at the system monitor
I see that the workload is not equally distributed in between CPUs (one of
them usually much lazier than the other.) I am sure parallezing the code run
would boost the processing speed. In my dataset I have 17 folders and each
folder is independent from each other. My script visits each folder and
calls the main external script via subprocess module's call function.
Processing starts with the first folder, and doesn't work on the next folder
unless the processing finishes with the previous folder. Basically, what I
really want is to put externally called scripts into separate threads, so
that I don't need to wait the previous job to be done during the processing
process.

>From the IPython parallel computing documentation, it seems like what I want
is doable in IPython. However I need some advice whether my understanding is
correct in this aspect. Also for the solution of the below warning messages.


Thanks.


[gsever@ccn Desktop]$  ipcluster local -n 4
/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead
  import sha
/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated
2009-10-17 18:59:37-0500 [-] Log opened.
2009-10-17 18:59:37-0500 [-] Process ['ipcontroller',
'--logfile=/home/gsever/.ipython/log/ipcontroller'] has started with
pid=11066
2009-10-17 18:59:37-0500 [-] Waiting for controller to finish starting...
2009-10-17 18:59:38-0500 [-]
'/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead\n  import sha\n'
2009-10-17 18:59:38-0500 [-]
'/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated\n'
2009-10-17 18:59:39-0500 [-] Controller started
2009-10-17 18:59:39-0500 [-] Process ['ipengine',
'--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
pid=11067
2009-10-17 18:59:39-0500 [-] Process ['ipengine',
'--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
pid=11068
2009-10-17 18:59:39-0500 [-] Process ['ipengine',
'--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
pid=11069
2009-10-17 18:59:39-0500 [-] Process ['ipengine',
'--logfile=/home/gsever/.ipython/log/ipengine11066-'] has started with
pid=11070
2009-10-17 18:59:39-0500 [-] Engines started with pids: [11067, 11068,
11069, 11070]
2009-10-17 18:59:39-0500 [-]
'/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead\n  import sha\n'
2009-10-17 18:59:39-0500 [-]
'/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead\n  import sha\n'
2009-10-17 18:59:39-0500 [-]
'/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated\n'
2009-10-17 18:59:40-0500 [-]
'/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated\n'
2009-10-17 18:59:40-0500 [-]
'/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead\n  import sha\n'
2009-10-17 18:59:40-0500 [-]
'/usr/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-i686.egg/twisted/python/filepath.py:12:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead\n  import sha\n'
2009-10-17 18:59:40-0500 [-]
'/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated\n'
2009-10-17 18:59:40-0500 [-]
'/usr/lib/python2.6/site-packages/foolscap-0.4.2-py2.6.egg/foolscap/banana.py:2:
DeprecationWarning: the sets module is deprecated\n'


Here is my system info:
================================================================================
Platform     :
Linux-2.6.29.6-217.2.3.fc11.i686.PAE-i686-with-fedora-11-Leonidas
Python       : ('CPython', 'tags/r26', '66714')
IPython      : 0.10
NumPy      : 1.4.0.dev
================================================================================

-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091017/ba60c135/attachment.html 


More information about the IPython-user mailing list