[IPython-user] Cannot start ipcluster

[Gökhan Sever]

On Sun, Oct 18, 2009 at 2:34 PM, Gökhan Sever <gokhansever@gmail.com> wrote:

>
>
> On Sun, Oct 18, 2009 at 2:08 PM, Gökhan Sever <gokhansever@gmail.com>wrote:
>
>>
>>
>> On Sun, Oct 18, 2009 at 12:27 PM, Brian Granger <ellisonbg.net@gmail.com>wrote:
>>
>>> Also, once you fix the import/name error, can you send me your script.
>>> That way I can see if there are any performance issues we can address.
>>>
>>> Cheers,
>>>
>>> Brian
>>
>>
>>   Will test with MultiEngineClient approach as well.
>>
>
> More speed-up :)
>
> From real    11m38.314s with TaskClient to real    10m17.769s with
> MultiEngineClient (user    0m0.515s, sys    0m0.158s) These last two is not
> significant I think for the comparisons? I started ipcluster with 2 local
> engines. Next step is to use multiprocessing module.
>
> One thing little bothers me that I cannot see what workers doing exactly
> except from looking at the log files. In regular processing my script
> outputs the state of the postprocessing in shell window while it
> accomplishes a task, like shown below:
>
> Processing the 09_03_17_13_13_42.sea file ....................... Done
> Processing the 09_03_17_13_13_42.analog.??? file ................ Done
> Creating 09_03_17_13_13_42.ballvario.1Hz ........................ Done
> Averaging the 09_03_17_13_13_42.physical.? file ................. Done
> Averaging the 09_03_17_13_13_42.physical.? file ................. Done
> Creating the 09_03_17_13_13_42.2dc.spec.1Hz ..................... Done
> Processing the 09_03_17_13_13_42.physical.? file ................ Done
> Creating 09_03_17_13_13_42.basic.10Hz ........................... Done
> Creating 09_03_17_13_13_42.basic.1Hz ............................ Done
> Processing the 09_03_17_13_13_42.counts.pcasp.??? ............... Done
> Processing the 09_03_17_13_13_42.counts.spp_fssp.raw ............ Done
> Creating 09_03_17_13_13_42.hotwire.raw .......................... Done
> Creating 09_03_17_13_13_42.2dc.shadowOR.raw ..................... Done
> Creating 09_03_17_13_13_42.hotwire.1Hz .......................... Done
> Creating 09_03_17_13_13_42.conc.spp_fssp.1Hz .................... Done
> Creating 09_03_17_13_13_42.serial.GPS.10sec ..................... Done
> Creating 09_03_17_13_13_42.conc_stp.pcasp.raw ................... Done
> Creating 09_03_17_13_13_42.conc_stp.spp_fssp.raw ................ Done
> Creating 09_03_17_13_13_42.2dc.comb.1Hz ......................... Done
> Creating 09_03_17_13_13_42.conc_stp.spp_fssp.1Hz ................ Done
> Creating 09_03_17_13_13_42.dmtccnc.combined.raw ................. Done
> Creating 09_03_17_13_13_42.550nm.scat.raw ....................... Done
> Creating 09_03_17_13_13_42.sau .................................. Done
> Creating 09_03_17_13_13_42.*.sau ................................ Done
> Creating 09_03_17_13_13_42.*.sau ................................ Done
> Creating 09_03_17_13_13_42.sau.nc ............................... Done
> Creating 09_03_17_13_13_42.kml .................................. Done
>
> These progress both output to screen and written into log files to see
> everything goes on well during processing. And it is helpful to see on the
> screen during the action. Could these be feasible with parallel processing
> approach?
>
>
>
> # Below is the code that I used with with MultiEngineClient approach
> # Almost identical to previous code except TaskClient parts replaced with
> MultiEngineClient
>
> ### BEGIN CODE
>
> from IPython.kernel.client import MultiEngineClient
> from subprocess import call
> import os
>
>
> def find_sea_files():
>
>     file_list, path_list = [], []
>     init = os.getcwd()
>
>     for root, dirs, files in os.walk('.'):
>         dirs.sort()
>         for file in files:
>             if file.endswith('.sea'):
>                 file_list.append(file)
>                 os.chdir(root)
>                 path_list.append(os.getcwd())
>                 os.chdir(init)
>
>     return file_list, path_list
>
>
> def process_all(path, file):
>     import os
>     from subprocess import call
>     os.chdir(path)
>     call(['postprocessing_saudi', file])
>
>
> if __name__ == '__main__':
>     mec = MultiEngineClient()
>     mec.execute('from subprocess import call')
>     mec.execute('import os')
>     files, paths = find_sea_files()
>     mec.map(process_all, paths, files)
>
> ### END CODE
>

I re-run this test since I forgot to remove module imports from within the
process_all function. To my surprise I got a longer processing time:

real    11m10.989s
user    0m0.517s
sys    0m0.130s

comparing to 10m17.769s.

One thing I observed in this run CPUs were not working at full speed. My
little CPU Frequency Scaling monitor was swinging around 1.6 and 2.5 GHz
levels. CPU fan is constantly only, maybe overheating not good for these
poor cores :)


-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/3458e75e/attachment.html