[IPython-user] Cannot start ipcluster

[Gökhan Sever]

On Sun, Oct 18, 2009 at 2:08 PM, Gökhan Sever <gokhansever@gmail.com> wrote:

>
>
> On Sun, Oct 18, 2009 at 12:27 PM, Brian Granger <ellisonbg.net@gmail.com>wrote:
>
>> Also, once you fix the import/name error, can you send me your script.
>> That way I can see if there are any performance issues we can address.
>>
>> Cheers,
>>
>> Brian
>
>
>   Will test with MultiEngineClient approach as well.
>

More speed-up :)

>From real    11m38.314s with TaskClient to real    10m17.769s with
MultiEngineClient (user    0m0.515s, sys    0m0.158s) These last two is not
significant I think for the comparisons? I started ipcluster with 2 local
engines. Next step is to use multiprocessing module.

One thing little bothers me that I cannot see what workers doing exactly
except from looking at the log files. In regular processing my script
outputs the state of the postprocessing in shell window while it
accomplishes a task, like shown below:

Processing the 09_03_17_13_13_42.sea file ....................... Done
Processing the 09_03_17_13_13_42.analog.??? file ................ Done
Creating 09_03_17_13_13_42.ballvario.1Hz ........................ Done
Averaging the 09_03_17_13_13_42.physical.? file ................. Done
Averaging the 09_03_17_13_13_42.physical.? file ................. Done
Creating the 09_03_17_13_13_42.2dc.spec.1Hz ..................... Done
Processing the 09_03_17_13_13_42.physical.? file ................ Done
Creating 09_03_17_13_13_42.basic.10Hz ........................... Done
Creating 09_03_17_13_13_42.basic.1Hz ............................ Done
Processing the 09_03_17_13_13_42.counts.pcasp.??? ............... Done
Processing the 09_03_17_13_13_42.counts.spp_fssp.raw ............ Done
Creating 09_03_17_13_13_42.hotwire.raw .......................... Done
Creating 09_03_17_13_13_42.2dc.shadowOR.raw ..................... Done
Creating 09_03_17_13_13_42.hotwire.1Hz .......................... Done
Creating 09_03_17_13_13_42.conc.spp_fssp.1Hz .................... Done
Creating 09_03_17_13_13_42.serial.GPS.10sec ..................... Done
Creating 09_03_17_13_13_42.conc_stp.pcasp.raw ................... Done
Creating 09_03_17_13_13_42.conc_stp.spp_fssp.raw ................ Done
Creating 09_03_17_13_13_42.2dc.comb.1Hz ......................... Done
Creating 09_03_17_13_13_42.conc_stp.spp_fssp.1Hz ................ Done
Creating 09_03_17_13_13_42.dmtccnc.combined.raw ................. Done
Creating 09_03_17_13_13_42.550nm.scat.raw ....................... Done
Creating 09_03_17_13_13_42.sau .................................. Done
Creating 09_03_17_13_13_42.*.sau ................................ Done
Creating 09_03_17_13_13_42.*.sau ................................ Done
Creating 09_03_17_13_13_42.sau.nc ............................... Done
Creating 09_03_17_13_13_42.kml .................................. Done

These progress both output to screen and written into log files to see
everything goes on well during processing. And it is helpful to see on the
screen during the action. Could these be feasible with parallel processing
approach?



# Below is the code that I used with with MultiEngineClient approach
# Almost identical to previous code except TaskClient parts replaced with
MultiEngineClient

### BEGIN CODE

from IPython.kernel.client import MultiEngineClient
from subprocess import call
import os


def find_sea_files():

    file_list, path_list = [], []
    init = os.getcwd()

    for root, dirs, files in os.walk('.'):
        dirs.sort()
        for file in files:
            if file.endswith('.sea'):
                file_list.append(file)
                os.chdir(root)
                path_list.append(os.getcwd())
                os.chdir(init)

    return file_list, path_list


def process_all(path, file):
    import os
    from subprocess import call
    os.chdir(path)
    call(['postprocessing_saudi', file])


if __name__ == '__main__':
    mec = MultiEngineClient()
    mec.execute('from subprocess import call')
    mec.execute('import os')
    files, paths = find_sea_files()
    mec.map(process_all, paths, files)

### END CODE
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/a290d681/attachment.html