[IPython-user] Cannot start ipcluster

[Brian Granger]

> One thing little bothers me that I cannot see what workers doing exactly
> except from looking at the log files. In regular processing my script
> outputs the state of the postprocessing in shell window while it
> accomplishes a task, like shown below:
>
>
Yes, right now IPython can't do this.  Also, when multiprocessing is run
over a network, I am not sure it still can do this.  If it can, I would love
to know how it does it!

Cheers,

Brian


> Processing the 09_03_17_13_13_42.sea file ....................... Done
> Processing the 09_03_17_13_13_42.analog.??? file ................ Done
> Creating 09_03_17_13_13_42.ballvario.1Hz ........................ Done
> Averaging the 09_03_17_13_13_42.physical.? file ................. Done
> Averaging the 09_03_17_13_13_42.physical.? file ................. Done
> Creating the 09_03_17_13_13_42.2dc.spec.1Hz ..................... Done
> Processing the 09_03_17_13_13_42.physical.? file ................ Done
> Creating 09_03_17_13_13_42.basic.10Hz ........................... Done
> Creating 09_03_17_13_13_42.basic.1Hz ............................ Done
> Processing the 09_03_17_13_13_42.counts.pcasp.??? ............... Done
> Processing the 09_03_17_13_13_42.counts.spp_fssp.raw ............ Done
> Creating 09_03_17_13_13_42.hotwire.raw .......................... Done
> Creating 09_03_17_13_13_42.2dc.shadowOR.raw ..................... Done
> Creating 09_03_17_13_13_42.hotwire.1Hz .......................... Done
> Creating 09_03_17_13_13_42.conc.spp_fssp.1Hz .................... Done
> Creating 09_03_17_13_13_42.serial.GPS.10sec ..................... Done
> Creating 09_03_17_13_13_42.conc_stp.pcasp.raw ................... Done
> Creating 09_03_17_13_13_42.conc_stp.spp_fssp.raw ................ Done
> Creating 09_03_17_13_13_42.2dc.comb.1Hz ......................... Done
> Creating 09_03_17_13_13_42.conc_stp.spp_fssp.1Hz ................ Done
> Creating 09_03_17_13_13_42.dmtccnc.combined.raw ................. Done
> Creating 09_03_17_13_13_42.550nm.scat.raw ....................... Done
> Creating 09_03_17_13_13_42.sau .................................. Done
> Creating 09_03_17_13_13_42.*.sau ................................ Done
> Creating 09_03_17_13_13_42.*.sau ................................ Done
> Creating 09_03_17_13_13_42.sau.nc ............................... Done
> Creating 09_03_17_13_13_42.kml .................................. Done
>
> These progress both output to screen and written into log files to see
> everything goes on well during processing. And it is helpful to see on the
> screen during the action. Could these be feasible with parallel processing
> approach?
>
>
>
> # Below is the code that I used with with MultiEngineClient approach
> # Almost identical to previous code except TaskClient parts replaced with
> MultiEngineClient
>
> ### BEGIN CODE
>
> from IPython.kernel.client import MultiEngineClient
> from subprocess import call
> import os
>
>
> def find_sea_files():
>
>     file_list, path_list = [], []
>     init = os.getcwd()
>
>     for root, dirs, files in os.walk('.'):
>         dirs.sort()
>         for file in files:
>             if file.endswith('.sea'):
>                 file_list.append(file)
>                 os.chdir(root)
>                 path_list.append(os.getcwd())
>                 os.chdir(init)
>
>     return file_list, path_list
>
>
> def process_all(path, file):
>     import os
>     from subprocess import call
>     os.chdir(path)
>     call(['postprocessing_saudi', file])
>
>
> if __name__ == '__main__':
>     mec = MultiEngineClient()
>     mec.execute('from subprocess import call')
>     mec.execute('import os')
>     files, paths = find_sea_files()
>     mec.map(process_all, paths, files)
>
> ### END CODE
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091018/74824e6d/attachment.html