<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>One thing little bothers me that I cannot see what workers doing exactly except from looking at the log files. In regular processing my script outputs the state of the postprocessing in shell window while it accomplishes a task, like shown below:<br>

<br></div></div></blockquote><div><br>Yes, right now IPython can&#39;t do this.  Also, when multiprocessing is run over a network, I am not sure it still can do this.  If it can, I would love to know how it does it!<br><br>
Cheers,<br><br>Brian<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Processing the 09_03_17_13_13_42.sea file ....................... Done<br>
Processing the 09_03_17_13_13_42.analog.??? file ................ Done<br>Creating 09_03_17_13_13_42.ballvario.1Hz ........................ Done<br>
Averaging the 09_03_17_13_13_42.physical.? file ................. Done<br>Averaging the 09_03_17_13_13_42.physical.? file ................. Done<br>Creating the 09_03_17_13_13_42.2dc.spec.1Hz ..................... Done<br>

Processing the 09_03_17_13_13_42.physical.? file ................ Done<br>Creating 09_03_17_13_13_42.basic.10Hz ........................... Done<br>Creating 09_03_17_13_13_42.basic.1Hz ............................ Done<br>

Processing the 09_03_17_13_13_42.counts.pcasp.??? ............... Done<br>Processing the 09_03_17_13_13_42.counts.spp_fssp.raw ............ Done<br>Creating 09_03_17_13_13_42.hotwire.raw .......................... Done<br>

Creating 09_03_17_13_13_42.2dc.shadowOR.raw ..................... Done<br>Creating 09_03_17_13_13_42.hotwire.1Hz .......................... Done<br>Creating 09_03_17_13_13_42.conc.spp_fssp.1Hz .................... Done<br>

Creating 09_03_17_13_13_42.serial.GPS.10sec ..................... Done<br>Creating 09_03_17_13_13_42.conc_stp.pcasp.raw ................... Done<br>Creating 09_03_17_13_13_42.conc_stp.spp_fssp.raw ................ Done<br>

Creating 09_03_17_13_13_42.2dc.comb.1Hz ......................... Done<br>Creating 09_03_17_13_13_42.conc_stp.spp_fssp.1Hz ................ Done<br>Creating 09_03_17_13_13_42.dmtccnc.combined.raw ................. Done<br>

Creating 09_03_17_13_13_42.550nm.scat.raw ....................... Done<br>Creating 09_03_17_13_13_42.sau .................................. Done<br>Creating 09_03_17_13_13_42.*.sau ................................ Done<br>

Creating 09_03_17_13_13_42.*.sau ................................ Done<br>Creating <a href="http://09_03_17_13_13_42.sau.nc" target="_blank">09_03_17_13_13_42.sau.nc</a> ............................... Done<br>Creating 09_03_17_13_13_42.kml .................................. Done<br>

<br>These progress both output to screen and written into log files to see everything goes on well during processing. And it is helpful to see on the screen during the action. Could these be feasible with parallel processing approach?<br>

<br><br><br># Below is the code that I used with with MultiEngineClient approach<br># Almost identical to previous code except TaskClient parts replaced with MultiEngineClient<br><br>### BEGIN CODE<div class="im"><br><br>
from IPython.kernel.client import MultiEngineClient<br></div><div class="im">
from subprocess import call<br>import os<br><br><br>def find_sea_files():<br><br>    file_list, path_list = [], []<br>    init = os.getcwd()<br><br>    for root, dirs, files in os.walk(&#39;.&#39;):<br>        dirs.sort()<br>

        for file in files:<br>            if file.endswith(&#39;.sea&#39;):<br>                file_list.append(file)<br>                os.chdir(root)<br>                path_list.append(os.getcwd())<br>                os.chdir(init)<br>

<br>    return file_list, path_list<br><br><br>def process_all(path, file):<br>    import os<br>    from subprocess import call<br>    os.chdir(path)<br>    call([&#39;postprocessing_saudi&#39;, file])<br><br><br>if __name__ == &#39;__main__&#39;:<br>
</div>
    mec = MultiEngineClient()<br>    mec.execute(&#39;from subprocess import call&#39;)<br>    mec.execute(&#39;import os&#39;)<div class="im"><br>    files, paths = find_sea_files()<br></div>    mec.map(process_all, paths, files)<br>
<br>### END CODE<br>
</div></div>
</blockquote></div><br>