[IPython-user] Cannot start ipcluster
Sun Oct 18 16:01:39 CDT 2009
On Sun, Oct 18, 2009 at 2:34 PM, Gökhan Sever <email@example.com> wrote:
> Moreeeeee speed-up :)
> Next step is to use multiprocessing module.
I did two tests since I was not sure which timing to believe:
which in result I figured out real is what I want to see. So the
improvement with respect to original linear 18m 5s run is 2.4 to 2.7X
speed-up in a Dual Core 2.5 Ghz laptop using Python's multiprocessing
module, which is great only adding a few line of code and slightly modifying
my original process_all wrapper script.
Here is the code:
Execute postprocessing_saudi script in parallel using multiprocessing
from multiprocessing import Pool
from subprocess import call
file_list, path_list = , 
init = os.getcwd()
for root, dirs, files in os.walk('.'):
for file in files:
return file_list, path_list
if __name__ == '__main__':
pool = Pool(processes=2) # start 2 worker processes
files, paths = find_sea_files()
pathfile = [[paths[i],files[i]] for i in range(len(files))]
The main difference is to change map call since Python's original map
supports only one iterable argument. This approach also shows execution
results on the terminal screen unlike IPython's. I am assuming like
IPython's, multiprocessing module should be able to run on external nodes.
Which means once I can set a few fast external machines I can perform a few
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-user