[IPython-user] Cannot start ipcluster
Mon Oct 19 00:04:55 CDT 2009
On Sun, Oct 18, 2009 at 11:47 PM, Brian Granger <firstname.lastname@example.org>wrote:
> You are right on your suspicion. I was making a clean run on each file.
>> That is deleting everything except the sea files in the folders. With this
>> configuration multiprocessing module's pooling approach doesn't work. It
>> cannot branch into the external script completely. However when I leave the
>> processed outputs in the folders and run the script it works and takes much
>> less than IPython's parallelism. Not the question is how to explain this
>> End of my 2.4 to 2.7X speed-up happiness :)
> I know this form experience. Every time I have had speedups that were too
> good to be true, there was always something hidden
> that ended the pipe-dream.
> But I just want to clarify a few points:
> * What is the len of the iterable that you pass to the various parallel
> versions of map?
4 elements, 4 file names and file path. In multiprocessing case I combine
these into one iterable
> * About how long on average does it take to compute things for one element
> of the iterable?
3 to 5 minutes depends on the length of the file. Actually each sea file
corresponds to measurements that were taken from aircraft flights containing
atmospheric measurement data -cloud microphysical data in specific.
> Because I don't have the data, I can't tell these things from your scripts.
I can send the data, however it would take at least a good 2 days to make
your computer fully processing the dataset :) providing that you will find
an access to IDL. In the project the code is open-source, but majority of it
written with IDL, and there are many mixed open languages. Still it takes
too much to make installation. That's why I keep saying to unify in our
research group and use Python :)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-user