[IPython-user] Cannot start ipcluster

Gökhan Sever gokhansever@gmail....
Mon Oct 19 00:04:55 CDT 2009


On Sun, Oct 18, 2009 at 11:47 PM, Brian Granger <ellisonbg.net@gmail.com>wrote:

>
> You are right on your suspicion. I was making a clean run on each file.
>> That is deleting everything except the sea files in the folders. With this
>> configuration multiprocessing module's pooling approach doesn't work. It
>> cannot branch into the external script completely. However when I leave the
>> processed outputs in the folders and run the script it works and takes much
>> less than IPython's parallelism. Not the question is how to explain this
>> behaviour.
>>
>> End of my 2.4 to 2.7X speed-up happiness :)
>>
>>
> I know this form experience.  Every time I have had speedups that were too
> good to be true, there was always something hidden
> that ended the pipe-dream.
>
> But I just want to clarify a few points:
>
> * What is the len of the iterable that you pass to the various parallel
> versions of map?
>

4 elements, 4 file names and file path. In multiprocessing case I combine
these into one iterable


>
> * About how long on average does it take to compute things for one element
> of the iterable?
>

3 to 5 minutes depends on the length of the file. Actually each sea file
corresponds to measurements that were taken from aircraft flights containing
atmospheric measurement data -cloud microphysical data in specific.


>
> Because I don't have the data, I can't tell these things from your scripts.
>

I can send the data, however it would take at least a good 2 days to make
your computer fully processing the dataset :) providing that you will find
an access to IDL. In the project the code is open-source, but majority of it
written with IDL, and there are many mixed open languages. Still it takes
too much to make installation. That's why I keep saying to unify in our
research group and use Python :)


>
> Cheers,
>
> Brian
>



-- 
Gökhan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20091019/feeb849b/attachment.html 


More information about the IPython-user mailing list