[IPython-User] MEC beats TC for "task farming" style parallelization: Huh?????
Sun Sep 26 08:19:17 CDT 2010
My student and I are running some "parameter sweep" type jobs using the IPython parallelization code.
For the code we are running, benchmarks indicate that a MultiEngineController based run wins over a TaskController based run (45 minutes to 75 minutes or thereabouts), even though the MEC run clearly has cpu starvation (mostly from breaks and continues in inner loops and so forth). Profiling the TC run shows that the vast majority of time is spent on the 'acquire' call, which I assume stems from the Twisted threading implementation or something.
I guess my basic question is, is there some tuning we can do to allow the TC to do what I think it is meant to do (i.e. avoid cpu starvation, and still beat the MEC based wall-clock times)?
Thanks for any tips you might be able to provide!
More information about the IPython-User