[IPython-User] engines dying due to excessive load

Fernando Perez fperez.net@gmail....
Fri Jun 29 13:57:31 CDT 2012


On Fri, Jun 29, 2012 at 9:22 AM, Robert Nishihara
<robertnishihara@gmail.com> wrote:
> Ok, so numpy uses Intel's Math Kernel Library, which tries to automatically
> parallelize things, which, on a cluster, can cause problems for the
> scheduler.
>
> Setting MLK_NUM_THREADS=1 on the engines appears to have completely fixed
> the problem.

This is the classic parallelization battle between libraries at
multiple levels, which to the best of my understanding, can only be
solved correctly if all codes delegate thread creation to a central
point such as apple's Grand Central Dispatch
(http://arstechnica.com/apple/2009/08/mac-os-x-10-6/12).

Cheers,

f


More information about the IPython-User mailing list