[SciPy-User] distributed optimisation package
Tue Mar 16 20:31:58 CDT 2010
Disclaimer: This isn't strictly about Scipy, but I think the people here
are probably the most likely to have something interesting to say about
this, as it's about scientific computing with Python.
Myself and a student in our lab have written a package that we're using
internally to distribute optimisation problems over several CPUs (or
GPUs using PyCUDA) over the small cluster of computers in our lab using
multiprocessing. It has two parts: some code for distributing work over
several computers and over several CPUs/GPUs on those computers. There's
no load balancing, we assume that each piece of work is roughly the
same, however it does handle using shared memory arrays with numpy
automatically (recently discussed on this list and the numpy one). It
can use named pipes or IP for communications (we had to use named pipes
because of the firewall setup in our lab).
The other part of the code is a framework for optimisation routines that
can be used with this. So far we've implemented a particle swarm
optimisation algorithm which is ideal for this sort of distributed
processing because it's very local and so splitting amongst several
machines is quite nice. We're also thinking about implementing a genetic
algorithm in the same framework.
Anyway, getting to the point. We've written this largely for our own
use, but it struck us that this might be of interest more widely. The
code is fairly simple (a few hundred lines), but there are maybe some
non-obvious things in there (at least for some people). We're thinking
about releasing it as an independent package, but there's some work
involved in that, so we were just looking to see if there was enough
interest to make it worth the effort really. Also, it's very likely that
our code is less than perfect, so if anyone would be interested in
working on it that could be good too.
Thanks for any comments!
More information about the SciPy-User