[SciPy-User] distributed optimisation package
Paul Anton Letnes
Tue Mar 16 21:02:27 CDT 2010
I'm not sure if I can use your algorithm, but I'm definitely curious / interested. (We've got a similar (I guess?) setup where I work.)
On 16. mars 2010, at 18.31, Dan Goodman wrote:
> Hi all,
> Disclaimer: This isn't strictly about Scipy, but I think the people here
> are probably the most likely to have something interesting to say about
> this, as it's about scientific computing with Python.
> Myself and a student in our lab have written a package that we're using
> internally to distribute optimisation problems over several CPUs (or
> GPUs using PyCUDA) over the small cluster of computers in our lab using
> multiprocessing. It has two parts: some code for distributing work over
> several computers and over several CPUs/GPUs on those computers. There's
> no load balancing, we assume that each piece of work is roughly the
> same, however it does handle using shared memory arrays with numpy
> automatically (recently discussed on this list and the numpy one). It
> can use named pipes or IP for communications (we had to use named pipes
> because of the firewall setup in our lab).
> The other part of the code is a framework for optimisation routines that
> can be used with this. So far we've implemented a particle swarm
> optimisation algorithm which is ideal for this sort of distributed
> processing because it's very local and so splitting amongst several
> machines is quite nice. We're also thinking about implementing a genetic
> algorithm in the same framework.
> Anyway, getting to the point. We've written this largely for our own
> use, but it struck us that this might be of interest more widely. The
> code is fairly simple (a few hundred lines), but there are maybe some
> non-obvious things in there (at least for some people). We're thinking
> about releasing it as an independent package, but there's some work
> involved in that, so we were just looking to see if there was enough
> interest to make it worth the effort really. Also, it's very likely that
> our code is less than perfect, so if anyone would be interested in
> working on it that could be good too.
> Thanks for any comments!
> SciPy-User mailing list
More information about the SciPy-User