[SciPy-User] distributed optimisation package

Paul Anton Letnes paul.anton.letnes@gmail....
Tue Mar 16 21:02:27 CDT 2010


Hi Dan,

I'm not sure if I can use your algorithm, but I'm definitely curious / interested. (We've got a similar (I guess?) setup where I work.)

-Paul

On 16. mars 2010, at 18.31, Dan Goodman wrote:

> Hi all,
> 
> Disclaimer: This isn't strictly about Scipy, but I think the people here 
> are probably the most likely to have something interesting to say about 
> this, as it's about scientific computing with Python.
> 
> Myself and a student in our lab have written a package that we're using 
> internally to distribute optimisation problems over several CPUs (or 
> GPUs using PyCUDA) over the small cluster of computers in our lab using 
> multiprocessing. It has two parts: some code for distributing work over 
> several computers and over several CPUs/GPUs on those computers. There's 
> no load balancing, we assume that each piece of work is roughly the 
> same, however it does handle using shared memory arrays with numpy 
> automatically (recently discussed on this list and the numpy one). It 
> can use named pipes or IP for communications (we had to use named pipes 
> because of the firewall setup in our lab).
> 
> The other part of the code is a framework for optimisation routines that 
> can be used with this. So far we've implemented a particle swarm 
> optimisation algorithm which is ideal for this sort of distributed 
> processing because it's very local and so splitting amongst several 
> machines is quite nice. We're also thinking about implementing a genetic 
> algorithm in the same framework.
> 
> Anyway, getting to the point. We've written this largely for our own 
> use, but it struck us that this might be of interest more widely. The 
> code is fairly simple (a few hundred lines), but there are maybe some 
> non-obvious things in there (at least for some people). We're thinking 
> about releasing it as an independent package, but there's some work 
> involved in that, so we were just looking to see if there was enough 
> interest to make it worth the effort really. Also, it's very likely that 
> our code is less than perfect, so if anyone would be interested in 
> working on it that could be good too.
> 
> Thanks for any comments!
> 
> Dan
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user



More information about the SciPy-User mailing list