[SciPy-User] distributed optimisation package

Debabrata Midya Debabrata.Midya@services.nsw.gov...
Tue Mar 16 20:59:34 CDT 2010

Thanks for the information.
I am interested.

>>> Dan Goodman <dg.gmane@thesamovar.net> 17/03/2010 12:31 pm >>>
Hi all,

Disclaimer: This isn't strictly about Scipy, but I think the people here 
are probably the most likely to have something interesting to say about 
this, as it's about scientific computing with Python.

Myself and a student in our lab have written a package that we're using 
internally to distribute optimisation problems over several CPUs (or 
GPUs using PyCUDA) over the small cluster of computers in our lab using 
multiprocessing. It has two parts: some code for distributing work over 
several computers and over several CPUs/GPUs on those computers. There's 
no load balancing, we assume that each piece of work is roughly the 
same, however it does handle using shared memory arrays with numpy 
automatically (recently discussed on this list and the numpy one). It 
can use named pipes or IP for communications (we had to use named pipes 
because of the firewall setup in our lab).

The other part of the code is a framework for optimisation routines that 
can be used with this. So far we've implemented a particle swarm 
optimisation algorithm which is ideal for this sort of distributed 
processing because it's very local and so splitting amongst several 
machines is quite nice. We're also thinking about implementing a genetic 
algorithm in the same framework.

Anyway, getting to the point. We've written this largely for our own 
use, but it struck us that this might be of interest more widely. The 
code is fairly simple (a few hundred lines), but there are maybe some 
non-obvious things in there (at least for some people). We're thinking 
about releasing it as an independent package, but there's some work 
involved in that, so we were just looking to see if there was enough 
interest to make it worth the effort really. Also, it's very likely that 
our code is less than perfect, so if anyone would be interested in 
working on it that could be good too.

Thanks for any comments!


SciPy-User mailing list

This email message, including any attached files, is confidential and
intended solely for the use of the individual or entity to whom it is

The Department of Services, Technology and Administration prohibits the 
right to publish, copy, distribute or disclose any information contained in 
this email, or its attachments, by any party other than the intended 
recipient. If you have received this email in error please notify the sender 
and delete it from your system.

No employee or agent is authorised to conclude any binding agreement 
on behalf of the Department of Services, Technology and Administration 
by email. The views or opinions presented in this email are solely those of 
the author and do not necessarily represent those of the Department, 
except where the sender expressly, and with authority, states them to be 
the views of the Department of Services, Technology and Administration.

The Department of Services, Technology and Administration accepts no 
liability for any loss or damage arising from the use of this email and 
recommends that the recipient check this email and any attached files for 
the presence of viruses.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100317/2a2e8dbe/attachment.html 

More information about the SciPy-User mailing list