[SciPy-User] Parallel code

Robin robince@gmail....
Mon Nov 30 06:39:41 CST 2009


If it is read only and you are on a platform with fork (ie not
windows) that multiprocessing is great for this sort of situation...
as long as the data is loaded before the fork, all the children can
read it fine (but be sure not to write to it - on write the page will
be copied for the child process leading to more memory use and changes
not visible between children).

Usually I put the variable to share in a module before calling pool... ie:

import mymodule # a blank module

mymodule.d = big_data_array

p = Pool(8)
p.map(function_which_does_something_to_mymodule.d, list_of_paraters)
p.close()

Cheers

Robin

On Mon, Nov 30, 2009 at 12:05 PM, Jose Gomez-Dans <jgomezdans@gmail.com> wrote:
> Hi!
> I want to run some code in parallel, and I have toyed with the idea of
> either using the multiprocessing module, or using ipython (which is quite
> easy to use). The main idea is to run a number of class methods in parallel
> (unsurprisingly!), fed with some arguments. However, these methods will need
> (read-)access to a rather large numpy array. Ideally (and since this is
> running on a SMP box), this could be a chunk of shared memory. I am aware of
> Sturla Molden's suggestion of using ctypes, but I guess that I was wondering
> whether some magic simple stuff is available off the shelf for this shared
> memory business?
>
> Thanks!
> J
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


More information about the SciPy-User mailing list