[Numpy-discussion] Using multiprocessing (shared memory) with numpy array multiplication

Christopher Barker Chris.Barker@noaa....
Mon Jun 13 12:16:30 CDT 2011


Brandt Belson wrote:
> Unfortunately I can't flatten the arrays. I'm writing a library where 
> the user supplies an inner product function for two generic objects, and 
> almost always the inner product function does large array 
> multiplications at some point. The library doesn't get to know about the 
> underlying arrays.

Now I'm confused -- if the user is providing the inner product 
implementation, how can you optimize that? Or are you trying to provide 
said user with an optimized "large array multiplication" that he/she can 
use?

If so, then I'd post your implementation here, and folks can suggest 
improvements.

If it's regular old element-wise multiplication:

a*b

(where a and b are numpy arrays)

then you are right, numpy isn't using any fancy multi-core aware 
optimized  package, so you should be able to make a faster version.

You might try numexpr also -- it's pretty cool, though may not help for 
a single operation. It might give you some ideas, though.

http://www.scipy.org/SciPyPackages/NumExpr


-Chris



> Thanks,
> Brandt
> 
> 
> 
>     Message: 2
>     Date: Fri, 10 Jun 2011 09:23:10 -0400
>     From: Olivier Delalleau <shish@keba.be <mailto:shish@keba.be>>
>     Subject: Re: [Numpy-discussion] Using multiprocessing (shared memory)
>            with numpy array multiplication
>     To: Discussion of Numerical Python <numpy-discussion@scipy.org
>     <mailto:numpy-discussion@scipy.org>>
>     Message-ID: <BANLkTikjppC90yE56T1mr+byAxXAw32YJA@mail.gmail.com
>     <mailto:BANLkTikjppC90yE56T1mr%2BbyAxXAw32YJA@mail.gmail.com>>
>     Content-Type: text/plain; charset="iso-8859-1"
> 
>     It may not work for you depending on your specific problem
>     constraints, but
>     if you could flatten the arrays, then it would be a dot, and you
>     could maybe
>     compute multiple such dot products by storing those flattened arrays
>     into a
>     matrix.
> 
>     -=- Olivier
> 
>     2011/6/10 Brandt Belson <bbelson@princeton.edu
>     <mailto:bbelson@princeton.edu>>
> 
>      > Hi,
>      > Thanks for getting back to me.
>      > I'm doing element wise multiplication, basically innerProduct =
>      > numpy.sum(array1*array2) where array1 and array2 are, in general,
>      > multidimensional. I need to do many of these operations, and I'd
>     like to
>      > split up the tasks between the different cores. I'm not using
>     numpy.dot, if
>      > I'm not mistaken I don't think that would do what I need.
>      > Thanks again,
>      > Brandt
>      >
>      >
>      > Message: 1
>      >> Date: Thu, 09 Jun 2011 13:11:40 -0700
>      >> From: Christopher Barker <Chris.Barker@noaa.gov
>     <mailto:Chris.Barker@noaa.gov>>
>      >> Subject: Re: [Numpy-discussion] Using multiprocessing (shared
>     memory)
>      >>        with numpy array multiplication
>      >> To: Discussion of Numerical Python <numpy-discussion@scipy.org
>     <mailto:numpy-discussion@scipy.org>>
>      >> Message-ID: <4DF128FC.8000807@noaa.gov
>     <mailto:4DF128FC.8000807@noaa.gov>>
>      >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>      >>
>      >> Not much time, here, but since you got no replies earlier:
>      >>
>      >>
>      >> >      > I'm parallelizing some code I've written using the built in
>      >> >     multiprocessing
>      >> >      > module. In my application, I need to multiply many
>     large arrays
>      >> >     together
>      >>
>      >> is the matrix multiplication, or element-wise? If matrix, then numpy
>      >> should be using LAPACK, which, depending on how its built, could be
>      >> using all your cores already. This is heavily dependent on your your
>      >> numpy (really the LAPACK it uses0 is built.
>      >>
>      >> >      > and
>      >> >      > sum the resulting product arrays (inner products).
>      >>
>      >> are you using numpy.dot() for that? If so, then the above applies to
>      >> that as well.
>      >>
>      >> I know I could look at your code to answer these questions, but I
>      >> thought this might help.
>      >>
>      >> -Chris
>      >>
>      >>
>      >>
>      >>
>      >>
>      >> --
>      >> Christopher Barker, Ph.D.
>      >> Oceanographer
>      >>
>      >> Emergency Response Division
>      >> NOAA/NOS/OR&R            (206) 526-6959
>     <tel:%28206%29%20526-6959>   voice
>      >> 7600 Sand Point Way NE   (206) 526-6329
>     <tel:%28206%29%20526-6329>   fax
>      >> Seattle, WA  98115       (206) 526-6317
>     <tel:%28206%29%20526-6317>   main reception
>      >>
>      >> Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
>      >>
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov


More information about the NumPy-Discussion mailing list