[Numpy-discussion] how to deal with large arrays

[Darren Dale]

Hello,

I have two 2D arrays, Q is 3-by-q and R is r-by-3. At each q, I need to sum 
q(r) over R, so I take a dot product RQ and then sum along one axis to get a 
1-by-q result.

I'm doing this with dot products because it is much faster than the equivalent 
for or while loop. The intermediate r-by-q array can get very large though 
(200MB in my case), so I was wondering if there is a better way to go about 
it?

If not, I can slice up R and deal with it one chunk at a time, then the 
intermediate arrays fit within the available system resources. Would somebody 
offer a suggestion of how to do this intelligently? Should the intermediate 
array be about the size of the processor cache, some fraction of the 
available memory, or is there something else I need to consider?

Thank you,
Darren