[Numpy-discussion] how to deal with large arrays
Darren Dale
dd55 at cornell.edu
Fri Oct 15 14:18:41 CDT 2004
Hello,
I have two 2D arrays, Q is 3-by-q and R is r-by-3. At each q, I need to sum
q(r) over R, so I take a dot product RQ and then sum along one axis to get a
1-by-q result.
I'm doing this with dot products because it is much faster than the equivalent
for or while loop. The intermediate r-by-q array can get very large though
(200MB in my case), so I was wondering if there is a better way to go about
it?
If not, I can slice up R and deal with it one chunk at a time, then the
intermediate arrays fit within the available system resources. Would somebody
offer a suggestion of how to do this intelligently? Should the intermediate
array be about the size of the processor cache, some fraction of the
available memory, or is there something else I need to consider?
Thank you,
Darren
More information about the Numpy-discussion
mailing list