[Numpy-discussion] histogramdd memory needs
Mon Feb 4 02:37:51 CST 2008
> 2) Is there a way to use another algorithm (at the cost of performance)
>> > that uses less memory during calculation so that I can generate bigger
>> > histograms?
> You could work through your array block by block. Simply fix the range and
> generate an histogram for each slice of 100k data and sum them up at the
Thank you for your answer.
I sliced the (original) data into blocks. However, when I do this, I
need at least twice the memory for the whole histogram (one for the
temporary result and one for accumulating the total result). Assuming my
histogram has a size of (280**3)*8 = 176 (megabytes) this does not help,
What I will try next is to compute smaller parts of the big histogram
and combine them at the end. (Slice the histogram into blocks) Is it
this, that you were recommending?
More information about the Numpy-discussion