[Numpy-discussion] Maximum available dimensions in numpy
Fernando.Perez at colorado.edu
Mon Jan 16 20:52:14 CST 2006
Paul Barrett wrote:
>>>How about setting the default case to 3 or 4 dimensions and then
>>>special casing the rare higher dimensional arrays, i.e. using malloc
>>>for these situations. The default dimension size could be a compile
>>>time option for those who routinely exceed the default size of 3 or 4.
>>This seems like premature optimization. In most cases, if you're in a
>>situation where the dimensional overhead matters (lot's of small arrays)
>>you are using Numeric/Numarray/NumPy poorly and your code is going to be
>>slow and bloated anyway. The way to get good efficiency with these
>>extensions is to do block operations on large matrices. This often
>>involves a little trickery and several extra dimensions. Reducing the
>>default matrix size down to 3 or 4 makes efficient code slower since
>>going through malloc will involve an extra dereference and probably some
> It also avoids the possibility of running up against the maximum number of
> dimensions, while conserving memory. For those users that create a multitude
> of small arrays, the wasted memory might become important. I only suggested
> 3 or 4 dimensions, because it would appear to cover 99% of the cases. I
> hope those users creating the other 1%, know what they are doing.
How about following blitz convention (since we use blitz for looping in C++
fast via weave)? Blitz sets the max at 11 (I don't know if this comes from a
Spinal Tap reference, string theory, or pure chance :).
I agree that 32 is perhaps excessive, but 4 is certainly too low a number, I
think. Given that most regular numerical algorithms scale exponentially with
the number of dimensions, it's hard to think of anything useful you can do
with a 32-dimensional array. But there are plenty of algorithms that
brute-force their way through problems with k^d scaling, for d=10 or so.
More information about the Numpy-discussion