[Numpy-discussion] dimensions too large error
Fri Mar 14 23:33:51 CDT 2008
On 14/03/2008, Dinesh B Vadhia <email@example.com> wrote:
> For the following code:
> I = 18000
> J = 33000
> filename = 'ij.txt'
> A = scipy.asmatrix(numpy.empty((I,J), dtype=numpy.int))
> for line in open(filename, 'r'):
> The following message appears:
> Traceback (most recent call last):
> File "C:\...\....py", line 362, in <module>
> A= scipy.asmatrix(numpy.empty((I,J), dtype=numpy.int))
> ValueError: dimensions too large.
> Is there a limit to array/matrix dimension sizes?
Yes. On 32-bit machines the hardware makes it exceedingly difficult
for one process to access more than two or three gigabytes of RAM, so
numpy's strides and sizes are all 32-bit integers. As a result you
can't make arrays bigger than about 2 GB. If you need this I'm afraid
you pretty much need a 64-bit machine.
> Btw, for numpy array's, ascontiguousarray() is available to set aside
> contiguous memory. Is there an equivalent for scipy matrix ie. an
That's not exactly what ascontiguousarray() is for. Normally if you
create a fresh numpy array it will be allocated as a single contiguous
block of memory, and the strides will be arranged in that special way
that numpy calls "contiguous". However, if you take a subarray of that
array - every second element, say - the resulting data is not
contiguous (in the sense that there are gaps between the elements).
Normally this is not a problem. But a few functions - mostly
handwritten C or FORTRAN code - needs contiguous arrays. The function
ascontiguousarray() will return the original array if it is contiguous
(and in C order), or make a copy if it isn't.
Numpy matrices are just a slight redefinition of the operators on an
array, so you can always convert an array to a matrix without copying.
Thus there's little cost (for large arrays) to just using
matrix(ascontiguousarray()) if you need matrices.
More information about the Numpy-discussion