[SciPy-user] sparse csr_matrix memory error

Dinesh B Vadhia dineshbvadhia@hotmail....
Tue Aug 12 00:31:17 CDT 2008


Hi Nathan

For future reference, how did you arrive at the 864MB?

Dinesh




--------------------------------------------------------------------------------
Message: 2
Date: Sun, 10 Aug 2008 18:39:13 -0400
From: "Nathan Bell" <wnbell@gmail.com>
Subject: Re: [SciPy-user] sparse csr_matrix memeory error
To: "SciPy Users List" <scipy-user@scipy.org>
Message-ID:
<d05265cb0808101539l862ece2i8e97894afc522f37@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Aug 10, 2008 at 5:29 PM, Dinesh B Vadhia
<dineshbvadhia@hotmail.com> wrote:
> I'm obtaining a memory error when creating a large sparse csr matrix as
> follows:
>
> I = 680000
> J = 900000
> nnz = 72000000
> row = numpy.empty(nnz, dtype='intc')
> column = numpy.empty(nnz, dtype='intc')
> # read (i,j) data into row and column
> data = scipy.ones(nnz, dtype='intc')

Together, the arrays for row,column, and data take 864MB of memory.
You need approximately 2x that to do the conversion to CSR.

> I'm running the program under Windows XP with over 2gb memory.  Any thoughts
> on what the problem is?

Yes, your matrix is simply too large.

-- 
Nathan Bell wnbell@gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/scipy-user/attachments/20080811/1e6617a8/attachment.html 


More information about the SciPy-user mailing list