[SciPy-user] Large matrix support
Dinesh B Vadhia
dineshbvadhia@hotmail....
Fri Apr 25 21:40:11 CDT 2008
Nathan / Alan
Here is how I'm constructing the sparse matrices:
> import numpy
> import scipy
> from scipy import sparse
> # create matrix A using the scipy.sparse coo_matrix method. A is very sparse <10%
> ij = numpy.array(numpy.zeros((nnz, 2), dtype=numpy.int))
<snip ... read data into ij>
> row = ij[:,0]
> column = ij[:,1]
> n = ij.shape[0]
> data = scipy.ones(n, dtype=numpy.int8)
> A = sparse.coo_matrix((data, (row, column)), shape=(I,J)) # changed from dims to shape for new svn
> A = sparse.csr_matrix(X)
> f = open('A.pkl', 'wb') # pickle the sparse matrix
> pickle.dump(A, f, 2)
> f.close()
> f = open('A.pkl', 'rb') # unpickle the sparse matrix
> A = pickle.load(f)
> f.close()
> I = A.shape[0]
> J = A.shape[1]
<snip ... a and b are two scalar vectors>
> b = A*x
------------------------------
Message: 4
Date: Fri, 25 Apr 2008 19:16:04 -0400
From: "Alan McIntyre" <alan.mcintyre@gmail.com>
Subject: Re: [SciPy-user] Large matrix support
To: "SciPy Users List" <scipy-user@scipy.org>
Message-ID:
<1d36917a0804251616q4babd57crf8acc2b47c6b9072@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On Fri, Apr 25, 2008 at 5:01 PM, Dinesh B Vadhia
<dineshbvadhia@hotmail.com> wrote:
> My matrix sizes can be large ie. > minimum 20,000 rows x 1m columns.
>
> When just creating these data sets using Python/Numpy/Scipy, I get
> pythonw.exe errors and the program aborts. I guess the Microsoft 32K limit
> has been reached somewhere.
>
> I haven't got any Python program to run with these data sets because of 'not
> enough memory' errors. I'm waiting for the int8 support in the sparse
> library to hopefully resolve that issue.
Dinesh,
I apologize if this is too basic a question to ask, but are the
matrices sparse? Could you post some sample code to show how you're
trying to construct these matrices?
Thanks,
Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/scipy-user/attachments/20080425/e3e094e5/attachment.html
More information about the SciPy-user
mailing list