[SciPy-User] fancy indexation of sparse matrices is terrible slow
Dmitrey
tmp50@ukr....
Mon Dec 14 14:51:20 CST 2009
От кого: Nathan Bell <wnbell@gmail.com>
The CSR and CSC formats also support fast fancy indexing. Something like:
>>> A = ... some sparse matrix
>>> csr_matrix(A)[I,J]
ought to work well.
I have tried all 3, lil works faster than any others, but casting to numpy ndarray and invoking it on the dense array obtained works several times faster, that is extremely important for me. Thus on my problem with sparse matrix of shape 151* 1178
each time casting it to dense before applying fancy indexing makes numerical optimization ipopt solver works faster from 130 to 60 sec. Here's the code that demonstrates the difference:
from scipy.sparse import lil_matrix, find
from numpy import where, zeros, prod, vstack, ones
from time import time
n, m = 1000, 20000
M = vstack((zeros((n, m)), ones((25, m))))
I, J = where(M)
# 1: time for lil
M = lil_matrix(M)
print 'sparsity of the matrix:', float(M.size)/prod(M.shape)
t = time()
M[I, J]
print('time elapsed with lil: %f' % (time()-t))
# 2: time for dense
t = time()
M = M.A # even with this time elapsed is less
M[I, J]
print('time elapsed with ndarray: %f' % (time()-t))
# output:
# sparsity of the matrix: 0.0243902439024
# time elapsed with lil: 17.006631
# time elapsed with ndarray: 0.710046
# as for csr and csc fancy indexation it works several times slower
So, I have filed a ticket for it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20091214/f02b8a35/attachment.html
More information about the SciPy-User
mailing list