[SciPy-User] SPARSE matrix dtypes, upcasting, sum function

David Cournapeau cournape@gmail....
Thu Sep 8 08:28:48 CDT 2011


On Thu, Sep 8, 2011 at 8:35 AM, Dinesh B Vadhia
<dineshbvadhia@hotmail.com> wrote:
> We have:
>
> I > 250000, J > 250000, nnz>10000000
>
> data = scipy.ones(nnz, dtype=numpy.uint8)
> A = sparse.csr_matrix((data, (xrow, xcolumn)), shape=(I,J))
>
> where xrow and xcolumn are int vectors of length nnz
>
> The row and column sums are:
> rowsum = A.sum(0)
> columnsum = A.sum(1)
>
> The max value given for each by Scipy are:
> rowsum .max() = 255
> columnsum .max() = 255
>
> But, the real values are:
> rowsum .max() = 41190
> columnsum .max() = 1080
>
> Can someone see what we are doing wrong?

It is at least a documentation bug, and I would have expected
upcasting as well. Note however that using integer will always have
some potential overflow issues, which are platform dependent (because
the default upcasting rules will use different sizes on different
platforms).

For example:

import numpy as np
a = 1024 * np.ones((4e6, 2), dtype=np.int16)
a.sum(0)

will give you the right answer on a 64 bits python on mac os x, but
the wrong one on 32 bits. As soon as you are doing operations which
can potentially overflow, I would advise to convert to float values.

cheers,

David


More information about the SciPy-User mailing list