[SciPy-User] 64-bit matrix indices

braingateway braingateway@gmail....
Tue Nov 16 16:17:14 CST 2010


Nathaniel Smith :
> On Tue, Nov 16, 2010 at 8:09 AM, Ioan-Alexandru Lazar <alexlz@lmn.pub.ro> wrote:
>   
>> I am trying to use SciPy for one of my HPC projects. A problem I am
>> currently facing is that 32-bit indices are too small for the matrix sizes
>> we require. Is there any way to use 64-bit ints for the indices?
>>     
> [...]
>   
>> I only need a few sparse operations and .mat file reading from it.
>>     
>
> You're asking specifically about the indices in scipy.sparse matrices, yes?
>
> At a quick glance, all the core matrix manipulation code in
> scipy.sparse seems to be templatized with respect to the type of the
> index -- you *might* be able to get 64-bit index support for all the
> core sparse matrix operations by editing
> scipy/sparse/sparsetools/sparsetools.i and adding the obvious stuff at
> around lines 145, 188, and 195, and then creating your matrices "by
> hand" (i.e., building your own indices and indptr arrays of 64-bit
> integers, and then passing them directly to the csc_matrix/csr_matrix
> constructors). The .mat file reader is potentially more tricky, but it
> sounds like you could read them in with 32-bit indices and then just
> convert them to 64-bit:
>   mymat.indptr = np.array(mymat.indptr, dtype=np.int64)
>   mymat.indices = np.array(mymat.indices, dtype=np.int64)
>
>   
>> If anyone is interested on the background story: the matrices themselves
>> aren't too big *at first*, but due to the peculiar structure they have,
>> the fill-in is mind-blowing. UMFPACK complaints that it doesn't have
>> enough memory for them; it does (our cluster's nodes have 24 GB of
>> memory), but once the number of indices blows past the 32-bit limit, it
>> hits the ceiling. Using a different solver is still not an option
>>     
>
> You'd also need a way to call UMFPACK's 64-bit functions (the "zl/dl"
> variants instead of the "zi/di" variants). It looks like
> scikits.umfpack might let you do this easily, but I'm not sure.
>
> -- Nathaniel
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>   
Humm, wait for the try-out result. I might encounter the same problem in 
near future.

LittleBigBrain


More information about the SciPy-User mailing list