[SciPy-User] efficient way to store and use a 4D redundant matrix

Emanuele Olivetti emanuele@relativita....
Fri Mar 18 05:01:30 CDT 2011

On 03/16/2011 08:31 PM, Daniel Lepage wrote:
> [...]
> The sparse matrix formats will only help you if you can rewrite A in
> terms of matrices that are mostly 0.

Correct. This is not my case, you are right.

> Do you need the results of slicing, reshaping, etc. to also be
> similarly compressed? If so, I can't see any way to implement this
> without an index array, because once you reshape or slice A you won't
> know which cells correspond to which indices in the original A.

I will have a deeper look to a solution with index array. Thanks for
pointing it out.

> If you're only taking small slices of this and then applying linear
> algebra operations to those, you might be better off writing a class
> that looks up the relevant values on the fly; you could overload
> __getitem__ so that e.g. A[:,1,:,3] would generate the correct float64
> array on the fly and return it.

Unfortunately I am not playing with small slices. So I guess that overloading
__getitem__ would be impractical.

> However, if the nonredundant part takes only ~4MB, then maybe I don't
> understand your layout - for a 100x100x100x100 and 64-bit floats, I
> think the nonredundant part should take ((100 choose 4) + ((100 choose
> 3) * 3) + ((100 choose 2) * 3) + (100 choose 1)) * 8 bytes = about
> 34MB. Was that a math error, or am I misunderstanding the question?

My fault. It is indeed ~34Mb. I missed one order of magnitude when
computing (100**4 * 8byte) / 24 .

Thanks again,


More information about the SciPy-User mailing list