[SciPy-User] efficient way to store and use a 4D redundant matrix
Mon Mar 28 15:48:34 CDT 2011
If the data is easily compressible maybe you can use carray? It
keep data compressed in memory and allow to make some operation
direction on the compressed data.
On Fri, Mar 18, 2011 at 6:01 AM, Emanuele Olivetti
> On 03/16/2011 08:31 PM, Daniel Lepage wrote:
>> The sparse matrix formats will only help you if you can rewrite A in
>> terms of matrices that are mostly 0.
> Correct. This is not my case, you are right.
>> Do you need the results of slicing, reshaping, etc. to also be
>> similarly compressed? If so, I can't see any way to implement this
>> without an index array, because once you reshape or slice A you won't
>> know which cells correspond to which indices in the original A.
> I will have a deeper look to a solution with index array. Thanks for
> pointing it out.
>> If you're only taking small slices of this and then applying linear
>> algebra operations to those, you might be better off writing a class
>> that looks up the relevant values on the fly; you could overload
>> __getitem__ so that e.g. A[:,1,:,3] would generate the correct float64
>> array on the fly and return it.
> Unfortunately I am not playing with small slices. So I guess that overloading
> __getitem__ would be impractical.
>> However, if the nonredundant part takes only ~4MB, then maybe I don't
>> understand your layout - for a 100x100x100x100 and 64-bit floats, I
>> think the nonredundant part should take ((100 choose 4) + ((100 choose
>> 3) * 3) + ((100 choose 2) * 3) + (100 choose 1)) * 8 bytes = about
>> 34MB. Was that a math error, or am I misunderstanding the question?
> My fault. It is indeed ~34Mb. I missed one order of magnitude when
> computing (100**4 * 8byte) / 24 .
> Thanks again,
> SciPy-User mailing list
More information about the SciPy-User