[SciPy-User] efficient way to store and use a 4D redundant matrix

Emanuele Olivetti emanuele@relativita....
Wed Mar 16 10:41:07 CDT 2011


Hi Everybody,

I have a 4D matrix "A" where entries differing only by a permutation of
indices are equal, i.e. A[1,2,3,4] = A[3,4,2,1]. Since each set of
of 4 indices I have then 24 permutations, my goal would be to avoid
storing 24 times the necessary space. I am looking at scipy.sparse but
I have no clear understanding whether it could handle my case.

Any idea?

For sake of clarity here is a simple code computing a matrix A like
the one above. I usually have A of shape (100,100,100,100) which requires
~800Mb when the dtype is double. Of course the non-redundant part is
just ~4Mb so you may understand my interest in this issue.

----
import numpy as np
from itertools import permutations
try:
     form itertools import combinations_with_replacement
except ImportError: # if Python < v2.7 that function is not available...
     form itertools import product
     def combinations_with_replacement(iterable, r):
         pool = tuple(iterable)
         n = len(pool)
         for indices in product(range(n), repeat=r):
             if sorted(indices) == list(indices):
                 yield tuple(pool[i] for i in indices)

rows = 10
columns = 20
x = np.random.rand(rows,columns)+1.0
A = np.zeros((rows, rows, rows, rows))

indices_rows = range(rows)
for i,j,k,l in combinations_with_replacement(indices_rows, 4):
     tmp = (x[i,:]*x[j,:]*x[k,:]*x[l,:]).sum()
     for a,b,c,d in permutations([i,j,k,l]):
         A[a,b,c,d] = tmp
----

In case you wonder which kind of operations do I need to do on A,
they are usual manypulations (slicing, reshaping, etc.), and common
linear algebra.

Best,

Emanuele



More information about the SciPy-User mailing list