[SciPy-User] operations on large arrays

Vincent Davis vincent@vincentdavis....
Sat Mar 6 23:22:56 CST 2010


I have arrays of 8-20 rows and 230,000 column, all the data is float64
I what to be able to find the difference in the correlation matrix between
arrays
let A and B be of size (10, 230000)
np.corrcoef(a)-np.corrcoef(b)

I can't seem to do this with more than 10000 columns at a time because of
memory limitations. (about 9GB usable to python)
Is there a better way?

I also have problem finding the column means which is surprising to me, I
was not able to get the column means for 10000 columns, but I can computer
 the corrcoeff ?
np.mean(a, axis=0)

Do I just need to divide up the job or is there a better approach?

Thanks

  *Vincent Davis
720-301-3003 *
vincent@vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100306/9a550d56/attachment.html 


More information about the SciPy-User mailing list