[SciPy-User] Correlation coefficient of large arrays

Vincent Davis vincent@vincentdavis....
Tue Mar 16 00:39:45 CDT 2010


>
> @Josef

how much memory does a

>>> 230000**2 = 52900000000L  float (double) array take ?



I guess I don't have a real appreciation for how large this is. I can do
this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
the memory I have available. But thats a long ways from 230,000X230,000. Of
course the array is symmetric.

Is it feasible to do writing it to the disk?
The end goal is to find the difference between two correlation arrays and
then calculate the mean of each column. Which then leaves me with an array
1X230,000

  *Vincent Davis
720-301-3003 *
vincent@vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd@gmail.com> wrote:

>
>
> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <vincent@vincentdavis.net>wrote:
>
>> I have an array 10 observations of 230,000 variables and what to find the
>> correlation coefficient between each variable.
>> numpy.corrcef(data) works except I can only do it with about 30,000
>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>> memory.
>> Is there a better way?
>>
>
>
> how much memory does a
> >>> 230000**2
> 52900000000L
>
> float (double) array take ?
>
> Josef
> (I'm not going to try)
>
>
>
>>
>>   *Vincent Davis
>> 720-301-3003 *
>> vincent@vincentdavis.net
>>  my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/1b285d4a/attachment.html 


More information about the SciPy-User mailing list