[SciPy-User] Correlation coefficient of large arrays
Vincent Davis
vincent@vincentdavis....
Tue Mar 16 00:39:45 CDT 2010
>
> @Josef
how much memory does a
>>> 230000**2 = 52900000000L float (double) array take ?
I guess I don't have a real appreciation for how large this is. I can do
this numpy.ones((100000,50000),dtype=np.float64) and it uses about 85% of
the memory I have available. But thats a long ways from 230,000X230,000. Of
course the array is symmetric.
Is it feasible to do writing it to the disk?
The end goal is to find the difference between two correlation arrays and
then calculate the mean of each column. Which then leaves me with an array
1X230,000
*Vincent Davis
720-301-3003 *
vincent@vincentdavis.net
my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
On Mon, Mar 15, 2010 at 11:16 PM, <josef.pktd@gmail.com> wrote:
>
>
> On Tue, Mar 16, 2010 at 1:04 AM, Vincent Davis <vincent@vincentdavis.net>wrote:
>
>> I have an array 10 observations of 230,000 variables and what to find the
>> correlation coefficient between each variable.
>> numpy.corrcef(data) works except I can only do it with about 30,000
>> variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of
>> memory.
>> Is there a better way?
>>
>
>
> how much memory does a
> >>> 230000**2
> 52900000000L
>
> float (double) array take ?
>
> Josef
> (I'm not going to try)
>
>
>
>>
>> *Vincent Davis
>> 720-301-3003 *
>> vincent@vincentdavis.net
>> my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20100315/1b285d4a/attachment.html
More information about the SciPy-User
mailing list