[SciPy-User] memory error - numpy mean - netcdf4

questions anon questions.anon@gmail....
Mon Aug 29 17:55:50 CDT 2011


Thanks for all of the responses. I have tried adding in the code you
mentioned (see below). I am not sure if I am putting it in the correct
place? and I am now receiving another error:
"UserWarning: Warning: converting a masked element to nan."
Not sure if that is bringing me any closer? Any feedback will be greatly
appreciated.

from netCDF4 import Dataset
import matplotlib.pyplot as plt
import numpy as N
from mpl_toolkits.basemap import Basemap
import os

MainFolder=r"E:/DSE_BushfireClimatologyProject/griddeddatasamples/GriddedData/T_SFC/"
all_TSFC=[]
for (path, dirs, files) in os.walk(MainFolder):
    for dir in dirs:
        print dir
    path=path+'/'
    for ncfile in files:
        if ncfile[-3:]=='.nc':
            ncfile=os.path.join(path,ncfile)
            ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
            TSFC=ncfile.variables['T_SFC'][4::24,:,:]
            LAT=ncfile.variables['latitude'][:]
            LON=ncfile.variables['longitude'][:]
            #TIME=ncfile.variables['time'][:]
            fillvalue=ncfile.variables['T_SFC']._FillValue
            ncfile.close()

            array=N.true_divide(TSFC[0],len(TSFC))

            for i in xrange(1, len(TSFC)-1,1):
                            array=N.add(array,
N.true_divide(array[i],len(TSFC)))

#plot output summary stats
map = Basemap(projection='merc',llcrnrlat=-40,urcrnrlat=-33,
              llcrnrlon=139.0,urcrnrlon=151.0,lat_ts=0,resolution='i')

x,y=map(*N.meshgrid(LON,LAT))
plt.title('TSFC Mean at 3pm')
ticks=[-5,0,5,10,15,20,25,30,35,40,45,50]
CS = map.contourf(x,y,array, cmap=plt.cm.jet)
l,b,w,h =0.1,0.1,0.8,0.8
cax = plt.axes([l+w+0.025, b, 0.025, h])
plt.colorbar(CS,cax=cax, drawedges=True)

plt.savefig((os.path.join(MainFolder, 'Mean.png')))
plt.show()
plt.close()





On Sat, Aug 27, 2011 at 10:54 AM, srean <srean.list@gmail.com> wrote:

>
> On Fri, Aug 26, 2011 at 2:33 PM, Phil Morefield <philmorefield@yahoo.com>wrote:
>
>>
>> The formula you have written looks like you're collapsing everything into
>> a single value. I think he's trying to average a bunch of 2D arrays into a
>> single 2D array.
>>
>
> You are correct, the form that I posted can be read as if it is  for
> updating single mean vector \mu, but you can use the same for an nd-array
> trivially. Just have \mu and t as nd-arrays. m can be one too.  Numpy
> broadcasting will take care of the rest.
>
> One advantage is that it requires only a constant amount of memory for the
> computation, you can even read the data in from an infinite pipe or
> generator that yields a single vector or a matrix at a time (or bundles them
> up m at a time). It will always be uptodate with the current estimate of the
> means. In fact will work for any moment too.
>
> --srean
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/scipy-user/attachments/20110830/7954d9d7/attachment.html 


More information about the SciPy-User mailing list