[Numpy-discussion] numpy.percentile multiple arrays

[questions anon]

I need some help understanding how to loop through many arrays to calculate
the 95th percentile.
I can easily do this by using numpy.concatenate to make one big array and
then finding the 95th percentile using numpy.percentile but this causes a
memory error when I want to run this on 100's of netcdf files (see code
below).
Any alternative methods will be greatly appreciated.


all_TSFC=[]
for (path, dirs, files) in os.walk(MainFolder):
    for dir in dirs:
        print dir
    path=path+'/'
    for ncfile in files:
        if ncfile[-3:]=='.nc':
            print "dealing with ncfiles:", ncfile
            ncfile=os.path.join(path,ncfile)
            ncfile=Dataset(ncfile, 'r+', 'NETCDF4')
            TSFC=ncfile.variables['T_SFC'][:]
            ncfile.close()
            all_TSFC.append(TSFC)

big_array=N.ma.concatenate(all_TSFC)
Percentile95th=N.percentile(big_array, 95, axis=0)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20120125/f0d2e515/attachment.html