[Numpy-discussion] Help using numPy to create a very large multi dimensional array

[Christopher Barker]

Lisandro Dalcin wrote:
> I am also +1 on this, but this functionality should be implemented in
> C, I think.

well,  maybe.

> I've just tested numpy.fromfile('name.txt', sep=' ')
> against pylab.load('name.txt') for a 35MB text file, the number are:
> 
> numpy.fromfile: 2.66 sec.
> pylab.load:  16.64 sec.

exactly that's expected. fromfile is designed to do the easy cases as 
fast as possible, pylab.load is designed to be be flexible, I'm not user 
you need both the speed and flexibility at the same time.

By the way, I haven't looked at pylab.load() for a while, but it could 
perhaps be sped up by using fromfile() and or fromstring internally. 
There may be some opportunity to special case the easy ones too (i.e. 
all columns desired, etc.)

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov