[Numpy-discussion] Efficient way to load a 1Gb file?

Russell E. Owen rowen@uw....
Mon Aug 15 15:25:50 CDT 2011

In article 
 Torgil Svensson <torgil.svensson@gmail.com> wrote:

> Try the fromiter function, that will allow you to pass an iterator
> which can read the file line by line and not preload the whole file.
> file_iterator = iter(open('filename.txt')
> line_parser = lambda x: map(float,x.split('\t'))
> a=np.fromiter(itertools.imap(line_parser,file_iterator),dtype=float)
> You have also the option to iterate the file twice and pass the
> "count" argument.

Thanks. That sounds great!

-- RUssell

More information about the NumPy-Discussion mailing list