[Numpy-discussion] Possible roadmap addendum: building better text file readers

[Erin Sheldon]

Excerpts from Warren Weckesser's message of Sun Feb 26 16:22:35 -0500 2012:
> Yes, thanks!   I'm working on a mmap version now.  I'm very curious to see
> just how much of an improvement it can give.

FYI, memmap is generally an incomplete solution for numpy arrays; it
only understands rows, not columns and rows.  If you memmap a rec array
on disk and try to load one full column, it still loads the whole file
beforehand.

This was why I essentially wrote my own memmap like interface with
recfile, the code I'm converting.  It allows working with columns and
rows without loading large chunks of memory.

BTW, I think we will definitely benefit from merging some of our codes.
When I get my stuff fully converted we should discuss.

-e
-- 
Erin Scott Sheldon
Brookhaven National Laboratory