[Numpy-discussion] Possible roadmap addendum: building better text file readers
Sun Feb 26 16:35:00 CST 2012
Excerpts from Warren Weckesser's message of Sun Feb 26 16:22:35 -0500 2012:
> Yes, thanks! I'm working on a mmap version now. I'm very curious to see
> just how much of an improvement it can give.
FYI, memmap is generally an incomplete solution for numpy arrays; it
only understands rows, not columns and rows. If you memmap a rec array
on disk and try to load one full column, it still loads the whole file
This was why I essentially wrote my own memmap like interface with
recfile, the code I'm converting. It allows working with columns and
rows without loading large chunks of memory.
BTW, I think we will definitely benefit from merging some of our codes.
When I get my stuff fully converted we should discuss.
Erin Scott Sheldon
Brookhaven National Laboratory
More information about the NumPy-Discussion