[SciPy-user] Reading in data as arrays, quickly and easily?

Sebastian Haase haase at msg.ucsf.edu
Fri Jul 9 15:17:02 CDT 2004


Hi Eric,
I assume you talk about Numeric, but in case you are open for numarray I use 
numarray's memmap quite successfully on files even larger than 1 GB (Linux; I 
think the effective limit for Windows might be lower ). It works for all 
datatypes and for byteswapped data too. You can skip any amount of bytes by 
having your mem-"slice" start at any offset you want. I actually  map the 
first part into a record-array so that I can read the parts of the 
"header"-information I'm interested in.

Regards,
Sebastian Haase

On Friday 09 July 2004 12:54 pm, Eric Jonas wrote:
> Hello! I'm trying to read in large chunks of binary data as arrays, but
> the file formats are complex enough that there is lots of junk that
> needs to be skipped over. I have a functioning datafile object in python
> with a read(N) method that returns the next N data points in the file,
> doing the various raw manipulations, endian conversions, and the like
> internally.
>
> The problem is that it's really really slow. I've spent most of today
> playing with boost.python, weave, raw numeric and numarray integration,
> and the like. And yet I still can't figure out what I should -really- be
> using.
>
> Does anyone have any suggestions for approaches that have worked for
> them? Ideally I'd love to read in my data and perform really basic
> preprocessing in c++ and then return the resulting arrays to manipulate
> using python. I'd rather not spend the weekend reinventing the wheel...
>
> Thanks!
> 			...Eric
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.net
> http://www.scipy.net/mailman/listinfo/scipy-user



More information about the SciPy-user mailing list