[Numpy-discussion] fromstring, tostring slow?

Christopher Barker Chris.Barker@noaa....
Tue Feb 13 18:07:03 CST 2007


Mark Janikas wrote:
> I don't think I can do that because I have heterogeneous rows of
> data.... I.e. the columns in each row are different in length.

like I said, show us your whole problem...

But you don't have to write.read all the data at once with from/tofile() 
anyway. Each of your "rows" has to be in a separate array anyway, as 
numpy arrays don't support "ragged" arrays, but each row can be written 
with tofile()

> Furthermore, when reading it back in, I want to read only bytes of the
> info at a time so I can save memory.  In this case, I only want to have
> one record in mem at once.

you can make multiple calls to fromfile(), thou you'll have to know how 
long each record is.

> Another issue has arisen from taking this routine cross-platform....
> namely, if I write the file on Windows I cant read it on Solaris.  I
> assume the big-little endian is at hand here.

yup.

> I know using the struct
> module that I can pack using either one.

so can numpy. see the "byteswap" method, and you can specify a 
particular endianess with a datatype when you read with fromfile():

a = N.fromfile(DataFile, dtype=N.dtype("<d"), count=20)

reads 20 little-endian doubles from DataFile, regardless of the native 
endianess of the machine you're on.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov


More information about the Numpy-discussion mailing list