[Numpy-discussion] genloadtxt: second serving

Pierre GM pgmdevlist@gmail....
Thu Dec 4 05:51:53 CST 2008


All,
Here's the second round of genloadtxt. That's a tad cleaner version  
than the previous one, where I tried to take  into account the  
different comments and suggestions that were posted. So, tabs should  
be supported and explicit whitespaces are not collapsed.
FYI, in the __main__ section, you'll find 2 hotshot tests and a timeit  
comparison: same input, no missing data, one with genloadtxt, one with  
np.loadtxt and a last one with matplotlib.mlab.csv2rec.

As you'll see, genloadtxt is roughly twice slower than np.loadtxt, but  
twice faster than csv2rec. One of the explanation for the slowness is  
indeed the use of classes for splitting lines and converting values.  
Instead of a basic function, we use the __call__ method of the class,  
which itself calls another function depending on the attribute values.  
I'd like to reduce this overhead, any suggestion is more than welcome,  
as usual.

Anyhow: as we do need speed, I suggest we put genloadtxt somewhere in  
numpy.ma, with an alias recfromcsv for John, using his defaults.  
Unless somebody comes with a brilliant optimization.

Let me know how it goes,
Cheers,
P.




-------------- next part --------------
A non-text attachment was scrubbed...
Name: _preview.py
Type: text/x-python-script
Size: 31694 bytes
Desc: not available
Url : http://projects.scipy.org/pipermail/numpy-discussion/attachments/20081204/f07f3c7a/attachment-0001.bin 
-------------- next part --------------



More information about the Numpy-discussion mailing list