[Numpy-discussion] load from text files Pull Request Review
Thu Sep 8 15:43:43 CDT 2011
On Wed, Sep 7, 2011 at 2:52 PM, Chris.Barker <Chris.Barker@noaa.gov> wrote:
> On 9/2/11 2:45 PM, Christopher Jordan-Squire wrote:
>> It doesn't have to parse the entire file to determine the dtypes. It
>> builds up a regular expression for what it expects to see, in terms of
>> dtypes. Then it just loops over the lines, only parsing if the regular
>> expression doesn't match. It seems that a regex match is fast, but a
>> regex fail is expensive.
> interesting -- I wouldn't have expected a regex to be faster that simple
> parsing, but that's why you profile!
>> Setting array elements is not as fast for the masked record arrays.
>> You must set entire rows at a time, so I have to build up each row as
>> a list, and convert to a tuple, and then stuff it in the array.
> hmmm -- that is a lot -- I was thinking of simple "set a value in an
> array". I"ve also done a bunch of this in C, where's it's really fast.
> However, rather than:
> build a row as a list
> build a row as a tuple
> stuff into array
> could you create an empty array scalar, and fill that, then put that in
> your array:
> In : dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)])
> In : dt
> Out: dtype([('x', '<f4'), ('y', '<i4'), ('z', '<f8')])
> In : temp = np.empty((), dtype=dt)
> In : temp['x'] = 3
> In : temp['y'] = 4
> In : temp['z'] = 5
> In : a = np.zeros((4,), dtype = dt)
> In : a = temp
> In : a
> array([(3.0, 4, 5.0), (0.0, 0, 0.0), (0.0, 0, 0.0), (0.0, 0, 0.0)],
> dtype=[('x', '<f4'), ('y', '<i4'), ('z', '<f8')])
> (and you could pass the array scalar into accumulator as well)
> maybe it wouldn't be any faster, but with re-using temp, and one less
> list-tuple conversion, and fewer python type to numpy type conversions,
> maybe it would.
I just ran a quick test on my machine of this idea. With
dt = np.dtype([('x',np.float32),('y', np.int32),('z', np.float64)])
temp = np.empty((), dtype=dt)
temp2 = np.zeros(1,dtype=dt)
In : def f():
...: l = 2.54
...: l = 4
...: l = 2.3645
...: j = tuple(l)
...: temp2 = j
In : def g():
...: temp['x'] = 2.54
...: temp['y'] = 4
...: temp['z'] = 2.3645
...: temp2 = temp
The timing results were 2.73 us for f and 3.43 us for g. So good idea,
but it doesn't appear to be faster. (Though the difference wasn't
nearly as dramatic as I thought it would be, based on Pauli's
>> it's even slower for the record arrays with missing data because I
>> must branch between adding missing data versus adding real data. Might
>> that be the reason for the slower performance than you'd expect?
> could be -- I haven't thought about the missing data part much.
>> I wonder if there are any really important cases where you'd actually
>> lose something by simply recasting an entry to another dtype, as Derek
> In general, it won't be a simple re-cast -- it will be a copy to a
> subset -- which may be hard to write the code, but would save having to
> re-parse the data.
> Anyway, you know the issues, this is good stuff either way.
> Christopher Barker, Ph.D.
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
> NumPy-Discussion mailing list
More information about the NumPy-Discussion