[Numpy-discussion] Array concatenation performance

[Christopher Barker]

Sturla Molden wrote:
> This thread remids me of something I've though about for a while: Would 
> NumPy benefit from an np.ndarraylist subclass of np.ndarray, that has an 
> O(1) amortized append like Python lists?

yes, it would.

> (Other methods of Python lists 
> (pop, extend) would be worth considering as well.) Or will we get the 
> same performance by combining a Python list and ndarray?

I don't think so.

In general, the advice is to accumulate in a Python list, then turn it 
into an array. And this works pretty well.

However, Python lists hold python objects, so it's bit inefficient, at 
least in terms of memory use.

I wrote such an accumulating array a while back. It uses a numpy array 
internally to store the data - oversized at first, and expanded as need 
by with ndarray.resize()

The result was that is was a bit slower than the list approach in most 
cases -- though should use less memory, and had an advantage is 
accumulating things more complex numdata types.

I've thought about trying to write it in C or Cython, rather than Python 
-- I think the extra function call overhead on indexing and all may be 
fairly expensive. There may be other ways to optimize it as well.

It's also only 1-d, though it wouldn't be hard to make it n-d, but 
expandable only on the first dimension.

I've enclosed my current draft, along with some code to test and help 
profile it.

Consider it a prototype, but I've used it successfully for a few things.

-Chris








-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Accumulator.zip
Type: application/zip
Size: 4701 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/numpy-discussion/attachments/20100715/dde88a0e/attachment.zip