[Numpy-discussion] Array concatenation performance
Thu Jul 15 18:31:52 CDT 2010
Sturla Molden wrote:
> This thread remids me of something I've though about for a while: Would
> NumPy benefit from an np.ndarraylist subclass of np.ndarray, that has an
> O(1) amortized append like Python lists?
yes, it would.
> (Other methods of Python lists
> (pop, extend) would be worth considering as well.) Or will we get the
> same performance by combining a Python list and ndarray?
I don't think so.
In general, the advice is to accumulate in a Python list, then turn it
into an array. And this works pretty well.
However, Python lists hold python objects, so it's bit inefficient, at
least in terms of memory use.
I wrote such an accumulating array a while back. It uses a numpy array
internally to store the data - oversized at first, and expanded as need
by with ndarray.resize()
The result was that is was a bit slower than the list approach in most
cases -- though should use less memory, and had an advantage is
accumulating things more complex numdata types.
I've thought about trying to write it in C or Cython, rather than Python
-- I think the extra function call overhead on indexing and all may be
fairly expensive. There may be other ways to optimize it as well.
It's also only 1-d, though it wouldn't be hard to make it n-d, but
expandable only on the first dimension.
I've enclosed my current draft, along with some code to test and help
Consider it a prototype, but I've used it successfully for a few things.
Christopher Barker, Ph.D.
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 4701 bytes
Desc: not available
Url : http://mail.scipy.org/pipermail/numpy-discussion/attachments/20100715/dde88a0e/attachment.zip
More information about the NumPy-Discussion