[Numpy-discussion] ragged array implimentation

[Christopher Barker]

On 3/10/11 9:51 AM, Francesc Alted wrote:
> A Thursday 10 March 2011 18:05:11 Christopher Barker escrigué:
>> NOTE: this looks like it could use a "growable" numpy array, much
>> like one I've written before -- maybe it's time to revive that
>> project and use it here, fixing some performance issues while I'm at
>> it.
>
> A growable array would be indeed very useful in many situations.  My
> carray project [1] is another attempt to create such an object.  It
> grows (or shrink) arrays by adding (removing) chunks, so they are fast
> beasts for performing these operations.  I suppose this is more or less
> like your approach.

I don't think so -- my approach is a lot simpler that I think carray is:

it acts much like a python list:

1) it pre-allocates extra space when an array is created.
2) when you append, it uses that extra space
3) when the extra space is used up, it re-allocates the entire array, 
with some more extra room

And to keep it really simple, I'm using a numpy array internally, and 
np.ndarray.resize to grow it. It can only be grown on the first axis 
(C-order).

IIUC, carray is doing a lot more smarts about keeping the array in 
chunks that can be individually manipulated, which is very cool.NOt to 
mention the compression!

> I also thought about implementing ragged arrays in carray, like those
> you are suggesting, but not sure when they will be ready for public
> consumption.

That would be very cool, and I think you are right, the facilities in 
carray could make a really nice ragged array.

> Perhaps it is a good time to join efforts and create a nice 'growable'
> array package?

Sure, but I don't know that I have much to offer, except maybe some test 
and timing code.

By the way, it would be great to have a growable array that could be 
efficiently used in Cython code -- so you could accumulate a bunch of 
native C datatype data easily.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker@noaa.gov