[Numpy-discussion] Accessing array members

David M. Cooke cookedm at arbutus.physics.mcmaster.ca
Mon Dec 2 17:47:01 CST 2002


On Mon, Dec 02, 2002 at 11:59:22AM +0100, Joachim Saul wrote:
> Hi there,
> 
> given the following (simplified) scenario:
> 
> typedef struct {
>         PyObject_HEAD
>         float bar[10];
> } FooObject;
> 
> I want to be able to set and retrieve the elements of bar from
> Python using e.g.
> 
> >>> foo = Foo()
> >>> foo.bar[4] = 1.23
> >>> x = foo.bar[4]
> 
> I have chosen an approach using 'PyArray_FromDimsAndData'. In fact
> I programmed it after studying 'arrayobject.c', namely the part in
> 'array_getattr' where the 'flat' attribute is accessed.
> 
> foo_getattr(FooObject *self, char *name)
> {
>         if (!strcmp(name, "bar")) {
>                 int n=10;
>                 PyObject *bar = PyArray_FromDimsAndData(1, &n,
>                                     PyArray_FLOAT, (char*)self->bar);
>                 if (bar == NULL) return NULL;
>                 return bar;
>         }
> 
>         return Py_FindMethod(foo_methods, (PyObject*)self, name);
> }
> 
> And it works! :-)
> 
> BUT how about refcounts here? 'PyArray_FromDimsAndData' will
> return an array which only contains a reference to foo's original
> bar array; that's why I can both set and access the latter the way
> described. And no memory leak is created.
> 
> But what if I create a reference to foo.bar, and later delete foo,
> i.e.
> 
> >>> b = foo.bar
> >>> del foo
> 
> Now the data pointer in b refers to freed data! In the mentioned
> 'array_getattr' this apeears to be solved by increasing the
> refcount; in the above example this would mean 'Py_INCREF(self)'
> before returning 'bar'. Then if deleting 'foo', its memory is not
> freed because the refcount is not zero. But AFAICS in this case
> (as well as in the Numeric code) the INCREF prevents the object
> from EVER being freed. Who would DECREF the object?

Something similiar came up a few weeks ago: how do you pass data owned
by something else as a Numeric array, while keeping track of when to
delete the data?

It's so simple I almost kicked myself when I saw it, from the code at
http://pobox.com/~kragen/sw/arrayfrombuffer/
which allows you to use memory-mapped files as arrays.

The idea is that a PyArrayObject has a member 'base', which is DECREF'd
when the array is deallocated. The idea is for when arrays are slices of
other arrays, deallocating the slice will decrease the reference count
of the original. However, we can subvert this by using our own base,
that knows how to deallocate our data.

In your case, the DECREF'ing is all you need, so you could use

foo_getattr(FooObject *self, char *name)
{
        if (!strcmp(name, "bar")) {
                int n=10;
                PyObject *bar = PyArray_FromDimsAndData(1, &n,
                                    PyArray_FLOAT, (char*)self->bar);
                if (bar == NULL) return NULL;
                /***** new stuff here *******/
                Py_INCREF(self);
                ((PyArrayObject *)bar)->base = self;
                /***********/
                return bar;
        }

        return Py_FindMethod(foo_methods, (PyObject*)self, name);
}

So, now with
>>> b = foo.bar
>>> del foo
b will still reference the original foo object. Now, do
>>> del b
and foo's data will be DECREF'd, freeing it if b had the only reference.

This can be extended: say you've allocated memory from some memory pool
that has to be freed with, say, 'my_pool_free'. You can create a Numeric
array from this without copying by

PyArrayObject *A = (PyArrayObject *)PyArray_FromDimsAndData(1, dims,
                                      PyArray_DOUBLE, (char *)data);
A->base = PyCObject_FromVoidPtr(data, my_pool_free);

Then A will be a PyArrayObject, that, when the last reference is
deleted, will DECREF A->base, which will free the memory.

Easy, huh?

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                 http://arbutus.physics.mcmaster.ca/cookedm/
|cookedm at physics.mcmaster.ca




More information about the Numpy-discussion mailing list