[Numpy-discussion] Cython issues w/ 1.4.0

[Pauli Virtanen]

ti, 2009-12-08 kello 21:52 +0100, Dag Sverre Seljebotn kirjoitti:
[clip] 
> How about this:
>   - Cache/store the format string in a bytes object in a global 
> WeakRefKeyDict (?), keyed by dtype
>   - The array holds a ref to the dtype, and the Py_buffer holds a ref to 
> the array (through the obj field).

Yep, storage in a static variable is the second alternative. We can even
handle allocation and deallocation manually.

I think I'll make it this way then.

> Alternatively, create a new Python object and stick it in the "obj" in 
> the Py_buffer, I don't think obj has to point to the actual object the 
> buffer was acquired from, as long as it keeps alive a reference to it 
> somehow (though I didn't find any docs for the obj field, it was added 
> as an afterthought by the implementors after the PEP...). But the only 
> advantage is not using weak references (if that is a problem), and it is 
> probably slower and doesn't cache the string.

The current implementation of MemoryView in Python assumes that you can
call PyObject_GetBuffer(view->obj).

But this can be changed, IIRC, the memoryview has also a ->base member
containing the same info.

> >> - We need to cache the buffer protocol format string somewhere,
> >>  if we do not want to regenerate it on each buffer acquisition.
> > 
> > My suspicion is that YAGNI. I would wait until it is actually in use
> > and we see whether it takes up a significant amount of time in actual
> > code.

Sure, it's likely that it won't be a real problem performance-wise, as
it's simple C code.

The point is that the format string needs to be stored somewhere for
later deallocation, and to work around bugs in Python, we cannot put it
in Py_buffer where it would naturally belong to.

But anyway, it may really be best to not pollute object structs because
of a need for workarounds -- I suppose if I submit patches to Python
soon, they may make it in releases before Numpy 1.5.0 rolls out. For
backward compatibility, we'll just make do with static variables.

Ok, the reserved-for-future pointers in structs may not then be needed
after all, at least for this purpose.

-- 
Pauli Virtanen