[Numpy-discussion] Getting C-function pointers from Python to C

Dag Sverre Seljebotn d.s.seljebotn@astro.uio...
Thu Apr 12 17:08:12 CDT 2012


On 04/12/2012 11:55 PM, Travis Oliphant wrote:
>
> On Apr 12, 2012, at 4:51 PM, Dag Sverre Seljebotn wrote:
>
>> On 04/12/2012 11:13 PM, Travis Oliphant wrote:
>>> Dag,
>>>
>>> Thanks for the link to your CEP.  This is the first time I've seen it.   You probably referenced it before, but I hadn't seen it.
>>>
>>> That CEP seems along the lines of what I was thinking of.    We can make scipy follow that CEP and NumPy as well in places that it needs function pointers.
>>>
>>> I can certainly get behind it with Numba and recommend it to SciPy (and write the scipy.integrate.quad function to support it).
>>>
>>> Thanks for the CEP.
>>
>> Great. I'll pass this message on to the Cython list and see if anybody
>> wants to provide input (but given the scope, it should be minor tweaks
>> and easy to accommodate in whatever code you write).
>>
>> You will fill in more of the holes as you implement this in Numba and
>> SciPy of course (my feeling is they will support it before Cython; let's
>> say I hope this happens within the next year).
>
> Very nice.  This will help immensely I think.    It's actually just what I was looking for.     Just to be clear, by " pad to sizeof(void*) alignment", you mean that after the first 2 bytes there are (sizeof(void*) - 2)  bytes before the first function pointer in the memory block pointed to by the PyCObject / Capsule?

You are only right if the starting address of the data is divisible by 
sizeof(void*). On 64-bit you would do something like

func_ptr = (func_ptr_t*)((char*)descriptor & 0xfffffffffffffff7) + 8)

Hmm, not sure if I like it any longer. I don't know a priori how much 
alignment really matters either on modern CPUs (but in the Cython case, 
we would like this to be somewhat competitive with compile-time binding, 
so it does merit checking I think).

Dag

>
> Thanks,
>
> -Travis
>
>
>>
>> Dag
>>
>>
>>
>>
>>>
>>> -Travis
>>>
>>>
>>>
>>>
>>>
>>> On Apr 12, 2012, at 2:08 PM, Dag Sverre Seljebotn wrote:
>>>
>>>> On 04/12/2012 07:24 PM, Nathaniel Smith wrote:
>>>>> On Wed, Apr 11, 2012 at 10:23 PM, Travis Oliphant<teoliphant@gmail.com>    wrote:
>>>>>>>> In the mean-time, I think we could do as Robert essentially suggested and just use Capsule Objects around an agreed-upon simple C-structure:
>>>>>>>>
>>>>>>>>       int   id   /* Some number that can be used as a "type-check" */
>>>>>>>>       void *func;
>>>>>>>>       char *string;
>>>>>>>>
>>>>>>>> We can then just create some nice functions to go to and from this form in NumPy ctypeslib and then use this while the Python PEP gets written and adopted.
>>>>>>>
>>>>>>> What is not clear to me is how one get from the Python callable to the
>>>>>>> capsule.
>>>>>>
>>>>>> This varies substantially based on the tool.   Numba would do it's work and create the capsule object using it's approach.   Cython would use a different approach.
>>>>>>
>>>>>> I would also propose to have in NumPy some basic functions that go back-and forth between this representation, ctypes, and any other useful representations that might emerge.
>>>>>>
>>>>>>>
>>>>>>> Or do you simply intend to pass a non-callable capsule as an argument in
>>>>>>> place of the callback?
>>>>>>
>>>>>> I had simply intended to allow a non-callable capsule argument to be passed in instead of another call-back to any SciPy or NumPy function that can take a raw C-function pointer.
>>>>>
>>>>> If the cython folks are worried about type-checking overhead, then
>>>>> PyCapsule seems sub-optimal, because it's unnecessarily complicated to
>>>>> determine what sort of PyCapsule you have, and then extract the actual
>>>>> C struct. (At a minimum, it requires two calls to non-inlineable
>>>>> functions, plus an unnecessary pointer indirection.)
>>>>
>>>> I think this discussion is moot -- the way I reverse-engineer Travis is
>>>> that there's no time for a cross-project discussion about this now.
>>>> That's not too bad, Cython will go its own way (eventually), and perhaps
>>>> we can merge in the future...
>>>>
>>>> But for the entertainment value:
>>>>
>>>> In my CEP [1] I descripe two access mechanisms, one slow (for which I
>>>> think capsules is fine), and a faster one.
>>>>
>>>> Obviously, only the slow mechanism will be implemented first.
>>>>
>>>> So the only things I'd like changed in how Travis' want to do this is
>>>>
>>>> a) Storing the signature string data in the struct, rather than as a char*;
>>>>
>>>>      void *func
>>>>      char string[1]; // variable-size-allocated and null-terminated
>>>>
>>>> b) Allow for multiple signatures in the same capsule, i.e. "dd->d",
>>>> "ff->f", in the same capsule.
>>>>
>>>>> A tiny little custom class in a tiny little library that everyone can
>>>>> share might be better? (Bonus: a custom class could define a __call__
>>>>> method that used ctypes to call the function directly, for interactive
>>>>> convenience/testing/etc.)
>>>>
>>>> Having NumPy and Cython depend on a common library, and getting that to
>>>> work for users, seems rather utopic to me. And if I propose that Cython
>>>> have a hard dependency of NumPy for a feature as basic as calli.ng a
>>>> callback object then certain people will be very angry.
>>>>
>>>> Anyway, in my CEP I went to great pains to avoid having to do this, with
>>>> a global registration mechanism for multiple such types.
>>>>
>>>> Regarding your idea for the __call__, that's the exact opposite of what
>>>> I'm doing in the CEP. I'm pretty sure that what I described is what we
>>>> want for Cython; we will never tell our users to pass capsules around.
>>>> What I want is this:
>>>>
>>>> @numba
>>>> def f(x): return 2 * x
>>>>
>>>> @cython.inline
>>>> def g(x): return 3 * x
>>>>
>>>> print f(3)
>>>> print g(3)
>>>> print scipy.integrate.quad(f, 0.2, 3) # fast!
>>>> print scipy.integrate.quad(g, 0.2, 3) # fast!
>>>>
>>>> # If you really want a capsule:
>>>> print f.__nativecall__
>>>>
>>>> Dag
>>>>
>>>> [1] http://wiki.cython.org/enhancements/cep1000
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion@scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



More information about the NumPy-Discussion mailing list