[Numpy-discussion] Getting C-function pointers from Python to C

[Travis Oliphant]

Dag, 

Thanks for the link to your CEP.  This is the first time I've seen it.   You probably referenced it before, but I hadn't seen it.  

That CEP seems along the lines of what I was thinking of.    We can make scipy follow that CEP and NumPy as well in places that it needs function pointers.   

I can certainly get behind it with Numba and recommend it to SciPy (and write the scipy.integrate.quad function to support it). 

Thanks for the CEP. 

-Travis





On Apr 12, 2012, at 2:08 PM, Dag Sverre Seljebotn wrote:

> On 04/12/2012 07:24 PM, Nathaniel Smith wrote:
>> On Wed, Apr 11, 2012 at 10:23 PM, Travis Oliphant<teoliphant@gmail.com>  wrote:
>>>>> In the mean-time, I think we could do as Robert essentially suggested and just use Capsule Objects around an agreed-upon simple C-structure:
>>>>> 
>>>>>      int   id   /* Some number that can be used as a "type-check" */
>>>>>      void *func;
>>>>>      char *string;
>>>>> 
>>>>> We can then just create some nice functions to go to and from this form in NumPy ctypeslib and then use this while the Python PEP gets written and adopted.
>>>> 
>>>> What is not clear to me is how one get from the Python callable to the
>>>> capsule.
>>> 
>>> This varies substantially based on the tool.   Numba would do it's work and create the capsule object using it's approach.   Cython would use a different approach.
>>> 
>>> I would also propose to have in NumPy some basic functions that go back-and forth between this representation, ctypes, and any other useful representations that might emerge.
>>> 
>>>> 
>>>> Or do you simply intend to pass a non-callable capsule as an argument in
>>>> place of the callback?
>>> 
>>> I had simply intended to allow a non-callable capsule argument to be passed in instead of another call-back to any SciPy or NumPy function that can take a raw C-function pointer.
>> 
>> If the cython folks are worried about type-checking overhead, then
>> PyCapsule seems sub-optimal, because it's unnecessarily complicated to
>> determine what sort of PyCapsule you have, and then extract the actual
>> C struct. (At a minimum, it requires two calls to non-inlineable
>> functions, plus an unnecessary pointer indirection.)
> 
> I think this discussion is moot -- the way I reverse-engineer Travis is 
> that there's no time for a cross-project discussion about this now. 
> That's not too bad, Cython will go its own way (eventually), and perhaps 
> we can merge in the future...
> 
> But for the entertainment value:
> 
> In my CEP [1] I descripe two access mechanisms, one slow (for which I 
> think capsules is fine), and a faster one.
> 
> Obviously, only the slow mechanism will be implemented first.
> 
> So the only things I'd like changed in how Travis' want to do this is
> 
> a) Storing the signature string data in the struct, rather than as a char*;
> 
>     void *func
>     char string[1]; // variable-size-allocated and null-terminated
> 
> b) Allow for multiple signatures in the same capsule, i.e. "dd->d", 
> "ff->f", in the same capsule.
> 
>> A tiny little custom class in a tiny little library that everyone can
>> share might be better? (Bonus: a custom class could define a __call__
>> method that used ctypes to call the function directly, for interactive
>> convenience/testing/etc.)
> 
> Having NumPy and Cython depend on a common library, and getting that to 
> work for users, seems rather utopic to me. And if I propose that Cython 
> have a hard dependency of NumPy for a feature as basic as calli.ng a 
> callback object then certain people will be very angry.
> 
> Anyway, in my CEP I went to great pains to avoid having to do this, with 
> a global registration mechanism for multiple such types.
> 
> Regarding your idea for the __call__, that's the exact opposite of what 
> I'm doing in the CEP. I'm pretty sure that what I described is what we 
> want for Cython; we will never tell our users to pass capsules around. 
> What I want is this:
> 
> @numba
> def f(x): return 2 * x
> 
> @cython.inline
> def g(x): return 3 * x
> 
> print f(3)
> print g(3)
> print scipy.integrate.quad(f, 0.2, 3) # fast!
> print scipy.integrate.quad(g, 0.2, 3) # fast!
> 
> # If you really want a capsule:
> print f.__nativecall__
> 
> Dag
> 
> [1] http://wiki.cython.org/enhancements/cep1000
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion