[Numpy-discussion] Cython numerical syntax revisited

[Dag Sverre Seljebotn]

Andrew Straw wrote:
> Dag Sverre Seljebotn wrote:
>> This is NOT yet discussed on the Cython list; I wanted to check with 
>> more numerical users to see if the issue should even be brought up there.
>>
>> The idea behind the current syntax was to keep things as close as 
>> possible to Python/NumPy, and only provide some "hints" to Cython for 
>> optimization. My problem with this now is that a) it's too easy to get 
>> non-optimized code without a warning by letting in untyped indices, b) I 
>> think the whole thing is a bit too "magic" and that it is too unclear 
>> what is going on to newcomers (though I'm guessing there).
> 
> These may be issues, but I think keeping "cython -a my_module.pyx" in 
> one's development cycle and inspecting the output will lead to great 
> enlightenment on the part of the Cython user. Perhaps this should be 
> advertised more prominently? I always do this with any Cython-generated 
> code, and it works wonders.

Well, I do so too (or rather just open the generated C file in emacs, 
but since I'm working on Cython I'm more used to read that garbage than 
others I suppose :-) ). But it feels like "one extra step" which must be 
done. A syntax highlighter in emacs highlighting C operations would help 
as well though.

>> My proposal: Introduce an explicit "buffer syntax":
>>
>> arr = np.zeros(..)
>> cdef int[:,:] buf = arr # 2D buffer
> 
> My initial reaction is that it seems to be a second implementation of 
> buffer interaction Cython, and therefore yet another thing to keep in 

Well, it would use much of the same implementation of course :-) It's 
more of a change in the parser and a few rules here and there than 
anything else.

> mind and it's unclear to me how different it would be from the 
> "traditional" Cython numpy ndarray behavior and how the behavior of the 
> two approaches might differ, perhaps in subtle ways. So that's a 
> disadvantage from my perspective. I agree that some of your ideas are 
> advantages, however. Also, it seems it would allow one to (more easily) 
> interact with buffer objects in sophisticated ways without needing the 
> GIL, which is another advantage.
> 
> Could some or all of this be added to the current numpy buffer 
> implementation, or does it really need the new syntax?

The new features I mention? Most of it could be wedged into the existing 
syntax, but at the expense of making things even less clear (or at least 
I feel so) -- for instance, if we decided that Cython was to take care 
of operations like "a + b" in a NumExpr-like fashion, then it would mean 
that all declared buffers would get their Python versions of their 
arithmetic operators hidden and fixed.

To make a point, it would mean that the NumPy matrix object would 
suddenly get componentwise multiplication when assigned to an 
ndarray[int] variable! (Granted, my feeling is that the matrix class 
should be better avoided anyway, but...)

Regarding passing buffers to C/Fortran, it's just a matter of coming up 
with a nice syntax.

> Also, is there anything possible with buffer objects that would be 
> limited by the choice of syntax you propose? I imagine this might not 
> work with structured data types (then again, it might...).

mystruct[:,:] should just work, if that is what you mean. It's simply a 
matter of a) adding something to the parser, and b) disable the current 
pass-through of "what the buffer cannot handle" to the Python runtime.

-- 
Dag Sverre