[Numpy-discussion] Cython numerical syntax revisited
Dag Sverre Seljebotn
Thu Mar 5 02:05:08 CST 2009
Andrew Straw wrote:
> Dag Sverre Seljebotn wrote:
>> This is NOT yet discussed on the Cython list; I wanted to check with
>> more numerical users to see if the issue should even be brought up there.
>> The idea behind the current syntax was to keep things as close as
>> possible to Python/NumPy, and only provide some "hints" to Cython for
>> optimization. My problem with this now is that a) it's too easy to get
>> non-optimized code without a warning by letting in untyped indices, b) I
>> think the whole thing is a bit too "magic" and that it is too unclear
>> what is going on to newcomers (though I'm guessing there).
> These may be issues, but I think keeping "cython -a my_module.pyx" in
> one's development cycle and inspecting the output will lead to great
> enlightenment on the part of the Cython user. Perhaps this should be
> advertised more prominently? I always do this with any Cython-generated
> code, and it works wonders.
Well, I do so too (or rather just open the generated C file in emacs,
but since I'm working on Cython I'm more used to read that garbage than
others I suppose :-) ). But it feels like "one extra step" which must be
done. A syntax highlighter in emacs highlighting C operations would help
as well though.
>> My proposal: Introduce an explicit "buffer syntax":
>> arr = np.zeros(..)
>> cdef int[:,:] buf = arr # 2D buffer
> My initial reaction is that it seems to be a second implementation of
> buffer interaction Cython, and therefore yet another thing to keep in
Well, it would use much of the same implementation of course :-) It's
more of a change in the parser and a few rules here and there than
> mind and it's unclear to me how different it would be from the
> "traditional" Cython numpy ndarray behavior and how the behavior of the
> two approaches might differ, perhaps in subtle ways. So that's a
> disadvantage from my perspective. I agree that some of your ideas are
> advantages, however. Also, it seems it would allow one to (more easily)
> interact with buffer objects in sophisticated ways without needing the
> GIL, which is another advantage.
> Could some or all of this be added to the current numpy buffer
> implementation, or does it really need the new syntax?
The new features I mention? Most of it could be wedged into the existing
syntax, but at the expense of making things even less clear (or at least
I feel so) -- for instance, if we decided that Cython was to take care
of operations like "a + b" in a NumExpr-like fashion, then it would mean
that all declared buffers would get their Python versions of their
arithmetic operators hidden and fixed.
To make a point, it would mean that the NumPy matrix object would
suddenly get componentwise multiplication when assigned to an
ndarray[int] variable! (Granted, my feeling is that the matrix class
should be better avoided anyway, but...)
Regarding passing buffers to C/Fortran, it's just a matter of coming up
with a nice syntax.
> Also, is there anything possible with buffer objects that would be
> limited by the choice of syntax you propose? I imagine this might not
> work with structured data types (then again, it might...).
mystruct[:,:] should just work, if that is what you mean. It's simply a
matter of a) adding something to the parser, and b) disable the current
pass-through of "what the buffer cannot handle" to the Python runtime.
More information about the Numpy-discussion