[Numpy-discussion] Proposed Roadmap Overview

Robert Kern robert.kern@gmail....
Sat Feb 18 16:03:39 CST 2012


On Sat, Feb 18, 2012 at 21:51, Matthew Brett <matthew.brett@gmail.com> wrote:
> On Sat, Feb 18, 2012 at 1:40 PM, Charles R Harris
> <charlesr.harris@gmail.com> wrote:
>>
>>
>> On Sat, Feb 18, 2012 at 2:17 PM, David Cournapeau <cournape@gmail.com>
>> wrote:
>>>
>>> On Sat, Feb 18, 2012 at 8:45 PM, Charles R Harris
>>> <charlesr.harris@gmail.com> wrote:
>>> >
>>> >
>>> > On Sat, Feb 18, 2012 at 1:39 PM, Matthew Brett <matthew.brett@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Sat, Feb 18, 2012 at 12:35 PM, Charles R Harris
>>> >> <charlesr.harris@gmail.com> wrote:
>>> >> >
>>> >> >
>>> >> > On Sat, Feb 18, 2012 at 12:21 PM, Matthew Brett
>>> >> > <matthew.brett@gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi.
>>> >> >>
>>> >> >> On Sat, Feb 18, 2012 at 12:18 AM, Christopher Jordan-Squire
>>> >> >> <cjordan1@uw.edu> wrote:
>>> >> >> > On Fri, Feb 17, 2012 at 11:31 PM, Matthew Brett
>>> >> >> > <matthew.brett@gmail.com> wrote:
>>> >> >> >> Hi,
>>> >> >> >>
>>> >> >> >> On Fri, Feb 17, 2012 at 10:18 PM, Christopher Jordan-Squire
>>> >> >> >> <cjordan1@uw.edu> wrote:
>>> >> >> >>> On Fri, Feb 17, 2012 at 8:30 PM, Sturla Molden
>>> >> >> >>> <sturla@molden.no>
>>> >> >> >>> wrote:
>>> >> >> >>>>
>>> >> >> >>>>
>>> >> >> >>>> Den 18. feb. 2012 kl. 05:01 skrev Jason Grout
>>> >> >> >>>> <jason-sage@creativetrax.com>:
>>> >> >> >>>>
>>> >> >> >>>>> On 2/17/12 9:54 PM, Sturla Molden wrote:
>>> >> >> >>>>>> We would have to write a C++ programming tutorial that is
>>> >> >> >>>>>> based
>>> >> >> >>>>>> on
>>> >> >> >>>>>> Pyton knowledge instead of C knowledge.
>>> >> >> >>>>>
>>> >> >> >>>>> I personally would love such a thing.  It's been a while since
>>> >> >> >>>>> I
>>> >> >> >>>>> did
>>> >> >> >>>>> anything nontrivial on my own in C++.
>>> >> >> >>>>>
>>> >> >> >>>>
>>> >> >> >>>> One example: How do we code multiple return values?
>>> >> >> >>>>
>>> >> >> >>>> In Python:
>>> >> >> >>>> - Return a tuple.
>>> >> >> >>>>
>>> >> >> >>>> In C:
>>> >> >> >>>> - Use pointers (evilness)
>>> >> >> >>>>
>>> >> >> >>>> In C++:
>>> >> >> >>>> - Return a std::tuple, as you would in Python.
>>> >> >> >>>> - Use references, as you would in Fortran or Pascal.
>>> >> >> >>>> - Use pointers, as you would in C.
>>> >> >> >>>>
>>> >> >> >>>> C++ textbooks always pick the last...
>>> >> >> >>>>
>>> >> >> >>>> I would show the first and the second method, and perhaps
>>> >> >> >>>> intentionally forget the last.
>>> >> >> >>>>
>>> >> >> >>>> Sturla
>>> >> >> >>>>
>>> >> >> >>
>>> >> >> >>> On the flip side, cython looked pretty...but I didn't get the
>>> >> >> >>> performance gains I wanted, and had to spend a lot of time
>>> >> >> >>> figuring
>>> >> >> >>> out if it was cython, needing to add types, buggy support for
>>> >> >> >>> numpy,
>>> >> >> >>> or actually the algorithm.
>>> >> >> >>
>>> >> >> >> At the time, was the numpy support buggy?  I personally haven't
>>> >> >> >> had
>>> >> >> >> many problems with Cython and numpy.
>>> >> >> >>
>>> >> >> >
>>> >> >> > It's not that the support WAS buggy, it's that it wasn't clear to
>>> >> >> > me
>>> >> >> > what was going on and where my performance bottleneck was. Even
>>> >> >> > after
>>> >> >> > microbenchmarking with ipython, using timeit and prun, and using
>>> >> >> > the
>>> >> >> > cython code visualization tool. Ultimately I don't think it was
>>> >> >> > cython, so perhaps my comment was a bit unfair. But it was
>>> >> >> > unfortunately difficult to verify that. Of course, as you say,
>>> >> >> > diagnosing and solving such issues would become easier to resolve
>>> >> >> > with
>>> >> >> > more cython experience.
>>> >> >> >
>>> >> >> >>> The C files generated by cython were
>>> >> >> >>> enormous and difficult to read. They really weren't meant for
>>> >> >> >>> human
>>> >> >> >>> consumption.
>>> >> >> >>
>>> >> >> >> Yes, it takes some practice to get used to what Cython will do,
>>> >> >> >> and
>>> >> >> >> how to optimize the output.
>>> >> >> >>
>>> >> >> >>> As Sturla has said, regardless of the quality of the
>>> >> >> >>> current product, it isn't stable.
>>> >> >> >>
>>> >> >> >> I've personally found it more or less rock solid.  Could you say
>>> >> >> >> what
>>> >> >> >> you mean by "it isn't stable"?
>>> >> >> >>
>>> >> >> >
>>> >> >> > I just meant what Sturla said, nothing more:
>>> >> >> >
>>> >> >> > "Cython is still 0.16, it is still unfinished. We cannot base
>>> >> >> > NumPy
>>> >> >> > on
>>> >> >> > an unfinished compiler."
>>> >> >>
>>> >> >> Y'all mean, it has a zero at the beginning of the version number and
>>> >> >> it is still adding new features?  Yes, that is correct, but it seems
>>> >> >> more reasonable to me to phrase that as 'active development' rather
>>> >> >> than 'unstable', because they take considerable care to be backwards
>>> >> >> compatible, have a large automated Cython test suite, and a major
>>> >> >> stress-tester in the Sage test suite.
>>> >> >>
>>> >> >
>>> >> > Matthew,
>>> >> >
>>> >> > No one in their right mind would build a large performance library
>>> >> > using
>>> >> > Cython, it just isn't the right tool. For what it was designed for -
>>> >> > wrapping existing c code or writing small and simple things close to
>>> >> > Python
>>> >> > - it does very well, but it was never designed for making core C/C++
>>> >> > libraries and in that role it just gets in the way.
>>> >>
>>> >> I believe the proposal is to refactor the lowest levels in pure C and
>>> >> move the some or most of the library superstructure to Cython.
>>> >
>>> >
>>> > Go for it.
>>>
>>> The proposal of moving to a core C + cython has been discussed by
>>> multiple contributors. It is certainly a valid proposal. *I* have
>>> worked on this (npymath, separate compilation), although certainly not
>>> as much as I would have wanted to. I think much can be done in that
>>> vein. Using the "shut up if you don't do it" is a straw man (and
>>> uncalled for).
>>
>>
>> OK, I was annoyed.
>
> By what?

Your misunderstanding of what was being discussed. The proposal being
discussed is implementing the core of numpy in C++, wrapped in C to be
usable as a C library that other extensions can use, and then exposed
to Python in an unspecified way. Cython was raised as an alternative
for this core, but as Chuck points out, it doesn't really fit. Your
assertion that what was being discussed was putting the core in C and
using Cython to wrap it was simply a non-sequitur. Discussion of
alternatives is fine. You weren't doing that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the NumPy-Discussion mailing list