[Numpy-discussion] NumPy nogil API
Dag Sverre Seljebotn
Mon Oct 31 06:01:33 CDT 2011
On 10/31/2011 11:48 AM, mark florisson wrote:
> On 31 October 2011 10:03, Dag Sverre Seljebotn
> <firstname.lastname@example.org> wrote:
>> Mark: I'm just wondering what you wanted to do with NumPy from Cython -- a
>> stopgap solution for SIMD, iterator support, or something else?
>> SIMD using NumPy really isn't the best idea long-term because of all the
>> temporaries needed in compound expressions, which is really bad on the
>> memory bus for anything but tiny arrays. For that I'd rather look at finding
>> a nogil core of numexpr or similar.
> Yes I'm aware of numexpr and the general problem with array
> expressions in NumPy. It's not just about SIMD or iterators, it's as
> you say below, there's lots of stuff that wouldn't be available even
> if we get SIMD. And if NumPy would get such an API, Cython could
> figure out how many (or if) temporaries are actually needed and call
> into the NumPy API with inplace operations.
> The thing is, how much of NumPy (and numexpr or theano) does Cython
> want to reimplement? Will you stop at SIMD with elemental functions?
> And will it run on my GPU?
> I suppose from a purity perspective I'd just like this functionality
> to be available in a library and have my language use the library
> efficiently behind my back, instead of implementing everything itself.
I do totally agree, but I'm also afraid that this is a neverending quest
as long as the GIL is present in CPython. There will always be stuff I'd
like to call without the GIL. Only NumPy is not sufficient; I'd also
like to use all the scientific libraries which relies on and extends
NumPy (all of SciPy for starters), and so on.
I do feel that what we have + SIMD covers just enough situations that it
is useful for writing "numerical cores", without needing the rest of
NumPy. If one starts to pull in more conveniences then I feel I might
equally likely need something in SciPy.
I'm not really against what you try to do; any progress at all on how
much one can do without the GIL is great, I'm just playing the devil's
advocate for a bit.
More information about the NumPy-Discussion