[Numpy-discussion] ndarray and lazy evaluation
Dag Sverre Seljebotn
Mon Feb 20 12:14:34 CST 2012
On 02/20/2012 10:04 AM, Francesc Alted wrote:
> On Feb 20, 2012, at 6:46 PM, Dag Sverre Seljebotn wrote:
>> On 02/20/2012 09:24 AM, Olivier Delalleau wrote:
>>> Hi Dag,
>>> Would you mind elaborating a bit on that example you mentioned at the
>>> end of your email? I don't quite understand what behavior you would like
>>> to achieve
>> Sure, see below. I think we should continue discussion on numpy-discuss.
>> I wrote:
>>> You need at least a slightly different Python API to get anywhere, so
>>> numexpr/Theano is the right place to work on an implementation of this
>>> idea. Of course it would be nice if numexpr/Theano offered something as
>>> convenient as
>>> with lazy:
>>> arr = A + B + C # with all of these NumPy arrays
>>> # compute upon exiting...
>> More information:
>> The disadvantage today of using Theano (or numexpr) is that they require
>> using a different API, so that one has to learn and use Theano "from the
>> ground up", rather than just slap it on in an optimization phase.
>> The alternative would require extensive changes to NumPy, so I guess
>> Theano authors or Francesc would need to push for this.
>> The alternative would be (with A, B, C ndarray instances):
>> with theano.lazy:
>> arr = A + B + C
>> On __enter__, the context manager would hook into NumPy to override it's
>> arithmetic operators. Then it would build a Theano symbolic tree instead
>> of performing computations right away.
>> In addition to providing support for overriding arithmetic operators,
>> slicing etc., it would be necesarry for "arr" to be an ndarray instance
>> which is "not yet computed" (data-pointer set to NULL, and store a
>> compute-me callback and some context information).
>> Finally, the __exit__ would trigger computation. For other operations
>> which need the data pointer (e.g., single element lookup) one could
>> either raise an exception or trigger computation.
>> This is just a rough sketch. It is not difficult "in principle", but of
>> course there's really a massive amount of work involved to work support
>> for this into the NumPy APIs.
>> Probably, we're talking a NumPy 3.0 thing, after the current round of
>> refactorings have settled...
>> Please: Before discussing this further one should figure out if there's
>> manpower available for it; no sense in hashing out a castle in the sky
>> in details.
> I see. Mark Wiebe already suggested the same thing some time ago:
Thanks, I didn't know about that (though I did really assume this was on
Mark's radar already).
>> Also it would be better to talk in person about this if
>> possible (I'm in Berkeley now and will attend PyData and PyCon).
> Nice. Most of Continuum crew (me included) will be attending to both conferences. Mark W. will make PyCon only, but will be a good occasion to discuss this further.
I certainly don't think I have anything to add to this discussion beyond
what Mark wrote up. But will be nice to meet up anyway.
More information about the NumPy-Discussion