[Numpy-discussion] Proposed Roadmap Overview

Lluís xscript@gmx....
Mon Feb 20 13:28:01 CST 2012

Francesc Alted writes:

> On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
>> You need at least a slightly different Python API to get anywhere, so 
>> numexpr/Theano is the right place to work on an implementation of this 
>> idea. Of course it would be nice if numexpr/Theano offered something as 
>> convenient as
>> with lazy:
>> arr = A + B + C # with all of these NumPy arrays
>> # compute upon exiting…

> Hmm, that would be cute indeed.  Do you have an idea on how the code in the with
> context could be passed to the Python AST compiler (à la numexpr.evaluate("A + B
> + C"))?

Well, I started writing some experiments to "almost transparently" translate
regular ndarray operations to numexpr strings (or others) using only python

The concept is very simple:

    # you only need the first one to start building the AST
    a = lazy(np.arange(16))
    b = np.arange(16)
    res = a + b + 3
    print evaluate(res)
    # the actual evaluation can be delayed to something like __repr__ or __str__
    print repr(res)
    print res
    # you could also delay evaluation until someone uses res to create a new array

My target was to use this to also generate optimized GPU kernels in-flight using
pycuda, but I think some other relatively recent project already performed
something similar (w.r.t. generating cuda kernels out of python expressions).

The supporting code for numexpr was something like:

    import numexpr
    import numpy as np

    def build_arg_expr (arg, args):
        if isinstance(arg, Expr):
           # recursively build the expression
           arg_expr, arg_args = arg.build_expr()
           return arg_expr
           # unique argument identifier
           arg_id = "arg_%d" % id(arg)
           args[arg_id] = arg
           return arg_id
    # generic expression builder
    class Expr:
          def evaluate(self):
              expr, args = self.build_expr()
              return numexpr.evaluate(expr, local_dict = args, global_dict = {})
          def __repr__ (self):
              return self.evaluate().__repr__()
          def __str__ (self):
              return self.evaluate().__str__()
          def __add__ (self, other):
              return ExprAdd(self, other)
    # expression builder for adds
    class ExprAdd(Expr):
          def __init__(self, arg1, arg2):
              self.arg1 = arg1
              self.arg2 = arg2
          def build_expr(self):
              args = {}
              expr1 = build_arg_expr(self.arg1, args)
              expr2 = build_arg_expr(self.arg2, args)
              return "("+expr1+") + ("+expr2+")", args
    # ndarray-like class to generate expression builders
    class LazyNdArray(np.ndarray):
          def __add__ (self, other):
              return ExprAdd(self, other)
    # build a LazyNdArray
    def lazy (arg):
        return arg.view(LazyNdArray)
    # evaluate with numexpr an arbitrary expression builder     
    def evaluate(arg):
        return arg.evaluate()

The thing here is to always return to the user something that looks like an

As you can see the whole thing is not very complex, but some less funny code had
to be written meanwhile for work and I just dropped this :)


 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom

More information about the NumPy-Discussion mailing list