[Numpy-discussion] Insights / lessons learned from NumPy design

Mike Anderson mike.r.anderson.13@gmail....
Wed Jan 9 04:57:27 CST 2013


On 4 January 2013 16:13, Dag Sverre Seljebotn <d.s.seljebotn@astro.uio.no>wrote:

> On 01/04/2013 09:00 AM, Dag Sverre Seljebotn wrote:
> > On 01/04/2013 07:29 AM, Mike Anderson wrote:
> <snip>
>
> Oh: Depending on your amibitions, it's worth thinking hard about i)
> storage format, and ii) lazy evaluation.
>
> Storage format: The new trend is for more flexible formats than just
> column-major/row-major, e.g., storing cache-sized n-dimensional tiles.
>

I'm hoping the API will be independent of storage format - i.e. the
underlying implementations can store the data any way they like. So the API
will be written in terms of abstractions, and the user will have the choice
of whatever concrete implementation best fits the specific needs. Sparse
matrices, tiled matrices etc. should all be possible options.

Has this kind of approach been used much with NumPy?


>
> Lazy evaluation: The big problem with numpy is that "a + b + np.sqrt(c)"
> will first make a temporary result for "a + b", rather than doing the
> whole expression on the fly, which is *very* bad for performance.
>
> So if you want immutability, I urge you to consider every operation to
> build up an expression tree/"program", and then either find out the
> smart points where you interpret that program automatically, or make
> explicit eval() of an expression tree the default mode.
>

Very interesting. Seems like this could be layered on top though? i.e. have
a separate DSL for building up the expression tree, then compile this down
to the optimal set of underlying operations?


>
> Of course this depends all on how ambitious you are.
>

A little ambitious, though mostly I'll be glad to get something working
that people find useful :-)

Thanks again for your comments Dag!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20130109/1e99a9e6/attachment-0001.html 


More information about the NumPy-Discussion mailing list