[Numpy-discussion] Which dtype are supported by numexpr ?

Francesc Altet faltet@carabos....
Wed Mar 14 16:28:32 CDT 2007


El dc 14 de 03 del 2007 a les 16:10 -0400, en/na David M. Cooke va
escriure:
> On Wed, 14 Mar 2007 13:02:10 +0100
> Francesc Altet <faltet@carabos.com> wrote:
> 
> > The info above is somewhat inexact. I was talking about the enhanced
> > numexpr version included in PyTables 2.0 (see [1]). The original version of
> > numexpr (see [2]) doesn't have support for int64 on 32-bit platforms and
> > also neither does for strings. Sorry for the confusion.
> 
> What are you doing with strings in numexpr? Ivan Vilata i Balaguer submitted
> a patch to add them, but I rejected it as I didn't have any use-cases for
> strings that needed the speed of numexpr, and were worth adding the
> complexity.

Well, basically we need them because we use an (modified) instance of
numexpr internally in the new PyTables 2.0 in order to allow selections
of data on tables (kind of recarrays, but on-disk). For example, if we
have a table with 3 columns (col1, col2 and col3) with three different
datatypes (say string, int64 and float64 repectively), the user will be
able to do things like:

expr = '(((col1 == "item1") & (col2 < 3)) | (col3 > 24.1e12))'
selected_rows = [ row.now for row in table.where(expr) ]

Of course, using numexpr for doing this kind of stuff is twofold:

- The expresion interpreter comes integrated
- The computations are done much faster than using plain numpy

However, as I've shown on a recent message:

http://thread.gmane.org/gmane.comp.python.numeric.general/14130

adding complexity to the numexpr kernel is hurting its performance (most
specially in CPUs with reduced L2 caches), so perhaps is not a good idea
to add this kind of stuff (I mean, string support) to the generic
numexpr.  In that sense, we will also see what we can do for reducing
the kernel computing in the pytables version. So, unfortunately enough,
it seems that we will have to end maintaining our own version of numexpr
for pytables purposes :-/

Cheers,

-- 
Francesc Altet    |  Be careful about using the following code --
Carabos Coop. V.  |  I've only proven that it works, 
www.carabos.com   |  I haven't tested it. -- Donald Knuth



More information about the Numpy-discussion mailing list