[Numpy-discussion] newbie question - large dataset

Steve Staneff staneff@constructiondatares....
Sat Apr 7 11:31:37 CDT 2007


Hi,

I'm looking for a better solution to managing a very large calculation. 
Set A is composed of tuples a, each of the form a = [float, string]; set B
is composed of tuples of similar structure (b = [float, string]).  For
each possible combination of a and b I'm calculating c, of the form c =
f(a,b) = [g(a[0], b[0]), h(a[1], b[1])] where g() and h() are non-trivial
functions.

There are now 15,000 or more tuples in A, and 100,000 or more tuples in B.
 B is expected to grow with time as the source database grows.  In
addition, there are many more elements in a and b than I've stated (and
many more functions operating on them).  I'm currently using python to
loop through each a in A and each b in B, which takes days.

If anyone can point me to a better approach via numpy ( or anything
else!), I'd be very appreciative.

Thanks in advance,

Steve




More information about the Numpy-discussion mailing list