[Numpy-discussion] newbie question - large dataset
Sun Apr 15 11:02:29 CDT 2007
Many thanks to all those who responded to my question. Not being a real programmer, sometimes it's good to be reminded
of the basics - such as timing various sections of code. It turns out that most of the time in the loop was spent on
two of the functions. I've rewritten the first to be pre-calculated (a real oversight on my part), and looked at
improving the second as much as possible.
Some of the other time in the loop is being spent going back to the db server for data, and I'll look into using
pytables as a way of speeding that up (possibly by initializing temporary tables on each of my client machines).
> On 4/7/07, Steve Staneff <firstname.lastname@example.org> wrote:
>>I'm looking for a better solution to managing a very large calculation.
>>Set A is composed of tuples a, each of the form a = [float, string]; set B
>>is composed of tuples of similar structure (b = [float, string]). For
>>each possible combination of a and b I'm calculating c, of the form c =
>>f(a,b) = [g(a, b), h(a, b)] where g() and h() are non-trivial
More information about the Numpy-discussion