mysql -> record array

Travis Oliphant oliphant at
Tue Nov 14 17:50:03 CST 2006

John Hunter wrote:

>>>>>>"John" == John Hunter <jdhunter at> writes:
>>>>>>"Erin" == Erin Sheldon <erin.sheldon at> writes:
>    Erin> The question I have been asking myself is "what is the
>    Erin> advantage of such an approach?".  It would be faster, but by
>    John> In the use case that prompted this message, the pull from
>    John> mysql took almost 3 seconds, and the conversion from lists
>    John> to numpy arrays took more that 4 seconds.  We have a list of
>    John> about 500000 2 tuples of floats.
>    John> Digging in a little bit, we found that numpy is about 3x
>    John> slower than Numeric here
>    John>   peds-pc311:~> python with dtype: 4.25 elapsed
>    John> seconds w/o dtype 5.79 elapsed seconds Numeric 1.58 elapsed
>    John> seconds 24.0b2 1.0.1.dev3432
>    John> Hmm... So maybe the question is -- is there some low hanging
>    John> fruit here to get numpy speeds up?
>And for reference, numarray is 5 times faster than Numeric here and 15
>times faster than numpy
>  peds-pc311:~> python
>  with dtype: 4.20 elapsed seconds
>  w/o dtype 5.71 elapsed seconds
>  Numeric  1.60 elapsed seconds
>  numarray  0.30 elapsed seconds
>  24.0b2
>  1.0.1.dev3432
>  1.5.1
>import numarray
>tnow = time.time()
>y = numarray.array(x, numarray.Float)
>tdone = time.time()
This sounds like it could be definitely be sped up, then.      
Assign_Array is the relevant code (it then calls PySequence_SetItem) so 
that basically

for k in a.shape[0]:
      a[k] = x[k]

is what is being done.  Thus, it might be the indexing code that is 
causing this to be a slower operation.  We should look at what numarray 
is doing --- it has provided important speed-ups in the past.

I still don't have time to look at this, but please file a ticket as we 
should fix this one.  Reference the faster numarray implementation.


Take Surveys. Earn Cash. Influence the Future of IT
Join's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash

More information about the Numpy-discussion mailing list