[SciPy-dev] Fwd: [SciPy-user] MemoryError in scipy_core

Travis Oliphant oliphant at ee.byu.edu
Wed Nov 16 15:18:32 CST 2005


Chris Fonnesbeck wrote:

>>On 11/16/05, Travis Oliphant <oliphant at ee.byu.edu> wrote:
>>    
>>
>>>>Good news: the leak appears to be fixed. cbm.py is chugging along at
>>>>constant memory usage (pre-burn-in, of course). Only took 41 messages
>>>>to sort it out!
>>>>
>>>>
>>>>        
>>>>
>>>Fantastic.   Your code is actually a pretty good test because apparently
>>>you are using a lot of scalars.  The performance of the array scalars
>>>will be improved as time goes on.  I'd be interested in hearing how your
>>>code performs relative to the Numeric version because you have a lot of
>>>small arrays in your code base --- and a lot of scalars.
>>>
>>>      
>>>
>
>Travis,
>
>I have profiled the Numeric-based and scipy-based versions of my code.
>The Numeric version is just under twice as fast. The slowest bits
>appear to be in the numeric.py and oldnumeric.py modules, although the
>f2py stuff appears a little slower also. Here are the two profiles:
>
>  
>
This seems about right given your heavy use of small arrays.    While in 
C, the scipy core array creation is a bit more complicated, the indexing 
is more complicated,  and the ufuncs require an attribute look up.  
These three things I suspect account for most of the slowdown although 
I'd be thrilled to find other low-hanging fruit optimizations.

There are still optimizations possible (for example defining math on the 
array scalars rather than going through ufuncs would probably be a big 
gain). 

The other optimization possible is to consider a different way to get 
the special computation flags rather than an attribute look up in the 
locals, globals, and builtin dictionaries.   Or, perhaps allow someway 
to bypass the standard lookup approach and fix the behavior globally.  

Profiling can help in the optimization process. 

-Travis


Original profiles for scipy developers:

>scipy code:
>
>         1399359 function calls in 58.010 CPU seconds
>
>   Ordered by: standard name
>
>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    57005    0.400    0.000    0.400    0.000 :0(append)
>       55    0.350    0.006    0.350    0.006 :0(array)
>    60008    1.610    0.000    1.610    0.000 :0(concatenate)
>     8000    0.060    0.000    0.060    0.000 :0(copy)
>       15    0.000    0.000    0.000    0.000 :0(isinstance)
>   165103    1.350    0.000    1.350    0.000 :0(len)
>    15000    0.220    0.000    0.220    0.000 :0(normal)
>     7544    0.100    0.000    0.100    0.000 :0(random_sample)
>        3    0.000    0.000    0.000    0.000 :0(range)
>    65010    0.470    0.000    0.470    0.000 :0(ravel)
>    60023    2.840    0.000    2.840    0.000 :0(reduce)
>    80010    0.720    0.000    0.720    0.000 :0(reshape)
>     7238    0.100    0.000    0.100    0.000 :0(round)
>        1    0.000    0.000    0.000    0.000 :0(setprofile)
>        9    0.020    0.002    0.020    0.002 :0(sort)
>    60015    1.370    0.000    1.370    0.000 :0(sum)
>       51    0.000    0.000    0.000    0.000 :0(time)
>        3    0.000    0.000    0.000    0.000 :0(transpose)
>    10020    0.220    0.000    0.220    0.000 :0(values)
>    15006    0.440    0.000    0.440    0.000 :0(zip)
>        1    0.000    0.000   58.010   58.010 <string>:1(?)
>    30003    3.410    0.000   28.380    0.001 MCMC.py:1176(poisson_like)
>        2    0.000    0.000    0.000    0.000 MCMC.py:1567(parameter)
>        1    0.000    0.000    0.000    0.000 MCMC.py:1588(node)
>        1    0.010    0.010    1.680    1.680 MCMC.py:1593(summary)
>        1    0.000    0.000    0.050    0.050 MCMC.py:1989(calculate_dic)
>        1    0.000    0.000    0.000    0.000 MCMC.py:2055(__init__)
>    10000    1.750    0.000   35.700    0.004 MCMC.py:2061(test)
>        1    0.860    0.860   58.010   58.010 MCMC.py:2086(sample)
>        1    0.000    0.000    0.000    0.000 MCMC.py:2197(__init__)
>    15002    2.380    0.000   49.710    0.003 MCMC.py:2217(calculate_likelihood)
>        1    0.000    0.000    0.000    0.000 MCMC.py:279(make_indices)
>        3    0.000    0.000    0.000    0.000 MCMC.py:335(__init__)
>    36009    0.280    0.000    0.280    0.000 MCMC.py:352(get_value)
>     8739    0.090    0.000    0.090    0.000 MCMC.py:357(set_value)
>    12000    0.640    0.000    0.920    0.000 MCMC.py:362(tally)
>       39    0.090    0.002    0.430    0.011 MCMC.py:384(get_trace)
>        4    0.000    0.000    0.050    0.012 MCMC.py:396(quantiles)
>       15    0.000    0.000    0.190    0.013 MCMC.py:444(mean)
>        8    1.170    0.146    1.370    0.171 MCMC.py:452(var)
>        4    0.000    0.000    0.710    0.178 MCMC.py:462(mcerror)
>        4    0.020    0.005    0.070    0.017 MCMC.py:470(hpd)
>    15000    0.210    0.000    0.430    0.000 MCMC.py:557(normal_deviate)
>        2    0.000    0.000    0.000    0.000 MCMC.py:592(__init__)
>    10000    0.710    0.000    1.700    0.000 MCMC.py:628(sample_candidate)
>    10000    0.710    0.000   38.470    0.004 MCMC.py:645(propose)
>       18    0.000    0.000    0.000    0.000 MCMC.py:676(tune)
>        1    0.000    0.000    0.000    0.000 MCMC.py:737(__init__)
>     7238    0.110    0.000    0.210    0.000 MCMC.py:756(set_value)
>        1    0.000    0.000    0.000    0.000 MCMC.py:775(__init__)
>    15002    3.550    0.000   18.850    0.001 MCMC.py:820(uniform_like)
>        1    0.000    0.000    0.000    0.000 Matplot.py:36(__init__)
>       15    0.000    0.000    0.010    0.001 function_base.py:127(average)
>   280046   14.090    0.000   14.090    0.000 numeric.py:67(asarray)
>    65008    1.670    0.000    3.120    0.000 oldnumeric.py:131(reshape)
>        3    0.000    0.000    0.000    0.000 oldnumeric.py:179(transpose)
>        9    0.000    0.000    0.020    0.002 oldnumeric.py:187(sort)
>    60008   10.490    0.000   21.470    0.000 oldnumeric.py:220(resize)
>    65010    1.500    0.000    3.300    0.000 oldnumeric.py:258(ravel)
>    45008    0.600    0.000    1.270    0.000 oldnumeric.py:270(shape)
>    60015    1.380    0.000    3.510    0.000 oldnumeric.py:289(sum)
>        9    0.000    0.000    0.000    0.000 oldnumeric.py:348(rank)
>        0    0.000             0.000          profile:0(profiler)
>        1    0.000    0.000   58.010   58.010
>profile:0(sampler=DisasterSampler();
>sampler.sample(iterations=5000,burn=1000,verbose=False,plot=False))
>    45005    2.020    0.000   13.600    0.000 shape_base.py:115(atleast_1d)
>
>Numeric code:
>
>         1236421 function calls in 34.990 CPU seconds
>
>   Ordered by: standard name
>
>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    12000    0.060    0.000    0.060    0.000 :0(append)
>    22787    0.270    0.000    0.270    0.000 :0(apply)
>    45062    4.630    0.000    4.630    0.000 :0(array)
>    60009    1.050    0.000    1.050    0.000 :0(concatenate)
>     4000    0.010    0.000    0.010    0.000 :0(copy)
>    22802    0.260    0.000    0.260    0.000 :0(isinstance)
>   277933    2.460    0.000    2.460    0.000 :0(len)
>        3    0.000    0.000    0.000    0.000 :0(range)
>   105038    1.930    0.000    1.930    0.000 :0(reduce)
>   130020    3.320    0.000    3.320    0.000 :0(reshape)
>     7440    0.100    0.000    0.100    0.000 :0(round)
>        1    0.000    0.000    0.000    0.000 :0(setprofile)
>        9    0.010    0.001    0.010    0.001 :0(sort)
>       51    0.000    0.000    0.000    0.000 :0(time)
>        3    0.000    0.000    0.000    0.000 :0(transpose)
>    10011    0.180    0.000    0.180    0.000 :0(values)
>    15006    0.200    0.000    0.200    0.000 :0(zip)
>        1    0.000    0.000   34.990   34.990 <string>:1(?)
>    30004    2.850    0.000   15.410    0.001 MCMC.py:1179(poisson_like)
>        2    0.000    0.000    0.000    0.000 MCMC.py:1579(parameter)
>        1    0.000    0.000    0.000    0.000 MCMC.py:1600(node)
>        1    0.000    0.000    0.660    0.660 MCMC.py:1605(summary)
>        1    0.000    0.000    0.020    0.020 MCMC.py:2001(calculate_dic)
>        1    0.000    0.000    0.000    0.000 MCMC.py:2067(__init__)
>    10000    0.650    0.000   19.840    0.002 MCMC.py:2073(test)
>        1    0.590    0.590   34.990   34.990 MCMC.py:2098(sample)
>        1    0.000    0.000    0.000    0.000 MCMC.py:2210(__init__)
>    15002    1.380    0.000   27.670    0.002 MCMC.py:2230(calculate_likelihood)
>        1    0.000    0.000    0.000    0.000 MCMC.py:271(make_indices)
>        3    0.000    0.000    0.000    0.000 MCMC.py:327(__init__)
>    40000    0.360    0.000    0.360    0.000 MCMC.py:344(get_value)
>     9014    0.060    0.000    0.060    0.000 MCMC.py:349(set_value)
>    12000    0.640    0.000    0.870    0.000 MCMC.py:354(tally)
>       39    0.080    0.002    0.380    0.010 MCMC.py:376(get_trace)
>        4    0.000    0.000    0.050    0.012 MCMC.py:388(quantiles)
>       15    0.000    0.000    0.140    0.009 MCMC.py:436(mean)
>        8    0.200    0.025    0.430    0.054 MCMC.py:444(var)
>        4    0.010    0.003    0.280    0.070 MCMC.py:454(mcerror)
>        4    0.000    0.000    0.030    0.008 MCMC.py:462(hpd)
>    15000    0.250    0.000    1.710    0.000 MCMC.py:544(normal_deviate)
>        2    0.000    0.000    0.000    0.000 MCMC.py:579(__init__)
>    10000    0.560    0.000    2.790    0.000 MCMC.py:622(sample_candidate)
>    10000    0.560    0.000   23.590    0.002 MCMC.py:639(propose)
>       18    0.010    0.001    0.010    0.001 MCMC.py:670(tune)
>        1    0.000    0.000    0.000    0.000 MCMC.py:731(__init__)
>     7440    0.150    0.000    0.250    0.000 MCMC.py:757(set_value)
>        1    0.000    0.000    0.000    0.000 MCMC.py:776(__init__)
>    15002    2.090    0.000   10.770    0.001 MCMC.py:821(uniform_like)
>        1    0.000    0.000    0.000    0.000 Matplot.py:36(__init__)
>    60009    1.210    0.000    2.260    0.000 Numeric.py:231(concatenate)
>        3    0.000    0.000    0.000    0.000 Numeric.py:243(transpose)
>        9    0.000    0.000    0.010    0.001 Numeric.py:252(sort)
>    60009    4.260    0.000   13.350    0.000 Numeric.py:415(resize)
>    65011    0.830    0.000    2.840    0.000 Numeric.py:583(ravel)
>    60016    2.030    0.000    3.020    0.000 Numeric.py:634(sum)
>        9    0.000    0.000    0.000    0.000 Numeric.py:738(rank)
>    45018    0.400    0.000    0.400    0.000 Numeric.py:744(shape)
>       15    0.000    0.000    0.000    0.000 Numeric.py:762(average)
>    22787    0.760    0.000    1.490    0.000
>RandomArray.py:37(_build_random_array)
>     7787    0.120    0.000    0.610    0.000 RandomArray.py:54(random)
>    15000    0.270    0.000    1.270    0.000 RandomArray.py:87(standard_normal)
>    15000    0.190    0.000    1.460    0.000 RandomArray.py:93(normal)
>        0    0.000             0.000          profile:0(profiler)
>        1    0.000    0.000   34.990   34.990
>profile:0(sampler=DisasterSampler();
>sampler.sample(iterations=5000,burn=1000,verbose=False,plot=False))
>
>
>--
>Chris Fonnesbeck
>Atlanta, GA
>  
>




More information about the Scipy-dev mailing list