[Numpy-discussion] Profiling numpy ? (parts written in C)
David Cournapeau
david at ar.media.kyoto-u.ac.jp
Wed Dec 20 00:13:17 CST 2006
John Hunter wrote:
>>>>>> "David" == David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:
> David> At the end, in the original context (speeding the drawing
> David> of spectrogram), this is the problem. Even if multiple
> David> backend/toolkits have obviously an impact in performances,
> David> I really don't see why a numpy function to convert an array
> David> to a RGB representation should be 10-20 times slower than
> David> matlab on the same machine.
>
> This isn't exactly right. When matplotlib converts a 2D grayscale
> array to rgba, a lot goes on under the hood. It's all numpy, but it's
> far from single function and it involves many passes through the
> data. In principle, this could be done with one or two passes through
> the data. In practice, our normalization and colormapping abstractions
> are so abstract that it is difficult (though not impossible) to
> special case and optimize.
>
Well, we managed to have more than a 100 % speed increase for to_rgba
function already, with the help from Eric Firing. Before, both functors
Normalize and Colormap were expensive; I think we got something like a
2-3x speed increase in normalize (now, the main problem is the clip
function, which lead to the discussion about numpy clip), and now, the
Colormap functor takes 75 % of the time of to_rgba function with an
array of 8000x256 samples:
1 0.000 0.000 0.832 0.832
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:275(expose_event)
1 0.010 0.010 0.831 0.831
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtkagg.py:71(_render_figure)
1 0.000 0.000 0.821 0.821
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_agg.py:385(draw)
1 0.000 0.000 0.819 0.819
/home/david/local/lib/python2.4/site-packages/matplotlib/figure.py:511(draw)
1 0.000 0.000 0.817 0.817
/home/david/local/lib/python2.4/site-packages/matplotlib/axes.py:1043(draw)
1 0.000 0.000 0.648 0.648
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:1927(imshow)
3 0.000 0.000 0.573 0.191
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:883(gca)
1 0.000 0.000 0.572 0.572
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:950(ishold)
1 0.007 0.007 0.510 0.510
/home/david/local/lib/python2.4/site-packages/matplotlib/image.py:173(draw)
1 0.109 0.109 0.503 0.503
/home/david/local/lib/python2.4/site-packages/matplotlib/image.py:109(make_image)
4 0.000 0.000 0.491 0.123
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:903(gcf)
1 0.000 0.000 0.491 0.491
/home/david/local/lib/python2.4/site-packages/matplotlib/pylab.py:818(figure)
1 0.000 0.000 0.491 0.491
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtkagg.py:36(new_figure_manager)
1 0.024 0.024 0.482 0.482
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:401(__init__)
1 0.000 0.000 0.458 0.458
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtkagg.py:25(_get_toolbar)
1 0.001 0.001 0.458 0.458
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:496(__init__)
1 0.000 0.000 0.458 0.458
/home/david/local/lib/python2.4/site-packages/matplotlib/backend_bases.py:1112(__init__)
1 0.010 0.010 0.458 0.458
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:557(_init_toolbar)
1 0.029 0.029 0.448 0.448
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:595(_init_toolbar2_4)
1 0.419 0.419 0.419 0.419
/home/david/local/lib/python2.4/site-packages/matplotlib/backends/backend_gtk.py:967(__init__)
1 0.002 0.002 0.393 0.393
/home/david/local/lib/python2.4/site-packages/matplotlib/cm.py:50(to_rgba)
1 0.111 0.111 0.307 0.307
/home/david/local/lib/python2.4/site-packages/matplotlib/colors.py:570(__call__)
Of this 300 ms spent in Colormap functor, 200 ms are taken by the take
function: this is the function which I think can be speed up considerably.
> The top-level routine is
>
> def to_rgba(self, x, alpha=1.0):
> '''Return a normalized rgba array corresponding to x.
> If x is already an rgb or rgba array, return it unchanged.
> '''
> if hasattr(x, 'shape') and len(x.shape)>2: return x
> x = ma.asarray(x)
> x = self.norm(x)
> x = self.cmap(x, alpha)
> return x
>
> which implies at a minimum two passes through the data, one for norm
> and one for cmap.
>
> In 99% of the use cases, cmap is a LinearSegmentedColormap though
> users can define their own as long as it is callable. My guess is
> that the expensive part is Colormap.__call__, the base class for
> LinearSegmentedColormap. We could probably write some extension code
> that does the following routine in one pass through the data. But it
> would be hairy. In a quick look and rough count, I see about 10
> passes through the data in the function below.
So with above points, I agree that self.norm and self.cmap are the slow
parts, but I think this can be much improved by improving the
corresponding numpy functions. I think there is space to improve things
without touching matplotlib,
cheers,
David
More information about the Numpy-discussion
mailing list