[Numpy-discussion] Proposed Roadmap Overview

Mark Wiebe mwwiebe@gmail....
Sun Feb 19 14:36:02 CST 2012


On Sun, Feb 19, 2012 at 4:03 AM, David Cournapeau <cournape@gmail.com>wrote:

> On Sun, Feb 19, 2012 at 9:28 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
>
> > Is there anyone who uses a blue gene or small device which needs
> up-to-date
> > numpy support, that I could talk to directly? We really need a list of
> > supported platforms on the numpy wiki we can refer to when discussing
> this
> > stuff, it all seems very nebulous to me.
>
> They may not need an up to date numpy version now, but if stopping
> support for them is a requirement for C++, it must be kept in mind. I
> actually suspect Travis to have more details on the big iron side of
> things. On the small side of things:
> http://projects.scipy.org/numpy/ticket/1969
>
> This may seem like not very useful - but that's part of what a open
> source project is all about in my mind.
>
> >
> > Particular styles of using templates can cause this, yes. To properly do
> > this kind of advanced C++ library work, it's important to think about the
> > big-O notation behavior of your template instantiations, not just the
> big-O
> > notation of run-time. C++ templates have a turing-complete language
> (which
> > is said to be quite similar to haskell, but spelled vastly different)
> > running at compile time in them. This is what gives template
> > meta-programming in C++ great power, but since templates weren't designed
> > for this style of programming originally, template meta-programming is
> not
> > very easy.
>
> scipy.sparse.sparsetools is quite straightforward in its usage of
> templates (would be great if you could suggest improvement BTW, e.g.
> scipy/sparse/sparsetools/csr.h), and does not by itself use any
> meta-template programming.
>

I took a look, and I think the reason this is so slow to compile and uses
so much memory is visible as follows:

[sparsetools]$ wc *.cxx | sort -n
   4039   13276  116263 csgraph_wrap.cxx
   6464   21385  189537 dia_wrap.cxx
  14002   45406  412262 coo_wrap.cxx
  32385  102534  963688 csc_wrap.cxx
  42997  140896 1313797 bsr_wrap.cxx
  50041  161127 1501400 csr_wrap.cxx
 149928  484624 4496947 total

That's almost 4.5MB of code, in 6 files. C/C++ compilers are not optimized
to compile this sort of thing fast, they are focused on more "human-style"
coding with smaller individual files. Looking at some of these
SWIG-generated files, the way they dispatch based on the input Python types
is bloated as well. Probably the main question I would ask is, does scipy
really need sparse matrix variants for all of int8, uint8, int16, uint16,
etc? Trimming away some of these might be reasonable, and would be a start
to improve compile times. The reason for the slowness is not C++ templates
in this example.

Cheers,
Mark


> I like that numpy can be built in a few seconds (at least without
> optimization), and consider this to be a useful feature.
>
> cheers,
>
> David
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/numpy-discussion/attachments/20120219/eb5bff55/attachment-0001.html 


More information about the NumPy-Discussion mailing list