[Numpy-discussion] Proposed Roadmap Overview
Sun Feb 19 14:36:02 CST 2012
On Sun, Feb 19, 2012 at 4:03 AM, David Cournapeau <email@example.com>wrote:
> On Sun, Feb 19, 2012 at 9:28 AM, Mark Wiebe <firstname.lastname@example.org> wrote:
> > Is there anyone who uses a blue gene or small device which needs
> > numpy support, that I could talk to directly? We really need a list of
> > supported platforms on the numpy wiki we can refer to when discussing
> > stuff, it all seems very nebulous to me.
> They may not need an up to date numpy version now, but if stopping
> support for them is a requirement for C++, it must be kept in mind. I
> actually suspect Travis to have more details on the big iron side of
> things. On the small side of things:
> This may seem like not very useful - but that's part of what a open
> source project is all about in my mind.
> > Particular styles of using templates can cause this, yes. To properly do
> > this kind of advanced C++ library work, it's important to think about the
> > big-O notation behavior of your template instantiations, not just the
> > notation of run-time. C++ templates have a turing-complete language
> > is said to be quite similar to haskell, but spelled vastly different)
> > running at compile time in them. This is what gives template
> > meta-programming in C++ great power, but since templates weren't designed
> > for this style of programming originally, template meta-programming is
> > very easy.
> scipy.sparse.sparsetools is quite straightforward in its usage of
> templates (would be great if you could suggest improvement BTW, e.g.
> scipy/sparse/sparsetools/csr.h), and does not by itself use any
> meta-template programming.
I took a look, and I think the reason this is so slow to compile and uses
so much memory is visible as follows:
[sparsetools]$ wc *.cxx | sort -n
4039 13276 116263 csgraph_wrap.cxx
6464 21385 189537 dia_wrap.cxx
14002 45406 412262 coo_wrap.cxx
32385 102534 963688 csc_wrap.cxx
42997 140896 1313797 bsr_wrap.cxx
50041 161127 1501400 csr_wrap.cxx
149928 484624 4496947 total
That's almost 4.5MB of code, in 6 files. C/C++ compilers are not optimized
to compile this sort of thing fast, they are focused on more "human-style"
coding with smaller individual files. Looking at some of these
SWIG-generated files, the way they dispatch based on the input Python types
is bloated as well. Probably the main question I would ask is, does scipy
really need sparse matrix variants for all of int8, uint8, int16, uint16,
etc? Trimming away some of these might be reasonable, and would be a start
to improve compile times. The reason for the slowness is not C++ templates
in this example.
> I like that numpy can be built in a few seconds (at least without
> optimization), and consider this to be a useful feature.
> NumPy-Discussion mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the NumPy-Discussion