[SciPy-dev] Splitting sparsetools_wrap source file ?
Sat Mar 15 04:22:53 CDT 2008
Nathan Bell wrote:
> While better than manually splitting the _wrap file, this approach is
> still cumbersome. There are ~35 functions in sparsetools, so a 1
> function : 1 file policy is not really scalable.
Well, having one file/function is overkill. I just think that 500-600 Mb
of Ram to build one source file is really not good for various reasons;
several MB source files is fundementaly a bad thing. Splitting it in
half would already be pretty good, and that can be done easily: you just
have to be careful about what functions to put in which file to get the
memory reduction, though.
> I tried lumping all the CSR functions together and found only modest savings.
With two files, I could get from 500 Mb to around 260 Mb, which is
already pretty good I think.
> Disabling the templates that unroll dense loops for the BSR matrix
> (see bsr_matvec) produced a measureable improvement in memory usage so
> I've committed this version of sparsetools.h to SVN.
the *matvec functions are indeed the ones which take a lot of memory for
compilation, which is not surprising if they use expression template. Is
it really useful ? I certainly do not want this to cause any performance
penalty or anything. Putting those functions in a different file is what
gives the most advantage in my experiment.
> Are you saying that g++ fails to compile on the VM, or that it starts
> swapping to disk?
Yes, that's what I am saying. I need to allocate 1 Gb of memory for
solaris when compiling sparsetools with sun compilers right now. Even by
today standards, that's an awful lot of memory.
More information about the Scipy-dev