[SciPy-dev] Splitting sparsetools_wrap source file ?

Nathan Bell wnbell@gmail....
Mon Mar 10 08:05:50 CDT 2008


On Sun, Mar 9, 2008 at 10:12 PM, David Cournapeau <cournape@gmail.com> wrote:
>  It is not so much the build time that the memory consumption which I found
>  problematic. It is becoming difficult to build scipy on my laptop with 512 Mb
>  (compiling the sparsetools_wrap.cxx with gcc -O2 takes around 400 Mb, and some
>  compilers are even worse), or on virtual machines.
>
>  Is the patch really that invasive ? Splitting in 2 or 3 would already be good if
>  the memory consumption reduction is decreased by the same order.

Splitting the file into multiple parts does reduce the memory usage,
but not by the expected fraction.  Aside from manually splitting the
SWIG output into multiple files (which would be tedious, time
consuming, and error-prone), I'm not sure how to remedy the situation.

In the era of $25/GB RAM, is it not more expedient to simply increase
your memory capacity?  Using SWIG and C++ templates is a major
convenience in sparsetools since adding new dtypes becomes trivial.
However this implies that each function is instantiated ~15 times
(once for each dtype) which results in high memory usage.

If there's a simple solution that addresses your concerns, I'd be
happy to make the necessary changes.  Otherwise, I don't think the
problem merits complicating sparsetools.

-- 
Nathan Bell wnbell@gmail.com
http://graphics.cs.uiuc.edu/~wnbell/


More information about the Scipy-dev mailing list