[SciPy-dev] Memory usage of scipy.io.loadmat
Tue Jun 30 17:19:14 CDT 2009
On Tue, Jun 30, 2009 at 17:16, Ariel Rokem<email@example.com> wrote:
> Hi everyone,
> I have stumbled on some interesting behavior of scipy.io.loadmat. The short
> of it - it looks like loadmat is gobbling up memory in some unjustified
> manner and releasing it under some strange circumstances.
> Here's the long story. This is all happening on a Mac OS 10.5.7, running EPD
> 4.0.30001 (Python 2.5.2), but with a relatively new version of scipy (see
> I start ipython -pylab in one terminal and run 'top' in another, in order to
> monitor the memory usage. Here's what I get initially:
> PhysMem: 417M wired, 449M active, 183M inactive, 1055M used, 3041M free.
> In : import scipy
> In : scipy.__version__
> Out: '0.8.0.dev5606'
> In : import scipy.io as sio
> Here's what it looks like now:
> PhysMem: 418M wired, 450M active, 183M inactive, 1058M used, 3038M free.
> So far, so good. I read in a large matfile with tons of data in it:
> In : a =
> PhysMem: 419M wired, 1024M active, 183M inactive, 1632M used, 2464M free.
> So - about 600 MB of memory is taken up by this new variable.
> Now to the wierdness:
> In : b
> NameError Traceback (most recent call last)
> /Users/arokem/<ipython console> in <module>()
> NameError: name 'b' is not defined
> Of course - 'b' doesn't exist!
> But now, the memory usage has dramatically gone down:
> PhysMem: 420M wired, 740M active, 183M inactive, 1350M used, 2746M free.
> So - just invoking an error in the ipython command line has freed up 300 MB.
> Where did they come from? I tried different things - assigning other
> variables doesn't seem to free up this memory. Neither do calls to other
> functions. Except "plot()", which does seem to do the trick for some reason.
> Interestingly, when I run all this in a python interactive session (and not
> ipython), I get a similar memory usage initially. Calling a non-existent
> variable does not free up the memory, but other things do. For example,
> import matplotlib.pylab into the namespace did the trick. Does anyone have
> any idea what is going on?
Garbage collection? Try the above after a call to gc.disable().
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
More information about the Scipy-dev