[SciPy-dev] Memory usage of scipy.io.loadmat

Robert Kern robert.kern@gmail....
Tue Jun 30 17:19:14 CDT 2009


On Tue, Jun 30, 2009 at 17:16, Ariel Rokem<arokem@berkeley.edu> wrote:
> Hi everyone,
>
> I have stumbled on some interesting behavior of scipy.io.loadmat. The short
> of it - it looks like loadmat is gobbling up memory in some unjustified
> manner and releasing it under some strange circumstances.
>
> Here's the long story. This is all happening on a Mac OS 10.5.7, running EPD
> 4.0.30001 (Python 2.5.2), but with a relatively new version of scipy (see
> below).
>
> I start ipython -pylab in one terminal and run 'top' in another, in order to
> monitor the memory usage. Here's what I get initially:
>
> PhysMem:  417M wired,  449M active,  183M inactive, 1055M used, 3041M free.
>
> Then:
>
> In [1]: import scipy
>
> In [2]: scipy.__version__
> Out[2]: '0.8.0.dev5606'
>
> In [3]: import scipy.io as sio
>
> Here's what it looks like now:
>
> PhysMem:  418M wired,  450M active,  183M inactive, 1058M used, 3038M free.
>
> So far, so good. I read in a large matfile with tons of data in it:
>
> In [4]: a =
> sio.loadmat('/Users/arokem/Projects/SchizoSpread/Scans/SMR033109_MC/Gray/Original/TSeries/Scan1/tSeries1.mat')
>
> PhysMem:  419M wired, 1024M active,  183M inactive, 1632M used, 2464M free.
>
> So - about 600 MB of memory is taken up by this new variable.
>
> Now to the wierdness:
>
> In [5]: b
> ---------------------------------------------------------------------------
> NameError                                 Traceback (most recent call last)
>
> /Users/arokem/<ipython console> in <module>()
>
> NameError: name 'b' is not defined
>
> Of course - 'b' doesn't exist!
>
> But now, the memory usage has dramatically gone down:
>
> PhysMem:  420M wired,  740M active,  183M inactive, 1350M used, 2746M free.
>
> So - just invoking an error in the ipython command line has freed up 300 MB.
> Where did they come from? I tried different things - assigning other
> variables doesn't seem to free up this memory. Neither do calls to other
> functions. Except "plot()", which does seem to do the trick for some reason.
> Interestingly, when I run all this in a python interactive session (and not
> ipython), I get a similar memory usage initially. Calling a non-existent
> variable does not free up the memory, but other things do. For example,
> import matplotlib.pylab into the namespace did the trick. Does anyone have
> any idea what is going on?

Garbage collection? Try the above after a call to gc.disable().

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the Scipy-dev mailing list