[SciPy-user] Arrays and strange memory usage ...

Francesc Alted faltet@pytables....
Wed Sep 3 05:36:44 CDT 2008


A Wednesday 03 September 2008, christophe grimault escrigué:
> OK, I understand now !
> Thanks very for the explanation.

You are welcome.  For the sort of situations that you are facing of, I 
normally query directly the Linux kernel so as to get a finer 
perception of the resources wasted.  Here it goes the function that I 
use:

def show_stats(explain, tref):
    "Show the used memory"
    # Build the command to obtain memory info (only for Linux 2.6.x)
    cmd = "cat /proc/%s/status" % os.getpid()
    sout = subprocess.Popen(cmd, shell=True, 
stdout=subprocess.PIPE).stdout
    for line in sout:
        if line.startswith("VmSize:"):
            vmsize = int(line.split()[1])
        elif line.startswith("VmRSS:"):
            vmrss = int(line.split()[1])
        elif line.startswith("VmData:"):
            vmdata = int(line.split()[1])
        elif line.startswith("VmStk:"):
            vmstk = int(line.split()[1])
        elif line.startswith("VmExe:"):
            vmexe = int(line.split()[1])
        elif line.startswith("VmLib:"):
            vmlib = int(line.split()[1])
    sout.close()
    print "Memory usage: ******* %s *******" % explain
    print "VmSize: %7s kB\tVmRSS: %7s kB" % (vmsize, vmrss)
    print "VmData: %7s kB\tVmStk: %7s kB" % (vmdata, vmstk)
    print "VmExe:  %7s kB\tVmLib: %7s kB" % (vmexe, vmlib)
    tnow = time()
    print "WallClock time:", round(tnow - tref, 3)
    return tnow

And here it is an example of use:

# declare this at the beginning of your module
profile = True
[clip]
if profile: tref = time()
if profile: show_stats("Entering initial_append", tref)
[your statements here...]
if profile: show_stats("Before creating idx", tref)
[more statements...]
if profile: show_stats("After creating idx", tref)

I hope you get the idea.

-- 
Francesc Alted


More information about the SciPy-user mailing list