[SciPy-user] MemoryError in scipy_core

Chris Fonnesbeck fonnesbeck at gmail.com
Fri Nov 4 08:12:41 CST 2005


In the course of moving PyMC from Numeric to scipy_core, I am running
into some pretty serious memory issues. For those of you unfamilair
with PyMC, it is simply a Bayesian simulation module that estimates
model paramters by iteratively sampling from the joint posterior
distribution of the model, and saving each sample to an array. Under
Numeric, I could safely run several hundered thousand iterations of
pretty complex models (i.e. lots of paramters) without trouble. Under
scipy_core, PyMC hogs most of the system resources (you really cant do
anything else while its running), and crashes after just over 10K
iterations, under a pretty simple model. Here is the end of the
output:

Iteration 14000 at 21696.1090701

Iteration 14100 at 21901.323216

Iteration 14200 at 22108.8079159

Iteration 14300 at 22316.150553
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
python(3910,0xa000ed68) malloc: *** vm_allocate(size=1069056) failed
(error code=3)
python(3910,0xa000ed68) malloc: *** error: can't allocate region
python(3910,0xa000ed68) malloc: *** set a breakpoint in szone_error to debug
Traceback (most recent call last):
  File "regs.py", line 1075, in ?
    results = sampler.sample(iterations,burn=burn)
  File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/PyMC/MCMC.py",
line 2143, in sample
    parameter.propose(debug)
  File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/PyMC/MCMC.py",
line 664, in propose
    if not self._sampler.test():
  File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/PyMC/MCMC.py",
line 2060, in test
    like = self.calculate_likelihood()
  File "regs.py", line 703, in calculate_likelihood
    like += self.poisson_like(int(c),exp(theta))
  File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/PyMC/MCMC.py",
line 1192, in poisson_like
    if sum(mu<=0): raise LikelihoodError
  File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy/base/oldnumeric.py",
line 312, in sum
    return asarray(x).sum(axis, rtype)
MemoryError

Not sure if this is a leak, or what. Any assistance most welcome.

--
Chris Fonnesbeck
Atlanta, GA



More information about the SciPy-user mailing list