[SciPy-user] Debugging memory exhaustion in Python?

David M. Cooke cookedm@physics.mcmaster...
Fri Jun 15 15:27:09 CDT 2007


On Jun 15, 2007, at 15:48 , Greg Novak wrote:

> Matthieu Brucher <matthieu.brucher@gmail.com> wrote:
>> Are you using a specific IDE ?
>
> Plain old IPython, but it happens when I run it in a bare python
> interpreter as well.
>
> On 6/15/07, Brian Granger <ellisonbg.net@gmail.com> wrote:
>> 1)  What version of python are you using?  Python 2.4 and below has
>> some issues with memory not  being released back to the OS.
>
> 2.5
>
>> 2)  What data structures are you using to represent the data?
>
> Lots of arrays...  It's mostly particle data, although I do flagrantly
> generate lots of temporaries.  I'm not careful at all about that.  I
> thought this could have speed implications, but I didn't realize it
> could have memory exhaustion implications, too.  Since I'm only
> handling 10's of MB at a time, I also thought that memory
> fragmentation wouldn't be a severe problem.  If I had GB arrays and
> started generating lots of temporary copies, I could see that that
> would lead to trouble...

When this happens to me its either because I screwed up handling the  
reference counts in a C extension, or I'm keeping old copies of  
arrays in a cache or a log object.

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm@physics.mcmaster.ca



More information about the SciPy-user mailing list