[IPython-User] out of memory error

Robert Nishihara robertnishihara@gmail....
Fri Jul 6 08:56:13 CDT 2012


I'm running multiple trials of the same experiment in a for loop.

    for i in range(10):
        run_experiment()

It behaves properly for the first several trials. Then it fails with the
error (this error goes to the controller's standard error)

    MemoryError
    FATAL ERROR: OUT OF MEMORY (epoll.cpp:57)

I've read this thread <
http://mail.scipy.org/pipermail/ipython-user/2012-March/009687.html>, and
so I am already clearing the caches between trials with this subroutine

    def clear_cache(rc, dview):
        rc.results.clear()
        rc.metadata.clear()
        dview.results.clear()
        assert not rc.outstanding, "don't clear history when tasks are
outstanding"
        rc.history = []
        dview.history = []

But given that the memory error occurs after multiple successful trials, it
seems like something must be accumulating. Are there other sources of
caching that I'm missing? Is anything cached on the engines for instance? I
do not store my results between trials, I use cPickle to dump them to files.

-Robert




The full error from the controller's standard error is included below
----------------------------

ERROR:root:Uncaught exception, closing connection.
Traceback (most recent call last):
  File
"/software/linux/x86_64/epd-7.3-1/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py",
line 391, in _handle_events
    self._handle_recv()
  File
"/software/linux/x86_64/epd-7.3-1/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py",
line 412, in _handle_recv
    msg = self.socket.recv_multipart(zmq.NOBLOCK, copy=self._recv_copy)
  File "socket.pyx", line 723, in zmq.core.socket.Socket.recv_multipart
(zmq/core/socket.c:6495)
  File "socket.pyx", line 616, in zmq.core.socket.Socket.recv
(zmq/core/socket.c:5961)
  File "socket.pyx", line 650, in zmq.core.socket.Socket.recv
(zmq/core/socket.c:5832)
  File "socket.pyx", line 120, in zmq.core.socket._recv_copy
(zmq/core/socket.c:1681)
  File "message.pyx", line 75, in zmq.core.message.copy_zmq_msg_bytes
(zmq/core/message.c:1082)
MemoryError
ERROR:root:Exception in I/O handler for fd <zmq.core.socket.Socket object
at 0x162a6b0>
Traceback (most recent call last):
  File
"/software/linux/x86_64/epd-7.3-1/lib/python2.7/site-packages/zmq/eventloop/ioloop.py",
line 330, in start
    self._handlers[fd](fd, events)
  File
"/software/linux/x86_64/epd-7.3-1/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py",
line 391, in _handle_events
    self._handle_recv()
  File
"/software/linux/x86_64/epd-7.3-1/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py",
line 412, in _handle_recv
    msg = self.socket.recv_multipart(zmq.NOBLOCK, copy=self._recv_copy)
  File "socket.pyx", line 723, in zmq.core.socket.Socket.recv_multipart
(zmq/core/socket.c:6495)
  File "socket.pyx", line 616, in zmq.core.socket.Socket.recv
(zmq/core/socket.c:5961)
  File "socket.pyx", line 650, in zmq.core.socket.Socket.recv
(zmq/core/socket.c:5832)
  File "socket.pyx", line 120, in zmq.core.socket._recv_copy
(zmq/core/socket.c:1681)
  File "message.pyx", line 75, in zmq.core.message.copy_zmq_msg_bytes
(zmq/core/message.c:1082)
MemoryError
FATAL ERROR: OUT OF MEMORY (epoll.cpp:57)
/usr/share/gridengine/hpc/spool/cloudcompute-5/job_scripts/1998: line 14:
31003 Aborted                 (core dumped) ipcontroller --profile=sge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20120706/d2c108a9/attachment.html 


More information about the IPython-User mailing list