[Numpy-discussion] saving incrementally numpy arrays

Robert Kern robert.kern@gmail....
Tue Aug 11 13:05:21 CDT 2009


On Mon, Aug 10, 2009 at 22:29, Juan Fiol<fiolj@yahoo.com> wrote:
> Hi, I am creating numpy arrays in chunks and I want to save the chunks while my program creates them. I tried to use numpy.save but it failed (because it is not intended to append data). I'd like to know what is, in your opinion, the best way to go. I will put a few thousands every time but building up a file of several Gbytes. I do not want to put into memory
> all previous data each time. Also I cannot wait until the program finishes, I must save partial results periodically. Thanks, any help will be appreciated

As others mentioned, PyTables is an excellent, complete solution. If
you still want to write your own, then you can pass an open file
object to numpy.save() in order to append. Just open it with the mode
'a+b' and seek to the end.

  f = open('myfile.npy', 'a+b')
  f.seek(0, 2)
  numpy.save(f, chunk)
  f.close()

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the NumPy-Discussion mailing list