[Numpy-discussion] saving incrementally numpy arrays

Keith Goodman kwgoodman@gmail....
Tue Aug 11 13:46:28 CDT 2009


On Tue, Aug 11, 2009 at 11:05 AM, Robert Kern<robert.kern@gmail.com> wrote:
> On Mon, Aug 10, 2009 at 22:29, Juan Fiol<fiolj@yahoo.com> wrote:
>> Hi, I am creating numpy arrays in chunks and I want to save the chunks while my program creates them. I tried to use numpy.save but it failed (because it is not intended to append data). I'd like to know what is, in your opinion, the best way to go. I will put a few thousands every time but building up a file of several Gbytes. I do not want to put into memory
>> all previous data each time. Also I cannot wait until the program finishes, I must save partial results periodically. Thanks, any help will be appreciated
>
> As others mentioned, PyTables is an excellent, complete solution. If
> you still want to write your own, then you can pass an open file
> object to numpy.save() in order to append. Just open it with the mode
> 'a+b' and seek to the end.
>
>  f = open('myfile.npy', 'a+b')
>  f.seek(0, 2)
>  numpy.save(f, chunk)
>  f.close()

That looks nice. What am I doing wrong?

>> x = np.array([1,2,3])
>> y = np.array([4,5,6])
>>
>> f = open('myfile.npy', 'a+b')
>> np.save(f, x)
>> f.seek(0, 2)
>> np.save(f, y)
>> f.close()
>>
>> xy = np.load('myfile.npy')
>> xy
   array([1, 2, 3])

I was expecting something like array([1, 2, 3, 4, 5, 6]).


More information about the NumPy-Discussion mailing list