[Numpy-discussion] saving incrementally numpy arrays
Tue Aug 11 13:46:28 CDT 2009
On Tue, Aug 11, 2009 at 11:05 AM, Robert Kern<email@example.com> wrote:
> On Mon, Aug 10, 2009 at 22:29, Juan Fiol<firstname.lastname@example.org> wrote:
>> Hi, I am creating numpy arrays in chunks and I want to save the chunks while my program creates them. I tried to use numpy.save but it failed (because it is not intended to append data). I'd like to know what is, in your opinion, the best way to go. I will put a few thousands every time but building up a file of several Gbytes. I do not want to put into memory
>> all previous data each time. Also I cannot wait until the program finishes, I must save partial results periodically. Thanks, any help will be appreciated
> As others mentioned, PyTables is an excellent, complete solution. If
> you still want to write your own, then you can pass an open file
> object to numpy.save() in order to append. Just open it with the mode
> 'a+b' and seek to the end.
> f = open('myfile.npy', 'a+b')
> f.seek(0, 2)
> numpy.save(f, chunk)
That looks nice. What am I doing wrong?
>> x = np.array([1,2,3])
>> y = np.array([4,5,6])
>> f = open('myfile.npy', 'a+b')
>> np.save(f, x)
>> f.seek(0, 2)
>> np.save(f, y)
>> xy = np.load('myfile.npy')
array([1, 2, 3])
I was expecting something like array([1, 2, 3, 4, 5, 6]).
More information about the NumPy-Discussion