Mon Oct 10 03:15:04 CDT 2011
On Sun, Oct 9, 2011 at 2:37 PM, Jean-Louis Durrieu
> Hi everyone,
> I was just wondering something: lately, I had to use the load function, to load arrays stored in npz files.
> During one session, I need to read quite a few times several files (or even the same files), for some model training. I however just found out that the batch processing I ran failed because of a "too many open files" problem.
> After checking, with lsof, it seems that the use of np.load(filename), where filename is a string (= path to the file), worked an unexpected way. When I do the following, in a ipython 0.11 session, with the --pylab option :
> In : np.__version__
> Out: '1.6.1'
> In : np.load
> Out: <function numpy.lib.npyio.load>
> In : struc = np.load('path/to/file.npz')
> In : ar1 = struc['ar1']
> I would expect to have opened a file, read the array in it, and closed it. However, 'lsof' proved me wrong, and I found out that I need to explicitly do 'struc.close()' in order to close the file.
This is a documentation bug. If you look into the sources of load, you
will see that in the case of zipfile, a NpzFile instance is returned
by load. This is a file-like object, and needs to be closed. The
rationale is that it enables lazy-loading (not all arrays are loaded
in memory, only the one you request).
So for now, closing the returned NpzFile instance is the correct
solution. I added a note about this in the load doc, and a context
manager to NpzFile so you can also do (python >= 2.5 only):
with load('yo.npz') as data:
More information about the NumPy-Discussion