[SciPy-User] format for chunked file save and read ?

Robert Kern robert.kern@gmail....
Wed Sep 22 10:12:26 CDT 2010


On Wed, Sep 22, 2010 at 10:03,  <josef.pktd@gmail.com> wrote:
> On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern <robert.kern@gmail.com> wrote:
>> On Wed, Sep 22, 2010 at 09:18,  <josef.pktd@gmail.com> wrote:
>>> What is the best file format for storing temporary data, for chunked
>>> saving and loading, that only uses numpy and scipy?
>>
>> An mmapped .npy file would probably serve the purpose well enough.
>
> Thanks, I will try.
>
> The old stats.models had one example with memmapped temporary storage,
> where I never managed to figure out how to find and close an open file
> handle (on Windows) that was not cleaned up on delete or when the
> instance went out of scope.

Use numpy.lib.format.open_memmap(). When the mmapped array goes out of
scope, the associated file object will also get finalized.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco


More information about the SciPy-User mailing list