[SciPy-User] format for chunked file save and read ?
Wed Sep 22 10:52:18 CDT 2010
On Wed, Sep 22, 2010 at 10:41, <email@example.com> wrote:
> On Wed, Sep 22, 2010 at 11:12 AM, Robert Kern <firstname.lastname@example.org> wrote:
>> On Wed, Sep 22, 2010 at 10:03, <email@example.com> wrote:
>>> On Wed, Sep 22, 2010 at 10:36 AM, Robert Kern <firstname.lastname@example.org> wrote:
>>>> On Wed, Sep 22, 2010 at 09:18, <email@example.com> wrote:
>>>>> What is the best file format for storing temporary data, for chunked
>>>>> saving and loading, that only uses numpy and scipy?
>>>> An mmapped .npy file would probably serve the purpose well enough.
>>> Thanks, I will try.
>>> The old stats.models had one example with memmapped temporary storage,
>>> where I never managed to figure out how to find and close an open file
>>> handle (on Windows) that was not cleaned up on delete or when the
>>> instance went out of scope.
>> Use numpy.lib.format.open_memmap(). When the mmapped array goes out of
>> scope, the associated file object will also get finalized.
> I will have to experiment more systematically. Now I have several
> memmapped files in my working directory that windows blocks as
> accessed, and I'm not able to close them until I shut down the python
> shell (which I don't want to do since I have other work in it.)
Are you using IPython? IPython can hold onto references to many of the
objects that you use.
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
More information about the SciPy-User