[SciPy-User] format for chunked file save and read ?

Robert Kern robert.kern@gmail....
Wed Sep 22 11:40:32 CDT 2010

On Wed, Sep 22, 2010 at 11:29, Nathaniel Smith <njs@pobox.com> wrote:
> On Wed, Sep 22, 2010 at 7:18 AM,  <josef.pktd@gmail.com> wrote:
>> What is the best file format for storing temporary data, for chunked
>> saving and loading, that only uses numpy and scipy?
>> I would like a file format that could be shared cross-platform and
>> across python/numpy versions if needed.
> Why not just use pickle? Mmap isn't giving you any advantages here
> that I can see, and pickles are much easier to handle when you want to
> write things out incrementally.

Large arrays are not written or read incrementally in a pickle. We
have some tricks in order to not duplicate memory, but they don't
always work.

Using np.save() and np.load() would probably also work fine. For most
data, they will use fromfile() and .tofile() so they will not
duplicate memory during the process.

Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco

More information about the SciPy-User mailing list