[Numpy-discussion] saving groups of numpy arrays to disk

Derek Homeier derek@astro.physik.uni-goettingen...
Fri Aug 26 07:04:20 CDT 2011


On 25.08.2011, at 8:42PM, Chris.Barker wrote:

> On 8/24/11 9:22 AM, Anthony Scopatz wrote:
>>    You can use Python pickling, if you do *not* have a requirement for:
> 
> I can't recall why, but it seem pickling of numpy arrays has been 
> fragile and not very performant.
> 
Hmm, the pure Python version might be, but, I've used cPickle for a long time 
and never noted any stability problems. And it is still noticeably faster than 
pytables, in my experience. Still, for the sake of a standardised format I'd 
go with HDF5 any time now (and usually prefer h5py now when starting 
anything new - my pytables implementation mentioned above likely is not 
the most efficient compared to cPickle). 

But with the usual disclaimers, you should be able to simply use cPickle 
as a drop-in replacement in the example below.

Cheers,
						Derek

On 21.08.2011, at 2:24PM, Pauli Virtanen wrote:

> You can use Python pickling, if you do *not* have a requirement for:
> 
> - real persistence, i.e., being able to easily read the data years later
> - a standard data format
> - access from non-Python programs
> - safety against malicious parties (unpickling can execute some code
>  in the input -- although this is possible to control)
> 
> then you can use Python pickling:
> 
> 	import pickle
> 
> 	file = open('out.pck', 'wb')
> 	pickle.dump(file, tree, protocol=pickle.HIGHEST_PROTOCOL)
> 	file.close()
> 
> 	file = open('out.pck', 'rb')
> 	tree = pickle.load(file)
> 	file.close()



More information about the NumPy-Discussion mailing list