[Numpy-discussion] Pickle, pytables, and sqlite - loading and saving recarray's

Vincent Nijs v-nijs@kellogg.northwestern....
Thu Jul 19 21:42:42 CDT 2007


I am interesting in using sqlite (or pytables) to store data for scientific
research. I wrote the attached test program to save and load a simulated
11x500,000 recarray. Average save and load times are given below (timeit
with 20 repetitions). The save time for sqlite is not really fair because I
have to delete the data table each time before I create the new one. It is
still pretty slow in comparison. Loading the recarray from sqlite is
significantly slower than pytables or cPickle. I am hoping there may be more
efficient ways to save and load recarray¹s from/to sqlite than what I am now
doing. Note that I infer the variable names and types from the data rather
than specifying them manually.

I¹d luv to hear from people using sqlite, pytables, and cPickle about their
experiences. 

saving recarray with cPickle:       1.448568 sec/pass
saving recarray with pytable:      3.437228 sec/pass
saving recarray with sqlite:         193.286204 sec/pass

loading recarray using cPickle:    0.471365 sec/pass
loading recarray with pytable:     0.692838 sec/pass
loading recarray with sqlite:        15.977018 sec/pass

Best,

Vincent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070719/20749e95/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: load_tables_test.py
Type: application/octet-stream
Size: 4154 bytes
Desc: not available
Url : http://projects.scipy.org/pipermail/numpy-discussion/attachments/20070719/20749e95/attachment.obj 


More information about the Numpy-discussion mailing list