[IPython-user] How to save a pylab workspace

Fernando Perez Fernando.Perez at colorado.edu
Wed Oct 12 15:23:36 CDT 2005


Jouni K Seppanen wrote:
> One Matlab feature I'm missing in pylab is the save and load functions
> for saving the workspace to disk. Ipython does have functions called
> load and save, but they use an ASCII format, which takes up a lot of
> space (and I imagine there would be some loss of accuracy for
> floating-point numbers). It also seems to handle only arrays, not e.g. 
> dicts. I have some ideas on how to do the equivalent of Matlab's
> load/save in pylab, and am writing to ask for the opinions of more
> experienced folks: am I missing something obvious, or otherwise doing
> something stupid?
> 
> What I've been doing is pack up all my variables in a dict, and use
> cPickle.dump(the_dict, file, -1) to put it in a file. This seems to
> work nicely, but when I load my dict back, I need to type x['foo'] to
> get at variable foo. So, I put a trivial class around the dict:

[...]

> So, does this seem reasonable, or should I be using some preexisting
> functionality in Python or pylab? Would it be useful to incorporate
> something like this in pylab?

The reason why I've never added this functionality internally is because it's 
_extremely_ brittle.  There are a lot of things that pickle won't serialize, 
and all it takes is one single variable that doesn't make it out cleanly to 
blow up a namespace serialization.  At that point, it's impossible to know 
whether the variables that didn't make it are critical to the user or not.

It would be easy to write a routine with a calling API of the type

savevars('myvars.pkl','a','b','c')

along with the corresponding

x = loadvars('myvars.pkl')

This would obviously blow up for anything unpickleable, but that would be up 
to the user, since there is no way to determine ahead of time what objects 
will be pickleable.

So this is just a thin wrapper around what you are already doing.  If this 
were implemented, it would probably be best done with a shelve instead of a 
pickle, since that allows you to provide better feedback (variable by 
variable), and upon loading you don't pay the price of reading the whole thing 
up front.

Note that you can always update your global namespace via

__IP.user_ns.update(mydict)

So to summarize:

1. your idea is sound, but IMHO too brittle to ship as a builtin.  Since it 
_will_ break randomly for people, I'd rather not include it in the default build.

2. with a little cleanup and shelve instead of pickle, it can be convenient 
for users who know exactly what its limitations are.  Feel free to contribute 
the recipe to the IPython cookbook at the wiki: 
http://projects.scipy.org/ipython/ipython/wiki/CookBook

Cheers,

f




More information about the IPython-user mailing list