[IPython-User] Howto organize your workflow? / saving code plus results
Keith C Smith
Fri Jan 7 17:54:36 CST 2011
On 1/7/2011 9:00 AM, Johannes Korn wrote:
> I´m having trouble organizing my work. I analyze large datasets. I
> basically do this by writing a .py-file with some code (normally 30-500
> lines) in an editor, then I run the code in ipython, afterwards I
> inspect the results by manually plotting them or doing a bit of
> Often I realize that I need to change my code slightly and I have to
> re-run it. This means that the results of the first run get lost. Of
> course I can save the plots or save the data but after a week or so it
> turns out that I can´t remember if the plot is for the full data set or
> only parts of it because I was too sloppy to name the plot properly or I
> missed to save a critical piece of information etc...
> It would great to be able to save the full ipython session with all
> variables plus a snapshot of the code I executed in one file so I can
> look-up quickly what source code or what parameter set produced a
> certain result.
> Is there a simple way to do so? If not, how do you deal with this kind
> of problems?
> Kind regards!
> IPython-User mailing list
There are several different ways ipythoners deal with this issue. For
what's its worth, here's my suggestions
.py-files of several hundred lines is getting kind of large. I've found
in my work that by taking an object oriented approach I can usually
develop a set of classes which help me deal with the programming at a
higher level of abstraction. In other words I create a module of
reusable, general purpose classes that make my custom .py-files much
smaller, easier to understand and modify. without understanding your
problem domain its difficult to suggest anything, but try to take a look
at anything that you are doing over and over again which could be
Secondly, I started using reportLab to document my work by creating PDF
files that capture the code and results. It gives me something like a
MATLAB notebook capability. You might look into a library like
ReportLab to do something similar. That means you'll have to add
reportLab programming statements in you .py-files, but if you are able
to simplify them as I describe above, and write cover classes or
functions for doing common reportLab documentation tasks, then you may
still end up with a much smaller, more manageable .py-file that
generates a PDF report.
I hope my suggestions help.
More information about the IPython-User