[IPython-User] Howto organize your workflow? / saving code plus results

[Keith C Smith]

On 1/7/2011 9:00 AM, Johannes Korn wrote:
> Hi,
>
> I´m having trouble organizing my work. I analyze large datasets. I 
> basically do this by writing a .py-file with some code (normally 30-500 
> lines) in an editor, then I run the code in ipython, afterwards I 
> inspect the results by manually plotting them or doing a bit of 
> post-processing.
>
> Often I realize that I need to change my code slightly and I have to 
> re-run it. This means that the results of the first run get lost. Of 
> course I can save the plots or save the data but after a week or so it 
> turns out that I can´t remember if the plot is for the full data set or 
> only parts of it because I was too sloppy to name the plot properly or I 
> missed to save a critical piece of information etc...
>
> It would great to be able to save the full ipython session with all 
> variables plus a snapshot of the code I executed in one file so I can 
> look-up quickly what source code or what parameter set produced a 
> certain result.
>
> Is there a simple way to do so? If not, how do you deal with this kind 
> of problems?
>
> Kind regards!
>
> Johannes
>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
Johannes,
There are several different ways ipythoners deal with this issue.  For
what's its worth, here's my suggestions

.py-files of several hundred lines is getting kind of large.  I've found
in my work that by taking an object oriented approach I can usually
develop a set of classes which help me deal with the programming at a
higher level of abstraction.  In other words I create a module of
reusable, general purpose classes that make my custom .py-files much
smaller, easier to understand and modify.  without understanding your
problem domain its difficult to suggest anything, but try to take a look
at anything that you are doing over and over again which could be
generalized somehow.

Secondly, I started using reportLab to document my work by creating PDF
files that capture the code and results.  It gives me something like a
MATLAB notebook capability.  You might look into a library like
ReportLab to do something similar.   That means you'll have to add
reportLab programming statements in you .py-files, but if you are able
to simplify them as I describe above, and write cover classes or
functions for doing common reportLab documentation tasks, then you may
still end up with a much smaller, more manageable .py-file that
generates a PDF report.

I hope my suggestions help.
Keith Smith