[IPython-User] Howto organize your workflow? / saving code plus results

Adrian Ratnapala adrian.ratnapala@gmail....
Sun Jan 9 23:24:40 CST 2011

For what it's worth, I feel Johannes' pain.  Read my detailed whinging
below, but the only concrete suggestion I have for iPython is that it
provides a slicker way to reload all modules.  For example an "%rrun"
magic which runs a .py file but recursively forces all modules it uses
to be reloaded - (if that's too slow, you can exclude paths which are
obviously system libraries).   I don't think it would solve Johannes'
problem, but it would solve a lot of mine. Also I  thank Keith for the
suggestion about ReportLab, I will check it out when I get the chance.

Now for some real whinging:

While Keith is right that Proper Programming Techniques will help tame
complexity, they are not always a good match for the world of data
analysis, where most of what you write will not be re-used the usual
sense of the word, and encapsulation / data hiding is a royal pain in
the nether.  Sometimes I find that *functions* are too much of an
abstraction, let alone classes.

You basically want to be able to poke around in absolutely
*everything*, at any moment.  Which is why it's tempting to write
(numeric) Python as if you are using MATLAB (which it resembles).  But
the reason I am using Python is that it's a Proper Programming
language that can handle the load when things get complicated.

So I find myself in a catch-22.

On 7 January 2011 23:54, Keith C Smith <kcsmith@raytheon.com> wrote:
> On 1/7/2011 9:00 AM, Johannes Korn wrote:
>> Hi,
>> I´m having trouble organizing my work. I analyze large datasets. I
>> basically do this by writing a .py-file with some code (normally 30-500

> .py-files of several hundred lines is getting kind of large.  I've found
> in my work that by taking an object oriented approach I can usually
> develop a set of classes which help me deal with the programming at a
> higher level of abstraction.  In other words I create a module of

> reusable, general purpose classes that make my custom .py-files much
> smaller, easier to understand and modify.  without understanding your
> problem domain its difficult to suggest anything, but try to take a look
> at anything that you are doing over and over again which could be
> generalized somehow.
> Secondly, I started using reportLab to document my work by creating PDF
> files that capture the code and results.  It gives me something like a
> MATLAB notebook capability.  You might look into a library like
> ReportLab to do something similar.   That means you'll have to add
> reportLab programming statements in you .py-files, but if you are able
> to simplify them as I describe above, and write cover classes or
> functions for doing common reportLab documentation tasks, then you may
> still end up with a much smaller, more manageable .py-file that
> generates a PDF report.
> I hope my suggestions help.
> Keith Smith
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user

Adrian Ratnapala

Centre for Cold Matter,
The Blackett Laboratory,
Imperial College London,
Prince Consort Road,
London SW7 2AZ.

Tel: +44-(0)2075947864
Fax: +44-(0)2075947714

More information about the IPython-User mailing list