[IPython-User] Howto organize your workflow? / saving code plus results
Sun Jan 9 23:24:40 CST 2011
For what it's worth, I feel Johannes' pain. Read my detailed whinging
below, but the only concrete suggestion I have for iPython is that it
provides a slicker way to reload all modules. For example an "%rrun"
magic which runs a .py file but recursively forces all modules it uses
to be reloaded - (if that's too slow, you can exclude paths which are
obviously system libraries). I don't think it would solve Johannes'
problem, but it would solve a lot of mine. Also I thank Keith for the
suggestion about ReportLab, I will check it out when I get the chance.
Now for some real whinging:
While Keith is right that Proper Programming Techniques will help tame
complexity, they are not always a good match for the world of data
analysis, where most of what you write will not be re-used the usual
sense of the word, and encapsulation / data hiding is a royal pain in
the nether. Sometimes I find that *functions* are too much of an
abstraction, let alone classes.
You basically want to be able to poke around in absolutely
*everything*, at any moment. Which is why it's tempting to write
(numeric) Python as if you are using MATLAB (which it resembles). But
the reason I am using Python is that it's a Proper Programming
language that can handle the load when things get complicated.
So I find myself in a catch-22.
On 7 January 2011 23:54, Keith C Smith <email@example.com> wrote:
> On 1/7/2011 9:00 AM, Johannes Korn wrote:
>> I´m having trouble organizing my work. I analyze large datasets. I
>> basically do this by writing a .py-file with some code (normally 30-500
> .py-files of several hundred lines is getting kind of large. I've found
> in my work that by taking an object oriented approach I can usually
> develop a set of classes which help me deal with the programming at a
> higher level of abstraction. In other words I create a module of
> reusable, general purpose classes that make my custom .py-files much
> smaller, easier to understand and modify. without understanding your
> problem domain its difficult to suggest anything, but try to take a look
> at anything that you are doing over and over again which could be
> generalized somehow.
> Secondly, I started using reportLab to document my work by creating PDF
> files that capture the code and results. It gives me something like a
> MATLAB notebook capability. You might look into a library like
> ReportLab to do something similar. That means you'll have to add
> reportLab programming statements in you .py-files, but if you are able
> to simplify them as I describe above, and write cover classes or
> functions for doing common reportLab documentation tasks, then you may
> still end up with a much smaller, more manageable .py-file that
> generates a PDF report.
> I hope my suggestions help.
> Keith Smith
> IPython-User mailing list
Centre for Cold Matter,
The Blackett Laboratory,
Imperial College London,
Prince Consort Road,
London SW7 2AZ.
More information about the IPython-User