[IPython-dev] Project idea: Automatic lab notebook for iPython
Thu Apr 18 19:45:11 CDT 2013
Is there something in your concept of provenance that git or a similar dvcs
does not cover?
It would seem that the kind of granular history tracking built into git
could be taken advantage of with a small amount of semantic glue on top to
give you what you want. Plus you get a large global user community.
There are well defined API's wrapping a git client lib that would make this
feasible. If this makes sense please feel free to connect via nborwankar
I've spent non-trivial amount of time exploring how git could be used as a
database-with-history for heterogenous data.
On Tue, Apr 16, 2013 at 4:38 PM, Peter Macko <firstname.lastname@example.org>wrote:
> Hi iPython developers,
> Here is a new project idea: automatic lab notebook for iPython and
> iPython Notebook, which would keep track of how each of your output
> files was produced, linking this "history" (or a "lineage") of an object
> across different iPython sessions and different iPython notebooks, and
> storing it persistently. This is frequently referred to in the Computer
> Science literature as "provenance."
> It will enable you to ask questions like "what did I do to produce this
> plot?" - and for example, it will tell you that you downloaded the input
> data set on Monday from such and such website, you ran all these
> commands to process the data on Tuesday, and then produced this plot on
> Thursday from a different iPython session. Note that this goes beyond
> (and is complementary in purpose to) iPython Notebook, since the history
> of a file is tracked across different sessions and Notebooks, and when
> you ask a question, you will get only the relevant information,
> suppressing any additional things that you did that are unrelated to the
> file in which you are interested.
> We are in touch with computational scientists all the way from
> bioinformatics to physics that are very interested in this feature! We
> met their needs partially by developing a cross-platform, multi-lingual
> library (https://code.google.com/p/core-provenance-library/) that they
> can use to annotate their Python (and non-Python) scripts in order to
> track the lineage of their objects.
> Our vision is that this will be all done fully automatically, without
> requiring the users to manually annotate their scripts. But
> unfortunately neither of us who are involved in this project has the
> resources or the knowledge of the iPython code-base to tackle this
> challenge. We need your help to make this happen! We have some ideas
> about how we might go about this, but we need someone who knows more
> about iPython to talk them over and to spearhead the actual development.
> Please let us know if you can help!
> Thank you,
> Peter Macko
> Harvard School of Engineering and Applied Sciences
> 33 Oxford St.
> Cambridge, MA 02138
> IPython-dev mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-dev