[IPython-User] PLEASE make autosave standard

David Warde-Farley d.warde.farley@gmail....
Wed Nov 7 04:48:36 CST 2012


I understand you're frustrated by losing your work. It's completely
understandable, and we've all been there. But please consider your tone
carefully, and try to remember that (nearly?) everyone working on IPython
is doing so on a volunteer basis, in their spare time.

The core developers on this project, Thomas and Matthias included, are a
clever bunch, and tend not to want to "half-ass" things. Despite IPython's
long history, the notebook is less than a year old -- and given the
reception it has received from the community, many would agree that it was
worth the wait -- but, despite the tremendous effort behind the transition
to a two process model and the implementation of the notebook
infrastructure and client, it is still not quite a finished product.
Matthias has briefly outlined some of the technical considerations involved
in getting autosave functionality done in a clean, transparent and
future-proof manner. I can think of a few other snags and possible
solutions but Matthias is far more familiar with the notebook's codebase
than I am. In an application with as many moving parts as the notebook, the
actual implementation is always going to be more complicated than it seems
in a high-level bullet-point description, and in an application with an
API, users expect stability of that API -- quickly and arbitrarily changing
things to accomodate autosave might be doable without changes to that API,
it might not, but either way it is not something that can be undertaken
lightly. Perhaps worse is the possibility of introducing new APIs,
realizing they suck, and having to deprecate them in favour of the "right"
solution, resulting in frustration on the part of people who have, in the
mean time, come to depend on those APIs.

I agree with you that it's a very important feature, one that I would love
to see. But between all of the currently open bugs and the limited amount
of developer bandwidth, plus the availability of Min RK's extension as a
short-term fix, it may be a while before the developers have the time to
devote in order to get it right.

But code talks. If you truly believe both that this is an important feature
(I'm with you on that one), and that the right design decisions are
self-evident/the implementation is straightforward (I'm a good deal less
certain about that), give it a go and open a pull request. It probably will
not suffice right off the bat, but it will get developers thinking about it
and paying attention to it sooner (it is much easier to reason about and
offer constructive criticism about a concrete implementation than the
abstract idea of one). Provided you have the time and patience, they can
guide you in the direction of improving it to the point that merging might
be an option. Importantly, do not think of a pull request as an
accept/reject patch submission. GitHub themselves[1] describe their use of
pull requests as a way to "start a conversation" about a feature and
receive feedback about it as it evolves, and in my experience this is a
healthy model of pull request-driven development.

I apologize if I've read more hostility into your posts than there actually
was, but in virtually all cases, you will catch more flies with honey than
with vinegar. That is to say, no one expects you to be a sycophant or a
suck-up, but please try to temper your understandable agitation with the
knowledge that the developers have only the best intentions for improving
and maintaining IPython for *all* of its users, and moreover that they do
so in spare moments of otherwise busy careers/lives, largely without
compensation of any kind. Approaching conversations like this with both a
calm tone and an open mind will invariably get us all further toward
solving the problem.


[1]: https://github.com/blog/1124-how-we-use-pull-requests-to-build-github

On Wed, Nov 7, 2012 at 4:55 AM, Tom <tmbdev@gmail.com> wrote:

> The take-home message from your response is that iPython Notebooks are not
> going to get any form of autosave or crash protection any time soon, and
> that if there is data loss from crashes, you consider it the user's fault.
> Since there hasn't been any other response to this, I assume that's
> official iPython policy.
> Thanks for taking the time to clarify this.
> Tom
> On Wed, Nov 7, 2012 at 10:15 AM, Matthias BUSSONNIER <
> bussonniermatthias@gmail.com> wrote:
>> Le 7 nov. 2012 à 03:01, Tom a écrit :
>> > "This is all solvable, but it would be a significant expenditure of
>> time and effort, and so far no-one's got round to doing it."
>> >
>> > It only becomes a significant expenditure of effort because people are
>> trying to reinvent the wheel and mix this issue up with unrelated and
>> complicated issues of backup files, multiple instances, version control,
>> and god knows what else. In fact, autosave is pretty simple:
>> Even if you consider 'just autosaving' it is a pretty big amount of work.
>> Especially if it is not done correctly, because we will have to deal with
>> legacy code, but let's forgot this one.
>> > - periodically, you save the notebook to an autosave file, but only if
>> it has been modified
>> Ok, let's say timer in javascript.
>> The API to do so does not exist. I once try to use the "save as" function
>> in JS periodically, but "save as" is actually "rename", and I lost a day of
>> work.
>> So you'll need to create a "save_copy(name)" API. (Potentially breaking
>> API that other client rely on, meaning at least 1 version with deprecation
>> warning and design legacy mode option?)
>> > - autosave only saves user input (which is all text fairly small), not
>> the output or images
>> Gaps, ok, I have to patch cell, codecell, markdowncell, header cell, raw
>> cell, in JS, which none of us really know.
>> Think of future cell types that might inherit and should also be saved.
>> Add a method which 'serialize input only' of notebook. Shouldn't be too
>> hard, design a API that send a notebook through the wire that have only
>> inputs.
>> Do you send only input or ipynb file strip of output ? are they "real
>> ipynb file"?, do you keep metadata ? Cell type ? Prompt number ?
>> If some fields are not existing or empty, do you save them or not ? You
>> might not care, but the server does. If you forget require field then
>> backup files are un readable, if you put unnecessary one, you are doing too
>> much work for nothing.
>> Now, for long notebook, serialization you will might feel a hang in the
>> browser every now and then, hopefully nobody will complain.
>> > - the autosave file for "filename.ipynb" is "#filename.ipynb#" in the
>> same directory
>> Whatever the filename, maybe leading dot to have it hidden. Still be
>> carefull with extension.
>> If it end with ipynb you have to take care of not showing it into the
>> dashboard.
>> Btw, we have many possible backend, how do you do when the backend is not
>> a filesystem base but a database ?
>> > - the autosave file is removed whenever a manual save takes place (and
>> only gets recreated once there are modifications)
>> This is a little more complicated as the server does not have notion of
>> wether the browser have a modified version, but doable.
>> The subject of backup files has been discussed on the ml before, the
>> deletion of them is a complex problem, even emacs and vi user are sometime
>> complaining about not deleted swap file. Let's assume everyone agree on a
>> how/where to save so that you don't have to make the all thing configurable.
>> > - when you open a file, you check whether there is an autosave file for
>> it and offer the user to recover it, open it as a new file, or delete it
>> Ok, now you need a full interaction between server and browser, with
>> dialog, handler, thinking of all possible actions.
>> Knowing that some other programs (like emacs) also speak to server, you
>> need to design an API, document it.
>> You probably wan't to add that in the dashboard.  Even it dashboard look
>> simple, the code is not that easy.
>> You probably want the user to be able to browse the 2 files independently
>> before choosing which one to keep.
>> (we won't try yet to show a diff between 2 ipynb files...).
>> So this means now that the server should be able to let you open
>> ipynbfiles that are not ipynb files, so that are not listed in some places,
>> but listed in others, and attach kernel to it (probably), otherwise you
>> need a RO view which is non functional right now.
>> Just to be sure, when you open a file for a fist time it started a kernel
>> ? right ? and to shut down this kernel, you... Click on shutdown on the
>> dashboard.
>> But backup files are not shown ! How do I shut down their kernel ?
>> (Btw this might be a bug, if you delete a file from the file explorer,
>> what become of its kernel, can you shut it down easily?)
>> > If doing autosave well over slow connections looks too complicated,
>> just turn it off for slow/remote conections and put up a warning indicator
>> in the menu bar; autosaving 95% of the time is a lot better than autosaving
>> 0% of the time, which is the current situation.
>> Then we need to define "slow", if you want to make it configurable (even
>> enable/diasble), then you have to design way to push config from server to
>> browser.
>> Which is right now not existing at all.
>> > "Perhaps it's better to focus on reducing the need for autofocus. As
>> others have said, Ctrl+S should work in all browsers to save the notebook.
>> The original post also mentions problems with long output"
>> >
>> > You cannot fix this by trying to fix bugs elsewhere. No matter how many
>> bugs you fix, computers crash, lose power, and get forced reboots from
>> Windows updates. Nor is manual saving a solution because people don't
>> expect to have to do that anymore and simply will not remember. I don't
>> remember to (even now that it works), which is why I keep losing work in
>> iPython.
>> Now you have something working, maybe configurable, maybe not.
>> It is not well designed but it does it job, it took you a full day, pull
>> request not included, you didn't add test_case, and even don't bother
>> testing on windows.
>> Now you want to redesign the notebook to handle files in multiple
>> directory.
>> Well basically now you curse yourself, because you have twice as much
>> things to redisign and it would take you 2 weeks instead of one, and you
>> will break other people code that rely on what you did.
>> > Implementing autosave is a small task and easily done if you decide to
>> do it, and it greatly alleviates the pain resulting from many other bugs.
>> We do care that people lost data some of their work, and we want to avoid
>> it as much as possible. The problem is that we have to balance between
>> doing things the right way, at the right time or hacking something in a
>> afternoon, and being able to relatively easily add new features.
>> Having a full blown autosave as part of IPython notebook is far from
>> being as 'easily done' as you think. We'll do our best make it as easy as
>> possible to do what you want. It will just require sometime to wait for the
>> feature to be ready, in the meantime, you will have to inject some of you
>> code directly into IPython.
>> Just to finish on a personal note.
>> In some class, we warn student, "please regularly save you work" we even
>> remind them every 10-20 minutes during the exams.
>> A colleague of my tell them "I will cut the power when the time is over",
>> and once  the time is over it count down from 10 to zero and does
>> effectively cut the power of the all room.
>> Believe me that after that, student **do** save their work regularly.
>> Teaching student not to rely on machines might be a good thing sometime.
>> > Tom
>> >
>> > On Wed, Nov 7, 2012 at 1:44 AM, Jon Wilson <jsw@fnal.gov> wrote:
>> > On 11/06/2012 06:19 PM, Carl Smith wrote:
>> > > it'd probably be more useful if it displayed
>> > > the amount of time since the last save
>> >
>> > Or even (also?) tell you how many cells have changed since the last
>> save.
>> > Regards,
>> > Jon
>> >
>> > _______________________________________________
>> > IPython-User mailing list
>> > IPython-User@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-user
>> >
>> > _______________________________________________
>> > IPython-User mailing list
>> > IPython-User@scipy.org
>> > http://mail.scipy.org/mailman/listinfo/ipython-user
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20121107/b3e6bf16/attachment-0001.html 

More information about the IPython-User mailing list