[IPython-User] slow startup time possibly because of sqlite
Wed Jan 30 10:57:08 CST 2013
Reason for my interest, was that I suspected my start time may suffer from
same problem, but I guess anyone mentioning start time problem will have my
attention as I have it too.
I thought to look at it, and browsed documentation for history manager. I
didn't saw writing to memory as option, but it's probably there somewhere.
Then I thought to check the timings and confirm that I have this issue
impact, although not as severe as I don't use such file system. But I
didn't noticed issue related to history by monitoring ipython-qtconsole
- it looks for sqlite library two times (0.1s and then 0.2s)
- it reads from history file in less then second ~0.8s
So there is no problem I guess.
Interesting to mention is that on Ubuntu I get qtconsole start in couple of
seconds, but on Windows it takes ~45s! Adobe applications doesn't take that
much time to start.
I browsed qtconsole process log (90K lines) but I can see obvious time jump
and it's hard to extract meaningful data, but I'll try to browse the log
again. I guess that is to be expected from complex Python app on Windows.
On Tue, Jan 29, 2013 at 7:05 PM, Thomas Kluyver <firstname.lastname@example.org> wrote:
> - Dumping to file on exit: sadly not as simple as just dumping the raw
> data in memory - as far as I know, you have to connect to a file, query the
> in-memory database, and insert the results into the file database. We don't
> want to be doing all that on shutdown, especially if the file is on NFS
> where it could have problems.
> - Loading other history files - you can create a separate HistoryAccessor
> already. Wrapping that in magic commands or whatever isn't a priority, at
> least for me.
> Both could probably be implemented as extensions if you want to play
> around with them.
> On 29 January 2013 07:21, klo uo <email@example.com> wrote:
>> Do you think it would be useful, to allow option to write current session
>> to memory as in Matthias example, and:
>> - allow dumping history from memory to file by appending, on exit
>> - allow IPython function to load stored history file, on demand
>> On Mon, Jan 28, 2013 at 11:32 PM, Thomas Kluyver <firstname.lastname@example.org>wrote:
>>> On 28 January 2013 19:33, Matthias BUSSONNIER <
>>> email@example.com> wrote:
>>>> IIRC we had some issues with history on NFS, but maybe it was only on
>>> It's not, unfortunately - SQLite over NFS is a general problem. And as
>>> far as we know, there's no way to easily detect whether a directory is on
>>> NFS. The workaround is to set HistoryManager.hist_file to a location that
>>> will be stored locally, or to :memory: (in which case, history will be lost
>>> between sessions).
>>> Thanks for the detailed analysis of the problem. From what you say,
>>> connecting from different threads may be the issue. We have a
>>> HistorySavingThread which opens its own connection to the database to write
>>> command history. Python's sqlite3 module has an undocumented option to
>>> share a single database connection between threads. We've been reluctant to
>>> play with it, but you might want to experiment with it.
>>> Best wishes,
>>> IPython-User mailing list
>> IPython-User mailing list
> IPython-User mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-User