[IPython-User] slow startup time possibly because of sqlite

klo uo klonuo@gmail....
Wed Jan 30 10:57:08 CST 2013


Thanks Thomas,

Reason for my interest, was that I suspected my start time may suffer from
same problem, but I guess anyone mentioning start time problem will have my
attention as I have it too.

I thought to look at it, and browsed documentation for history manager. I
didn't saw writing to memory as option, but it's probably there somewhere.
Then I thought to check the timings and confirm that I have this issue
impact, although not as severe as I don't use such file system. But I
didn't noticed issue related to history by monitoring ipython-qtconsole
processes:

 - it looks for sqlite library two times (0.1s and then 0.2s)
 - it reads from history file in less then second ~0.8s

So there is no problem I guess.

Interesting to mention is that on Ubuntu I get qtconsole start in couple of
seconds, but on Windows it takes ~45s! Adobe applications doesn't take that
much time to start.

I browsed qtconsole process log (90K lines) but I can see obvious time jump
and it's hard to extract meaningful data, but I'll try to browse the log
again. I guess that is to be expected from complex Python app on Windows.


On Tue, Jan 29, 2013 at 7:05 PM, Thomas Kluyver <takowl@gmail.com> wrote:

> - Dumping to file on exit: sadly not as simple as just dumping the raw
> data in memory - as far as I know, you have to connect to a file, query the
> in-memory database, and insert the results into the file database. We don't
> want to be doing all that on shutdown, especially if the file is on NFS
> where it could have problems.
> - Loading other history files - you can create a separate HistoryAccessor
> already. Wrapping that in magic commands or whatever isn't a priority, at
> least for me.
>
> Both could probably be implemented as extensions if you want to play
> around with them.
>
> Thomas
>
>
> On 29 January 2013 07:21, klo uo <klonuo@gmail.com> wrote:
>
>> Do you think it would be useful, to allow option to write current session
>> to memory as in Matthias example, and:
>>
>>  - allow dumping history from memory to file by appending, on exit
>>  - allow IPython function to load stored history file, on demand
>>
>>
>> On Mon, Jan 28, 2013 at 11:32 PM, Thomas Kluyver <takowl@gmail.com>wrote:
>>
>>> On 28 January 2013 19:33, Matthias BUSSONNIER <
>>> bussonniermatthias@gmail.com> wrote:
>>>
>>>> IIRC we had some issues with history on NFS, but maybe it was only on
>>>> windows.
>>>
>>>
>>> It's not, unfortunately - SQLite over NFS is a general problem. And as
>>> far as we know, there's no way to easily detect whether a directory is on
>>> NFS. The workaround is to set HistoryManager.hist_file to a location that
>>> will be stored locally, or to :memory: (in which case, history will be lost
>>> between sessions).
>>>
>>> Thanks for the detailed analysis of the problem. From what you say,
>>> connecting from different threads may be the issue. We have a
>>> HistorySavingThread which opens its own connection to the database to write
>>> command history. Python's sqlite3 module has an undocumented option to
>>> share a single database connection between threads. We've been reluctant to
>>> play with it, but you might want to experiment with it.
>>>
>>> Best wishes,
>>> Thomas
>>>
>>> _______________________________________________
>>> IPython-User mailing list
>>> IPython-User@scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>>
>>>
>>
>> _______________________________________________
>> IPython-User mailing list
>> IPython-User@scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-user
>>
>>
>
> _______________________________________________
> IPython-User mailing list
> IPython-User@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.scipy.org/pipermail/ipython-user/attachments/20130130/9ad3b25b/attachment.html 


More information about the IPython-User mailing list