332934 - extremely slow startup as bookmarks_history.sqlite file grows

Status: RESOLVED FIXED

(Firefox :: Bookmarks & History, defect, P1)

Description

[Asa Dotzler [:asa]]

I've got history set to expire after 20 days (rather than the default 9.) I've got a reasonably sized bookmarks file (fewer than 300 bookmarks) and my bookmarks_history.sqlite file is up to about 6MB.

I'm on the latest 1.8 boncho nightlies and startup takes about 12 seconds with this file. If I remove the bookmarks_history.sqlite file, let the app create a new one, and restart, Firefox launches in about 4 seconds.

I think this is a serious performance degredation in startup time holding this many history/bookmark items compared to the pre-places infrastructure.

Comment 1

[Brett Wilson]

I have a preload patch for bug 331158 that will go in in a few days that improves startup time (I'm not positive how much, but potentially a lot). Either Darin or I plan to do some startup time profiling soon as well.

The real benchmark is pre-places with similar history. Mork degraded significantly with history size as well. It seem to be highly dependent on machine performance attributes. Darin's laptop performance is similar to yours. On some machines startup time is lower with places and large histories.

Comment 2

[Asa Dotzler [:asa]]

I can try to get some better measurements and against pre-places. I'll need to build up some history on a pre-places profile first, and then I can do some measures on that, then install a current boncho nightly and get measures there. Pre places I let history go to 99 days and the file would often get to be greater than 10MB and I don't recall startup ever getting as bad as it is right now for me. I know that's just anecdotal, so I'll try to get something better.

Comment 3

[Brett Wilson]

With your startup time, I would expect places to still be noticably slower. It would be interesting to hear your tests, but don't spend a lot of time on it at this point.

Comment 4

[Asa Dotzler [:asa]]

OK. I won't spend too much time. would be nice to have something of a baseline, even if rough, to see what kind of improvement comes from bug 331158. I can probably rack up a rather large history pretty quickly with the Link Visitor extension :-)

Comment 5

[Brett Wilson]

Now that bug 331158 has landed, I'm interested in people's startup experience.

Comment 6

[Brett Wilson]

Attachment: Patch

This patch should fix the slow startup time problem. The issue was that finding bookmark redirects did a statement that wasn't using an index. This resulted in a linear scan of every web page you visited for each bookmark you have. This was fixed by adding an index on moz_historyvisit.from_visit.

I also fixed some of the bookmark redirect statements which were returning nulls, and added a transaction around committing lazy messages which I noticed improves runtime performance in some cases when I was creating thousands of URLs to make my database very large (shouldn't affect Tp much, unfortunately).

My tests were on a 41MB database. Hash generation time (only part of startup time):

Before patch: 6.188 seconds
After patch : 0.000781 seconds

I think startup time with large histories should be faster than 1.5 now, though I haven't tested it.

Comment 7

[Doug Halamay]

I can confirm that the patch decreases startup time by a considerable (~10 s) amount of time...I have a 9.75MB bookmarks_history.sqlite file.  Wow...what an improvement!

Comment 8

[Doug Halamay]

One other thing...in the patch for nsNavHistory.cpp, the following line:

+  // page. For final release, if we thing everybody running alpha1 has run

should probably read:

+  // page. For final release, if we think everybody running alpha1 has run

(note the change from "thing" to "think".

Comment 9

[Brett Wilson]

Fixed on trunk. Waiting for branch to reopen...

Note: the first time you run a build with this patch, it will generate a new index. If you have a large history, this can take a while. My 40MB .sqlite test file took 20 seconds or so on my fast computer. It will only do this once.

Comment 10

[Jim Jeffery not reading bug-mail 1/2/11]

Could this be checked into the branch now ?  On my older machine its taking 45 secs for Firefox to open, really looking forward to testing. 

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20060424 BonEcho/2.0a1 ID:2006042405

Comment 11

[Brett Wilson]

On branch.

Comment 12

[u88484]

Latest trunk at work and startup is still slow as molasses (sp?), I spend roughly 10 hours 6 days a weeks surfing. History expires after 14 days.

Comment 13

[Brett Wilson]

(In reply to comment #12)
> Latest trunk at work and startup is still slow as molasses (sp?), I spend
> roughly 10 hours 6 days a weeks surfing. History expires after 14 days.

Can you give me some idea what these numbers are? Do you know how fast 1.5 will start on your system with similar history?

Also, if your bookmarks_history.sqlite is larger than 6% of your memory some things will be slower. You can set the preference "browser.history_cache_percentage" to a larger integer like 20 to see if it helps.

Comment 14

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv