Closed Bug 701145 Opened 13 years ago Closed 12 years ago

Firefox 8 has ridiculously slow bookmark searching and address bar dropdown with thousands of tags

Categories

(Firefox :: Bookmarks & History, defect)

Product:
Firefox
Component:
Bookmarks & History
Version:
8 Branch
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: thisisnotanapple, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: regression) 

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2

Steps to reproduce:

Initiate a bookmark search through the bookmark sidebar (ctrl-b) or the bookmark manager window, or click on the dropdown arrow at the right of the address bar.


Actual results:

Firefox 8 stops responding, I get the blue spinning cursor in Windows 7, and only after several minutes does the search finish or the dropdown populate and Firefox 8 becomes responsive again. This started only in the last few beta builds but persists


Expected results:

Should have finished the action instantly or within a few seconds. This started only in the last few beta builds of Firefox 8. It did not happen in 7.01 or previous beta 8 builds. But it persists in the release build of Firefox 8. There was a similar issue several versions ago which was eventually fixed. This is extremely frustrating as it renders Firefox nearly useless for me and it is now constantly trying to force me to upgrade from 7.01 to 8.0 on all my computers. 

I've run Firefox 8 in safe mode to rule out an extension but this does not help. 

Note also, even re-installing 7.01 does not immediately fix the issue. However, I've found that deleting the files "urlclassifier3.sqlite" and "places.sqlite" (40 and 20 Mb, respectively) will fix the issue after downgrading to 7.01. The first time Firefox loads after deleting these files it will remain hidden and hang indefitely, but if I kill the process and reload it then works fine and i can restore my bookmark library by downloading it from Xmarks, after which performance is just fine. If I do this same process using Firefox 8, it continues to have poor performance even after rebuilding the bookmark library.
try cleaning up your database with this addon https://addons.mozilla.org/firefox/addon/places-maintenance/
The add-on should also give you statistics you can paste here.

I'm not sure what's up there, but seeing you use xmarks it's possible your database is no as we expect it to be. You should really not have hangs, especially clicking the dropdown in the locationbar. If you wish you can compress your places.sqlite db and send it to me privately to check what's up there, just don't attach it here since it contains your sensitive data.
the fact it hangs when you delete places.sqlite and restart is also unexpected, may you check in the profile/bookmarksbackup.folder how large are your json backups? do you have xmarks enabled when you do this?
I ran the cleanup but it didn't help. Here's the output from the plugin:
> Integrity check
+ The database is sane
> Reindex
+ The database has been reindexed
> Orphans expiration
+ Database cleaned up
> Coherence check
+ The database is coherent
> Vacuum
Initial database size is 20480 KiB
+ The database has been vacuumed
Final database size is 20480 KiB
> Statistics
Database size is 20480 KiB
user_version is 12
page_size is 32768
cache_size is 15710
journal_mode is wal
synchronous is 1
History can store a maximum of 257394 unique pages
Table moz_places has 5296 records
Table moz_historyvisits has 24 records
Table moz_inputhistory has 0 records
Table moz_bookmarks has 87769 records
Table moz_bookmarks_roots has 5 records
Table moz_keywords has 16 records
Table sqlite_sequence has 1 records
Table moz_favicons has 789 records
Table moz_annos has 0 records
Table moz_anno_attributes has 6 records
Table moz_items_annos has 5594 records
Table sqlite_stat1 has 15 records
Index sqlite_autoindex_moz_inputhistory_1
Index sqlite_autoindex_moz_bookmarks_roots_1
Index sqlite_autoindex_moz_keywords_1
Index sqlite_autoindex_moz_favicons_1
Index sqlite_autoindex_moz_anno_attributes_1
Index moz_places_faviconindex
Index moz_places_hostindex
Index moz_places_visitcount
Index moz_places_frecencyindex
Index moz_places_lastvisitdateindex
Index moz_historyvisits_placedateindex
Index moz_historyvisits_fromindex
Index moz_historyvisits_dateindex
Index moz_bookmarks_itemindex
Index moz_bookmarks_parentindex
Index moz_bookmarks_itemlastmodifiedindex
Index moz_places_url_uniqueindex
Index moz_places_guid_uniqueindex
Index moz_bookmarks_guid_uniqueindex
Index moz_annos_placeattributeindex
Index moz_items_annos_itemattributeindex
Trigger moz_bookmarks_beforedelete_v1_trigger

I'm a little reluctant to send out my bookmark database but if I can't find a solution I'll consider it.

I'll contact Xmarks and see if they are aware of any issues with Firefox 8.

Also, it looks like I don't get the initial hang after deleting places.sqlite alone, but only if I also delete urlclassifier3.sqlite. Initially I wasn't sure where all the bookmarks and history were stored so i deleted both files as they are both sqlite databases and were the largest files in my profile. From what I understand now, it's only places.sqlite where bookmarks and history are stored and the other database has to do with security.

Thanks.
hm, this is a bit insane:
Table moz_bookmarks has 87769 records

Do you really have so many bookmarks, or do you use a lot tags? or maybe you have some add-on that tags bookmarks automatically? Like some time ago StumbleUpon was used to tag each single bookmark with his own tags.

urlclassifier is likely a separate bug, it's completely unrelated to the bookmarks/history service.
But the hang on startup can likely be due to us reimporting 87769 bookmarks from a backup... I'd really like to take a look at your bookmarks, the json backup from the backupBookmarks folder may be enough (as usual, by private means, like by mail).
I tried creating a new profile and importing bookmarks from the last autobacked up .json file but had the same problem so I don't think it's an extension or Xmarks that is the problem.

There seem to be about 5300 bookmarks and about 12000 tags. Many are years old and transferring them from my old bookmark manager to Firefox (once Firefox supported tags) was quite the ordeal. I don't have any desire to go back and try to manually curate the mess, but up until the latest version of Firefox it has been working fine.

I'll email you the json backup.

Thanks.
yeah the problem are likely the 12000 tags, not your fault clearly, we have a known issue with how tags are stored in bookmarks.
When you say that it worked fine until the latest version, you mean Firefox 7 is fast while firefox 8 is slow?
Yes, that's correct. As I mentioned, if I load Firefox 8, then downgrade to 7.01 using the same profile, I get the same slow behaviour in 7.01. BUT, if I delete the places.sqlite file and reconstruct my bookmark library by downloading again from Xmarks (while in 7.01) it goes back to being fast. Also, it's only the release and the last (or last few?) beta builds of 8 that actually show this problem. Earlier beta builds of 8 were as fast as 7.01. I can't tell exactly which build broke it. As soon as I noticed the issue i went back to 7.01 and had hoped that it was just a transient bug that would be fixed by the 8.0 release.

Thanks.
this is a longstanding problem, so the fact it "disappears" reconstructing the database is puzzling. Or better, I may think of a reason but seems a bit crazy.
How did you create these thousands tags? manually?

Btw, I received your json, will do some test on it.
Well, I just know from my experience that this problem (or something similar) was an issue a while ago (like Firefox 3 or 4) then it got fixed (at least for me) and everything worked fine again until the latest builds of Firefox 8. Maybe something else changed in Firefox 8 that has exacerbated the underlying issue?

How did I get so many tags? I used to use a bookmark manager called Powermarks, which advocated a flat database and use of tags instead of folders. They also had a sync server. In many ways the developer was way ahead of his time, but unfortunately didn't keep up. If i recall, that program tried to automatically suggest tags based on a variety of metadata, including page descriptions where it would assign a tag to each word in a description. Most likely, the majority of those tags were created by that feature, so most of it is not very useful but still there from my legacy imports. Today I only add tags manually when I add a new bookmark. I usually add on average 3 - 7 tags, trying to anticipate how I might search for it in the future once I've forgotten about it.
(In reply to Magritte from comment #8)
> How did I get so many tags? I used to use a bookmark manager called
> Powermarks, which advocated a flat database and use of tags instead of
> folders.

ah, I see, this created the performance problem.
While it's possible that there is a perf regression in 8 (will try to track that here) you should probably find a way to remove some of the redundant tags you don't use. I understand it may be annoying considered the many you have.
Assignee: nobody → mak77
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Trying to manage the tags is not something I see myself doing anytime soon. While it's a good idea, in theory, frankly Firefox's bookmark manager is not very good and doesn't provide the kind of sorting and analysis I'd need to do anything with the tags in an efficient manner. Sitting down and going through 12000 tags one by one isn't a viable option. Xmarks tag management isn't any better. I suppose at some point I can sit down and try to figure out how bookmarks and tags are managed in the json format or in the sqlite database itself and try to script something.

I was about to write that the issue seemed to have been fixed in 9.0b1. I downloaded the beta, then deleted the places.sqlite file and downloaded again from Xmarks and initially I was able to perform searches instantly again. Unfortunately, somewhere between the few minutes where it worked and when I tested it again before posting here, something happened and its back to behaving like 8.0 where searches are unacceptably slow. It looks like it's back to 7.01 for me until this gets fixed. 

Thanks.
got another report in the #sync channel that this is happening with 1600 tags.
It's likely a regression in this hot code path.
Keywords: regression
Summary: Firefox 8 has ridiculously slow bookmark searching and address bar dropdown. → Firefox 8 has ridiculously slow bookmark searching and address bar dropdown with thousands of bookmarks
Ok, I analyzed the 1600 tags case (I'm still trying to import the 12 000 tags backup, but I have been unable so far, due to our tags implementation), and I have various perf improvements in different parts of the code.
Specifically for this bug, I've identified an index exclusion that, in debug mode, reduced the search time from 5 seconds to 30ms. This specific fix could be backported, I think.
To simplify the review process I'll now split each improvement to its own patch.
I have filed the perf improvement I found so far, most of them are trivial and I have all of the patches ready. I'll keep this as sort of a tracking bug for the "Firefox is slow with lots of tags" issue.
The bug we may backport to solve the specific reported instance is bug 707954.
Thanks for your work on this!
Adding bug 424160 that is the long term fix, all the other simpler optimizations landed on mozilla-inbound and will be in mozilla-central in the next days.
Unassigning since there isn't more I can do, will nominate bug 707954 for Aurora (and ev. beta) after some days in Nightly.

Thank you for sending me your json backup, it has been really useful to profile performances.
Assignee: mak77 → nobody
Depends on: 424160
Summary: Firefox 8 has ridiculously slow bookmark searching and address bar dropdown with thousands of bookmarks → Firefox 8 has ridiculously slow bookmark searching and address bar dropdown with thousands of tags
Status: ASSIGNED → NEW
(In reply to Marco Bonardo [:mak] from comment #15)
> Adding bug 424160 that is the long term fix, all the other simpler
> optimizations landed on mozilla-inbound and will be in mozilla-central in
> the next days.
> Unassigning since there isn't more I can do, will nominate bug 707954 for
> Aurora (and ev. beta) after some days in Nightly.
> 
> Thank you for sending me your json backup, it has been really useful to
> profile performances.

No prob. I'm a little confused as to when/how the fixes will get worked into Firefox. At what point can I upgrade and still get reasonable performance? 

Thanks!
Bug 707954 has landed in mozilla-inbound, which corresponds with Firefox 11 (assuming it doesn't get backed out); you can also look at the target milestone in that bug (mozilla11). If the patch gets accepted for Aurora, it'll make it into Firefox 10. That will go into beta as soon as Firefox 9 is released.
In a couple weeks from now, part of the fix (bug 707954) should be in beta (Firefox 10), the next merge will happen around 20th of December.
The other bugs are too deep in code to backport them.
So does that mean the other bugs will never be fixed? Is 707954 sufficient restore Firefox ver 7.01 bookmark performance?

Thanks.
(In reply to Magritte from comment #19)
> So does that mean the other bugs will never be fixed? Is 707954 sufficient
> restore Firefox ver 7.01 bookmark performance?

It should be the most important fix among those. The other bugs will be fixed in Firefox 11.
Just tried Aurora 10.0a2 = speedy, speedy bookmark searches! 
Good work!
(In reply to Magritte from comment #21)
> Just tried Aurora 10.0a2 = speedy, speedy bookmark searches! 
> Good work!

Glad to hear that! Thanks for confirming the fix was effective.
Would you be fine with sharing the bookmarks backup you sent me with our QA team so that they can verify the fix? Otherwise I'll have to build a similar profile with fake data (would just take a bit more time)
That would be okay so long as its not publicly available. There are years of bookmarks there and no doubt things I've forgotten about that will destroy any future political aspirations...
no, surely it won't be made public, we care about our users!
I'd call this WFM for the ridiculous part.  More work can be done though this bug is likely not useful to track it.
Thanks!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.