1172828 - Awesomebar suggestions are very slow with a lot of tags

Status: NEW

(Firefox :: Address Bar, defect, P3)

Description

[0RUBn0]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
Build ID: 20150525141253

Steps to reproduce:

Type in a word in the awesomebar and wait (or press "down") (normal mode or safe mode).


Actual results:

Only after 30-60 seconds I get a dropdown with suggestions. 
Sometimes I get 3 results with no results from bookmarks tags fitting the word, sometimes I get a lot of results with a lot of tags fitting the same word.  (As if the script sometimes says "This takes too long, I will only display other resulst")




Expected results:

Much faster reaction of the suggestions dropdown.

I suspect this behavior to be related to the fact that I have a lot of bookmarks (1861) and especially a lot of tags (522).

Note: I did recently update my bookmarks with sql.
I can't imagine a firefox sql query or script running slow because of this. If someone would think so, I am laying  the question forward that this could be poor "DB querying design".
This is the script if anyone thinks that could be related:
http://attic.e-motiv.net/ff-delicious/delicious-to-firefox

I do not experience the same slowliness with Firefox Developer Edition 40, with about the same amount of bookmarks and tags after having done the same sql script. (There, it's about 5-12 seconds. Still to slow, but "useable".)

Comment 1

[Marco Bonardo [:mak]]

that's not a lot of bookmarks, looks like you might have database coherence issues?

By issuing manual queries to the database, you might have broken data coherency. We have various explicit and implicit triggers to control data.
There is an API to do additions/removal to tags and bookmarks, any other way of directly editing the database is very risky and should be avoided cause it won't activate those triggers. Your script could easily be converted into a Firefox Add-on, and then it would go through supported APIs.

Though, you might also have a large number of tags and maybe there's a relation there. Our tags schema is not very nice and we'll try to change it as soon as we have the resources to do that.

If you are willing to share your bookmarks, you can send me a compressed json file with them by mail, and I can then do some profiling to see if we can improve any query to speed it up.

Comment 2

[0RUBn0]

First off: Thank you for your comment and willingness to profile.
------
I would have doubted coherence issues (with a good logic design, that is), if you didn't mention triggers.  I realized risk before attempting, but I have read some docs about it and have been very careful, but well.. that's not always fullproof, I get that.  Anyway, let's see first if it is related or not by providing my json. Thank you for that!  I don't have a clue myself how to profile the awesomebar or similar.
--> I will share my bookmarks via e-mail after this. I don't like it to be public.  

API: I'm sorry to say that I didn't have time to go through that. I'm debugging and developing too much open source projects. (Getting crazy sometimes.) and finding out about firefox sql structure was already more than enough. (But necessary and my only option: There is an open bug here somewhere about that. Tags not being imported right etc.. which I also suffer from. Hence the sql approach.)

Tags schema not very nice: I have noticed that! O my! But well...maybe I will have the time to be the developer on that ..once. (Feel free to give me a link to the project or documents where this is discussed if you would know, for if I find the time later.)

Comment 3

[0RUBn0]

Marco Bonardo, did you have a chance to check the file I sent you?

Comment 4

[Marco Bonardo [:mak]]

Sorry for the delay, I can confirm the problem is due to a large number of tags in your profile.
Unfortunately tags are not well optimized at this time, we are working on that but resources are limited.
The default query in your profile takes 13s, if I remove tags from the query it takes 52ms.

I'll see if there's anything I can do as a workaround, otherwise we will have to wait for bug 424160.

Comment 5

[0RUBn0]

For the record, I found a workaround that speeds up awesome bar suggestions and other searches*.
    * I noticed later that this super delay also happened in other places (such as bookmark search etc..).

-> I backed-up all my bookmarks (via json, doesn't work other possible  ways), exit firefox, delete any places.sql related files in profile, restarted firefox (which creates empty sql) and then restored them.
I guess Firefox makes a better database because of it (and I did have some manual sql "inefficiencies").

Comment 6

[Adrian Florinescu [:aflorinescu]]

:mak, do you have a sharable usecase for this scenario or an idea if it's still as bad as intially reported?

Comment 7

[Marco Bonardo [:mak]]

(In reply to Adrian Florinescu [:aflorinescu] from comment #6)

:mak, do you have a sharable usecase for this scenario or an idea if it's still as bad as intially reported?

What do you mean by a sharable usecase? We have a few users with thousands of tags, and likely this is still slow for various reasons, mostly that retrieving tags is slow (bug 424160), and then matching on them is also slow because we build a giant string and linearly match on it. It would be useful to have a tags table with an index.
There may have been some improvement on the frontend side with QuantumBar, but it's likely still slow because the backend didn't change around tags.

Comment 8

[Mandy Cheang [:mcheang][she/her]]

There needs to be further investigation in how we optimally search tags. Currently, we are joining the tags in a string and matching against the string. A possible solution is to put the tags in memory, or put the tags in its own index table, there are many possibilities and need further investigation is needed. We also want to match full tags, not partial tags.