788146 - Awesomescreen frecency logic often seems illogical

Status: RESOLVED DUPLICATE of bug 934030

(Firefox for Android Graveyard :: Awesomescreen, defect)

Description

[David Tenser [:djst]]

This is one of those things that has bugged me for more than a year, but has been hard to actually file a bug for because I can't come up with a solid test case to prove anything. I've now decided to file it anyway with the hope that it might be duped against an already known bug or that I'll be pointed to better ways to produce reliable STR.

I would very much welcome any pointers to ways I can provide better test cases. I'd also be fairly happy with sharing some actual awesomebar data if that can help troubleshoot all the issues I'm having, as long as we can separate out cookies and passwords. 
The vague problem statement is that the priorities of items in "Top Sites" and "Bookmarks" seem random and unpredictable. Some examples of the issues I run:

* When setting up sync for the first time and syncing with your tried-and-true desktop Firefox install, the top sites don't really seem to match what you're seeing on the desktop. The items are typically slightly offset, and sometimes one of those truly top sites that you visit daily doesn't even show up on the first couple of pages of scrolling.

* Often, just surfing to a website once can be enough to bump up the frecency of that website to have it appear on the first page of the awesome screen. This seems odd to me since it competes with top sites I've visited daily for the last few years.

* I have some bookmarks which it doesn't matter how many times I scroll down in the awesome screen and pick -- they never bump up in the list. One example of mine is www.engadget.com, which is two pages down in my bookmarks list, even though I visit it daily. It simply won't move up in the list like it would on my desktop Firefox.

* Often, the "wrong" flavor of a URL becomes the higher prioritized item. For example, I always visit www.aftonbladet.se, but one day a particular article on that news site, say http://www.aftonbladet.se/nyheter/article15385982.ab, shows up in the Top Sites for no good reason. Typically, I only read that particular news article once.

* Sometimes, top sites simply get down-prioritized for no reason. Using the same example as above, www.aftonbladet.se no longer shows up on my first page. This just suddenly happened after a restart or upgrade, I think. Nowadays, I have to always type "af" and then it will appear. It never seems to make it back in the frecency, despite the number of times I visit it after this happened.

* When typing "af" on the desktop Firefox, www.aftonbladet.se *always* comes up first, both in the URL autocomplete and in the awesome bar results. It has been this case since Firefox 3.0. 

* The Awesome screen has a tendency to record all variants of a URL and treat them equally. See http://farm9.staticflickr.com/8319/7929444198_69aa0fd5f9_z.jpg for an example of how my current Awesome screen is flooded with various Zimbra URLs. Note that I never navigate to any of those URLs manually except for the base one -- all the other variants appear automatically after surfing around on the site.

All in all, I'm wondering if I have missed the point of Sync and what it's supposed to actually synchronize. I always assumed that the frecency information would also be synced, not just the actual list of websites I've visited/bookmarked over the years. Is this not the idea?

Anothing thing: is it possible to get a "debug" view of the awesome screen where I can see how the frecency score is actually calculated? That would really make it easier for me to determine what the problem(s) is/are.

Comment 1

[Johnathan Nightingale [:johnath]]

We did the current "frecency" work in 704977 which does some math with "is it a bookmark" and "how recently have we seen it" because our history DB doesn't have the right fields to be able to do proper frecency. That doesn't mean we can't improve, though, or find some novel hack for storing visit count and other things that might make the world better.

Comment 2

[David Tenser [:djst]]

Thanks for the pointer to how the math works. This does suggest to me that some of the problems I'm experiencing are due to bugs in my data. For example, by the logic of that formula used, the problem I have with the engadget.com never climbing the frecency latter shouldn't happen, since it should have a "days since last visit"=0 and numVisits>1000.

Is there any way I can see how the score breaks down for each of my history/bookmark entries? E.g. an add-on or similar that reveals the numbers behind the parameters of the formula?

Comment 3

[:Margaret Leibovic]

Thank you for the detailed report!

(In reply to David Tenser [:djst] from comment #2)
> Is there any way I can see how the score breaks down for each of my
> history/bookmark entries? E.g. an add-on or similar that reveals the numbers
> behind the parameters of the formula?

That doesn't exist, but it would be possible for someone to make one. I'll be sitting on a plane for most of tomorrow, so maybe I can try then ;) Or if anyone following along would like to give it a try, last week I was messing around with running queries against my browser.db from an add-on, and the code is here:
https://github.com/leibovic/skeleton-addon-fxandroid/blob/sql/bootstrap.js

As an aside, I've started to research some performance problems with the awesomescreen in bug 785945, and we may end up needing to tweak this algorithm again to avoid some pretty terrible performance problems.

Comment 4

[David Tenser [:djst]]

(In reply to Margaret Leibovic [:margaret] from comment #3)
> Thank you for the detailed report!
> 
> (In reply to David Tenser [:djst] from comment #2)
> > Is there any way I can see how the score breaks down for each of my
> > history/bookmark entries? E.g. an add-on or similar that reveals the numbers
> > behind the parameters of the formula?
> 
> That doesn't exist, but it would be possible for someone to make one. I'll
> be sitting on a plane for most of tomorrow, so maybe I can try then ;) 

That would be totally awesome. :) A theory that I have is that for whatever reason, the numVisits is reset for some of my bookmarks, or never incremented. That would explain why some once-in-a-lifetime pages bubble up above some really frequently visited sites. Having that kind of visibility into the parameters would help me confirm that theory and hopefully provide more useful STRs.

Comment 5

[:Margaret Leibovic]

(In reply to David Tenser [:djst] from comment #4)
> (In reply to Margaret Leibovic [:margaret] from comment #3)
> > Thank you for the detailed report!
> > 
> > (In reply to David Tenser [:djst] from comment #2)
> > > Is there any way I can see how the score breaks down for each of my
> > > history/bookmark entries? E.g. an add-on or similar that reveals the numbers
> > > behind the parameters of the formula?
> > 
> > That doesn't exist, but it would be possible for someone to make one. I'll
> > be sitting on a plane for most of tomorrow, so maybe I can try then ;) 
> 
> That would be totally awesome. :) A theory that I have is that for whatever
> reason, the numVisits is reset for some of my bookmarks, or never
> incremented. That would explain why some once-in-a-lifetime pages bubble up
> above some really frequently visited sites. Having that kind of visibility
> into the parameters would help me confirm that theory and hopefully provide
> more useful STRs.

In the meantime, I also made an add-on to copy your profile to your sdcard [1], and if you have adb set up you can pull that to your desktop (or maybe you can somehow share it with yourself using Android's file system app) and take a look at your database with SQLite Manager [2]. This is definitely a more DIY approach to looking at the data, but it's there if you're curious :)

[1] https://addons.mozilla.org/en-US/android/addon/copy-profile/
[2] https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/

Comment 6

[David Tenser [:djst]]

Margaret, thanks for the pointers to the two add-ons above. I installed them and copied over my Android (nightly) profile over to my desktop computer and launched the SQLite manager, but I don't know which file I'm supposed to open. I was expecting a "places.sqlite" file somewhere, but I can't find it. 

In which db file are bookmarks and history stored? Thanks!

Comment 7

[David Tenser [:djst]]

Nevermind, it's browser.db. For some reason the dialog only wanted to filter by *.sqlite, but I see now that it happily eats the .db files too.

Comment 8

[Wesley Johnston (:wesj)]

I think some of this is due to redirects we get that desktop doesn't. For instance, having www.engadget.com bookmarked might not help if it always redirects to m.engadget.com.

Comment 9

[David Tenser [:djst]]

Wow, this is really interesting. This provided me with more information about a new illogical Awesomescreen behavior that started about two weeks ago.

I'm often visiting yr.no, which is a great weather website for Sweden even though it's made by Norwegians (which is a story I'm saving for later, but suffice it to say they're not our little brother anymore!!). I have a few bookmarks to their site for cities I care about, such as my home town, and e.g. San Francisco. 

Two weeks ago I visited one of these bookmarks and accidentally tapped on a radar map, and all of a sudden my #1 visited website in the Awesomebar was this new page that I've never visited before.

Here are some STR (but I don't know if it was just a glitch or if this will indeed reproduce it):

1. Visit http://www.yr.no/sted/Sverige/S%C3%B6dermanland/Eskilstuna/
2. Click the radar map on the right side. It just looks like a gray map with a tiny Play button in the lower-left corner. Click anywhere on this map.
3. yr.no will now load another page -- this page ended up in my awesomescreen at the top.

After loading browser.db into the SQLite Manager, I was able to see that Firefox had recorded a whopping 177 visits to this page! That explains why it suddenly appeared at the top of the list, and it *might* explain why this has happened before on some other pages where suddenly a page appears at the top, or is suddenly vanished from the top 20 view: the visits column of the db seems a bit unstable?

Comment 10

[David Tenser [:djst]]

Attachment: Screenshot of SQLite Manager showing my most visited history items

Here's my most visited history items in my Android profile. As you can see, the one-time visit to that yr.no URL made it climb up to the top, and after two weeks of hitting that Zimbra URL, yr.no has only made it down one step in the list.

I suspect it will remain in the front page view for months if I don't simply remove it (which I will now that I have this db dump on my desktop!).

Comment 11

[Richard Newman [:rnewman]]

Hey old bug! Bug 934030 and friends is relevant here.

Comment 12

[:Margaret Leibovic]

As defined, it's too hard for this bug to get traction, I'm just going to dupe this to the effort in bug 934030.