723257 - Extend BrowserProvider with some notion of frecency

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: General, defect, P2)

Description

[Richard Newman [:rnewman]]

This has a few aspects.

Firstly, we'd like to be able to set the sortindex of Sync's uploaded records, so that -- for example -- another Fennec device can pull down the most important records first. Fennec isn't able to "play nice" here. Sync needs to be able to find out Fennec's frecency, or compute it ourselves. (I'm OK with computing it ourselves; after all, we know every visit with more granularity than Fennec does!)

Secondly, in a world with history truncation, we need a way to say to Fennec "this record has importance X" so that it has a way to throw away the least important records, rather than the oldest. It does nobody any good if Fennec throws away my years-old Twitter history entry, despite it having massive frecency, when Sync downloads and applies it.

See Bug 721898, Bug 708005, Bug 723241, Bug 723240, and Bug 723235.

Comment 1

[Lucas Rocha (:lucasr)]

(In reply to Richard Newman [:rnewman] from comment #0)
> This has a few aspects.
> 
> Firstly, we'd like to be able to set the sortindex of Sync's uploaded
> records, so that -- for example -- another Fennec device can pull down the
> most important records first. Fennec isn't able to "play nice" here. Sync
> needs to be able to find out Fennec's frecency, or compute it ourselves.
> (I'm OK with computing it ourselves; after all, we know every visit with
> more granularity than Fennec does!)

I recommend using a URI query arg to force sort order to use frecency on history queries. That probably involves moving some bits from LocalBrowserDB's getTopSites() and filter() into our ContentProvider.

> Secondly, in a world with history truncation, we need a way to say to Fennec
> "this record has importance X" so that it has a way to throw away the least
> important records, rather than the oldest. It does nobody any good if Fennec
> throws away my years-old Twitter history entry, despite it having massive
> frecency, when Sync downloads and applies it.

This is most likely related to the fix to bug 704977. Maybe we should pre-compute the frecency score and store it on DB instead of computing it in the query. That would probably help you here.

Comment 2

[Richard Newman [:rnewman]]

(In reply to Lucas Rocha (:lucasr) from comment #1)

> I recommend using a URI query arg to force sort order to use frecency on
> history queries. That probably involves moving some bits from
> LocalBrowserDB's getTopSites() and filter() into our ContentProvider.

Can we actually get the frecency value that way? Otherwise we'll need to steal the frecency logic.


> This is most likely related to the fix to bug 704977. Maybe we should
> pre-compute the frecency score and store it on DB instead of computing it in
> the query. That would probably help you here.

Yeah, either that or the pruning would need to calculate frecencies in order to prune...

Comment 3

[Richard Newman [:rnewman]]

I mis-spoke. We don't know the frecency of records that we download; indeed, we can't even trust that different clients would have the same concept of frecency, so it's not included in the record payload.

Either:

1. we need an implementation of Fennec's frecency algorithm, or one of our own, that we can use alongside the known values in the database to decide whether to insert, or

2. the Fennec browser CP will take care of frecency-based pruning decisions, and we'll blindly pass in records just as we do now.

I think I'd prefer #2 (#1 requires additional queries, and support from the CP anyway).

Consequently, my requests are:

* Calculate and use some frecency-like score when deciding on pruning, including during insertion. Inserted records will not include a frecency value. Fennec gets to pick which records it cares about.

* If you can, return that frecency value when you return records from the CP.

Independently, I will explore implementing Places' frecency algorithm in Java, because Sync knows all of your history visits. I'll use this to set sortindex in uploaded records.

The reason why I'm not exclusively pursuing that avenue is that this requires history access to compute bookmark frecency, and I'm not sure what changes would need to be made to make that happen. As a fallback, or as additional data, we can use the stored frecency from Fennec.

Does that make sense to you, Lucas?

Comment 4

[Richard Newman [:rnewman]]

Oh good, I cleared flags. Sorry chaps!

Comment 5

[Mark Finkle (:mfinkle) (use needinfo?)]

This still might be wanted to improve the awesomebar results

Comment 6

[Mark Finkle (:mfinkle) (use needinfo?)]

We will look at this if needed, after other performance fixes land.

Comment 7

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.