1466267 - Recalculate frecency in chunks on idle

Status: RESOLVED DUPLICATE of bug 1806666

(Toolkit :: Places, task, P2)

Description

[Lina Butler [:lina]]

Currently, we either chunk and recalculate frecencies asynchronously at startup, after a migration (https://searchfox.org/mozilla-central/rev/38bcf897f1fa19c1eba441a611cf309482e0d6e5/toolkit/components/places/Database.cpp#2410), or after a sync (https://searchfox.org/mozilla-central/rev/38bcf897f1fa19c1eba441a611cf309482e0d6e5/toolkit/components/places/SyncedBookmarksMirror.jsm#623,4535,4538). This is expensive if we need to do it for many pages.

In bug 1462046, comment 18, and bug 1463938, comment 1, Mak suggests a more general approach:

> What I'm suggesting as a long term fix instead, is to add a frecency_recalc field to moz_places and just set it to true everywhere we must recalculate frecency, that would preserve the previous frecency value, that likely won't be so wrong, and then recalculate frecency in chunks using something like idleDispatch, still using WHERE NOT frecencyIsValid ORDER BY frecency DESC LIMIT N.

Comment 1

[Marco Bonardo [:mak]]

In general we have a frecency performance problem due to 2 facts:
1. every sigle change (history/bookmarks) changes frecency
2. frecency must be decayed daily.

The only way to fix 2) would be a new algorithm, something that was suggested in bug 704025.

But we could reduce the implications of 1) by just marking a frecency as "invalid" when any change happens, and then recalculating a chunk of frecency on idle (if necessary).
Pretty much that approach could probably be applied to any frecency calculation need, and in the meanwhile the awesomebar could keep using the existing value, that even if it's not up-to-date, it's also not horrible.

We can evaluate whether it's more efficient to add a bool field to moz_places, or a moz_invalid_frecencies table with just a place_id column.