1421485 - Imported chrome browsing history ignores visit_count and shows recently visited history instead of top sites

Status: RESOLVED FIXED

(Firefox :: Migration, enhancement, P2)

Description

[Ed Lee :Mardak]

A user reported importing from chrome and seeing Top Sites that had pages visited from today but seemed to be missing an actual Facebook top site. Some imported top sites might have only been visited once in Chrome.

Comment 1

[Tim Spurway [:tspurway]]

Let's check to see if this is a specific issue with Activity Stream, or if it belongs in another component.

Comment 2

[Andrei Oprea [:andreio]]

I've tried importing chrome data into a fresh profile to see what happens. I can take a guess as to why the user would complain about that behavior: a website visited today will a frecency value that can be an order of magnitude greater than another that you haven't visited today (because of time decay). We have a custom sorting function [0] in which frecency values take precedence. Most likely all of the websites the user accesses frequently were selected but their order did not fit her/his mental model, they probably got pushed to the second row which is hidden by default.
I also found it important that the same navigation behavior in both browsers leads to the same frequency value (after you import). The history import helper [1] gets all the navigation details (typed, or transition link, counts etc).


[0] https://searchfox.org/mozilla-central/rev/f5f1c3f294f89cfd242c3af9eb2c40d19d5e04e7/toolkit/modules/NewTabUtils.jsm#1111-1115
[1] https://searchfox.org/mozilla-central/rev/f5f1c3f294f89cfd242c3af9eb2c40d19d5e04e7/browser/components/migration/ChromeProfileMigrator.js#264-334

Comment 3

[Ed Lee :Mardak]

And what were the frecency values for the imported data compared to the page's visit count and type? If a user visits a page many times but not today, it should have a higher frecency value.

Comment 4

[Andrei Oprea [:andreio]]

Good catch. This is the "urls" table we query when importing history:

0|id|INTEGER|0||1
1|url|LONGVARCHAR|0||0
2|title|LONGVARCHAR|0||0
3|visit_count|INTEGER|1|0|0
4|typed_count|INTEGER|1|0|0
5|last_visit_time|INTEGER|1||0
6|hidden|INTEGER|1|0|0 
(this is just the output given by table_info comand)

We check "typed_count > 0" [0] to decide what kind of visit was it, typed or navigation. There doesn't seem to be any query to fetch the value of the "visit_count" field.

In Google Chrome, a query for url, visit_count returns:
https://news.ycombinator.com/|14
After we import in moz_places I see:
https://news.ycombinator.com/|1

Perhaps this can be easily fixed by inserting "visit_count" entries in the visits array [1] (plus some additional logic to figure out what the visit type was).

[0] https://searchfox.org/mozilla-central/rev/ed212c79cfe86357e9a5740082b9364e7f6e526f/browser/components/migration/ChromeProfileMigrator.js#327
[1] https://searchfox.org/mozilla-central/rev/ed212c79cfe86357e9a5740082b9364e7f6e526f/browser/components/migration/ChromeProfileMigrator.js#333

Comment 5

[Ed Lee :Mardak]

I believe we could just copy the visit_count value from Chrome's urls table to Firefox's moz_places visit_count, and frecency should compute a relatively better number. (I believe it's [visit frequency] * [recent visit types] ?)

mak, any idea if visit_count was previously used but removed? Or just never used to begin with?

Comment 6

[Marco Bonardo [:mak]]

the computes in moz_places are actual, thus they reflect the real entries in moz_historyvisits. We don't write fake numbers there and we cannot generate visits because Chrome doesn't even store the visit type and date for each visit afaik.
Keeping fake numbers would involve quite a few changes to avoid coherency problems and possible privacy hits.

Comment 7

[Ed Lee :Mardak]

Looks like "visits" is similar to moz_historyvisits:

CREATE TABLE visits(id INTEGER PRIMARY KEY,url INTEGER NOT NULL,visit_time INTEGER NOT NULL,from_visit INTEGER,transition INTEGER DEFAULT 0 NOT NULL,segment_id INTEGER,visit_duration INTEGER DEFAULT 0 NOT NULL, is_indexed BOOLEAN);

id = id
url = place_id
visit_time = visit_date
from_visit = from_visit
transition = visit_type https://chromium.googlesource.com/chromium/src/+/master/ui/base/page_transition_types.h
segment_id = session

Comment 8

[Marco Bonardo [:mak]]

Thanks, that means we should import more visits, that is something we don't do because it's too slow. Doug Thayer is working on improving the time to insert many visits in bug 1332225

Comment 9

[Niklas Baumgardner [:niklas]]

Attachment: Bug 1421485 - Import multiple visits per url from Chrome history. r=#migration,jhirsch

Previously each Chrome URL was imported with one visit at its last visit
time, typed if it had ever been typed. Now up to
browser.migrate.chrome.history.visit.limit of a URL's most recent visits
are imported with their own dates, and their Chrome transitions mapped to
Places ones, so frecency reflects recent usage. The number of imported URLs is
unchanged, so frecency recalculation, which is per page and accelerated after
large imports since bug 1873629, does about the same amount of work. Only the
number of inserted visits grows.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a5b23cf9ecb7
https://hg.mozilla.org/integration/autoland/rev/28fa2dd49208
Import multiple visits per url from Chrome history. r=jhirsch,mconley

Comment 11

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/28fa2dd49208

Comment 12

[Pascal Chevrel (relman team) -> :pascalc]

Did you want to nominate this for the Fx161 relnotes? If so, set the relnote-firefox flag to "?"
https://wiki.mozilla.org/Release_Management/Release_Notes_Nomination

Possible wording:

Browsing history imported from Chrome now includes how often pages were visited, so the most visited sites appear in shortcuts and address bar suggestions after importing.

Thanks

Comment 13

[Niklas Baumgardner [:niklas]]

Thanks for the needinfo Pascal!

Release Note Request (optional, but appreciated)
[Why is this notable]:
Without visits, Places can't calculate frecency for imported Chrome history, so those pages ranked as if never visited in the address bar; importing the visits fixes that and lets frequently and recently visited Chrome sites show up in suggestions right away.

[Suggested wording]:
Browsing history imported from Chrome now includes how often you visited each page, so the sites you use most rank higher in address bar suggestions and shortcuts.

Comment 14

[Pascal Chevrel (relman team) -> :pascalc]

Thanks, added to our 161 notes.