1999939 - Track origin redirects more efficiently

Status: NEW

(Firefox :: Address Bar, task, P2)

Description

[Marco Bonardo [:mak]]

After the update to the new frecency, the origins scores have a much wider distribution, that causes our origins threshold to ignore redirecting origins.
These are sites like gmail.com or maps.google.com that are permanently redirecting to a different domain (mail.google.com or www.google.com/maps).
We must address this issue.

Comment 1

[Marco Bonardo [:mak]]

[Tracking Requested - why for this release]: This is affecting a critical urlbar feature (autofill).

Comment 2

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1986285

:jteow, since you are the author of the regressor, bug 1986285, could you take a look?

For more information, please visit BugBot documentation.

Comment 3

[James Teow [:jteow]]

We discussed the issue on Slack.

:mak came up with the following ideas to address this issue of autofill being primarily frecency based. One major issue in the migration to new frecency is the scale is pretty different.

Solution 1 - Make origin frecency an average of values instead of a sum. This would make the distribution of the values more narrow, but we'd lose the visit frequency signal since we would no longer give a boost to origins with many visits.

Soultion 2: Use a threshold that's a median or percentile. The issue is we don't have a way to experiment which percentile works the best and it's unclear if it'd solve the problem with Gmail without having a lot of false positives as we'd likely have to set the threshold very low to accommodate the difference between it and the redirect target origin which could have a substantially higher score (e.g. gmail.com vs www.google.com).

Solution 3: For permanent redirects, set its frecency to the destination origin. It could work the best but might be expensive.

Solution 4: Calculate frecency of an origin similar to calculating frecency of a page, but instead of visits to a specific url we consider visits to any url in the origin. This is more expensive and it's unclear if it'd solve the gmail issue as it should still consider visits to the redirect target origin.

Adaptive autofill partially workarounds the issue where if you type gm and pick gmail.com it will start filling. But if you are more likely to pick mail.google.com (which has a higher score), it wouldn't activate autofill in the future because it doesn't start with gm. It is one of things that should be improved.

---

Solution 3 seems like the way to go. One potential drawback is we could autofill something the user doesn't know about (e.g. a user clicks a link and it redirects, well now that link has a boosted frecency). We can avoid this by checking that the visit was a typed visit.

Additionally, in moz_origins we should have a redirects_to_id field that points to the final chain of redirect destination only if the root redirects (e.g. fb.com -> facebook.com) and they differ by more than the protocol or www (e.g. not http://example.com -> https://example.com. We also need to keep this updated efficiently.

Comment 4

[Marco Bonardo [:mak]]

tldr; per each domain we must walk up the redirect chains and see if there is an original typed domain, that domain should get the score of the destination one. Having the redirects info at hand would make calculation faster, though I'm not sure if we can directly store it when visits are added by History (when visits are added, maybe by a trigger?) or if we should calculate it later (when?).

Comment 5

[James Teow [:jteow]]

Attachment: Bug 1999939 - Part 1: Modify moz_origin schema and migrate database - r?mak!

Comment 6

[James Teow [:jteow]]

Attachment: Bug 1999939 - Part 2: Cache redirects in history - r?mak!

Comment 7

[James Teow [:jteow]]

Attachment: WIP: - Bug 1999939 - Part 3: Enable autofill to use redirect target frecency score - r?mak!

Comment 8

[James Teow [:jteow]]

Hi Dave, we created this bug because of a conversation we had in Slack where typing maps wouldn't give you the right entry. However, this might have been fixed by some work we did in Bug 2001284, is it better for you?

Comment 9

[Dave Townsend [:mossop]]

(In reply to James Teow [:jteow] from comment #8)

Hi Dave, we created this bug because of a conversation we had in Slack where typing maps wouldn't give you the right entry. However, this might have been fixed by some work we did in Bug 2001284, is it better for you?

Yes the maps case does appear fixed

Comment 10

[James Teow [:jteow]]

Thank you.

I'm going to rework this bug to instead be about a broader discussion about how we can track redirects on origins more efficiently so that we can do additional work on better suggesting origins when a redirect source is what users type instead of the redirect target.

We feel okay with lowering the priority of this work.

Comment 11

[James Teow [:jteow]]

Attachment: Bug 1999939 - Add moz_places_redirects - r?mak!

Comment 12

[James Teow [:jteow]]

Attachment: WIP: - Bug 1999939 - Part 2: Exploration of using the redirects table - r?mak!

This is just an exploration of how the table could be used, the real
patches will likely be in a separate bug with tests, etc.