1046074 - Improve post-filtering of dupes in UnifiedComplete

Status: VERIFIED FIXED

(Toolkit :: Places, defect, P2)

Description

[Marco Bonardo [:mak]]

we can probably slightly improve the post-filtering of duplicate entries by also using the finalcomplete value of the autoFill entry and maybe trying to strip www.

Comment 1

[u428464]

This is still an issue with current unified autocomplete implementation.

Comment 2

[u428464]

Any news on this ?

Comment 3

[u428464]

I'm glad to say that Firefox 53 greatly improved deduplication.

Comment 4

[Marco Bonardo [:mak]]

yes, the second part of this has been handled in bug 1322747.
We could still investigate merging www. and non-www. entries, but that's not a priority atm, I think the most problematic part has been handled. So, I'm dropping this since I'm overloaded already and doubt I can fix this anytime soon.
Patches with tests are welcome though!

Comment 7

[Marco Bonardo [:mak]]

Here the remaining problem is we have lots of duplicates in the form
http://site.com/path
http://www.site.com/path
https://site.com/path
https://www.site.com/path

One thing we should likely do is deduping https and http, and we should prefer the https entry. An alternative may be to actually fetch whether the origin supports https, that can be done by using moz_origins.prefix. So, if we want to be more aggressive on https (ask Connor) we could modify all the queries to additionally fetch the prefix from moz_origins and force that prefix on returned urls.

The other thing is www, and here we must confirm whether we want to dedupe or not, we ignore www when searching and we don't show it in results after Bug 1614957, that means in the end we'll show 2 identical urls without a way to distinguish them if we don't dedupe.

Comment 9

[Harry Twyford [:harry]]

We discussed that this bug should encompass deduping any results with different prefixes that are otherwise identical. This means prioritizing www. and http/https differently for the purposes of deduping, surfacing results with these preferences:

https:// > https://www. > http:// > http://www.

For example, if the user has both http://site.com/path and https://www.site.com/path in their results, we show only https://www.site.com/path.

We should not dedupe if the results have different titles; this is to mitigate against deduping www.site.com when it is an entirely different site from site.com. We should also not dedupe if a result with a lower-priority prefix is the heuristic result. In the example above, we would show both http://site.com/path and https://www.site.com/path if http://site.com/path was the heuristic result.

I'm bumping the points on this due to the increase in scope and because quite a few tests will need updating as well.

This is all still pending Product approval.

Comment 10

[Harry Twyford [:harry]]

Attachment: Bug 1046074 - Improve post-filtering of dupes in UnifiedComplete. r?mak

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/604ae25cad30
Improve post-filtering of dupes in UnifiedComplete. r=mak

Comment 12

[Razvan Maries]

Backed out for lint failures on test_swap_protocol.js.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=294402630&resultStatus=testfailed%2Cbusted%2Cexception&revision=604ae25cad3059e5b0b23285717116fe5ef67b8c&searchStr=linting%2Copt%2Cjavascript%2Cchecks%2Csource-test-mozlint-eslint%2Cjs%28es%29

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=294402630&repo=autoland&lineNumber=277

Backout: https://hg.mozilla.org/integration/autoland/rev/fc4c4e983d42d9bf0600385276e6e23c22a7eaa9

Comment 13

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b320c59d42db
Backed out changeset 604ae25cad30 for Lint failure on test_swap_protocol.js. CLOSED TREE

Comment 14

[Harry Twyford [:harry]]

Fixed lint issues and queued a new patch for landing.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e1c0b3f826d
Improve post-filtering of dupes in UnifiedComplete. r=mak

Comment 16

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/8e1c0b3f826d

Comment 17

[Adrian Florinescu [:aflorinescu]]

Updating Fx75 tracking flags to reflect QA triage decision taken with :mdeboer in QA-Search weekly sync meeting.

Comment 18

[Cristian Comorasu, QA [:ccomorasu], Release Desktop QA]

Harry, could you please provide some STR in order to manually verify this issue?
Thank you!

Comment 19

[Harry Twyford [:harry]]

STR:

1. Open Firefox with a new profile.
2. Visit http://example.com, http://www.example.com, https://example.com, and https://www.example.com, in that order.
3. Open a new tab and close the tab that you were opening example.com in.
4. Type "ex" in the address bar.
5. Verify that the first result (the one that would be selected if you hit Enter), is https://www.example.com.
6. Verify that there is one and only one other history result for example.com. Use the arrow keys to move down and highlight it. The address bar should be filled with https://example.com.
7. Visit https://example.com several more times. Open a new tab and close the tab you were opening https://example.com in.
8. Repeat steps 4-6, except the order of the results in steps 5 and 6 should be swapped. https://example.com should be the first result and https://www.example.com should be the other result.

Comment 20

[Cristian Comorasu, QA [:ccomorasu], Release Desktop QA]

Thank you for the steps.
I can confirm this issue is fixed, I verified using Fx 77.0a1 and Fx 76.0b8 on Windows 10 x64, macOS 10.13 and Ubuntu 18.04.
Updating the flags accordingly.