1735894 - Ignore timestamps in URLs when de-duping Firefox Suggest results and history results

Status: VERIFIED FIXED

(Firefox :: Address Bar, enhancement, P3)

Description

[Drew Willcoxon :adw]

The URLs of Firefox Suggest results contain a %YYYYMMDDHH% timestamp template that we fill in when a suggestion is fetched. That means the URL of the same suggestion can slightly change depending on the time at which the user triggers it.

That interferes with the de-duping logic in the muxer that discards URL results that dupe one another -- in this case, history results that dupe Firefox Suggest results. So if you pick a Firefox Suggest result so it's added to your history and then later trigger the same result again, it's possible you'll see both the history result and the Firefox Suggest result.

We should make the de-duping logic smarter specifically for this case so that the history result is not shown. Possible fixes:

1. Modify the muxer logic to ignore the timestamp substrings in the URLs
2. Use Places annotations to annotate (visited) Firefox Suggest URLs in the Places database, and then discard annotated URLs. This has the benefit of being agnostic to all sorts of URL differences, but it's probably harder to implement and there may be perf or lag concerns due to accessing the Places database.

Comment 1

[Drew Willcoxon :adw]

Working on a patch for this. We can convert the QS result's templated URL string to a regexp and compare it to history URLs, discarding history URLs that match (or the QS result, if we want to prefer history URLs instead).

Comment 2

[Drew Willcoxon :adw]

Attachment: Bug 1735894 - Ignore quick suggest URL parameters when de-duping them from history results.

This creates a regular expression pattern from the parameterized quick suggest
URL and then uses the regexp to discard matching history URLs in the same query.

Depends on D128665

Comment 3

[Drew Willcoxon :adw]

Attachment: Bug 1735894 - Discard history results that dupe the quick suggest result.

This adds UrlbarProviderQuickSuggest.isURLEquivalentToResultURL() to tell
whether a URL is equivalent to a quick suggest URL. URLs are equivalent if they
are the same except for their timestamp substrings. The muxer uses the new
method to discard history results whose URLs are equivalent to the quick suggest
URL.

Depends on D130943

Comment 4

[Drew Willcoxon :adw]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9c8ec84105bc160947393afb46f00676c080a603

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/663c0c074a74
Discard history results that dupe the quick suggest result. r=nanj,mak

Comment 6

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/663c0c074a74

Comment 7

[Drew Willcoxon :adw]

STR for QA

1. Type "amazon" in the urlbar and verify there's an Amazon quick suggest result
2. Use the arrow keys to select the result but don't press enter
3. Copy-paste the URL somewhere so you can refer to it below
4. Press enter to visit the result
5. Open a new tab and close the Amazon tab
6. Set your system clock to a new hour. e.g. if you're testing at 11:XX, set it to 12:XX.
7. Type "amazon" again and verify there's an Amazon quick suggest result
8. Verify the Amazon history result does not appear. Only the quick suggest result should appear.
9. Use the arrow keys to select the result but don't press enter
10. Verify that the last 10 digits in the result's URL aren't identical to the last 10 in the URL from step 3.

e.g. the two URLs might be:

https://www.amazon.com/?tag=admarketus-20&ref=pd_sl_a318361095US592021111708
https://www.amazon.com/?tag=admarketus-20&ref=pd_sl_a318361095US592021111709

These URLs pass verification because "2021111708" != "2021111709"

Comment 8

[Monica Chiorean]

(In reply to Drew Willcoxon :adw from comment #7)

STR for QA

1. Type "amazon" in the urlbar and verify there's an Amazon quick suggest result
2. Use the arrow keys to select the result but don't press enter
3. Copy-paste the URL somewhere so you can refer to it below
4. Press enter to visit the result
5. Open a new tab and close the Amazon tab
6. Set your system clock to a new hour. e.g. if you're testing at 11:XX, set it to 12:XX.
7. Type "amazon" again and verify there's an Amazon quick suggest result
8. Verify the Amazon history result does not appear. Only the quick suggest result should appear.
9. Use the arrow keys to select the result but don't press enter
10. Verify that the last 10 digits in the result's URL aren't identical to the last 10 in the URL from step 3.

e.g. the two URLs might be:

https://www.amazon.com/?tag=admarketus-20&ref=pd_sl_a318361095US592021111708
https://www.amazon.com/?tag=admarketus-20&ref=pd_sl_a318361095US592021111709

These URLs pass verification because "2021111708" != "2021111709"

Hi Drew,
Is there a pref that needs to be set in order to get the hour in the URL? I tried to verify the issue on 96.0b4, but I not able to see the hour.
Eg: this is the link suggestion that I am getting after a search (https://www.amazon.com/s?k=gaming+keyboard&pd_rd_r=dd003c44-2aa9-448a-959f-5d2bdf83bd53&pd_rd_w=tCKhC&pd_rd_wg=7fuCQ&pf_rd_p=12129333-2117-4490-9c17-6d31baf0582a&pf_rd_r=NTYMBYDCK8DDRJDABA5N&ref=pd_gw_unk).
Thank you.

Comment 9

[Drew Willcoxon :adw]

Attachment: screenshot

Hi Monica, are you testing in the U.S. or with a VPN inside the U.S., and with Firefox Suggest enabled? When I try on 96.0b4, I get an Amazon URL similar to the ones in my previous comment. I'll attach a screenshot of what everything should look like.

Comment 10

[Monica Chiorean]

(In reply to Drew Willcoxon :adw from comment #9)

Created attachment 9255158 [details]
screenshot

Hi Monica, are you testing in the U.S. or with a VPN inside the U.S., and with Firefox Suggest enabled? When I try on 96.0b4, I get an Amazon URL similar to the ones in my previous comment. I'll attach a screenshot of what everything should look like.

I was able to verify the issue as you mentioned only after setting the VPN, browser.search.region=US and "browser.newtabpage.activity-stream.discoverystream.spocs-endpoint" set to "https://spocs.getpocket.com/spocs?country=US&region=CA".
Issue is fixed on Win10/Ubuntu 20.4 using build 96.0b4(20211212185725). Thank you so much for your help.

https://www.amazon.com/?tag=admarketus-20&ref=pd_sl_a318361095US592021121410
https://www.amazon.com/?tag=admarketus-20&ref=pd_sl_a318361095US592021121407