1389229 - Filter duplicate google docs from awesomebar fuzzy matches

Status: VERIFIED FIXED

(Firefox :: Address Bar, task, P2)

Description

[Stuart Philp :sphilp]

Attachment: Screen Shot 2017-08-10 at 3.33.24 PM.png

I'm not sure how realistic this is to fix, but often when trying to get back to a google doc via the awesomebar I'll get multiple entries of the same docs showing up for a given query (see screenshot). 

It would be great to limit these so each entry is *more* unique (ignore fragments maybe, or possibly a distance algorithm for similar looking URIs?). This would make it easier to find docs that have similar names or possibly show more variety when the fuzziness isn't *quite* right.

Comment 1

[Wil Clouser [:clouserw]]

Attachment: raci.png

I was also going to file this with a suggestion to just collapse all fragments into the latest fragment.  So, whatever tab/location you were at in the document last, that's what gets put in your awesomebar.

Attaching a screenshot showing a single document blowing out all the results for RACI in my awesomebar.  Same thing happens with other queries like OKR or screenshots or whatever other project we're working on at the time.

Comment 2

[Marco Bonardo [:mak]]

we can start investigating deduplication on Google Docs and see if it can easily apply to other top origins.

Comment 3

[Marco Bonardo [:mak]]

Dao pointed out risks here, because of history pushstate being used with refs to move to completely different pages. I wonder if we could use the page title to dedupe for those cases?

Comment 4

[Dão Gottwald [:dao]]

Bug 1632804 makes us ignore the trailing #. I'm not sure that more can be done here. Some URLs include the user id, which is relevant if you use multiple google accounts, and some URLs seem to point to headings within the document or something, which I assume isn't redundant information.

Comment 5

[Marco Bonardo [:mak]]

While discussing on Slack with Connor, we were evaluating whether it would be possible to detect pushState() that change the document using refs, and annotate that per origin (in places.sqlite for example). Then we could probably dedupe origins that didn't see it.
There's some limitation related to permanent pb, and initial blank state, where we may not have enough info yet.

Comment 6

[Marco Bonardo [:mak]]

A possible alternative approach may be to base the decision on last visit date.
Basically, the top result would always be added, for subsequent urls that only differ by the ref, we skip them if the last visit is older than a given time threshold, for example 7 days. We could use a pref for the threshold, where of course a very large value would basically filter nothing. We'll have to add the last visit date in the payload of each url result.
One downside is that UnifiedComplete doesn't know how many results may be skipped by this, so it would have to return more than necessary to allow late filtering...

Comment 8

[Mike Kaply [:mkaply] (PTO until Sept 14)]

FYI, google docs doesn't use rel="canonical, but they do have this:

<meta property="og:url" content="https://docs.google.com/document/d/1EMgFtS805zxxU4v9llU1pMTO0mj0D3HbrYf3AZVmHaI/edit?usp=embed_facebook">

Which we could theoretically use as a "canonical" url (with the ? part removed)

Comment 11

[BugBot [:suhaib / :marco]]

The severity field for this bug is relatively low, S3. However, the bug has 3 duplicates.
:adw, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 13

[Moritz Beier [:mbeier]]

Attachment: Bug 1389229 - Deduplicate urlbar history results that only differ by their URL fragment. r?mak

Comment 14

[Mike Kaply [:mkaply] (PTO until Sept 14)]

So we looked into this for Flowstate, and you can't just generically use the fragment for everything as some sites are highly dependent on the fragment.

See:

https://docs.google.com/document/d/1t13dAW6yMyxKys2Emthe0tZE1Q9j-CTsjRb2fa-LNL0/edit

Comment 15

[Moritz Beier [:mbeier]]

We are not just using the fragment but fragment and title and only removing results that were not visited in the last 7 days. I think this works for all examples in that document but there likely are some websites where this doesn't work and too much gets deduplicated when there are no recent visits.
Do you know of any websites where this approach would deduplicate too much?

Comment 16

[Daniel Holbert [:dholbert]]

Attachment: image.png

Do you know of any websites where this approach would deduplicate too much?

Arguably any site where there's just a bunch of in-page anchors, e.g. CSS spec documents.

Here's a screenshot of an example from my own history - I've got various parts of https://drafts.csswg.org/css-text-4/ in my history, which are useful to have preserved in my awesomebar suggestions.

(e.g. I can type white-space-property into my awesomebar right now and get https://drafts.csswg.org/css-text-4/#white-space-property as an awesomebar suggestion, which is nice.)

Comment 17

[Mike Kaply [:mkaply] (PTO until Sept 14)]

I've also seen news sites where as you scroll it's multiple articles and they change the fragment as you hit the new article.

Comment 18

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #16)

Do you know of any websites where this approach would deduplicate too much?

Arguably any site where there's just a bunch of in-page anchors, e.g. CSS spec documents.

I guess Google Docs -- the motivating site here -- is also in this^ category (site with a bunch of in-page anchors).. The special things about Google Docs here are that (a) its fragment IDs are gibberish rather than human-readable, which means users can't meaningfully distinguish among them in the list of Firefox Suggest suggestions; and (b) users inadvertently "visit" its links by simply moving the cursor through the document (which makes Google Docs update the URLbar to reflect your current heading, and which makes you inadvertently get a bunch of history entries without actually clicking any links or realizing that your'e navigating).

Unfortunately, I'm not sure it's tractable to programmatically differentiate between a site like Google Docs vs. a site like drafts.csswg.org, in terms of fragments being something the user might care about or might want deduplicated. If we can think of a way, though, that's great. :)

In my case in my duplicate bug 1753747, I think the thing that prompted me to file was that the flood of history-entries drowned out "Switch to tab" for me and prevented that from showing up (for a Google Doc that I actually had open in a tab).

Comment 19

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #18)

In my case in my duplicate bug 1753747, I think the thing that prompted me to file was that the flood of history-entries drowned out "Switch to tab" for me and prevented that from showing up (for a Google Doc that I actually had open in a tab).

Completing this thought: my use-case in that dupe bug could be addressed without needing to fully de-duplicate the history-entry-URLs, if we simply bumped up the priority of switch-to-tab showing up in awesomebar suggestions (at least, when weighted against N history-entries that are identical aside from fragment ID).

Comment 20

[Moritz Beier [:mbeier]]

I can type white-space-property into my awesomebar right now and get https://drafts.csswg.org/css-text-4/#white-space-property as an awesomebar suggestion, which is nice.

This deduplication doesn't change that. If you type e.g. CSS Text, you will get only one result instead of 10 (only the first one) but when typing white-space-property , there would only be one result with or without the deduplication. Chrome also appears to do the same in this case.

I've also seen news sites where as you scroll it's multiple articles and they change the fragment as you hit the new article.

They likely also change the title of the page so they won't be deduplicated.

that dupe bug could be addressed without needing to fully de-duplicate the history-entry-URLs, if we simply bumped up the priority of switch-to-tab showing up in awesomebar suggestions (at least, when weighted against N history-entries that are identical aside from fragment ID).

That would probably be safer but it would only solve issues with switch to tab and the result list would still be cluttered.
Another problem with the implementation I submitted is that it doesn't work when you recently went through the whole Google Docs document.

However I feel like it would almost never deduplicate too much. Of course it's impossible to know every site so we wanted to disable it by default for now to test it on more pages without affecting users. @:mak came up with this logic so maybe he can say more.

Comment 21

[Marco Bonardo [:mak]]

I hink Moritz already did a great job at answering your concerns.

Our intent here is not just to declutter, but also to increase diversity of results, as having them dominated by a site just because it makes large use of fragments without appropriate titles is not exactly valuable, in most cases. Otherwise we'd not get frequent feedback about better handling duplicates.

We are fully aware of the issues involved, and that's why we're starting with a very conservative approach of just refs and only if the title is the same. In addition to that we have two other conditions: 1. frecency will ensure the most interesting url remains, 2. we're retaining duplicates visited recently (we guessed 7 days, could be extended eventually).

May this cause issues? Of course it could, that's why we're landing behind a pref to study it in Nightly, and we consider this just the first step. We must start from somewhere.
Leaving alone fancy pages that use the ref to change content leaving the title alone (I consider that a Website Bug), the annoyance we are potentially causing here is the user having to go back to the index or scroll back to a position. That seems, on a first stance, acceptable, or at least well balanced by the advantage of avoiding results dominance.

To go further, we are evaluating further things:

1. instead of discarding results completely, we could queue them up. This doesn't guarantee they will be shown, but if there's no other results, why not?
2. we could offer a "Some results were deduped, would you like to get full results" option, like most search engines do, then we'd rerun the query without deduplication.
3. we are evaluating how we can measure through telemetry when the user is lacking these results, but it's not trivial. Offering an option to disable deduping for a search could help.
4. maybe the "recent" approach is not sufficient, and we should instead try a frecency-distance approach (how less relevant are page2/3/4 compared to the first one?)

Comment 22

[Daniel Holbert [:dholbert]]

(In reply to Moritz Beier [:mbeier] from comment #20)

I can type white-space-property into my awesomebar right now and get https://drafts.csswg.org/css-text-4/#white-space-property as an awesomebar suggestion, which is nice.

This deduplication doesn't change that. If you type e.g. CSS Text, you will get only one result instead of 10 (only the first one) but when typing white-space-property , there would only be one result with or without the deduplication

That's great, thanks. I misunderstood the deduplication to be happening at an earlier state where this sort of differentiation wouldn't be possible. No concerns from me on the drafts.csswg.org scenario then.

(I don't personally have any context on the flowstate learnings, so I can't speak to whether or not this ends up fine vs. troublesome regarding those; but maybe it's fine?)

(In reply to Marco Bonardo [:mak] from comment #21)

2. we're retaining duplicates visited recently (we guessed 7 days, could be extended eventually).

May this cause issues? Of course it could, that's why we're landing behind a pref to study it in Nightly, and we consider this just the first step. We must start from somewhere.

This probably means the scenario in my dupe-bug 1753747 won't be addressed (switch-to-tab being drowned out by history entries from a Google doc that you've got open and have been actively working with). But that's fine -- we can un-dupe or file a more specific bug.

Comment 23

[Marco Bonardo [:mak]]

It's possible in the future we'll experiment "coalesced" results, that may include a sub list of "duplicates", reachable through some action. Then we can coalesce more results while still providing access to them. That would also allow us to measure how often duplicates are picked.
This first filtering opens the door to further experiments.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e3c65d4e3de6
Deduplicate urlbar history results that only differ by their URL fragment. r=mak,urlbar-reviewers

Comment 25

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/e3c65d4e3de6

Comment 26

[Donal Meehan [:dmeehan]]

:mbeier might be worth nominating this for a release note? (Process info)

Comment 27

[Marco Bonardo [:mak]]

the feature is disabled by default behind a pref (we are investigating UI treatments yet).

Comment 28

[Petruta Horea [:phorea]]

This fix was covered and subsequently verified by the Deduplicate Results from the Same Page QA-3671 feature run.