1601362 - Decide what kind of telemetry retained results should send

Status: RESOLVED FIXED

(Firefox :: Address Bar, task, P2)

Description

[Harry Twyford]

+++ This bug was initially created as a clone of Bug #1601052 +++

openViewOnFocus sends engagement telemetry and retained results does not. openViewOnFocus functionality is very similar to retained results and the user might not understand which of the two features is firing. If the point of the engagement telemetry is to learn about how users interact with the Urlbar, it doesn't make sense to not record telemetry for an action that is indistinguishable (from the user's perspective) from another feature that does send it.

Marco suggested sending a "returning" engagement event.

Comment 1

[Harry Twyford]

We just could record the same telemetry event for retained results as we do for openViewOnFocus. This would give us less granular telemetry than an approach that records two different types of events, but it's probably more representative of the user's intent: "I am clicking on the Urlbar because I want to go somewhere". I don't think there would really be two different intents warranting two different types of telemetry. That is, the user doesn't really know when they're triggering openViewOnFocus and when they're triggering retained results.

Comment 2

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Well, in one case (retained) it's returning to a previously interrupted engagement, in the other case (openView) it's starting a new engagement.
We store the initial interaction type as "typed", "dropped", "pasted", "topsites". openViewOnFocus falls under topsites or typed depending whether there's a search string. Retained results could act the same (basically it would always be "typed" because there must be a search string), but I think it may be nicer to count it apart with a different initial interaction.
May be a good question for Data.

Comment 3

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Let me sum up the facts and questions.

We have 2 features:

1. openViewOnFocus is behind a pref (disabled by default, but enabled in the top sites experiment) is expected to replace the dropdown marker and open the top sites list. Currently it also opens a normal search if the user focuses the urlbar while a search string is typed (we must determine if this behavior should be removed). It returns "topsites" or "typed" as initial engagement.

2. retained results, happens when the user abandoned a search, but then goes back to it. We are not currently accounting for this. The question is whether this should be accounted with a "typed" initial engagement, or it should have its own "returning" initial engagement.

Comment 4

[Teon Brooks [:teon]]

wrt 1. could you elaborate more on the switch from openViewOnFocus, is this when a user hovers over, clicks on the urlbar? is this part of the megabar redesign?

wrt 2. when the user abandons the search, is the content preserved? if so, is it labeled as "return" if add to it? what happens if they delete the content then continue?

at the moment, I don't see a strong need for this but i think i am missing some of the broader context.

Comment 5

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Teon Brooks [:teon] from comment #4)

wrt 1. could you elaborate more on the switch from openViewOnFocus, is this when a user hovers over, clicks on the urlbar? is this part of the megabar redesign?

openViewOnFocus is a separate pref we introduced for the TopSites experiment, it is currently not enabled in the redesign, it's only enabled in the experiment, because we were waiting for experiment results to make a call about it.
When enabled, if the urlbar is NOT focused and the user focuses it, by clicking or with the keyboard shortcut, we open the results pane.
It registers "typed" or "topsites" depending on the string in the urlbar, but we are debating about making it only open when it would show top sites (so it would only register "topsites").

wrt 2. when the user abandons the search, is the content preserved? if so, is it labeled as "return" if add to it? what happens if they delete the content then continue?

First, "retained results" is part of the redesign and enabled with it. So it's only enabled in Nightly atm.
We keep the last search results cached, if the user abandons a search, and then returns to it, we reopen the results pane with the cached results. If there are no cached results for the string in the urlbar, we do nothing.
For example if the user is typing something, switches to another tab or another window, and comes back, this kicks-in.

Comment 6

[Teon Brooks [:teon]]

hey, when do you need to have a decision for this. with the topsites study that just completed, we are currently checking with data engineering to make sure that the increase in the event telemetry is ok to proceed. could this be addressed after we have an answer from them which is pending on my report that will be completed next week?

Comment 7

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Teon Brooks [:teon] from comment #6)

hey, when do you need to have a decision for this. with the topsites study that just completed, we are currently checking with data engineering to make sure that the increase in the event telemetry is ok to proceed. could this be addressed after we have an answer from them which is pending on my report that will be completed next week?

No strict schedule, the new design won't be released in 73... though if we want to run some experiment on 73 we may need this, that pretty much makes this a requirement at the beginning of the next week (with holidays in the middle it may be complicate later).

I'm not sure why the increase matters here, event telemetry is still behind a pref, and this change is only adding a few events, basically when the user returns to the last executed search. I don't think it will move the needle by much.

Comment 8

[Teon Brooks [:teon]]

i think it is more that if we were to recommend that the event telemetry is preffed to default on. it's possible we might need to restructure it to keyed histograms based on the volume. but that is actually not the question at hand.

for the retained results, would we know if they continue to add to the results versus cleared them all out and began a new search. I think if we were to know that, that would let us know if the caching is useful or not.

i think that the distinction would be useful between the openViewOnFocus and Retained Results. it would give us greater granularity in the interaction and if it could also be lumped together if we were just looking at the generalized action between the two.

Comment 9

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Based on latest discussion with Verdi, openViewOnFocus and Retained Results are being merged into a single feature.
We will always open the panel on focus, unless we are on about:home or about:newtab with an empty string.

At this point, also based on comment 8, the only thing that is interesting to measure apart, is what the user does when returning to an abandoned search.

This is my proposal:

• Focus input field when it's empty or contains the currently loaded url:
  value is "topsites", numChars can be used to tell if the user picked a topsite or typed a search string and picked something else.
  This is already implemented.
• Focus input field when it contains an abandoned search string, and continue that search (also removing part of the value, but not all of it):
  value is "returning"
  Must be implemented.
• Focus input field when it contains an abandoned search string, and clear it:
  value is "resetting", numChars can be used to tell if the user picked a topsite or typed a search string and picked something else.
  Must be implemented.

Comment 10

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Attachment: Bug 1601362 - Add more specific retained results telemetry. r=adw

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:mak, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b049edb3d213
Add more specific retained results telemetry. r=adw

Comment 13

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/b049edb3d213