1570683 - Define Engagement Success for the TopSites (and future) experiment

Status: RESOLVED FIXED

(Data Science :: Experiment Collaboration, task, P1)

Description

[Marco Bonardo [:mak]]

Now we have Event Telemetry in addition to the existing probes.

Event Telemetry will report the following information
EVENTS:
urlbar#engagement#click
urlbar#engagement#enter
urlbar#engagement#drop_go
urlbar#engagement#paste_go
urlbar#abandonment#blur
EVENT VALUE:
The value for the event is the type of initial interaction: "typed", "pasted", "dropped", "topsites"
EXTRA DATA:
elapsed time, number of characters
only for engagement: selection index, selection type

We already report keyed scalars counting the number of each event, the above event name is the key, even if recording is disabled.

To get events, recording must be enabled by flipping the browser.urlbar.eventTelemetry.enabled pref, or using the browser.urlbar.engagementTelemetry WebExtension API.

Based on this data, data-science and product management should come to a definition of success and a way to visualize it.

Please reach engineering if you are missing something.

I'm assigning this to Ryan, should likely be discussed with Teon and Connor.

Comment 1

[Marco Bonardo [:mak]]

Setting a few heads-up needinfo.

Comment 2

[Teon Brooks [:teon]]

thanks ::mak. is there an experiment that is in the works. i didn't see anything in Experimenter and just found the meta experiment bug, which just seems to be a tracking one.

Comment 3

[Marco Bonardo [:mak]]

The bug blocked by this one (Bug 1547279) tracks the relase of the first experiment, TopSites.
Though, it would be nice if we could have some confirmation that we are in a good shape with the available telemetry to measure success for that experiment, once it will be released. I know, it's a bit abstract without a running experiment, but once 69 is released, and experiments run on it, we won't be able to make changes to the collected data.

Comment 4

[Teon Brooks [:teon]]

Some of the metrics I see worth understanding are:

• Selection of top site in the QuantumBar. My understanding is that this will be pre-filled topsites at initial interaction.
• We would need to look at the clicking behavior of the TopSites on the ActivityStream via PingCenter. We would like to understand if the behavior in the QuantumBar is additive to the NewTab TopSite engagement, subtractive, or has no effect.

::mak will there be a way to tell when the TopSite is presented versus the search suggestions as the metrics stand at the moment? At the moment, that is not captured in the northstar engagement metrics, but this would be valuable to have so we can understand topsite behaviors.

Comment 5

[Marco Bonardo [:mak]]

(In reply to Teon Brooks [:teon] from comment #4)

Some of the metrics I see worth understanding are:

• Selection of top site in the QuantumBar. My understanding is that this will be pre-filled topsites at initial interaction.

Yes, initial interaction (thus the value of the event) will be "topsites", that means the user either pressed down or clicked on the dropmarker, without typing anything.
Note, it's possible the user initiates the engagement like that, but then types something, it will still be marked as "topsites" because that's the initial interaction. To differentiate between the 2 actions one can check numChars.

Thus, to get topsites picks, it should be enough to check value == "topsites" and extra.numChars == "0"

Comment 6

[Drew Willcoxon :adw]

Ben had some comments on an experiment control for top sites that I found very helpful, over in the similar search-nudges bug: bug 1572174 comment 6

Comment 7

[Teon Brooks [:teon]]

The data science peer should be able to answer these questions, on the Data Science bug in Bugzilla:

What is the goal of the effort the experiment is supporting?

We are introducing new features to the Address bar and would like to understand if this improves the overall experience.
The Quantum Bar, which is the HTML based implementation of the Address bar, now allows our designers to test out new UX for this experience. We have developed a set of metrics to evaluate the experience within the address bar, and these have now been instrumented. We aim to validate these probes, and begin to have a better picture of the user's interaction with the Address bar. This includes the proportion of search abandonment to all initiated searches, the time to complete searches, and the manner of interacting with the address bar (e.g. copy/paste, click, enter, escape).

Is an experiment a useful next step towards this goal?

Yes, it gives us an opportunity to validate our new metrics.

What is the hypothesis or research question? Are the consequences for the top-level goal clear if the hypothesis is confirmed or rejected?

This study is two-fold: 1. to understand if users would benefit from having access to their topsites from the address bar. 2. validate new probes and metrics.

Which measurements will be taken, and how do they support the hypothesis and goal? Are these measurements available in the targeted release channels? Has there been data steward review of the collection?

The measurements to be studied are search volumes, interactions with topsites, search abandonment, and engagement time

Is the experiment design supported by an analysis plan? Is it adequate to answer the experimental questions?

yes. part of the experiment is exploratory and validation of new probes. it would give us a meaningful understanding of the interactions.

Is the requested sample size supported by a power analysis that includes the core product metrics?

no. a power analysis was not conducted as the metrics of interests are new and have no basis for comparison.

If the experiment is deployed to channels other than release, is it acceptable that the results will not be representative of the release population?

this is to release channel.

Comment 8

[Experimenter Bot]

[Experiment]Activity Stream Search TopSites status has been changed to: Live
            url:https://experimenter.services.mozilla.com/experiments/search-topsites/

Comment 9

[Experimenter Bot]

[Experiment]Activity Stream Search TopSites status has been changed to: Complete
            url:https://experimenter.services.mozilla.com/experiments/search-topsites/

Comment 10

[Teon Brooks [:teon]]

final report here: https://iodide.telemetry.mozilla.org/notebooks/390/?viewMode=report