1572174 - Define engagement success for the search nudges add-on experiment

Status: RESOLVED DUPLICATE of bug 1575953

(Firefox :: Address Bar, task, P1)

Description

[Drew Willcoxon :adw]

+++ This bug was initially created as a clone of Bug #1570683 +++

This is similar to bug 1570683 but it's for the quantumbar search nudges add-on experiment. In addition to the event telemetry Marco mentions in bug 1570683, we will probably add a new type of urlbar#engagement for clicks on the search nudges' "OK, got it" button, so that clicks on that button are accurately recorded in telemetry. (That new telemetry will be tracked/implemented in bug 1571144.)

Based on this data, data-science and product management should come to a definition of success and a way to visualize it.

Comment 1

[Ryan Harter [:harter]]

Passing to Ben, who I believe worked on the last instantiation of nudges. Ben please consult with Teon if necessary.

Comment 2

[Ben Miroglio [:bmiroglio]]

ack

I can handle the core search metrics for this experiment, using the following metrics for success:

At least a 1% increase in SAP search volume in general, and from the URL bar source. Split out by:

• overall
• saw the prompt
• engaged with the prompt

A nice-to-have is also a complementary decrease in organic search.

@teon, did you have specific URL Bar engagement metrics in mind for this experiment?

Comment 3

[Teon Brooks [:teon]]

I think we would just like to see if the event telemetry is working as it should. we haven't seen it collected in the wild yet so it would be great just to assess if it is within expectations and to get a better understanding of it.

Comment 4

[Ben Miroglio [:bmiroglio]]

Is the nudges functionality the same as before in that it sends "phantom" nudge events for the control group? This is essential for getting a true apples-to-apples comparison between the groups, which for analysis ultimately become:

Treatment: those who saw at least one nudge
Control: those who would have seen at least one nudge, but didn't since they are in the control

Comment 5

[Drew Willcoxon :adw]

I'm not familiar with the previous experiment for search/urlbar nudges, but we're still writing the add-on and the APIs, so we will certainly keep that in mind.

Do we need a control for the top-sites experiment, too (bug 1570683)? e.g., record clicks in the urlbar, which would trigger the top-sites popup on the treatment branch but not do anything on the control branch.

Comment 6

[Ben Miroglio [:bmiroglio]]

If the treatment condition is hinged on a user action, i.e. clicking on the URL Bar, then we need to capture that user action (and hypothetical delivery of a popup) across all branches when possible. Suppose that only 10% of users (in both branches) perform the action that triggers the popup--that means 90% of the treatment group never actually saw the treatment, and we need a way to differentiate them from the control.

The ideal case in the above scenario is that we can isolate the 10% of control clients that would have triggered the popup, and compare them to the 10% of users that did actually see the popup. If we have sufficient event telemetry for this, that that will suffice, but I believe for nudges the pop-up is shown when a user navigates to google.com? (which isn't specifically captured in telemetry)

See the dashboard for the previous nudges experiment as a reference: https://sql.telemetry.mozilla.org/dashboard/search-nudges-experiment?p_study_name_58658=searchNudgesExperiment&p_start_date_58658=20180827&p_version_58658=1.8.2.

Note the charts at the bottom, that show certain events for the noshow (control) branch.

Comment 7

[Drew Willcoxon :adw]

Thank you, Ben, that's very helpful. This is something I hadn't considered before. Yes, the treatment condition for top sites (if I understand that term correctly) is that the user focuses the urlbar and then the top-sites popup opens as a result. I'll need to make sure we're capturing the correct engagement event telemetry for control, in particular not inadvertently recording a treatment "topsites" event, which I think might be what would happen. That might require adding another event type for control, I'll see.

(In reply to Ben Miroglio [:bmiroglio] from comment #6)

If we have sufficient event telemetry for this, that that will suffice, but I believe for nudges the pop-up is shown when a user navigates to google.com? (which isn't specifically captured in telemetry)

True, we'll need to capture that specifically somehow. Note to self on the implementation: I'm guessing we can do it in the add-on itself via the browser.telemetry API, without needing to land anything in Firefox.

Comment 8

[Ben Miroglio [:bmiroglio]]

Yes, I believe for nudges in the past, all telemetry was sent the same way for both branches, the popup was just hidden for the control branch

Comment 9

[Ben Miroglio [:bmiroglio]]

Hey Drew--I'm doing some workload organization and have a request:

Would you mind filing a bug under Data Science > Experiment Collaboration

A template is there for that component that helps our team size / triage these requests. You can put my name as the Data Scientist, its more so that my team/managers have this experiment on their radar as well. You can see also this bug to ensure our conversation is preserved--sorry for the hassle.

Comment 10

[Drew Willcoxon :adw]

I'll close this bug since it doesn't seem like we need both open. We can still refer to the discussion here, as you said.