Closed Bug 1572174 Opened 3 years ago Closed 3 years ago

Define engagement success for the search nudges add-on experiment

Categories

(Firefox :: Address Bar, task, P1)

task

Tracking

()

RESOLVED DUPLICATE of bug 1575953

People

(Reporter: adw, Assigned: bmiroglio)

References

Details

+++ This bug was initially created as a clone of Bug #1570683 +++

This is similar to bug 1570683 but it's for the quantumbar search nudges add-on experiment. In addition to the event telemetry Marco mentions in bug 1570683, we will probably add a new type of urlbar#engagement for clicks on the search nudges' "OK, got it" button, so that clicks on that button are accurately recorded in telemetry. (That new telemetry will be tracked/implemented in bug 1571144.)

Based on this data, data-science and product management should come to a definition of success and a way to visualize it.

Flags: needinfo?(teon)
Flags: needinfo?(rharter)

Passing to Ben, who I believe worked on the last instantiation of nudges. Ben please consult with Teon if necessary.

Assignee: nobody → bmiroglio
Flags: needinfo?(rharter) → needinfo?(bmiroglio)

ack

I can handle the core search metrics for this experiment, using the following metrics for success:

At least a 1% increase in SAP search volume in general, and from the URL bar source. Split out by:

  • overall
  • saw the prompt
  • engaged with the prompt

A nice-to-have is also a complementary decrease in organic search.

@teon, did you have specific URL Bar engagement metrics in mind for this experiment?

Flags: needinfo?(bmiroglio)

I think we would just like to see if the event telemetry is working as it should. we haven't seen it collected in the wild yet so it would be great just to assess if it is within expectations and to get a better understanding of it.

Flags: needinfo?(teon)

Is the nudges functionality the same as before in that it sends "phantom" nudge events for the control group? This is essential for getting a true apples-to-apples comparison between the groups, which for analysis ultimately become:

Treatment: those who saw at least one nudge
Control: those who would have seen at least one nudge, but didn't since they are in the control

Flags: needinfo?(adw)

I'm not familiar with the previous experiment for search/urlbar nudges, but we're still writing the add-on and the APIs, so we will certainly keep that in mind.

Do we need a control for the top-sites experiment, too (bug 1570683)? e.g., record clicks in the urlbar, which would trigger the top-sites popup on the treatment branch but not do anything on the control branch.

Flags: needinfo?(adw)

If the treatment condition is hinged on a user action, i.e. clicking on the URL Bar, then we need to capture that user action (and hypothetical delivery of a popup) across all branches when possible. Suppose that only 10% of users (in both branches) perform the action that triggers the popup--that means 90% of the treatment group never actually saw the treatment, and we need a way to differentiate them from the control.

The ideal case in the above scenario is that we can isolate the 10% of control clients that would have triggered the popup, and compare them to the 10% of users that did actually see the popup. If we have sufficient event telemetry for this, that that will suffice, but I believe for nudges the pop-up is shown when a user navigates to google.com? (which isn't specifically captured in telemetry)

See the dashboard for the previous nudges experiment as a reference: https://sql.telemetry.mozilla.org/dashboard/search-nudges-experiment?p_study_name_58658=searchNudgesExperiment&p_start_date_58658=20180827&p_version_58658=1.8.2.

Note the charts at the bottom, that show certain events for the noshow (control) branch.

Thank you, Ben, that's very helpful. This is something I hadn't considered before. Yes, the treatment condition for top sites (if I understand that term correctly) is that the user focuses the urlbar and then the top-sites popup opens as a result. I'll need to make sure we're capturing the correct engagement event telemetry for control, in particular not inadvertently recording a treatment "topsites" event, which I think might be what would happen. That might require adding another event type for control, I'll see.

(In reply to Ben Miroglio [:bmiroglio] from comment #6)

If we have sufficient event telemetry for this, that that will suffice, but I believe for nudges the pop-up is shown when a user navigates to google.com? (which isn't specifically captured in telemetry)

True, we'll need to capture that specifically somehow. Note to self on the implementation: I'm guessing we can do it in the add-on itself via the browser.telemetry API, without needing to land anything in Firefox.

Yes, I believe for nudges in the past, all telemetry was sent the same way for both branches, the popup was just hidden for the control branch

Status: NEW → ASSIGNED

Hey Drew--I'm doing some workload organization and have a request:

Would you mind filing a bug under Data Science > Experiment Collaboration

A template is there for that component that helps our team size / triage these requests. You can put my name as the Data Scientist, its more so that my team/managers have this experiment on their radar as well. You can see also this bug to ensure our conversation is preserved--sorry for the hassle.

Flags: needinfo?(adw)
See Also: → 1575953

I'll close this bug since it doesn't seem like we need both open. We can still refer to the discussion here, as you said.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(adw)
Resolution: --- → DUPLICATE
Duplicate of bug: 1575953
You need to log in before you can comment on or make changes to this bug.