1606913 - Ensure telemetry for Interventions is sufficient and add new telemetry if necessary

Status: RESOLVED FIXED

(Firefox :: Address Bar, task, P2)

Description

[Harry Twyford]

Because Event Telemetry can only be enabled for a subset of the population, may want to add some keyed scalars.

Comment 1

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Drew, do you have any insights about discussions you had with Data while working on the experiment?
What do we care about, what they asked us to measure in addition to normal engagement?

Comment 2

[Drew Willcoxon :adw]

afaik, setting aside event telemetry, we want to record the same things the experiment does: which interventions were shown in an engagement, and which interventions the user picked. That's what Data was interested in for the experiment. The experiment uses keyed scalars for both, and that seems fine here too.

Comment 3

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Considered that for production we can't rely on event telemetry due to its costs, we need to come up with scalars or histograms that we can keep enabled indefinitely.
Based on what you said, we had keyed scalars for shown and picked interventions, next step is to have Data confirm those are what we care about long term.

Comment 4

[Teon Brooks [:teon]]

i think the keyed scalar would be sufficient for our needs.

I think from the discussion we had with Ben, the one thing we would like to see phantom interventions for the control group to get an estimate of how many interventions a user might see if it had been presented.

does the intervention come with a button to execute an action or does it only have a dismiss button? I think the one thing we would want to see is if the intervention message leads to the intended action.

Comment 5

[Harry Twyford]

(In reply to Teon Brooks [:teon] from comment #4)

does the intervention come with a button to execute an action or does it only have a dismiss button?

Interventions have two buttons: one to execute an action, and a help/"what is this?" button. They don't have a dismiss button. Rather, they are dismissed when the user modifies their query.

Comment 6

[Harry Twyford]

Interventions are live and preffed off in Nightly, if you want to demo them. Enable browser.urlbar.update1.interventions and search "firefox slow" in the Urlbar to trigger one.

Comment 7

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Teon Brooks [:teon] from comment #4)

I think from the discussion we had with Ben, the one thing we would like to see phantom interventions for the control group to get an estimate of how many interventions a user might see if it had been presented.

This seems related to the experiment add-on, not the product (ni Drew as an heads-up).

Comment 8

[Drew Willcoxon :adw]

Thanks, we'd discussed that previously and the experiment does have telemetry for phantom interventions.

Comment 9

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Some discussion related to this is ongoing in bug 1608461, we're likely implementing all tips telemetry at once due to similarities.

Comment 10

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

I'll just resolve this as an investigation bug, the result of the investigation is being implemented in bug 1608461