1213041 - Telemetry Measurements for Beta Doorhanger

Status: RESOLVED WONTFIX

(DevTools :: General, defect)

Description

[Helen V. Holmes (:helenvholmes) (:✨)]

It'd be nice to be able to do some A/B testing for the beta doorhanger to see which copy versions drive engagement the best.

Comment 1

[Helen V. Holmes (:helenvholmes) (:✨)]

Relevant line: https://dxr.mozilla.org/mozilla-central/source/devtools/client/shared/doorhanger.js#78

Comment 2

[Brian Grinstead [:bgrins]]

So at the minimum we need to add a measurement for when the doorhanger is shown, i.e. https://dxr.mozilla.org/mozilla-central/source/devtools/client/shared/doorhanger.js#78 and when it is clicked, i.e. https://dxr.mozilla.org/mozilla-central/source/devtools/client/shared/doorhanger.js#113.

I guess those are two booleans, like BETA_DOORHANGER_SHOWN and BETA_DOORHANGER_CLICKED. We would want to have some way to also correspond the click back to a variation of the doorhanger if we end up A/B testing two versions, so maybe those should be ints..  Helen, any other thoughts/details about what data you want to gather?

Jordan, I don't know how to do this.  I believe you've worked with telemetry for the perf tool and also the beta doorhanger, so maybe you can give some guidance on this.

1) What kind of histogram should we use for this?
2) Should it use this._telemetry.logOncePerBrowserVersion or this._telemetry.log or this._telemetry.logKeyed?  I guess it's only likely to fire once per profile since once we show the doorhanger the pref gets flipped

Comment 3

[Helen V. Holmes (:helenvholmes) (:✨)]

(In reply to Brian Grinstead [:bgrins] from comment #2)
> I guess those are two booleans, like BETA_DOORHANGER_SHOWN and
> BETA_DOORHANGER_CLICKED. We would want to have some way to also correspond
> the click back to a variation of the doorhanger if we end up A/B testing two
> versions, so maybe those should be ints..  Helen, any other thoughts/details
> about what data you want to gather?
I'd like to collect the percentage of people clicking on the green button (goes to the DE download page) out of the total number of people seeing the doorhanger. I'd like to sub in different copy and see what performs the best. Ultimately I'd like to collect that same data but subbing in different copy all over the doorhanger, but that feels like a reasonable, first start.

Comment 4

[Jordan Santell [:jsantell] [@jsantell]]

Flag would probably be a good type -- it'll only record once, and can be used for any type of data. I think if we have BETA_DOORHANGER_SHOWN_FLAG and BETA_DOORHANGER_CLICKED_FLAG, each with an int representing a version of the doorhanger, I think that'd work fine. Using `telemetry.log("BETA_DOORHANGER_SHOWN_FLAG", 1)` should do it. More info here if you haven't seen: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Keyed_Histograms

Comment 5

[Jordan Santell [:jsantell] [@jsantell]]

(ignore the hash in the URL, woops, this wouldn't be a keyed histo)

Comment 6

[Patrick Brosset <:pbro>]

The beta doorhanger is no more.