1240187 - Make DEVTOOLS_TOOLBOX_OPENED_BOOLEAN histogram opt-out

Status: RESOLVED DUPLICATE of bug 1247985

(Toolkit :: Telemetry, defect)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

In order to understand better how developers use our products, we need more information from the release channel. For example, we'd like to start using "opened devtools" as a factor in metrics about developer retention in Firefox. So, 2 questions...

1) Is DEVTOOLS_TOOLBOX_OPENED_BOOLEAN the right histogram to use for that measure?
2) Can we make the DEVTOOLS_TOOLBOX_OPENED_BOOLEAN histogram opt-out in release?

Comment 1

[Justin Crawford [:hoosteeno] [:jcrawford]]

Advice in #telemetry suggested I needinfo a data peer for review. :bsmedberg, would you consider this change? 

Advocating for it: Some of our big company-wide objectives this year require measurement of active (daily/monthly) users. In the developer space there are a variety of places a user could be active. The two big populations of active developer users are MDN (where we can measure) and devtools in the browser (where, without this change, I think we cannot).

Comment 2

[Benjamin Smedberg]

In order to make a metric opt-out, you need answers to the following questions:

* How will collecting this data provide user value? This can either be direct value (optimizing the experience for a particular person), indirect value (improving the product through collecting certain data), or data used to correlate other metrics which provide user value.
* Who is responsible for monitoring the data, and what data artifacts (dashboards, alerts, custom reports) will they use to do this? Especially if this is permanent (expires "never"), you need a clear plan for monitoring and providing the user value identified above.

Comment 3

[Justin Crawford [:hoosteeno] [:jcrawford]]

Thanks Benjamin. Answers below.

> * How will collecting this data provide user value? This can either be direct value (optimizing the experience for a particular person), indirect value (improving the product through collecting certain data), or data used to correlate other metrics which provide user value.

The primary goal is to make sure our features work as intended so that we can make a better product.  Specific examples of this can include the following:
 
a) Improve their experience at that specific touchpoint (for example, if we discover through this data that a huge population of developers uses the release channel, we might be inclined to optimize our documentation or product features for those people).
b) Relate to them via that touchpoint about other things they might find valuable (for example, beta and nightly currently include a one-time message to devtools users to consider the Developer Edition. Should it be in release? Maybe, if there are lots of developers using release.)
c) Understand their needs better (for example, learning that more developers use release than any pre-release channel would be a valuable insight that might guide product strategy going forward).
d) Put developer tools where they are needed (for example, if we discover through this data that a huge population of developers uses the release channel, we might change our development process to get those tools on release sooner).

> * Who is responsible for monitoring the data, and what data artifacts (dashboards, alerts, custom reports) will they use to do this? Especially if this is permanent (expires "never"), you need a clear plan for monitoring and providing the user value identified above.

The developer marketing team and the devtools product management team would share this responsibility. I would represent the developer marketing team, and Bryan Clark would represent the product management team.

Bryan is subscribed to the telemetry-alerts group where all alerts for large changes in probe values are sent, so he will notice major differences. I have a specific need for the data this tool will generate: I want to produce and watch retention/churn graphs for our developer users, similar to those the growth team has been working on (e.g. https://dataviz.mozilla.org/views/FirefoxDesktopCohortAnalysis/ByAcquisitionPeriod#2), so I will be paying close attention. When the change is made, I will schedule a meeting with Bryan to review the numbers in the context of the user values identified above, and we will together work to make any changes the data suggest.

Comment 4

[Bryan Clark (DevTools PM) [:clarkbw]]

Merging this with bug 1247985 because it is also requesting to make DEVTOOLS_TOOLBOX_OPENED_BOOLEAN histogram opt-out but also changing it to DEVTOOLS_TOOLBOX_OPENED_COUNT which is a much more useful metric.