1247985 - Convert *_OPENED_BOOLEAN probes to count instead

Status: VERIFIED FIXED

(DevTools :: General, defect, P1)

Description

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Our use of *_OPENED_BOOLEAN Telemetry probes that only record false is not very optimal.

We should convert all of these to count probes instead.

Comment 1

[Bryan Clark (DevTools PM) [:clarkbw]]

All of these probes should be the opt-out / release version.

Comment 2

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: count-probes

Comment 3

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e609f65b212d

Comment 4

[Benjamin Smedberg]

Comment on attachment 8720321 [details] [diff] [review]
count-probes

The changes except for the opt-out bits are fine.

For opt-out histograms, you need to provide a clear statement of how collecting this data will provide user value.

Please re-request feedback? for data review when that is available.

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

:clarkbw, can you provide the additional info here?

Comment 6

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

Comment on attachment 8720321 [details] [diff] [review]
count-probes

Review of attachment 8720321 [details] [diff] [review]:
-----------------------------------------------------------------

DEVTOOLS_XXX_OPENED_BOOLEAN - How many times has a tool been opened?
DEVTOOLS_XXX_OPENED_PER_USER_FLAG - How many users have opened a tool?
DEVTOOLS_XXX_TIME_ACTIVE_SECONDS - How long has a tool has been opened?

DEVTOOLS_XXX_OPENED_COUNT replaces DEVTOOLS_XXX_OPENED_BOOLEAN and shows how many times has a tool been opened. When we first added DEVTOOLS_XXX_OPENED_BOOLEAN the count histogram type did not exist.

Makes perfect sense to switch to the newer histogram type, r+ as long as the telemetry tests still pass on try.

Comment 7

[Bryan Clark (DevTools PM) [:clarkbw]]

bringing this from bug 1240187 which we may want to merge into this bug now.

> * How will collecting this data provide user value? This can either be direct value (optimizing the experience for a particular person), indirect value (improving the product through collecting certain data), or data used to correlate other metrics which provide user value.

The primary goal is to make sure our features work as intended so that we can make a better product.  Specific examples of this can include the following:
 
a) Improve their experience at that specific touch point (for example, if we discover through this data that a huge population of developers uses the release channel, we might be inclined to optimize our documentation or product features for those people).

b) Relate to them via that touch point about other things they might find valuable (for example, beta and nightly currently include a one-time message to DevTools users to consider the Developer Edition. Should it be in release? Maybe, if there are lots of developers using release.)

c) Understand their needs better (for example, learning that more developers use release than any pre-release channel would be a valuable insight that might guide product strategy going forward).

d) Put developer tools where they are needed (for example, if we discover through this data that a huge population of developers uses the release channel, we might change our development process to get those tools on release sooner, i.e. JSON Viewer).

> * Who is responsible for monitoring the data, and what data artifacts (dashboards, alerts, custom reports) will they use to do this? Especially if this is permanent (expires "never"), you need a clear plan for monitoring and providing the user value identified above.

The developer marketing team and the devtools product management team would share this responsibility. Justin Crawford represents the developer marketing team, and Bryan Clark would represent the product management team.

The DevTools team is creating a public and mozilla private dashboard for these metrics to monitor and use as a guide.

All probes send to the dev-developer-tools@ mailing list.  Additionally Bryan is subscribed to the telemetry-alerts group filtered for DEVTOOLS probes. 

Justin has a specific need for the data this tool will generate: producing and watching retention/churn graphs for our developer users, similar to those the growth team has been working on (e.g. https://dataviz.mozilla.org/views/FirefoxDesktopCohortAnalysis/ByAcquisitionPeriod#2), so he will be paying close attention. When the change is made, Justin will schedule a meeting with Bryan to review the numbers in the context of the user values identified above, and we will together work to make any changes the data suggest.

Comment 8

[Bryan Clark (DevTools PM) [:clarkbw]]

Comment on attachment 8720321 [details] [diff] [review]
count-probes

f+
However ending all the descriptions in a ? bothers me in a deep, dark place I didn't realize existed until I saw them all.

"How many times has the devtool's toolbox been opened?" is a statement about the probe, not a question.  amirite?

Comment 10

[Benjamin Smedberg]

Comment on attachment 8720321 [details] [diff] [review]
count-probes

Please note: the automated telemetry alerts are totally useless for what you're trying to do. They alert on scalar regressions in particular builds (performance data, basically), and aren't useful for this. Also what you want is almost exclusively user-based, not session-based. So you'll really have to end up doing your own analysis and not using the builtins.

data-review=me
Expect me to ping back every 6 months to make sure you're still using these, since the user benefit here is in terms of specific product improvements that may no longer be relevant in a year.

Comment 11

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

:clarkbw, do you want the *_OPENED_PER_USER_FLAG probes to be opt-out as well?  If so, let's file a new bug to avoid confusing things too much here.

Comment 12

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #11)
> :clarkbw, do you want the *_OPENED_PER_USER_FLAG probes to be opt-out as
> well?  If so, let's file a new bug to avoid confusing things too much here.

No, those are dead to me now.  We should probably file another bug to review them and remove them as I don't think they'll provide any value beyond the counts here.

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=69d59b0dc619

Comment 14

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/668e08174667

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/668e08174667

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Mihai, I sent an email yesterday to the address you supplied, but it seems like did not receive it.

I'll add my reply here.

> Can you please confirm that on this issue all I have to do is to verify that in about:telemetry - 
> Histograms, the probes from the patch (36 probes) are displayed _COUNT instead of _BOOLEAN and 
> there are displayed only 0 and 1 counters (I supposed that 0 is for counter and 1 is TRUE).

I expect the *_OPENED_COUNT probes to keep increasing each time to repeat the action they are logging. So for example with DEVTOOLS_TOOLBOX_OPENED_COUNT, each time you open the toolbox, it should add 1 again to DEVTOOLS_TOOLBOX_OPENED_COUNT.

Beyond that, I don't think there is more to be tested. When 47 is in release, we could test it is logged there as well, but that doesn't happen for several months.

DEVTOOLS_FONTINSPECTOR_OPENED_COUNT

The font panel was actually disabled recently, see bug 1247723. You could flip "devtools.fontinspector.enabled" in about:config if you want to test this. It appears as another side panel in the inspector tool.

DEVTOOLS_JSBROWSERDEBUGGER_OPENED_COUNT

This should be logged when opening the Browser Toolbox. https://developer.mozilla.org/en-US/docs/Tools/Browser_Toolbox

DEVTOOLS_PICKER_EYEDROPPER_OPENED_COUNT

Open the inspector tool, and find a CSS rule that uses a color value. It should have a small circle next to color value. Clicking the circle opens a color picker. Inside the picker is a eyedropper icon. Clicking the eyedropper icon should start the eyedropper and trigger this probe.

DEVTOOLS_ABOUTDEBUGGING_OPENED_COUNT

This is a new tool that has no UI button yet. I think you just open a tab and go to "about:debugging".

DEVTOOLS_WEBIDE_PROJECT_EDITOR_SAVE_COUNT

Start a new app project in WebIDE. Save a file in the file editor

DEVTOOLS_WEBIDE_NEW_PROJECT_COUNT

Create a new app project in WebIDE.

DEVTOOLS_WEBIDE_IMPORT_PROJECT_COUNT

Import an existing app project into WebIDE from disk.

DEVTOOLS_CUSTOM_OPENED_COUNT

This is used whenever a DevTools panel from an add-on is opened. For example, install https://addons.mozilla.org/en-US/firefox/addon/websocket-monitor/. Then, open the toolbox on some page. If you do not see a WebSocket tab in the toolbox, check the toolbox options to ensure the new tool is turned on. Once you've clicked on the toolbox tab for the add-on's tool, the probe should record.

Comment 17

[Mihai Boldan, Desktop QA [:mboldan]]

Thanks Ryan for helping me with the questions.
I managed to enable all the probes from the patch in about:telemetry - Histograms.
I found a potential issue:
- The tools that are enabled from the Default Firefox Developer Tools and then selected from the Toggle Tools bar, the count indicates that the selected tool was opened two times.
This issue is reproducible only if the tool is selected right after it was enabled.
Note that the issue is not reproducible for the default enabled tools - Performance and Network.
Here are the probes that are counted two times when they are opened:
DEVTOOLS_SHADEREDITOR_OPENED_COUNT
DEVTOOLS_CANVASDEBUGGER_OPENED_COUNT
DEVTOOLS_MEMORY_OPENED_COUNT
DEVTOOLS_STORAGE_OPENED_COUNT
DEVTOOLS_WEBAUDIOEDITOR_OPENED_COUNT
DEVTOOLS_SCRATCHPAD_OPENED_COUNT

Should I log a bug for the issue described above?

Comment 18

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Mihai Boldan, QA [:mboldan] from comment #17)
> Thanks Ryan for helping me with the questions.
> I managed to enable all the probes from the patch in about:telemetry -
> Histograms.
> I found a potential issue:
> - The tools that are enabled from the Default Firefox Developer Tools and
> then selected from the Toggle Tools bar, the count indicates that the
> selected tool was opened two times.
> This issue is reproducible only if the tool is selected right after it was
> enabled.
> Note that the issue is not reproducible for the default enabled tools -
> Performance and Network.
> Here are the probes that are counted two times when they are opened:
> DEVTOOLS_SHADEREDITOR_OPENED_COUNT
> DEVTOOLS_CANVASDEBUGGER_OPENED_COUNT
> DEVTOOLS_MEMORY_OPENED_COUNT
> DEVTOOLS_STORAGE_OPENED_COUNT
> DEVTOOLS_WEBAUDIOEDITOR_OPENED_COUNT
> DEVTOOLS_SCRATCHPAD_OPENED_COUNT
> 
> Should I log a bug for the issue described above?

Whoa...  That's pretty bad!  I was able to reproduce locally.  This was not caused by this change.  It appears to be a longstanding issue. :(

I filed bug 1253125 for this, since it's not specific to the RDM project.

Thanks for catching this!

Comment 19

[Mihai Boldan, Desktop QA [:mboldan]]

Since the issue found while testing this bug was logged separately as Bug 1253125, I'm marking this issue Verified-Fixed.
Note that the bug was verified on Firefox 47.0a1 (2016-03-01) and on Windows 10 x86.