1250171 - Remove *_PER_USER_FLAG probes

Status: RESOLVED FIXED

(DevTools :: General, defect, P2)

Description

[Bryan Clark (DevTools PM) [:clarkbw]]

bug 1247985 is implementing the various *_OPENED_PER_USER_FLAG probes as COUNT probes.  We need to review the existing FLAG probes to determine if they are still needed.

We likely need some transition time between the two probes so setting the old probes to expire a couple releases after the new probes have been available in the release channel may be an option to transition away.

Comment 1

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Filter on TEAPOT-SPLINES.

Comment 2

[Bryan Clark (DevTools PM) [:clarkbw]]

I've done a review of the *_OPENED_PER_USER_FLAG metrics and I believe we need to keep them. And I believe these metrics, just like the COUNT metrics from bug 1247985 need to be opt-out.

*_OPENED_PER_USER_FLAG metrics offer a view into long term user retention as they are only triggered once per release.  Meaning that as this number goes up it should reflect that more users are opening the toolbox per release, indicating overall user growth.

As demonstrated in our metrics dashboard the COUNT metrics will do well to provide DevTools with a system for tracking DAU (Daily Active Users) while the PER_USER_FLAG metrics also provide a similarly important metric for tracking MAU (Monthly Active Users) or more accurately in our case RAU (Release Active Users).  Without both COUNT and PER_USER_FLAG metrics both opt-out we cannot have an accurate understanding of our engagement ratio across all channels. 

The PER_USER_FLAG doesn't collect any more information than the COUNT metric, only the same data collected in a different method.

Metrics Dashboard demo:
https://clarkbw.github.io/devtools-metrics/

Comment 3

[Bryan Clark (DevTools PM) [:clarkbw]]

Bumping this up from the previous triage.  Without the release MAU numbers we'll have no way to create an accurate engagement ratio of the release population.

Comment 4

[Bryan Clark (DevTools PM) [:clarkbw]]

As I'm watching the COUNT metrics go into production we need to this to move forward as well or our release graphs will be useless.  Can you uplift this to catch up to the COUNT metrics?

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Bryan Clark (DevTools PM) [@clarkbw] from comment #4)
> As I'm watching the COUNT metrics go into production we need to this to move
> forward as well or our release graphs will be useless.  Can you uplift this
> to catch up to the COUNT metrics?

The only changes to be made would be marking the *_OPENED_PER_USER_FLAG probes as enabled for release?

Comment 6

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5)
> (In reply to Bryan Clark (DevTools PM) [@clarkbw] from comment #4)
> > As I'm watching the COUNT metrics go into production we need to this to move
> > forward as well or our release graphs will be useless.  Can you uplift this
> > to catch up to the COUNT metrics?
> 
> The only changes to be made would be marking the *_OPENED_PER_USER_FLAG
> probes as enabled for release?

Yes, the metrics are good as is for seeing users per release.  We just need it to be opt-out.

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Okay, let's give it a try.

Comment 8

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Bug 1250171 - Remove *_PER_USER_FLAG probes from DevTools.

Review commit: https://reviewboard.mozilla.org/r/43205/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43205/

Comment 9

[Bryan Clark (DevTools PM) [:clarkbw]]

Comment on attachment 8736303 [details]
Bug 1250171 - Remove *_PER_USER_FLAG probes from DevTools.

https://reviewboard.mozilla.org/r/43205/#review39795

::: toolkit/components/telemetry/Histograms.json:7105
(Diff revision 1)
>    "DEVTOOLS_TOOLBOX_OPENED_PER_USER_FLAG": {
> +    "alert_emails": ["[e-mail]"],
>      "expires_in_version": "never",
>      "kind": "flag",
> -    "description": "Number of users that have opened the DevTools toolbox."
> +    "bug_numbers": [1250171],
> +    "description": "Number of users that have opened the DevTools toolbox.",

I would recommend changing the descriptions to use the following format:

"Number of users, per release, that have opened ${x}"

Because this flag is actually reset with every Firefox release via the prefs. I had no idea that was the case until I looked at the code.

Comment 10

[Benjamin Smedberg]

The per-user-flag metrics are pretty hacky in general. They were the only way to get per-user metrics before we had a clientID. Now that we have both a clientID and longitudinal datasets, it really shouldn't be necessary to have all of this complexity on the client. You can instead look at the feature usage from the per-session counts and flags. That also lets you compute the engagement ratio for particular features.

Also we're really trying to get the descriptions to say what and when: so something like: "This flag is recorded the first time a user uses feature X in each version."

Comment 11

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

We're now planning to remove these probes and compute the values through other means as suggested in comment 10.  However, we don't want to remove the probes yet, as that would delete their data for all channels.

Reducing priority since there's no longer a quick change needed.

Comment 12

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Comment on attachment 8736303 [details]
Bug 1250171 - Remove *_PER_USER_FLAG probes from DevTools.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/43205/diff/1-2/

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Comment on attachment 8736303 [details]
Bug 1250171 - Remove *_PER_USER_FLAG probes from DevTools.

Don't think we need :bsmedberg to review, we're only removing probes now.

Comment 14

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=03d3aef0e831

Comment 15

[Bryan Clark (DevTools PM) [:clarkbw]]

Comment on attachment 8736303 [details]
Bug 1250171 - Remove *_PER_USER_FLAG probes from DevTools.

https://reviewboard.mozilla.org/r/43203/#review68492

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Comment on attachment 8736303 [details]
Bug 1250171 - Remove *_PER_USER_FLAG probes from DevTools.

:clarkbw was perplexed by MozReview and gave r+ on IRC.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/00f781f21da3
Remove *_PER_USER_FLAG probes from DevTools. r=clarkbw

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/00f781f21da3