1313151 - Remove opt-in histograms from longitudinal dataset

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Comment 1

[Ryan Harter [:harter]]

Hey Alessio,

Looking at histograms.json[1] I only see two histograms explicitly marked as opt-in[2]. Both of these appear to be for testing. If a histogram is not explicitly marked as opt-out is it opt-in?

[1] https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json
[2] https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json#5796

Comment 2

[Alessio Placitelli [:Dexter]]

(In reply to Ryan Harter [:harter] from comment #1)
> [...] If a histogram is not explicitly marked as opt-out is it opt-in?

Hey Ryan! Yes, if not specified, you can safely assume an histogram is "opt-in" (see [1]).

[1] - https://dxr.mozilla.org/mozilla-central/rev/86f702229e32c6119d092e86431afee576f033a1/toolkit/components/telemetry/histogram_tools.py#130

Comment 3

[Georg Fritzsche [:gfritzsche]]

(Commenting on User Story)
> Given that the longitudinal dataset is used as a representative dataset of
> our user population and that it's only a 1% sample, I think we should remove
> the opt-in measurements as they don't add value.

This seems like a potentially disruptive change, there should probably at least reasonable advance notice be given for this? 
On the pre-release channels (where the opt-in measurements are collected from everyone by default) i would expect them to be used.

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #3)
> (Commenting on User Story)
> > Given that the longitudinal dataset is used as a representative dataset of
> > our user population and that it's only a 1% sample, I think we should remove
> > the opt-in measurements as they don't add value.
> 
> This seems like a potentially disruptive change, there should probably at
> least reasonable advance notice be given for this? 

We should announce the intent of doing this on fhr-dev and fx-data-platform to see if there are any objections.

> On the pre-release channels (where the opt-in measurements are collected
> from everyone by default) i would expect them to be used.

In my experience we can't generally make statistically meaningful claims using only 1% of pre-release.

Comment 5

[Ryan Harter [:harter]]

> > This seems like a potentially disruptive change, there should probably at
> > least reasonable advance notice be given for this? 
> 
> We should announce the intent of doing this on fhr-dev and fx-data-platform
> to see if there are any objections.

I'll send out an email this afternoon.

> > On the pre-release channels (where the opt-in measurements are collected
> > from everyone by default) i would expect them to be used.
> 
> In my experience we can't generally make statistically meaningful claims
> using only 1% of pre-release.

Looks like the current dataset has ~250k clients in pre-release. What types of claims do we try to make with these data? It seems like we should be able to answer some questions with that number of users.

Comment 6

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Ryan Harter [:harter] from comment #5)
> Looks like the current dataset has ~250k clients in pre-release. What types
> of claims do we try to make with these data? It seems like we should be able
> to answer some questions with that number of users.

Right, but our users usually apply some filtering on top of that which can bring the number of eligible users quickly down to something which isn't very interesting. 

Furthermore, mixing opt-in and opt-out measurements makes self-served analysis more error prone as one could easily run a query based on an opt-in measure (and not knowing it's opt-in) and mistakenly think that the result applies to our population as a whole.

Comment 7

[Ryan Harter [:harter]]

Keeping this bug updated, we've identified all users/queries which depend on these histograms. I've emailed these users and am waiting on a response.

Comment 8

[Bryan Clark (DevTools PM) [:clarkbw]]

Jumping in to say that I agree with this choice in the general case.  I think it would likely be safer for those of us creating queries to not mistakenly use opt-in probes. 

I do have certain probes which are only useful to me in Nightly or Aurora because I'm tracking developer tools there.  But I believe I'm going to be the outsider here and I can look for another solution.  Hopefully someone can help me. :)  Given the size of the Nightly and Aurora population it would actually be nice to have a larger sample set anyway.

Comment 9

[Benjamin Smedberg]

Note that the original plan for longitudinal was to have separate longitudinals for certain subgroups, so I'd like us to continue to explore having a nightly-longitudinal with 100% of nightly, beta-longitudinal with 10% of beta, etc. I agree we should make this change for the current longitudinal because it's a footgun.

Comment 10

[Ryan Harter [:harter]]

Attachment: https://github.com/mozilla/telemetry-batch-view/pull/148

Sounds like we want to move forward with this change and pursue the 100% pre-release longitudinal set. 

PR attached.

Comment 11

[Frank Bertsch [:frank]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #4)
> In my experience we can't generally make statistically meaningful claims
> using only 1% of pre-release.

Any particular reason we kept opt-in scalars? This argument applies to those as well.

Comment 12

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Frank Bertsch [:frank] from comment #11)
> (In reply to Roberto Agostino Vitillo (:rvitillo) from comment #4)
> > In my experience we can't generally make statistically meaningful claims
> > using only 1% of pre-release.
> 
> Any particular reason we kept opt-in scalars? This argument applies to those
> as well.

Opt-in scalars should be removed as well.