843807 - Track telemetry uptake on beta + release once the new FHR UI goes in

Status: RESOLVED FIXED

(Mozilla Metrics :: Frontend Reports, defect)

Description

[(dormant account)]

There is huge upcoming change in telemetry opt-in process due to FHR. Opting into telemetry requires atleast 2 clicks and telemetry isn't even visible until the first click to show the options dialog.

We need to verify that our telemetry opt-in rates do not drop significantly and act quickly if they do.

Comment 1

[Alex Fowler]

How are things going on this potential issue?

Comment 2

[(dormant account)]

(In reply to Alex Fowler from comment #1)
> How are things going on this potential issue?

I was just about to ask Annie? Annie, can you give us an update on this?

Comment 3

[Lawrence Mandel [:lmandel] (use needinfo)]

I don't think anything is needed from me so I'm going to clear my needinfo flag.

Comment 4

[Mike Connor [:mconnor]]

I suspect we'll be unable to make any sort of conclusive determination on the basis of Fx21, since we don't have strong numbers for previous versions.  What I'd like to see is a) % of users with Telemetry enabled per channel and b) how that trend is going over time.

Taras, Lawrence, I know there's been a bunch of discussion about a target %/minimum viable population, did that ever reach a conclusion?  If we're going to have a dashboard/dashboard entry for this, we should really have a target.  The last numbers I've seen were 0.82% (release) and 0.92% (beta) based on submissions, so I think that's what we should be looking to see on the beta population.

Comment 5

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Mike Connor [:mconnor] from comment #4)
> I suspect we'll be unable to make any sort of conclusive determination on
> the basis of Fx21, since we don't have strong numbers for previous versions.
> What I'd like to see is a) % of users with Telemetry enabled per channel and
> b) how that trend is going over time.

I think this is a good point. At the very least, we don't have a consistent measurement on which we can easily draw from previous versions. Perhaps we should treat Fx21 as a baseline and watch for changes over the next few releases.

> Taras, Lawrence, I know there's been a bunch of discussion about a target
> %/minimum viable population, did that ever reach a conclusion?  If we're
> going to have a dashboard/dashboard entry for this, we should really have a
> target.  The last numbers I've seen were 0.82% (release) and 0.92% (beta)
> based on submissions, so I think that's what we should be looking to see on
> the beta population.

I'm not aware of a conclusion on the minimum viable population. I expect that FHR data should be able to help with this determination.

I think the initial concern is a substantial decrease in the Telemetry release population. (Perhaps the concern is really a substantial decrease in the % opt-in uptake per release.) If this is to be a long term measure, I agree that we need to think about if and how it will manifest in the health report.

Comment 6

[Gregory Szorc [:gps]]

Perhaps we should establish targets for what kinds of users we'd like to have Telemetry for, see where reality doesn't meet that desire, and employ "targeted advertising" (product announcements, about:home, etc) to attract those specific, under-represented user populations until we have sufficient opt-in to Telemetry.

Comment 7

["Saptarshi Guha[:joy]"]

Some points

1.We're not checking opt in unless we measure how many have actually seen the notice. What we're measuring are number of installations with telemetry enabled - call it telemetry enabled rate

As of today
telemOn       beta    nightly
      0 0.93809898 0.11981727
      1 0.06190102 0.88018273

telemOn is a longitudinal variable. It's value is the latest value in the installations longitudinal data store.

b) I wouldn't /at all/ compare with numbers in comment [4] - the process to arrive at these numbers is completely different. 

c) keep in mind, whatever be enabled rate, telemetry it is user opted in and hence (in the absence of any other information) *biased*

d) this number will fluctuate as more Firefox installations move to a version with FHR - it should (after that) stay steady.

Code for future reference(mine):
v <- doFHR({
  if(json[['version']]==2){
    channel <- json[['geckoAppInfo']][["updateChannel"]]
    if(channel %in% c("nightly","beta")){
      recentDay <- isn(json$data$days [ sort(names(json$data$days),decreasing=TRUE) ] [[1]])
      telem <- isn(recentDay[['org.mozilla.appInfo.appinfo']][['isTelemetryEnabled']])
      rhcollect(list(channel, telem),1)
}
  }
},reduce=summer,debug='collect')

Comment 8

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Gregory Szorc [:gps] from comment #6)
> Perhaps we should establish targets for what kinds of users we'd like to
> have Telemetry for, see where reality doesn't meet that desire, and employ
> "targeted advertising" (product announcements, about:home, etc) to attract
> those specific, under-represented user populations until we have sufficient
> opt-in to Telemetry.

In fact Mike proposed this type of scheme to me a while back. I think this makes sense once we have the ability to dissect Telemetry instances in a meaningful way.

Comment 9

[(dormant account)]

joy, I think you should look at idle-daily telemetry submission volume vs our ADI on beta vs previous beta releases. FHR data might interesting for other reasons, but doesn't help us see if our telemetry rates are plummeting(due to decreased optin).

I suppose you can use FHR to determine what proportion of beta profiles should be new telemetry opt-ins and how much that corresponds to decreases in telemetry ping rates

Comment 10

[Mike Connor [:mconnor]]

While I think it's worth comparing blocklist pings and Telemetry submission counts, at least as a basic validation step, I'd expect that over the long term we'll want to track the data via FHR since the validation should be much stronger (and we'll be measuring the user's choice directly, rather than attempting to infer from submission counts).

Taras, to partially address your concerns about opt-in rates, I suspect that relatively few Beta users are on new profiles, and thus would have seen the old Telemetry popup.  Joy, can you give us a sense of how many reports are from new profiles created with 21 beta?  And from that subset, can you tell us what percentage have Telemetry enabled?

As a point of concern, if >6% of Beta users have telemetry enabled (according to comment 7), I'd expect that the numbers for pings vs. submissions should be in that same range.  However, the last time we compared the two we got a number (0.92% of BL ADI) of only ~15% of that population.  Obviously we should repeat those comparisons on 21 Beta, but if the pattern holds it would seem that we're losing a lot of data somewhere.

Comment 11

[(dormant account)]

Attachment: downward telemetry trend

Hi taras,

I run this automated report for myself everyday. Looked at recently and saw the steady
downward trend. See pages 4 and 5.

Page 4: bottom panel, nightly and top panel is aurora
Page 5: bottom panel, beta and top panel is release.
The graphs plot the number of telemetry submissions (idle+saved) vs date.
Look at Beta and Aurora: there has been a steady downward submission, quite out of pattern.
Nightly appears to have been fixed.

*Ignore pages 1:3* (page 1 is all channels and is driven by release which is okay)

Regards,
Saptarshi

I asked Saptarshi to quantify this downward trend, waiting on that.

Comment 12

[Mike Connor [:mconnor]]

I filed bug 862599 on that issue, I believe that to be fallout from bug 844331.  More info in that bug.

Comment 13

["Saptarshi Guha[:joy]"]

Not sure how you would quantify loss, but the mean in that period before the drop (in nightly, starting Mar20) is about 280K/day and it dropped to 100K/day, i went back to normal starting April8(ish)

Aurora seems to back on track (wrnt from 380K/day to 130K/day)

Beta is still on the low,whereas it was hovering 700K /day region before the drop it is now roughly 180K/day

Comment 14

[:Irving Reid (No longer working on Firefox)]

Saptarshi, we'd like to get some estimates from FHR about how much Telemetry data we *should* be getting. Can you pull a report from the FHR database for me, covering # of complete sessions, # of aborted sessions, and whether telemetry is enabled, for about a week? I can compare that to what the new Telemetry servers are receiving.

Comment 15

["Saptarshi Guha[:joy]"]

Will be sending you the report shortly.

Comment 16

["Saptarshi Guha[:joy]"]

Attachment: bug.pdf

Hello,

Here is a report of what you asked. The attached PDF is graph of the columns.

ch: channel (aurora/nightly *no version/buildID filter*
dt: date
n: total profiles active on that day
ncomp: mean number of completed sessions per active profile
naborted: mean number of aborted sessions per active profile
propTelemOn: proportion of profiles with Telemetry turned on


         ch         dt      n    ncomp     nabort propTelemOn
 1:  aurora 2013-11-23 133359 2.970208 0.21556100   0.8807116
 2:  aurora 2013-11-24 127993 2.953794 0.20887080   0.8807622
 3:  aurora 2013-11-25 143621 2.890329 0.19828577   0.8850442
 4:  aurora 2013-11-26 141608 2.928719 0.20111152   0.8875246
 5:  aurora 2013-11-27 139583 2.917669 0.19672883   0.8904630
 6:  aurora 2013-11-28 135809 2.904741 0.19791030   0.8921049
 7:  aurora 2013-11-29 132008 2.904127 0.19792740   0.8938343
 8:  aurora 2013-11-30 119651 2.946645 0.20506306   0.8971859
 9:  aurora 2013-12-01 115742 2.868259 0.19547787   0.8965905
10:  aurora 2013-12-02 130832 2.605914 0.17475847   0.9000797
11:  aurora 2013-12-03 120316 1.174000 0.09671199   0.9081126
12: nightly 2013-11-23 145350 3.061163 0.18039904   0.9088358
13: nightly 2013-11-24 143332 3.052954 0.17341557   0.9116158
14: nightly 2013-11-25 157800 2.955925 0.16789607   0.9142437
15: nightly 2013-11-26 155081 2.982487 0.16542323   0.9186192
16: nightly 2013-11-27 152860 2.956280 0.16596232   0.9219804
17: nightly 2013-11-28 148262 2.923318 0.16214539   0.9251332
18: nightly 2013-11-29 144431 2.904674 0.16879340   0.9274738
19: nightly 2013-11-30 131555 2.978040 0.16861389   0.9317694
20: nightly 2013-12-01 130877 2.932685 0.16575105   0.9346514
21: nightly 2013-12-02 143949 2.665166 0.14744111   0.9394044
22: nightly 2013-12-03 132157 1.206429 0.08091134   0.9498501
         ch         dt      n    ncomp     nabort propTelemOn

Comment 17

[Mike Connor [:mconnor]]

Can we get the beta/release numbers as well/instead? I'm glad to see Telemetry adoption is solid on dev channels, but the open question is how much we've impacted usage on channels where we got rid of the super aggressive opt-in UI.

Comment 18

["Saptarshi Guha[:joy]"]

Attachment: Numbers updated for all channels

Hello,

Four pages, with 4 channels per page

1. Active profiles by date
2. Mean Aborted Sessions per Active Profile
3. Mean Complete Sessions per Active Profile
4. Prop. of Active profiles with Telemetry turned on (on that date)

Comment 19

["Saptarshi Guha[:joy]"]

Also, this is as of Dec 12th so ignore the last few days(they will change a lot if this is run again in the future)