Closed Bug 1376979 Opened 3 years ago Closed 2 years ago

Around 20% of Nightly 55 users have not updated to version 56

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox56+ fix-optional)

RESOLVED WORKSFORME
Tracking Status
firefox56 + fix-optional

People

(Reporter: kbrosnan, Unassigned)

Details

(Whiteboard: [pi-monitoring] )

Comparing the total usage hours for all reporting versions of Nightly  https://sql.telemetry.mozilla.org/queries/3771#7467 vs the current release usage hours from the bottom of https://telemetry.mozilla.org/crashes/

During v55 development we had around a 6% of the reports from users on previous versions. In the v56 development cycle we have seen around 26% using older versions. Assuming that 6% is a normal floor for people running old versions of nightly. This bug is to try and find out what caused 20% of nightly not to update to v56.

Need to investigate if this is OS specific or something else like aus rules or certificate issues.
Whiteboard: [monitoring]
Hi Chris, are you aware of any problems with Nightly updates?
Flags: needinfo?(catlee)
Nightly updates on OSX have been mostly disabled from June 23rd-June 28th.

Windows updates have been disabled a few times this cycle as well.

Would be good to know if this is OS specific or not. Where can we get that data?
Flags: needinfo?(catlee)
This is about older data I had used June 7th and June 20th as two data points.

I had asked chutten about that in slack. Just to make sure the question does not get lost. Chutten is there a way to get a graph similar to the /crashes graph however break it down by OS type [windows, os x, linux, other]?
Flags: needinfo?(chutten)
Here's a query that shows uptake rates on Nightly for the past few releases this year: https://sql.telemetry.mozilla.org/queries/5300/source#10811

We're not outrageously low, and all of the missing users appear to be happily hanging out in the previous versions for the time being.
Flags: needinfo?(chutten)
So it sounds like Nightly 56 usage is lower than 55 because 56's Mac and Windows updates have been disabled a few times so some 55 users haven't been upgraded yet?
If you look back to beginning of February, you can see a very similar pattern.

https://sql.telemetry.mozilla.org/queries/5367/source#10929
I edited some queries that show what I want to highlight.

Windows nightly kilo usage hours over the last 180 days - https://sql.telemetry.mozilla.org/queries/5366/source#10965
Linux & OS X kilo usage hours over the last 180 days - https://sql.telemetry.mozilla.org/queries/5384/source#10967

Windows usage on 56 is down significantly vs 55 however overall usage of nightly is flat. This means that we have users stuck on v55. How to get them unstuck? Generate a new partial between the last v55 nightly and a current v56 nightly? Something else? Just wait it out?
There are two things here:

1) To catlee and chutten's points, we're still pretty early in the cycle, so 56 Nightly doesn't look terribly different from 55 Nightly at this point.

More importantly, 

2) we have an issue with 56 Nightly where firefox.exe is not shutting down, and this prevents the update from being applied (there are a few bugs[1] on this, but the best one to follow right now is https://bugzilla.mozilla.org/show_bug.cgi?id=1374043). Therefore we're going to see users hanging back on previous 56 Nightly versions more than happened on 55 Nightly.

It's possible that there's more than one issue blocking update, so we have to patch what we find.

rstrong pushed a patch in bug 1375549 to allow the update to be applied without trying to restart firefox. We should keep watching these graphs, but to expect some older Nightly versions sticking around until a patch is in for 1374043.


[1] 1375242, 1374364, 1373918, 1374043, 1375614, 1375320...
Also, #c7 is not quite what the summary indicates, but I think between #c2 and #c8, we're covered on this. We wait for a few days.
Keeping an eye on this for 56. It looks like some issues with shutdown and the updater were just fixed around July 6. I'd expect to see more 55 users updating to 56 over this week.
David, Kevin, what do you think of the data now? Do you still think that users are stuck on 55 nightly? Or is this a normal summer dip in nightly usage?
Flags: needinfo?(kbrosnan)
Flags: needinfo?(ddurst)
I don't think windows users are stuck (we're almost certain that problem is 100% solved).

[I can't speak to the summer dip, but linux and darwin seem to be up from previous two versions.] Someone else would have to weigh in on whether this is consistent with, say, last year.
Flags: needinfo?(ddurst)
Nightly channel's usage hours, irrespective of version, in https://sql.telemetry.mozilla.org/queries/3771#7467 look pretty flat with no drop. I modified chutten's graph of Nightly version user counts by version (https://sql.telemetry.mozilla.org/queries/5300/source#10811) to look back from January to the present. Nightly 56's usage is lower than 54's and 55's, but 56 looks like it has climbed nearly back to 55's usage level. I think the issue with some users not upgrading from 55 to 56 in a timely manner has been resolved.
I agree we are close enough. It does highlight that if Nightly is unusable we loose a significant percentage of our user base.
Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(kbrosnan)
Resolution: --- → WORKSFORME
Whiteboard: [monitoring] → [pi-monitoring]
You need to log in before you can comment on or make changes to this bug.