1277925 - Users stuck on beta build 44.0b1 (20151217102820)

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[:Harald Kirschner :digitarald]

Beta build 20151217102820, probably 44.0b1, generates still 1mil usage hours per day with a UT main crash rate of over 10. The numbers are stable since end of 2015 which implies that users have no way of updating and probably don't know that they are stuck on an old build.

Usage Hours: https://sql.telemetry.mozilla.org/queries/469/source#772
Crash Rate: https://sql.telemetry.mozilla.org/queries/469/source#773

Comment 1

[:Harald Kirschner :digitarald]

ni? list via :lizzard to investigate

Comment 2

[Justin Wood (:Callek)]

FWIW this is the sha1 watershed release. Which also is pointed at by our downloads from the website for Windows versions unable to be identified as supporting sha2.

See Bug 1234277.

Comment 3

[Benjamin Smedberg]

Harald, can you confirm whether these users are largely on WinXP? If so, this is expected.

Comment 4

[:Harald Kirschner :digitarald]

All the current users are on Windows NT 6.1, aka Windows 7: https://sql.telemetry.mozilla.org/queries/469#774

Did we consider getting those users on an ESR release? Having users stuck on an old Beta does not help testing, skews our data and potentially impacts user sentiment.

Comment 5

[Justin Wood (:Callek)]

The "watershed" is just that, a version of Firefox that users we can't confirm have Operating Systems that support a Sha2 signing format have to pass through.

Once 44.0b1 is installed, if their OS supports the features we need (win7 should fit this bill, as we have understood it) the in-app updater should download the latest version and update those users to that version.

Do we have data on what Win7 Service pack these users have?

Comment 6

[Benjamin Smedberg]

Can release QA test and make sure that users are still being issued updates from this build? If that's the case, we should hand this to the install/update team to diagnose.

Comment 7

[Sylvestre Ledru [:Sylvestre]]

sure, Florin, Andrei, could you help? Merci

Comment 8

[Bogdan Maris, Desktop Test Engineering]

Hello, I tested on Windows 7 x32 and x64 bit with Firefox 44 beta 1 build 2 and 44 beta 9 and I did not received an update to a newer build. I am wondering if there is a timelimit when the update is triggered since I waited for more than 4 hours and I did not received an update. I did not force the update in any way from About Firefox or any some prefs. (I noticed that Nightly or DevEdition receives an update in seconds after I open an old version of the build).

Comment 9

[Robert Strong (they/them - no direct email)]

Bogdan, could you set app.update.log to true, open the about dialog, and copy / paste into this bug the lines in the browser console that start with *** AUS ? Thanks!

Comment 10

[Bogdan Maris, Desktop Test Engineering]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #9)
> Bogdan, could you set app.update.log to true, open the about dialog, and
> copy / paste into this bug the lines in the browser console that start with
> *** AUS ? Thanks!

I posted the console output in pastebin: http://pastebin.com/v0sxYtBe
I tested using Firefox 44beta1 on Windows 7 x86 Professional Version 6.1 (Build 7601: Service Pack 1).

Comment 11

[Robert Strong (they/them - no direct email)]

According to the log it downloaded
http://download.cdn.mozilla.net/pub/firefox/candidates/47.0-candidates/build3/update/win32/en-US/firefox-47.0.complete.mar

and the update was successfully staged
AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: applied-service, status: applied

The update was also in the downloading state when you started
AUS:SVC Downloader:_selectPatch - found existing patch with state: downloading
AUS:SVC Downloader:_selectPatch - resuming download

It appears that all is well with updating on your system. Do you have any indication that the update didn't succeed?

Comment 12

[Bogdan Maris, Desktop Test Engineering]

As I said in comment 8, I did not force an update using About dialog. I just started the browser and left it be, than I waited for an update. Wille waiting I did saw a few http://download.cdn.mozilla.net/pub/firefox/candidates/47.0-candidates/build3/update/win32/en-US/firefox-47.0.complete.mar lines in browser console but no notification about an update or update after restart could be seen during my session (~4hours).
Also I stated above that I am not sure if there is a time set via prefs when the update will be applied or not.

Comment 13

[Robert Strong (they/them - no direct email)]

Thanks. It appears that your system is updating properly and won't be of help here.

Comment 14

[Benjamin Smedberg]

Bogdan, after quitting and relaunching Firefox did it actually update correctly?

This is mysterious to me, but I'll start by creating a dataset of the recent update telemetry for users on this build to see if there are common patterns.

Comment 15

[Bogdan Maris, Desktop Test Engineering]

If I *don't* access about dialog page, Firefox will not prompt me with an update even if I restart, exit Firefox, I'll still get 44 beta 1. 
If I *do* access about dialog page and wait for the update to download, after a restart or exit I do get Firefox 47.0.

Comment 16

[Benjamin Smedberg]

That is strange. The log you gave rstrong shows us staging a 47beta update. This should be applied at next startup without any prompting (no prompting is normal and expected). So if we're staging the update but not applying it that's a problem.

rstrong, could you have a look please?

Comment 17

[Robert Strong (they/them - no direct email)]

Bogdan, could you reproduce the case you described with the app.update.log pref set to true and after a few hours copy / paste the values that start with *** AUS into this bug? It might be possible to see if something is wrong with a few of the following "http://download.cdn.mozilla.net/pub/firefox/candidates/47.0-candidates/build3/update/win32/en-US/firefox-47.0.complete.mar lines in browser console" but please just leave it open in case an error is reported at a later time.

Comment 18

[Bogdan Maris, Desktop Test Engineering]

Well, I retested again and now I do get the update from 44beta1 to 48beta1 after exit and relaunch. Here is the requested console output (http://pastebin.com/4DsDxN90).

Comment 19

[:Harald Kirschner :digitarald]

This build shows no change in usage hours. Can we explore other avenues for debugging this?

Comment 20

[Robert Strong (they/them - no direct email)]

Matthew and Stephen, any ideas as to how to research this? Perhaps telemetry?

Comment 21

[Molly Howell (she/her) (mostly inactive)]

Seems like a good chance this is the same issue as bug 1284915; 43.0.1 is also a SHA-2 watershed. But I have no idea yet what that issue might be; telemetry on release 43.0.1 has not shown me anything to implicate that specific version and, as with this bug, there's no OS version correlation either. Still working on it.

Comment 22

[Stephen A Pohl [:spohl] (OOO until 4/29)]

I get the impression that the affected users are either all running Windows 7 with KB3123479 applied, or not applied[1].

I don't believe we can detect whether or not KB3123479 is applied via telemetry.

Bogdan, could you:
1. Install Windows 7.
2. Install Beta build 20151217102820 (probably 44.0b1 per comment 0).
3. Apply all Windows updates.
4. Ensure that KB3123479 was installed.
5. Set app.update.log to true in prefs.js in your browser profile.
6. Run Firefox.
7. Open Browser Console.
8. Wait for update check to occur.
9. Copy/paste everything with *** AUS in this bug or pastebin.

Please repeat the above, but uninstall KB3123479 before opening Firefox and letting it check for updates.

Thanks!

[1] https://support.microsoft.com/en-us/kb/3123479

Comment 23

[Bogdan Maris, Desktop Test Engineering]

(In reply to Stephen A Pohl [:spohl] from comment #22)
> I get the impression that the affected users are either all running Windows
> 7 with KB3123479 applied, or not applied[1].
> 
> I don't believe we can detect whether or not KB3123479 is applied via
> telemetry.
> 
> Bogdan, could you:
> 1. Install Windows 7.
> 2. Install Beta build 20151217102820 (probably 44.0b1 per comment 0).
> 3. Apply all Windows updates.
> 4. Ensure that KB3123479 was installed.
> 5. Set app.update.log to true in prefs.js in your browser profile.
> 6. Run Firefox.
> 7. Open Browser Console.
> 8. Wait for update check to occur.
> 9. Copy/paste everything with *** AUS in this bug or pastebin.
> 
> Please repeat the above, but uninstall KB3123479 before opening Firefox and
> letting it check for updates.
> 
> Thanks!
> 
> [1] https://support.microsoft.com/en-us/kb/3123479

Here are the results:
- With KB3123479 installed, Firefox 44 beta 1 will update to 48 beta 6 (when I tested that was the latest) in a few hours. 
-- Pastebin link: http://pastebin.com/qikM8WK5
- Without KB3123479 installed, Firefox 44 beta 1 updated to 48 beta 1 in a few hours and did not update further during that session.
-- Pastebin link: http://pastebin.com/ta2ntkT8

Let me know if there is anything I can help with.

Comment 24

[Stephen A Pohl [:spohl] (OOO until 4/29)]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #23)
> (In reply to Stephen A Pohl [:spohl] from comment #22)
> > I get the impression that the affected users are either all running Windows
> > 7 with KB3123479 applied, or not applied[1].
> > 
> > I don't believe we can detect whether or not KB3123479 is applied via
> > telemetry.
> > 
> > Bogdan, could you:
> > 1. Install Windows 7.
> > 2. Install Beta build 20151217102820 (probably 44.0b1 per comment 0).
> > 3. Apply all Windows updates.
> > 4. Ensure that KB3123479 was installed.
> > 5. Set app.update.log to true in prefs.js in your browser profile.
> > 6. Run Firefox.
> > 7. Open Browser Console.
> > 8. Wait for update check to occur.
> > 9. Copy/paste everything with *** AUS in this bug or pastebin.
> > 
> > Please repeat the above, but uninstall KB3123479 before opening Firefox and
> > letting it check for updates.
> > 
> > Thanks!
> > 
> > [1] https://support.microsoft.com/en-us/kb/3123479
> 
> Here are the results:
> - With KB3123479 installed, Firefox 44 beta 1 will update to 48 beta 6 (when
> I tested that was the latest) in a few hours. 
> -- Pastebin link: http://pastebin.com/qikM8WK5
> - Without KB3123479 installed, Firefox 44 beta 1 updated to 48 beta 1 in a
> few hours and did not update further during that session.
> -- Pastebin link: http://pastebin.com/ta2ntkT8
> 
> Let me know if there is anything I can help with.

It looks like we're making progress, but there are two things that don't make sense. Is it possible that the two pastebin links got mixed up? The second one actually shows a successful download and apply of the update, while the first one shows a failure. Could you please clarify which one belongs to which scenario?

Also, the first pastebin link shows that a download was already in progress when the test ran. Could you please retest both scenarios with a new/clean profile? Thanks!

Comment 25

[Rail Aliiev [:rail]]

Also the first pastebin doesn't have the aus5 link, so we can verify the actual size declared in the update XML.

Comment 26

[Stephen A Pohl [:spohl] (OOO until 4/29)]

You will probably have to copy/paste the console output several times for each scenario. Otherwise, since you need to restart Firefox to apply updates, the previous console output will be lost. If you could create two pastebins (one for each scenario), where you append all the console output just before you restart Firefox, that would be really helpful.

Comment 27

[Stephen A Pohl [:spohl] (OOO until 4/29)]

If the first pastebin[1] in comment 23 is representative for the issue here, we should be seeing an excessive number of error code 14 for UPDATE_DOWNLOAD_CODE_COMPLETE in Beta 44. We're currently unable to verify this via our telemetry dashboard v4[2] due to bug 1286580.

[1] http://pastebin.com/qikM8WK5
[2] https://mzl.la/29vGCs9

Comment 28

[Bogdan Maris, Desktop Test Engineering]

(In reply to Stephen A Pohl [:spohl] from comment #26)
> You will probably have to copy/paste the console output several times for
> each scenario. Otherwise, since you need to restart Firefox to apply
> updates, the previous console output will be lost. If you could create two
> pastebins (one for each scenario), where you append all the console output
> just before you restart Firefox, that would be really helpful.

Finally I managed to do that \o/

I edited the bins from comment 23 with final results, and also commented inline:
- With KB3123479 installed:
-- Pastebin link: http://pastebin.com/qikM8WK5
- Without KB3123479 installed:
-- Pastebin link: http://pastebin.com/ta2ntkT8

Let me know if there is something I can help with further on.

Comment 29

[Stephen A Pohl [:spohl] (OOO until 4/29)]

Both of these pastebins now show successful updates, so we have not been able to show that KB3123479 has anything to do with this issue yet.

I am a bit concerned about the fact that we seemingly don't run any background update checks the first time after an update has been applied and I filed bug 1287851 for it.

We've released 48.0b7 recently and we fixed an issue along the way that tried to incorrectly update users to 48.0b6 directly, even though they should have been updated to 48.0b1 (a watershed build) before being updated to 48.0b6. I've rerun the reports in comment 0 and comment 4 and it looks like since the beginning of July, the situation is finally improving. Not sure why the crash rate was skyrocketing recently though. Harald, can you confirm? Or is this "improvement" only due to the fact that data is still coming in and we might not have an accurate picture for the month of July yet?

Comment 30

[:Harald Kirschner :digitarald]

> Harald, can you confirm?

Looks like the numbers are going down \o/. I'll keep an eye on it.

> Not sure why the crash rate was skyrocketing recently though.

The crash spike in the end is just data stabilizing with pings still missing.

Comment 31

[Rail Aliiev [:rail]]

Is there anything left here?

Comment 32

[:Harald Kirschner :digitarald]

It seems the trend downwards didn't turn out to be real. The usage hours reported are pretty constant. Was the expectation that this number should have been 0 by now?

Comment 33

[Stephen A Pohl [:spohl] (OOO until 4/29)]

Agreed. I'm now very certain that we're dealing with the equivalent of bug 1284915 here (see comment 21). Matt has been looking into this, but unfortunately, I don't believe we have a lead yet.

Comment 34

[Rail Aliiev [:rail]]

Maybe bug 1284484 will improve the situation - since this PT morning we are pointing XP beta downloads to 49.0b8, see https://bugzilla.mozilla.org/show_bug.cgi?id=1284484#c17

Release is still the same (43.0.1), but plan to serve 49.0 to XP users when it's released (Sep 13, IIRC).

Comment 35

[Stephen A Pohl [:spohl] (OOO until 4/29)]

Per comment 4, this affects Windows 7 users, not XP.

Comment 36

[Rail Aliiev [:rail]]

Ah, I see... It shouldn't affect this bug in this case. At least https://github.com/mozilla-services/go-bouncer/blob/master/handlers.go#L29-L30 tells me that "NT 6.1" shouldn't be affected.

Comment 37

[Hannes Verschore [:h4writer]]

IIUC we don't know why firefox doesn't update itself right?
I see a lot of crashes on "js::jit::EnterBaselineMethod" on 44.0b1 (20151217102820).
In the last 30 days 1926 to be exact. 22 persons have left a "valid" email during crashing.
Could we contact the persons that are stuck on it and try to debug it that way?

Comment 38

[Rail Aliiev [:rail]]

Robert, is this still the case? Maybe we should generate multiple partials (due to the watersheds) to try to update theses users?

Comment 39

[Robert Strong (they/them - no direct email)]

As with all updates partials would help. I think that getting partials for release users using versions prior to 47.0.2 would have a bigger impact overall though.

Comment 40

[Marco Castelluccio [:marco]]

This is still a problem: https://sql.telemetry.mozilla.org/queries/2327.
During the past week, the number of users on 44b was around half the users of 51b.

Comment 41

[Sylvestre Ledru [:Sylvestre]]

David, can you help with that?

Comment 42

[David Durst [:ddurst]]

Well, we have partials for 43.0.1 (bug 1309130) and 47.0.2 (bug 1319905). Do we have partials for things in between?

We know that between watersheds and general slowness to update, we are looking for ways to encourage users to upgrade (whether they are not updating or not able to update is harder to divine). Partials for all the things probably wouldn't hurt, but I defer to rstrong and rail, as I thought that we do this as a rule.

Comment 43

[Molly Howell (she/her) (mostly inactive)]

I've just filed bug 1334220 for issues downloading updates to the next watershed after 44, which is 48.0b2. That could be a cause of this bug.

Comment 44

[Robert Strong (they/them - no direct email)]

It definitely looks like this bug is due to bug 1334220.

I filed bug 1334419 in the hope of preventing this from happening in the future.

Comment 45

[Molly Howell (she/her) (mostly inactive)]

Bug 1334220 is now resolved. Hopefully the numbers should start improving very soon.

Comment 46

[Robert Strong (they/them - no direct email)]

Note: these clients will need to first update to 48 beta and then to 52 beta and releng has only created complete updates for this so it will take some time.

Comment 47

[Marco Castelluccio [:marco]]

I'll re-rerun the query on Telemetry in a week/two weeks.

Comment 48

[Chris H-C :chutten]

Beta 44 DAU is falling off a cliff in a very satisfactory fashion: https://sql.telemetry.mozilla.org/queries/2428/source#4497

Comment 49

[Rail Aliiev [:rail]]

(In reply to Chris H-C :chutten from comment #48)
> Beta 44 DAU is falling off a cliff in a very satisfactory fashion:
> https://sql.telemetry.mozilla.org/queries/2428/source#4497

\o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ 

I owe :mhowell a drink for findings in Bug 1334220. :)

Comment 50

[Sylvestre Ledru [:Sylvestre]]

Great news, many thanks to all!
I reported bug 1335732 to avoid this from happening in the future

Comment 51

[[:philipp]]

there are also a couple of other old beta versions which are eye-catching in crash stats data and might have users stranded on them - i've filed bug 1335736 for this...

Comment 52

[Rail Aliiev [:rail]]

I'm going to kill this bug with fire. Thanks all!