Closed
Bug 1277925
Opened 9 years ago
Closed 8 years ago
Users stuck on beta build 44.0b1 (20151217102820)
Categories
(Release Engineering :: Release Requests, defect)
Release Engineering
Release Requests
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: Harald, Unassigned)
References
(Depends on 1 open bug)
Details
Beta build 20151217102820, probably 44.0b1, generates still 1mil usage hours per day with a UT main crash rate of over 10. The numbers are stable since end of 2015 which implies that users have no way of updating and probably don't know that they are stuck on an old build.
Usage Hours: https://sql.telemetry.mozilla.org/queries/469/source#772
Crash Rate: https://sql.telemetry.mozilla.org/queries/469/source#773
Reporter | ||
Comment 1•9 years ago
|
||
ni? list via :lizzard to investigate
Flags: needinfo?(mhowell)
Flags: needinfo?(lhenry)
Flags: needinfo?(benjamin)
Comment 2•9 years ago
|
||
FWIW this is the sha1 watershed release. Which also is pointed at by our downloads from the website for Windows versions unable to be identified as supporting sha2.
See Bug 1234277.
Comment 3•9 years ago
|
||
Harald, can you confirm whether these users are largely on WinXP? If so, this is expected.
Flags: needinfo?(mhowell)
Flags: needinfo?(lhenry)
Flags: needinfo?(hkirschner)
Flags: needinfo?(benjamin)
Reporter | ||
Comment 4•9 years ago
|
||
All the current users are on Windows NT 6.1, aka Windows 7: https://sql.telemetry.mozilla.org/queries/469#774
Did we consider getting those users on an ESR release? Having users stuck on an old Beta does not help testing, skews our data and potentially impacts user sentiment.
Flags: needinfo?(hkirschner) → needinfo?(benjamin)
Comment 5•9 years ago
|
||
The "watershed" is just that, a version of Firefox that users we can't confirm have Operating Systems that support a Sha2 signing format have to pass through.
Once 44.0b1 is installed, if their OS supports the features we need (win7 should fit this bill, as we have understood it) the in-app updater should download the latest version and update those users to that version.
Do we have data on what Win7 Service pack these users have?
Comment 6•9 years ago
|
||
Can release QA test and make sure that users are still being issued updates from this build? If that's the case, we should hand this to the install/update team to diagnose.
Flags: needinfo?(benjamin) → needinfo?(sledru)
Comment 7•9 years ago
|
||
sure, Florin, Andrei, could you help? Merci
Flags: needinfo?(sledru)
Flags: needinfo?(florin.mezei)
Flags: needinfo?(andrei.vaida)
Comment 8•9 years ago
|
||
Hello, I tested on Windows 7 x32 and x64 bit with Firefox 44 beta 1 build 2 and 44 beta 9 and I did not received an update to a newer build. I am wondering if there is a timelimit when the update is triggered since I waited for more than 4 hours and I did not received an update. I did not force the update in any way from About Firefox or any some prefs. (I noticed that Nightly or DevEdition receives an update in seconds after I open an old version of the build).
Flags: needinfo?(florin.mezei)
Flags: needinfo?(benjamin)
Flags: needinfo?(andrei.vaida)
![]() |
||
Comment 9•9 years ago
|
||
Bogdan, could you set app.update.log to true, open the about dialog, and copy / paste into this bug the lines in the browser console that start with *** AUS ? Thanks!
Flags: needinfo?(bogdan.maris)
Comment 10•9 years ago
|
||
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #9)
> Bogdan, could you set app.update.log to true, open the about dialog, and
> copy / paste into this bug the lines in the browser console that start with
> *** AUS ? Thanks!
I posted the console output in pastebin: http://pastebin.com/v0sxYtBe
I tested using Firefox 44beta1 on Windows 7 x86 Professional Version 6.1 (Build 7601: Service Pack 1).
Flags: needinfo?(bogdan.maris) → needinfo?(robert.strong.bugs)
![]() |
||
Comment 11•9 years ago
|
||
According to the log it downloaded
http://download.cdn.mozilla.net/pub/firefox/candidates/47.0-candidates/build3/update/win32/en-US/firefox-47.0.complete.mar
and the update was successfully staged
AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. state: applied-service, status: applied
The update was also in the downloading state when you started
AUS:SVC Downloader:_selectPatch - found existing patch with state: downloading
AUS:SVC Downloader:_selectPatch - resuming download
It appears that all is well with updating on your system. Do you have any indication that the update didn't succeed?
Flags: needinfo?(robert.strong.bugs) → needinfo?(bogdan.maris)
Comment 12•9 years ago
|
||
As I said in comment 8, I did not force an update using About dialog. I just started the browser and left it be, than I waited for an update. Wille waiting I did saw a few http://download.cdn.mozilla.net/pub/firefox/candidates/47.0-candidates/build3/update/win32/en-US/firefox-47.0.complete.mar lines in browser console but no notification about an update or update after restart could be seen during my session (~4hours).
Also I stated above that I am not sure if there is a time set via prefs when the update will be applied or not.
Flags: needinfo?(bogdan.maris)
![]() |
||
Comment 13•9 years ago
|
||
Thanks. It appears that your system is updating properly and won't be of help here.
Comment 14•9 years ago
|
||
Bogdan, after quitting and relaunching Firefox did it actually update correctly?
This is mysterious to me, but I'll start by creating a dataset of the recent update telemetry for users on this build to see if there are common patterns.
Flags: needinfo?(benjamin) → needinfo?(bogdan.maris)
Comment 15•9 years ago
|
||
If I *don't* access about dialog page, Firefox will not prompt me with an update even if I restart, exit Firefox, I'll still get 44 beta 1.
If I *do* access about dialog page and wait for the update to download, after a restart or exit I do get Firefox 47.0.
Flags: needinfo?(bogdan.maris)
Comment 16•9 years ago
|
||
That is strange. The log you gave rstrong shows us staging a 47beta update. This should be applied at next startup without any prompting (no prompting is normal and expected). So if we're staging the update but not applying it that's a problem.
rstrong, could you have a look please?
Assignee: nobody → robert.strong.bugs
Flags: needinfo?(robert.strong.bugs)
![]() |
||
Comment 17•9 years ago
|
||
Bogdan, could you reproduce the case you described with the app.update.log pref set to true and after a few hours copy / paste the values that start with *** AUS into this bug? It might be possible to see if something is wrong with a few of the following "http://download.cdn.mozilla.net/pub/firefox/candidates/47.0-candidates/build3/update/win32/en-US/firefox-47.0.complete.mar lines in browser console" but please just leave it open in case an error is reported at a later time.
Flags: needinfo?(robert.strong.bugs) → needinfo?(bogdan.maris)
Comment 18•9 years ago
|
||
Well, I retested again and now I do get the update from 44beta1 to 48beta1 after exit and relaunch. Here is the requested console output (http://pastebin.com/4DsDxN90).
Flags: needinfo?(bogdan.maris)
Reporter | ||
Comment 19•9 years ago
|
||
This build shows no change in usage hours. Can we explore other avenues for debugging this?
Flags: needinfo?(robert.strong.bugs)
![]() |
||
Comment 20•9 years ago
|
||
Matthew and Stephen, any ideas as to how to research this? Perhaps telemetry?
Flags: needinfo?(spohl.mozilla.bugs)
Flags: needinfo?(robert.strong.bugs)
Flags: needinfo?(mhowell)
Comment 21•9 years ago
|
||
Seems like a good chance this is the same issue as bug 1284915; 43.0.1 is also a SHA-2 watershed. But I have no idea yet what that issue might be; telemetry on release 43.0.1 has not shown me anything to implicate that specific version and, as with this bug, there's no OS version correlation either. Still working on it.
Flags: needinfo?(mhowell)
Comment 22•9 years ago
|
||
I get the impression that the affected users are either all running Windows 7 with KB3123479 applied, or not applied[1].
I don't believe we can detect whether or not KB3123479 is applied via telemetry.
Bogdan, could you:
1. Install Windows 7.
2. Install Beta build 20151217102820 (probably 44.0b1 per comment 0).
3. Apply all Windows updates.
4. Ensure that KB3123479 was installed.
5. Set app.update.log to true in prefs.js in your browser profile.
6. Run Firefox.
7. Open Browser Console.
8. Wait for update check to occur.
9. Copy/paste everything with *** AUS in this bug or pastebin.
Please repeat the above, but uninstall KB3123479 before opening Firefox and letting it check for updates.
Thanks!
[1] https://support.microsoft.com/en-us/kb/3123479
Flags: needinfo?(spohl.mozilla.bugs) → needinfo?(bogdan.maris)
Comment 23•9 years ago
|
||
(In reply to Stephen A Pohl [:spohl] from comment #22)
> I get the impression that the affected users are either all running Windows
> 7 with KB3123479 applied, or not applied[1].
>
> I don't believe we can detect whether or not KB3123479 is applied via
> telemetry.
>
> Bogdan, could you:
> 1. Install Windows 7.
> 2. Install Beta build 20151217102820 (probably 44.0b1 per comment 0).
> 3. Apply all Windows updates.
> 4. Ensure that KB3123479 was installed.
> 5. Set app.update.log to true in prefs.js in your browser profile.
> 6. Run Firefox.
> 7. Open Browser Console.
> 8. Wait for update check to occur.
> 9. Copy/paste everything with *** AUS in this bug or pastebin.
>
> Please repeat the above, but uninstall KB3123479 before opening Firefox and
> letting it check for updates.
>
> Thanks!
>
> [1] https://support.microsoft.com/en-us/kb/3123479
Here are the results:
- With KB3123479 installed, Firefox 44 beta 1 will update to 48 beta 6 (when I tested that was the latest) in a few hours.
-- Pastebin link: http://pastebin.com/qikM8WK5
- Without KB3123479 installed, Firefox 44 beta 1 updated to 48 beta 1 in a few hours and did not update further during that session.
-- Pastebin link: http://pastebin.com/ta2ntkT8
Let me know if there is anything I can help with.
Flags: needinfo?(bogdan.maris) → needinfo?(spohl.mozilla.bugs)
Comment 24•9 years ago
|
||
(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #23)
> (In reply to Stephen A Pohl [:spohl] from comment #22)
> > I get the impression that the affected users are either all running Windows
> > 7 with KB3123479 applied, or not applied[1].
> >
> > I don't believe we can detect whether or not KB3123479 is applied via
> > telemetry.
> >
> > Bogdan, could you:
> > 1. Install Windows 7.
> > 2. Install Beta build 20151217102820 (probably 44.0b1 per comment 0).
> > 3. Apply all Windows updates.
> > 4. Ensure that KB3123479 was installed.
> > 5. Set app.update.log to true in prefs.js in your browser profile.
> > 6. Run Firefox.
> > 7. Open Browser Console.
> > 8. Wait for update check to occur.
> > 9. Copy/paste everything with *** AUS in this bug or pastebin.
> >
> > Please repeat the above, but uninstall KB3123479 before opening Firefox and
> > letting it check for updates.
> >
> > Thanks!
> >
> > [1] https://support.microsoft.com/en-us/kb/3123479
>
> Here are the results:
> - With KB3123479 installed, Firefox 44 beta 1 will update to 48 beta 6 (when
> I tested that was the latest) in a few hours.
> -- Pastebin link: http://pastebin.com/qikM8WK5
> - Without KB3123479 installed, Firefox 44 beta 1 updated to 48 beta 1 in a
> few hours and did not update further during that session.
> -- Pastebin link: http://pastebin.com/ta2ntkT8
>
> Let me know if there is anything I can help with.
It looks like we're making progress, but there are two things that don't make sense. Is it possible that the two pastebin links got mixed up? The second one actually shows a successful download and apply of the update, while the first one shows a failure. Could you please clarify which one belongs to which scenario?
Also, the first pastebin link shows that a download was already in progress when the test ran. Could you please retest both scenarios with a new/clean profile? Thanks!
Flags: needinfo?(spohl.mozilla.bugs) → needinfo?(bogdan.maris)
Comment 25•9 years ago
|
||
Also the first pastebin doesn't have the aus5 link, so we can verify the actual size declared in the update XML.
Comment 26•9 years ago
|
||
You will probably have to copy/paste the console output several times for each scenario. Otherwise, since you need to restart Firefox to apply updates, the previous console output will be lost. If you could create two pastebins (one for each scenario), where you append all the console output just before you restart Firefox, that would be really helpful.
Comment 27•9 years ago
|
||
If the first pastebin[1] in comment 23 is representative for the issue here, we should be seeing an excessive number of error code 14 for UPDATE_DOWNLOAD_CODE_COMPLETE in Beta 44. We're currently unable to verify this via our telemetry dashboard v4[2] due to bug 1286580.
[1] http://pastebin.com/qikM8WK5
[2] https://mzl.la/29vGCs9
Depends on: 1286580
Comment 28•9 years ago
|
||
(In reply to Stephen A Pohl [:spohl] from comment #26)
> You will probably have to copy/paste the console output several times for
> each scenario. Otherwise, since you need to restart Firefox to apply
> updates, the previous console output will be lost. If you could create two
> pastebins (one for each scenario), where you append all the console output
> just before you restart Firefox, that would be really helpful.
Finally I managed to do that \o/
I edited the bins from comment 23 with final results, and also commented inline:
- With KB3123479 installed:
-- Pastebin link: http://pastebin.com/qikM8WK5
- Without KB3123479 installed:
-- Pastebin link: http://pastebin.com/ta2ntkT8
Let me know if there is something I can help with further on.
Flags: needinfo?(bogdan.maris) → needinfo?(spohl.mozilla.bugs)
Comment 29•9 years ago
|
||
Both of these pastebins now show successful updates, so we have not been able to show that KB3123479 has anything to do with this issue yet.
I am a bit concerned about the fact that we seemingly don't run any background update checks the first time after an update has been applied and I filed bug 1287851 for it.
We've released 48.0b7 recently and we fixed an issue along the way that tried to incorrectly update users to 48.0b6 directly, even though they should have been updated to 48.0b1 (a watershed build) before being updated to 48.0b6. I've rerun the reports in comment 0 and comment 4 and it looks like since the beginning of July, the situation is finally improving. Not sure why the crash rate was skyrocketing recently though. Harald, can you confirm? Or is this "improvement" only due to the fact that data is still coming in and we might not have an accurate picture for the month of July yet?
Flags: needinfo?(spohl.mozilla.bugs) → needinfo?(hkirschner)
Reporter | ||
Comment 30•9 years ago
|
||
> Harald, can you confirm?
Looks like the numbers are going down \o/. I'll keep an eye on it.
> Not sure why the crash rate was skyrocketing recently though.
The crash spike in the end is just data stabilizing with pings still missing.
Flags: needinfo?(hkirschner)
Comment 31•9 years ago
|
||
Is there anything left here?
Reporter | ||
Comment 32•9 years ago
|
||
It seems the trend downwards didn't turn out to be real. The usage hours reported are pretty constant. Was the expectation that this number should have been 0 by now?
Flags: needinfo?(spohl.mozilla.bugs)
Comment 33•9 years ago
|
||
Agreed. I'm now very certain that we're dealing with the equivalent of bug 1284915 here (see comment 21). Matt has been looking into this, but unfortunately, I don't believe we have a lead yet.
Flags: needinfo?(spohl.mozilla.bugs)
Comment 34•9 years ago
|
||
Maybe bug 1284484 will improve the situation - since this PT morning we are pointing XP beta downloads to 49.0b8, see https://bugzilla.mozilla.org/show_bug.cgi?id=1284484#c17
Release is still the same (43.0.1), but plan to serve 49.0 to XP users when it's released (Sep 13, IIRC).
Comment 35•9 years ago
|
||
Per comment 4, this affects Windows 7 users, not XP.
Comment 36•9 years ago
|
||
Ah, I see... It shouldn't affect this bug in this case. At least https://github.com/mozilla-services/go-bouncer/blob/master/handlers.go#L29-L30 tells me that "NT 6.1" shouldn't be affected.
Updated•9 years ago
|
Summary: Users stuck on beta build 20151217102820 → Users stuck on beta build 44.0b1 (20151217102820)
Comment 37•9 years ago
|
||
IIUC we don't know why firefox doesn't update itself right?
I see a lot of crashes on "js::jit::EnterBaselineMethod" on 44.0b1 (20151217102820).
In the last 30 days 1926 to be exact. 22 persons have left a "valid" email during crashing.
Could we contact the persons that are stuck on it and try to debug it that way?
Comment 38•8 years ago
|
||
Robert, is this still the case? Maybe we should generate multiple partials (due to the watersheds) to try to update theses users?
Flags: needinfo?(robert.strong.bugs)
![]() |
||
Comment 39•8 years ago
|
||
As with all updates partials would help. I think that getting partials for release users using versions prior to 47.0.2 would have a bigger impact overall though.
Flags: needinfo?(robert.strong.bugs)
Comment 40•8 years ago
|
||
This is still a problem: https://sql.telemetry.mozilla.org/queries/2327.
During the past week, the number of users on 44b was around half the users of 51b.
Comment 42•8 years ago
|
||
Well, we have partials for 43.0.1 (bug 1309130) and 47.0.2 (bug 1319905). Do we have partials for things in between?
We know that between watersheds and general slowness to update, we are looking for ways to encourage users to upgrade (whether they are not updating or not able to update is harder to divine). Partials for all the things probably wouldn't hurt, but I defer to rstrong and rail, as I thought that we do this as a rule.
Flags: needinfo?(robert.strong.bugs)
Flags: needinfo?(rail)
Flags: needinfo?(ddurst)
Comment 43•8 years ago
|
||
I've just filed bug 1334220 for issues downloading updates to the next watershed after 44, which is 48.0b2. That could be a cause of this bug.
![]() |
||
Updated•8 years ago
|
Flags: needinfo?(robert.strong.bugs)
![]() |
||
Comment 44•8 years ago
|
||
It definitely looks like this bug is due to bug 1334220.
I filed bug 1334419 in the hope of preventing this from happening in the future.
Flags: needinfo?(rail)
![]() |
||
Updated•8 years ago
|
Assignee: robert.strong.bugs → nobody
Comment 45•8 years ago
|
||
Bug 1334220 is now resolved. Hopefully the numbers should start improving very soon.
![]() |
||
Comment 46•8 years ago
|
||
Note: these clients will need to first update to 48 beta and then to 52 beta and releng has only created complete updates for this so it will take some time.
Comment 47•8 years ago
|
||
I'll re-rerun the query on Telemetry in a week/two weeks.
Flags: needinfo?(mcastelluccio)
Comment 48•8 years ago
|
||
Beta 44 DAU is falling off a cliff in a very satisfactory fashion: https://sql.telemetry.mozilla.org/queries/2428/source#4497
Updated•8 years ago
|
Flags: needinfo?(mcastelluccio)
Comment 49•8 years ago
|
||
(In reply to Chris H-C :chutten from comment #48)
> Beta 44 DAU is falling off a cliff in a very satisfactory fashion:
> https://sql.telemetry.mozilla.org/queries/2428/source#4497
\o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/
I owe :mhowell a drink for findings in Bug 1334220. :)
Comment 50•8 years ago
|
||
Great news, many thanks to all!
I reported bug 1335732 to avoid this from happening in the future
Comment 51•8 years ago
|
||
there are also a couple of other old beta versions which are eye-catching in crash stats data and might have users stranded on them - i've filed bug 1335736 for this...
Comment 52•8 years ago
|
||
I'm going to kill this bug with fire. Thanks all!
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•