Closed Bug 735784 Opened 12 years ago Closed 12 years ago

Workaround for 13.0a1 users who are getting version downgrade errors

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox12 unaffected, firefox13 wontfix, firefox14 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox12 --- unaffected
firefox13 --- wontfix
firefox14 --- fixed

People

(Reporter: bbondy, Assigned: bbondy)

References

Details

Attachments

(1 file, 1 obsolete file)

Patch for Fix2 v2.
2.24 KB, patch
robert.strong.bugs
: review+
Details | Diff | Splinter Review
Attached patch Patch for fix2 (obsolete) — — Splinter Review 
People on 13.0a1 are getting version downgrade errors to go to 14.0a1 because of Bug 735713.  There are 2 fixes that I can think of, possibly someone else has other suggestions too:

------
Fix 1:
------

The following though is another plan that would leave no users:

Could we return a special MAR from the AUS server if the current version is v13 and the release channel is mozilla-central?

1) Strip the official win32 x86 MAR's signature:
mar -r Official_Signed_MC_Mar_With_Ehsans_fix.mar not_signed.mar

2) Hard code 13.0a1 into that official MAR:
mar -H firefox-mozilla-central -V 13.0a1 -i not_signed.mar

3) Re-sign that MAR:
mar -d NSSConfigDir -n certname -s not_signed.mar special_signed.mar

When a request comes in from a v13 x86 build running on m-c direct the user to special_signed.mar


------
Fix 2:
------

Land a patch on m-c that hardcodes MAR files with 13.0a1.
Leave this patch on m-c for whatever period seems safe.
People who upgrade to 14.0a1 build from March 14th will get a version downgrade error until we revert the task.  People with 13.0a1 builds will get upgraded and working properly.  There is a patch attached for this.
Attachment #605858 - Flags: feedback?(robert.bugzilla)
Comment on attachment 605858 [details] [diff] [review]
Patch for fix2

I *think* it would be better to create a single mar to fix the installations affected instead of going this route and since productVersion can be changed on the command line this patch shouldn't be necessary. As stated previously, what would be needed is to direct all nightly users using 13.* or the subset that require the version check to this mar. I'd like nthomas to weigh in on this before going other routes. We have also done similar things where we have restricted a set of builds to a specific mar.
Attachment #605858 - Flags: feedback?(robert.bugzilla) → feedback-
Agree fix1 is better if it is possible. Fix2 is just if needed.
Summary: Work around for 13.0a1 users who are getting version downgrade errors → Workaround for 13.0a1 users who are getting version downgrade errors
Relating to the signmar command line options. You can also use:
signmar -T inputmar.mar

This will give you information about the signature block and product information block and their values.  You can use this on the original MAR and the new created MAR to ensure that they have the same product information block except for the version.
Just to make sure I have no current action items on this bug, nthomas you are going to look into fix 1?
It might be helpful to identify the date range for builds affected. I've had to do this in the past for a vaguely similar problem and actually had to provide build ID's for the builds affected which might be necessary. Other than the date range we'll need releng input on this which will likely be from nthomas since he has worked on these issues from the releng side previously.
Ok I'll get that info.  By the way if anyone is wondering, for the people that did get this error before nightly m-c updates were turned off, they got a link to manually download the build on the error that came up.  Also once we work this out they will get automatically updated.
The faulty patch from bug 708690 which caused this to happen landed on 2/24.
Depends on: 708690
Info for  nthomas:


The important range are builds:
Nightly 2012-02-25 - Nightly 2012-03-13

Affecting bug info:
-------------------
Bug 708690
Landed on mozilla-central in:
http://hg.mozilla.org/mozilla-central/rev/4de2d63b21cf
Was uplifted to Aurora in: 
http://hg.mozilla.org/releases/mozilla-aurora/rev/18840d884c97
Aurora (and hence now beta) had a fallback though to use updates without this check if this check fails.


Build info:
-----------

Nightly 2012-02-25: First Nightly build the affecting bug landed in
Built from: http://hg.mozilla.org/mozilla-central/rev/ce20e9b47e9c
Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120225 Firefox/13.0a1
BuildID: 20120225031723
Downloaded from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-02-25-03-17-23-mozilla-central/

Nightly 2012-03-14: The Nightly build that is v14.0a1 without ehasn's fix 
Built from: http://hg.mozilla.org/mozilla-central/rev/c71845b3b2a6
Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20120314 Firefox/14.0a1
BuildID: 20120314031139
This build can be optionally included to use the special MAR file.
If not included and someone updates from this exact build to 15.0a1 in a long time they will get an error.
But it is highly unlikely someone would be in this case.
Downloaded from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-03-13-09-04-04-mozilla-central/

Nightly 2012-03-13: The last nightly build that was v13.0a1 without ehsan's fix
Built from: http://hg.mozilla.org/mozilla-central/rev/8d1c74566a0b
Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120313 Firefox/13.0a1
BuildID: 20120313090404
Downloaded from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-03-14-03-11-39-mozilla-central/


Notes for QA:
-------------

I'll follow up with further notes for QA a bit later today.
The special MAR only needs to be made for mozilla-central, so the above info only applies to that.

It is my understanding that we will re-spin the Aurora and Beta builds since no one has those updates yet.
The way updates work for Nightly makes it problematic to point a subset of requests at a special mar. The update server keeps track of a list of builds for each locale, always serving the latest complete update and possibly an partial update if the  buildID in the request is right.

Is there a version we can bake into the mar at compile time that will satisfy all the code in the wild ?
So the bug is that of -1 vs 1 check.  
Which is equivalent to version downgrades and equal versions are allowed, version upgrades are disallowed.

So to fix this we can hardcode v13.0a1 and I would suggest to also disable the version check completely so when people get onto the correct check they won't have to wait until it is backed out to get updates.
Sounds good to me, assuming I track down where in the signing server we set the version.
We set the version implicitly unless explicitly specified (and we don't explicitly specify it currently). 

The patch above is what needs to change.  I will add a check to disable the version compare as well. This only needs to land on m-c and be backed out at some point when we feel comfortable.   1-2 weeks would even be fine since we aren't doing the version compare check we won't be locking anyone out.  

Only people with v13 and v14 have the version check backwards.  So setting it to v13 and disabling the fix will be safe for everyone.

Beta is being respun so the first person to upgrade on beta will have the already fixed version from ehsan. 

Aurora will rely on the fallback.  People will be prompted to upgrade, it will try to use the service and fail.  Then it will fallback to no service and no security check and pass on the next browser restart.   Since it is a fallback we should keep an eye out for it though.
As long as rs is good with that plan, please don't re-enable nightly builds on m-c until after this lands.
Thanks for the great summary and for working on the fix. If you'd like to verify the Aurora behaviour you can use the auroratest channel (while we have Aurora updates disabled).

One last query, will Thunderbird etc be OK with a hardcoded 13.0a1 ?
No problem, thanks to everyone for their support in getting this fixed.

Yes other products are fine. The security checks are turned on in browser/confvars.sh which is firefox only as far as I know, and inside there we check only windows and only not x64.

Where are auroratest Nightly builds? I don't see them here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/
Also can I push to it and is it a diff repo than aurora?
We do the Aurora nightlies from the aurora repo as usual (if there have been en-US or locale changes) but publish the updates onto auroratest until QA has signed off on them. Then they get copied onto the aurora channel.
Relating my previous message:

> I'll follow up with further notes for QA a bit later today.

Tests for mozilla-central Nightly builds
----------------------------------------

These tests should be run on the Nightly builds once the patch in this bug lands.

Test 1: Test upgrades from 13.0a1 to 14.0a1
- Download & Install the 13.0a1 Nightly build from 2012-03-13
- Verify it is indeed 13.0a1 by going to about: 
- Go to about:buildconfig and note the revision URL that you are on.
- Perform an update by going to Help | About
- Verify that you are now upgraded to 14.0a1 and that your about:buildonfig revision URL is different.

Test 2: Test upgrades from the 14.0a1 that landed on 2012-03-14
- Download & Install the 14.0a1 Nightly build from 2012-03-14
- Verify it is indeed 14.0a1 by going to about:
- Go to about:buildconfig and note the revision URL that you are on.
- Perform an update by going ot Help | About
- Verify that you are now upgraded to 14.0a1 and that your about:buildonfig revision URL is different.

Test 3:
- The day after you run Test 2, and once the next Nightly is available, verify that a Nightly update works as normal silently.

Tests 4..N:
- Whatever your normal tests would be.

Tests for Aurora Nightly builds
-------------------------------

This repository will not get the patch in this bug.
So these tests can be tested before the patch lands on mozilla-central.

Test 1:
- Download & Install a Nightly auroratest on 2012-03-13 or slightly before 2012-03-13 if its not available. 
- Go to about:buildconfig and note the revision URL that you are on.
- Do an update
- Once the update is complete, verify that you have not been upgraded (by checking the revision URL in about:buildconfig again)
- Close the browser, wait a second or two.
- Open the browser, it should prompt you with a UAC prompt to do an update.
- Accept and install.
- Verify that you are now on v13 and that no updates are available.


Test 2:
- Once Aurora Nightly updates are turned back on, please test the same thing again but this time using aurora builds.

Test 3:
- The day after you run Test 2, and once the next Nightly is available, verify that a Nightly update works as normal silently.

Tests 4..N:
- Whatever your normal tests would be.

Tests for Beta
--------------

Test 1..N: 
- I have no special requests, whatever your normal tests would be.
- This patch will eventually be reverted
- This patch will only land on m-c
- No patches are needed for aurora and beta
- See Comment 10 for why Fix1 isn't possible.
- See Comment 13 for further details on the new Fix2 that will work for everyone
Assignee: nobody → netzen
Attachment #605858 - Attachment is obsolete: true
Attachment #606063 - Flags: review?(robert.bugzilla)
Comment on attachment 606063 [details] [diff] [review]
Patch for Fix2 v2.

So, instead of hardcoding 13.0a1 what do you think about hardingcoding 14 (perhaps 15) so systems that have upgraded to 14 will continue to get updates even without this reverted?
duh... my head is a tad foggy
Group 1: People that are before 13.0a1 do not have these security checks.
Group 2: People that are on 13.0a1 can consume MARs of 13.0a1 or anything smaller than that.
Group 3: People that are on 14.0a1 can consume MARS of 14.0a1 or anything smaller than that.

This is only related to m-c and all users fall into one of the 3 above categories.
So if you put 14 or 15 then you would lock groups 2 and 3 out.
Correction on my last message:

> So if you put 14 or 15 then you would lock groups 2 and 3 out.

If you put 14 you lock out people from Group 2. 
If you put 15 you lock out people from Group 2 and Group 3.
(In reply to Nick Thomas [:nthomas] from comment #10)
> The way updates work for Nightly makes it problematic to point a subset of
> requests at a special mar. The update server keeps track of a list of builds
> for each locale, always serving the latest complete update and possibly an
> partial update if the  buildID in the request is right.
> 
> Is there a version we can bake into the mar at compile time that will
> satisfy all the code in the wild ?
Nick, what was done in the past to deal with one-offs like this where we have a different update snippet for specific builds?
I know you already know this but just to be extra explicit: 

Once all groups get an upgrade they will continue to get updates after that because the version check is no longer done.  We will keep this in for 1-2 weeks.  Then we will revert it and all of our Nightly users that upgrade within a couple weeks (which I assume is most of the Nightly users total) will be fixed.
Right and in the past we have had to revisit a couple of these bugs due to there being around as little as a thousand stragglers.
Understood. My Comment 25 was not meant to be a deterrent to Fix1 nor Comment 24. 

I just wanted to make sure my description was clear for Fix2 in case Fix1 doesn't work out.
(In reply to Robert Strong [:rstrong] (do not email) from comment #24)
> Nick, what was done in the past to deal with one-offs like this where we
> have a different update snippet for specific builds?

Perhaps you're thinking of bug 602275 ?
Attachment #606063 - Flags: review?(robert.bugzilla) → review+
If we keep this workaround in place for 1 week, from past data that nthomas sent about 6% of Windows x86 users won't be updated.

If we keep this workaround in place for 2 weeks almost everyone (looks to me like < 1%) of Windows x86 users will have to manually update.  So I recommend 2 weeks since there is no downside.

If someone is using an hourly build and manually downloaded Ehsan's fixed build, then the fix won't work but these users will know how to fix themselves for sure and the number of such users will be extremely small.
status-firefox13: I marked as wontfix because the problem exists there but the fallback will be used so we don't need to do anything there.

http://hg.mozilla.org/mozilla-central/rev/eb7d13ddeeaf
Status: NEW → RESOLVED
Closed: 12 years ago
status-firefox12: --- → unaffected
status-firefox13: --- → wontfix
status-firefox14: --- → fixed
Resolution: --- → FIXED
Thanks Brian for taking care of this.  Please file a bug to revert this?
Blocks: 735969
(In reply to Nick Thomas [:nthomas] from comment #28)
> (In reply to Robert Strong [:rstrong] (do not email) from comment #24)
> > Nick, what was done in the past to deal with one-offs like this where we
> > have a different update snippet for specific builds?
> 
> Perhaps you're thinking of bug 602275 ?
Yes though I think there are others as well. If there are a significant number of users that fall into that category I think it is possible to do a rewrite request and point them at a custom update.xml that only has a complete mar with the fix etc.
status-firefox12: unaffected → ---
status-firefox13: wontfix → ---
status-firefox14: fixed → ---
No problem, thanks for all your help Ehsan.

I posted these:
Bug 735969 - Revert the disable version downgrade check workaround from Bug 735784 
Bug 735970 - Add xpcshell tests for version upgrades and version downgrades inside updater code
(In reply to Brian R. Bondy [:bbondy] from comment #30)
> status-firefox13: I marked as wontfix because the problem exists there but
> the fallback will be used so we don't need to do anything there.
> 
> http://hg.mozilla.org/mozilla-central/rev/eb7d13ddeeaf

...

(In reply to Brian R. Bondy [:bbondy] from comment #25)
> I know you already know this but just to be extra explicit: 
> 
> Once all groups get an upgrade they will continue to get updates after that
> because the version check is no longer done.  We will keep this in for 1-2
> weeks.  Then we will revert it and all of our Nightly users that upgrade
> within a couple weeks (which I assume is most of the Nightly users total)
> will be fixed.

Have you considered how this will affect non-Firefox or non-Windows builds? They don't use the updater executable, or certainly not the signed mar version.
Comment on attachment 606063 [details] [diff] [review]
Patch for Fix2 v2.

>-    infoBlock.productVersion = productVersion;
>+    // Temporarily hardcoded - see Bug 735784
>+    infoBlock.productVersion = "13.0a1";//productVersion;

Is this applied to all MARs of any kind?
What effect does it have on SeaMonkey, which uses the same MAR code but different versioning (trunk is 2.11a1 there)? Of course, if this only affects signed MARs (I have no clue), then SeaMonkey dodges this right now due to not having signing infrastructure.
> Is this applied to all MARs of any kind?
Yes.

> What effect does it have on SeaMonkey?
None. Please see Comment 16.
(In reply to Brian R. Bondy [:bbondy] from comment #29)
> If we keep this workaround in place for 1 week, from past data that nthomas
> sent about 6% of Windows x86 users won't be updated.
> 
> If we keep this workaround in place for 2 weeks almost everyone (looks to me
> like < 1%) of Windows x86 users will have to manually update.  So I
> recommend 2 weeks since there is no downside.
> 
> If someone is using an hourly build and manually downloaded Ehsan's fixed
> build, then the fix won't work but these users will know how to fix
> themselves for sure and the number of such users will be extremely small.

So-called "hourly" builds aren't on the nightly channel anymore anyways, they don't get any updates.
Nightly updates can be turned on by the way.
> Nightly updates can be turned on by the way.

This was meant for mozilla-central.
Sorry for another message, but it should be turned on once today's nightly builds are available which I think is in an hour or so.
It seems the Nightly builds didn't include the fix from this patch so I'll start a new Nightly once the current one completes. The mozilla-central Windows x86 Nightlies that are currently disabled should be re-enabled after this.
The new Nightly will be spun based on: 
https://hg.mozilla.org/mozilla-central/rev/eb7d13ddeeaf
bhearsum temporarily turned on Nightly updates for me to test.
I tested both of these upgrade scenarios and they both passed.

The upgrades are being turned off for a couple of hours until the language repacks are done. 

Pass: 13.0a1 from 2 days ago -> 14.0a1 today:
2012-03-13-09-04-04-mozilla-central -> 2012-03-15-06-49-03-mozilla-central

Pass: 14.0a1 from yesterday -> 14.0a1 today
2012-03-14-03-11-39-mozilla-central -> 2012-03-15-06-49-03-mozilla-central

Someone started a Nightly build after ehsan's fix but before this workaround.  This was in the afternoon yesterday but updates were disabled at that point and I don't think many people would have gotten it.

Otherwise I think we're good. 
I would still like QA to verify the steps mentioned in Comment 18 though.
Members of QA, please use the builds mentioned in this message specifically since there were 2 builds on the 03-14 that you could choose from.
Verified on Win 7 32-bit using the steps from Comment 18 and the builds indicated in Comment 42. 

Both updates scenarios passed. 

Test 1: Test upgrades from 13.0a1 to 14.0a1
- Download & Install the 13.0a1 Nightly build from 2012-03-13
- Verify it is indeed 13.0a1 by going to about: 
- Go to about:buildconfig and note the revision URL that you are on.
- Perform an update by going to Help | About
- Verify that you are now upgraded to 14.0a1 and that your about:buildonfig revision URL is different.

Results test 1:
- The about:buildconfig URL after install is: http://hg.mozilla.org/mozilla-central/rev/8d1c74566a0b 
- The about:buidlconfig URL after the update is: http://hg.mozilla.org/mozilla-central/rev/082d016c341f
- The update was done silently from 13a1 (Build id: 20120313090404) to 14a1 (Build id:  20120315064903)

Test 2: Test upgrades from the 14.0a1 that landed on 2012-03-14
- Download & Install the 14.0a1 Nightly build from 2012-03-14
- Verify it is indeed 14.0a1 by going to about:
- Go to about:buildconfig and note the revision URL that you are on.
- Perform an update by going ot Help | About
- Verify that you are now upgraded to 14.0a1 and that your about:buildonfig revision URL is different.

Results test 2:
- The about:buildconfig Url after the install is: http://hg.mozilla.org/mozilla-central/rev/c71845b3b2a6
- The about:buidlconfig URL after the update is: http://hg.mozilla.org/mozilla-central/rev/082d016c341f
- The update was done silently from 14a1 (Build id: 20120314031139 ) to 14a1 (Build id: 20120315064903).

I'll verify on Win Vista and Win XP and post the results.
Verified also on Windows Vista, Windows XP and Windows Server 2003 - all the updates were made through the service.

Test 1 -> Pass: 13.0a1 March13 -> 14.0a1 the last available build.
2012-03-13-09-04-04-mozilla-central -> 2012-03-15-06-49-03-mozilla-central
or
2012-03-13-09-04-04-mozilla-central -> 2012-03-16-03-11-51-mozilla-central

Test 2 ->Pass: 14.0a1 from March14 -> 14.0a1 the last available build.
2012-03-14-03-11-39-mozilla-central -> 2012-03-15-06-49-03-mozilla-central
or
2012-03-14-03-11-39-mozilla-central ->  2012-03-16-03-11-51-mozilla-central.
-----------------------------------------------------------------------------
Verified also using 32-bit builds on Windows 7 x64, Windows Vista x64, Windows XP x64 and Windows Server 2003 x64 - all the updates were made through the service.

Test 1 -> Pass: 13.0a1 March13 -> 14.0a1 the last available build.
2012-03-13-09-04-04-mozilla-central -> 2012-03-15-06-49-03-mozilla-central
or
2012-03-13-09-04-04-mozilla-central -> 2012-03-16-03-11-51-mozilla-central

Test 2 -> Pass: 14.0a1 from March14 -> 14.0a1 the last available build.
2012-03-14-03-11-39-mozilla-central -> 2012-03-15-06-49-03-mozilla-central
or
2012-03-14-03-11-39-mozilla-central ->  2012-03-16-03-11-51-mozilla-central.
------------------------------------------------------------------------------
Verified also using 64-bit builds that on Windows Vista x64 and Windows 7 x64 -Nightly 13.0a1 properly updates to Nightly 14.0a1, and Nightly 14.0a1 updates to the last Nightly 14.0a1.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: