Closed Bug 998721 Opened 10 years ago Closed 6 years ago

Consider serving updates for 3.x and lower as minor updates to avoid displaying the billboard.

Categories

(Release Engineering :: Release Requests, defect, P3)

Product:
Release Engineering
Component:
Release Requests
Platform:
x86_64
Windows 8.1
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: robert.strong.bugs, Assigned: nthomas)

References

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2333] [fce-active-legacy]) 

There are still quite a few systems running 3.x and below. For example, on 2014-04-14 there were 100187 aus pings from Firefox 3.0.19 from Windows 5.1 systems. These versions of Firefox require and opt-in for the update as well as showing the billboard.

For the versions of Firefox that checked extension compatibility we could also lie about the version that Firefox is being updated to in order to avoid prompting the user for incompatible extensions.
What happens if the user has incompatible extensions?
The extensions will be disabled.
We can certainly modify the update snippets for these old releases. I've set up 
  http://people.mozilla.org/~nthomas/update-watch/release/older-releases.html
to illustrate the current situation.

You can see that we have a complicated multi-hop setup. Looks like 
* 3.6.13 to 3.6.28 get a minor update to 12.0, with extensionVersion set to same 3.6.x as request
* 3.6 through 3.6.12 get a minor update to 3.6.28, with  extensionVersion set to 3.6.28
! 3.5.1 to 3.5.18 get a minor update to 3.6.18, with extensionVersion of 3.6.18
! 3.0.19 gets a major update to 3.6.13, extVersion 3.6.13
* older 3.0.x get a minor update to 3.0.19

Is it the last two ! cases you're looking at here ?

Do we know how many users have operating systems we've deprecated along the way. My go to place for looking that up is
  http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/inc/config-dist.php#538
  (the unsupportedPlatforms array)
As a first step I'm looking at Windows only with Win2K and above having updates to Firefox 12 since that would get them on the latest required hop to release. This avoids the issue with deprecation for now and I'll check those numbers as I have time to.

So, Win2K and above using Firefox 2 (1st version of app update with OS info) through Firefox 3.x (e.g. everything prior to Firefox 4) would have an update advertised to Firefox 12 since that is the latest Firefox version they can be updated to since they need to be updated to a version that contains the app update directory removal code, service pack version code, etc. The extensionVersion for these versions should be the same as the version the user is using and the update type should be minor.

Would that be acceptable for you as a first step? We should be able to get an idea of how affective this will be from that by doing a percentage comparison before and after.
I had assumed we were modifying existing snippets, but we can probably generate new ones easily enough.

We can limit to just Windows, but not to particular OS versions. Which means we can change the offer to be 12.0 for those still supported, and no update for de-supported. Given the long time the existing paths have been available that doesn't seem like a big deal to me.

ashughes, when could QA fit in testing for this ? I presume this is lower priority than current releases and we fit it in when we can.
Flags: needinfo?(anthony.s.hughes)
(In reply to Nick Thomas [:nthomas] from comment #5)
> ashughes, when could QA fit in testing for this?

It's really hard for me to say given how many different responsibilities we have on our plate right now. We won't be able to test this with automation so I assume it's going to take at least a few days to test the various permutations of update paths. The best I can tell you is to stage something for us to test and we'll get it done as soon as we can.
Flags: needinfo?(anthony.s.hughes)
Nick, is there a chance of this happening sometime soonish?
Flags: needinfo?(nthomas)
Likely to happen mid-release cycle, at this point.
Flags: needinfo?(nthomas)
Distilled request:
* version: 2.0 through to 3.6.28, updated to 12.0
* platforms: windows
* locales: all  (implicit)
* channels: release
* in update.xml
 * type="minor"
 * extVersion="<app version>", eg extVersion="3.6.13" in the snippet for 3.6.13

RelEng notes:
* need to check if we have any symlinks in play on aus3.m.o
* scrape up details of old releases from hg & cvs history
* hack snippet generator to set extVersion correctly
* releasetest channel for testing (check identical to release)
* if we need to keep this going in Balrog we'll need a 'fakeExtVersion' in the v1 blob, and code in the xml
 creation to handle that.
Assignee: nobody → nthomas
Priority: -- → P3
Nick, any chance of doing this soonish?
Flags: needinfo?(nthomas)
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2330]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2330] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2333]
Sorry for the delay here, there always seems to be something more important than this sliver of really old ADI. It should be much easier in Balrog, which should get into production for the release channel early in 2015. I've filed bug 1113475 to do the small amount of coding to support faking the extVersion.

Notes to me/ben: there's a Firefox-12.0-build1 release in Balrog already, schema_version 2 with cturra in the fileUrl so probably from some testing. We'll need to double check it, and copy to a schema_version 1 blob with fakeExtVersion (or whatever we end up with from bug 1113475). Then an appropriate rule for testing and prod.
Flags: needinfo?(nthomas)
Though this is only for a sliver of ADI's once this is done it will inform us regarding how many users we can get to the latest with newer versions of Firefox. If it is significant then the same will be done with the newer versions to pick up a much greater piece of the ADI's.
When we enabled Balrog for the release channel (~36h ago) we meant to leave 3.6.13 and earlier pointing at aus3. Unfortunately, that exception was missed and we ended up serving those users Firefox 12.0 snippets like:
<updates>
  <update type="minor" displayVersion="12.0" appVersion="12.0" platformVersion="12.0" buildID="20120420145725" detailsURL="https://www.mozilla.com/en-US/firefox/12.0/releasenotes/" actions="silent">
    <patch type="complete" URL="http://download.mozilla.org/?product=firefox-12.0-complete&os=osx&lang=en-US" hashFunction="sha512" hashValue="d0011a4eb72f9c3230acc7c29042474daa46fd8aaabf847b8cd5c93887efb2a5abf3adfa4d03be641482efbbe75ef447d32a250239df83e50837fb81a19a8a6b" size="30700210"/>
  </update>
</updates>

I added a rule to block updates for those users completely, because it wasn't immediately clear if serving them 12.0 would be bad or not.
Continuing to serve them updates as they were previously isn't bad at all. This bug is about serving them the same mar file and disabling extension compatibility checking by serving that update as the same Firefox version in the xml.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #14)
> Continuing to serve them updates as they were previously isn't bad at all.
> This bug is about serving them the same mar file and disabling extension
> compatibility checking by serving that update as the same Firefox version in
> the xml.

They were being served 3.6.28 prior to that change, not 12.0.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #16)
> I just checked updates for Firefox 3.6.12 and there is no update advertised.
> 
> URL sent by Firefox
> https://aus2.mozilla.org/update/3/Firefox/3.6.12/20101026210630/WINNT_x86-
> msvc/en-US/release/Windows_NT%206.2/default/default/update.xml
> 
> End result URL
> https://aus3.mozilla.org/update/3/Firefox/3.6.12/20101026210630/WINNT_x86-
> msvc/en-US/release/Windows_NT%206.2/default/default/update.xml
> 
> No updates. :(

Yeah, that's expected at the moment. Prior to the switch to Balrog we were serving them 3.6.28. After the Balrog switchover they were getting 12.0 (xml like https://aus4.mozilla.org/update/3/Firefox/5.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml), and we weren't sure if that was a good thing to do or not. IIRC, Nick tested and found that this update XML didn't work because it served new-style versions instead of extv and appv.

If this bug is still correct, the right thing to do is to serve them 12.0 with extv and appv faked out to something that prevents the compatibility check from firing. Is that right, Robert? If so, I assume we can set the version to 3.6.anything (rather than the version matching the minor version of the incoming request).
Flags: needinfo?(bhearsum) → needinfo?(robert.strong.bugs)
(In reply to Ben Hearsum [:bhearsum] from comment #17)
> (In reply to Robert Strong [:rstrong] (use needinfo to contact me) from
> comment #16)
> > I just checked updates for Firefox 3.6.12 and there is no update advertised.
> > 
> > URL sent by Firefox
> > https://aus2.mozilla.org/update/3/Firefox/3.6.12/20101026210630/WINNT_x86-
> > msvc/en-US/release/Windows_NT%206.2/default/default/update.xml
> > 
> > End result URL
> > https://aus3.mozilla.org/update/3/Firefox/3.6.12/20101026210630/WINNT_x86-
> > msvc/en-US/release/Windows_NT%206.2/default/default/update.xml
> > 
> > No updates. :(
> 
> Yeah, that's expected at the moment. Prior to the switch to Balrog we were
> serving them 3.6.28. After the Balrog switchover they were getting 12.0 (xml
> like
> https://aus4.mozilla.org/update/3/Firefox/5.0/20141202185629/Darwin_x86_64-
> gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml), and we
> weren't sure if that was a good thing to do or not. IIRC, Nick tested and
> found that this update XML didn't work because it served new-style versions
> instead of extv and appv.
> 
> If this bug is still correct, the right thing to do is to serve them 12.0
> with extv and appv faked out to something that prevents the compatibility
> check from firing. Is that right, Robert? If so, I assume we can set the
> version to 3.6.anything (rather than the version matching the minor version
> of the incoming request).
In Firefox 3.x (2.x too I believe) it always checks the entire value of extensionVersion to verify that it is not trying to update to an earlier version of the app and it also extensionVersion to check add-on compatibility if it is different than the current app version so the extensionVersion needs to match the client's app version making the request and the update type should be minor to avoid the billboard.
Flags: needinfo?(robert.strong.bugs)
bug 1113475 has a patch now. Once that's landed I think we just have to do this:

(In reply to Nick Thomas [:nthomas] from comment #11)
> Notes to me/ben: there's a Firefox-12.0-build1 release in Balrog already,
> schema_version 2 with cturra in the fileUrl so probably from some testing.
> We'll need to double check it, and copy to a schema_version 1 blob with
> fakeExtVersion (or whatever we end up with from bug 1113475). Then an
> appropriate rule for testing and prod.

I think we can set the rule to version<4.0. We may need a higher priority rule to block 1.5.0.x from getting updates depending on what's going on in bug 1026827.
Depends on: 1113475
Hi Nick, any idea when this can be completed? After this is done we are going to look at the affect to see if it would be helpful for the Firefox 4 and above case. Thanks!
Flags: needinfo?(nthomas)
Rob, I'll try to squeeze it in this week.

bhearsum, could you run your eye over this rule change for me. We have this right now
 priority   version   build target    mapping
  100                                 No-Update   (for deprecated OS)
   95       <4.0                      No-Update   (this was <3.6.13 until yesterday, see bug 1139249)
   93       <10.0                     Firefox-12.0-build1  (ReleaseBlobV2, displayVersion etc)
   91       >=36.0                    Firefox-36.0.1-build2-no-whatsnew
   90                                 Firefox-36.0.1-build2

And this is a set of rules I think should work, with changes marked as !
 priority  version    build target    mapping
  100                                 No-Update   (for deprecated OS)
!  97         <2.0                    No-Update
!  96         <4.0    WINNT_x86-msvc  Firefox-12.0-build1-schema1  (ReleaseBlobV1, appv etc, fake extv)
!  95         <4.0                    No-Update     (for mac & linux)
   93         <10.0                   Firefox-12.0-build1  (ReleaseBlobV2, displayVersion etc)
   91         >=36.0                  Firefox-36.0.1-build2-no-whatsnew
   90                                 Firefox-36.0.1-build2
Flags: needinfo?(nthomas) → needinfo?(bhearsum)
http://people.mozilla.org/~nthomas/IMG_20150310_231743.JPG is a sketch of a Venn diagram of that, leaving out the OS deprecation. I guess that wraps around everything - I'm just playing around with a way to visualize & reason about complicated rules.
(In reply to Nick Thomas [:nthomas] from comment #21)
> bhearsum, could you run your eye over this rule change for me. We have this
> right now
>  priority   version   build target    mapping
>   100                                 No-Update   (for deprecated OS)
>    95       <4.0                      No-Update   (this was <3.6.13 until yesterday, see bug 1139249)
>    93       <10.0                     Firefox-12.0-build1  (ReleaseBlobV2, displayVersion etc)
>    91       >=36.0                    Firefox-36.0.1-build2-no-whatsnew
>    90                                 Firefox-36.0.1-build2
> 
> And this is a set of rules I think should work, with changes marked as !
>  priority  version    build target    mapping
>   100                                 No-Update   (for deprecated OS)
> !  97         <2.0                    No-Update
> !  96         <4.0    WINNT_x86-msvc  Firefox-12.0-build1-schema1 (ReleaseBlobV1, appv etc, fake extv)
> !  95         <4.0                    No-Update     (for mac & linux)
>    93         <10.0                   Firefox-12.0-build1  (ReleaseBlobV2, displayVersion etc)
>    91         >=36.0                  Firefox-36.0.1-build2-no-whatsnew
>    90                                 Firefox-36.0.1-build2

I'm not quite sure why we're only targeting Windows here, but taking that as a given, this all looks fine to me. It won't have any effect until bug 1138443 is fixed, though.
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum [:bhearsum] from comment #23)
>...
> I'm not quite sure why we're only targeting Windows here, but taking that as
> a given, this all looks fine to me. It won't have any effect until bug
> 1138443 is fixed, though.
Targeting all platforms is fine. I went with Windows only since Windows requires updating to 12.
No longer blocks: diesnippets
Sorry for the delay here, I'm running into an end-of-quarter time crunch.

Sticking to windows actually avoids extra complications with the build targets changing for mac, https://wiki.mozilla.org/User:NThomas:Mac_BUILDTARGET
I'm fine with that and thanks!
Ping?
Flags: needinfo?(nthomas)
Depends on: 1172372
The following is mainly aimed at bhearsum & RelEng:

I think I figured out a way to include all platforms, with rules like this (for testing on release-cdntest):

+----------+--------------------------------+---------+------------------------------+
| priority | mapping                        | version | channel                      | Other restrictions
+----------+--------------------------------+---------+------------------------------+
|      150 | No-Update                      | NULL    | release*                     | Desupported OS
|      140 | No-Update                      | <2.0    | release*                     |
|      122 | No-Update                      | <4.0    | release*                     | OSX PPC
|      120 | Firefox-12.0-build1-schema1    | <4.0    | release-cdntest              |
|      116 | No-Update                      | <4.0    | release                      |    
|      110 | Firefox-12.0-build1            | <10.0   | release*                     |
|       96 | Firefox-38.0.6-build1-whatsnew | <38.0.5 | release-cdntest-cck-mozilla* | Pocket locales
|       95 | Firefox-38.0.6-build1          | NULL    | release-cdntest-cck-mozilla* |
|       92 | Firefox-38.0.5-build4-whatsnew | NULL    | release-cdntest              | Pocket locales
|       90 | Firefox-38.0.5-build4          | NULL    | release-cdntest              |
+----------+--------------------------------+---------+------------------------------+

* if you have a deprecated OS (eg Darwin 8, Win2K), Firefox older than 2.0, or older than 4.0 on PPC, then you get nothing
* for older than 4.0
 * on release-cdntest you get the special 12.0 with faked extVersion
 * on release you get nothing until we've finished testing  
* otherwise less than 10.0 gets 12.0, and then we're into the modern rules (much worse than it usually is!).

Firefox-12.0-build1-schema1 is Firefox-12.0-build1 with the appropriate renaming, cleaned up fileUrls, and adding aliases for Darwin_Universal-gcc3 and Darwin_x86-gcc3-u-ppc-i386. setExtvToIncomingVersion and fakePartials are set, but the latter doesn't affect xml (only snippets).

To move this into production, we'll modify the priority 120 rule to be for 'release*', and delete the rule with priority 116.
Some quick tests on the release-cdntest channel.

1, 3.6.28 on a modern mac:

Query (captured in-app) is:
https://aus2.mozilla.org/update/3/Firefox/3.6.28/20120306064154/Darwin_x86-gcc3-u-ppc-i386/en-US/release-cdntest/Darwin%2014.3.0/default/default/update.xml returns
<?xml version="1.0"?>
<updates>
    <update type="minor" version="12.0" extensionVersion="3.6.28" buildID="20120420145725" detailsURL="https://www.mozilla.com/en-US/firefox/12.0/releasenotes/">
        <patch type="complete" URL="http://download.mozilla.org/?product=firefox-12.0-complete&amp;os=osx&amp;lang=en-US" hashFunction="sha512" hashValue="d0011a4eb72f9c3230acc7c29042474daa46fd8aaabf847b8cd5c93887efb2a5abf3adfa4d03be641482efbbe75ef447d32a250239df83e50837fb81a19a8a6b" size="30700210"/>
    </update>
</updates>

NB extensionVersion is faked properly. Can successfully update to 12.0 and on to 38.0.5. No prompting for incompatible extensions, but I didn't specifically set any up.

2, 3.5.17 on Win XP
Can successfully update to 12.0 and on to 38.0.5, although I couldn't catch the update.xml in the first step.

3, Verify PPC is blocked (by faking the useragent, we're just testing for 'PPC' being present):
$ curl -L -A 'foo PPC bar' https://aus2.mozilla.org/update/3/Firefox/3.6.28/20120306064154/Darwin_ppc-gcc3-u-ppc-i386/en-US/release-cdntest/Darwin%2014.3.0/default/default/update.xml
<?xml version="1.0"?>
<updates>

4, 3.6.10 on mac
No updates found, nothing logged to browser console, or launching with -console. Any ideas ?  I specifically chose this because it should be querying with Darwin_Universal-gcc3.


One question, do we need to fake partial updates by copying the <patch type="complete" ... ? I recall that being an issue with some older versions where the updater gets stuck if there is not a partial and a complete.
Flags: needinfo?(nthomas) → needinfo?(robert.strong.bugs)
(In reply to Nick Thomas [:nthomas] from comment #30)
> 4, 3.6.10 on mac
> No updates found, nothing logged to browser console, or launching with
> -console. Any ideas ?  I specifically chose this because it should be
> querying with Darwin_Universal-gcc3.

Found the app.update.log.Checker pref, which yielded a query of
https://aus2.mozilla.org/update/3/Firefox/3.5.10/20100504085753/Darwin_Universal-gcc3/en-US/release-cdntest/Darwin%2014.3.0/default/default/update.xml?force=1

Which let me trace an error in the server config, a trailing space on 'Darwin_Universal-gcc3 '. I get 12.0 then 38.0.5 now.
Let's go with a fake partial alongside the complete both pointing at the complete.

I checked a few version and came up with the following to prevent the billboard as well as skip the add-ons compatibility check:

2.0.0.20 and 3.0 - the version equal must the existing app version and the extensionVersion should be omitted.

3.5 - the version can equal 12 and the extensionVersion should be omtitted.

3.6 and above - the version can equal 12 and the extensionVersion should equal the app version that is checking for updates.
Flags: needinfo?(robert.strong.bugs)
We'll need some more update server changes, I'll file a separate bug for that. Is that 2.0.0.20 specifically, or does it apply to all of 2.0.0.x ?
Depends on: 1174605
I'm away for a few weeks. Once bug 1174605 is deployed we can update the Firefox-12.0-build1-schema1 release in prod (s/setExtvToIncomingVersion/oldVersionSpecialCases) and retest.
Assignee: nthomas → nobody
(In reply to Nick Thomas [:nthomas] (away Dec 21-Jan 24) from comment #34)
> I'm away for a few weeks. Once bug 1174605 is deployed we can update the
> Firefox-12.0-build1-schema1 release in prod
> (s/setExtvToIncomingVersion/oldVersionSpecialCases) and retest.

I updated this release as noted above, don't have time to do any testing myself though.
I changed the rule priority to 10 in balrog on release-cdntest
I've set up release-cdntest again. While kicking the tires I noticed:
* we're not faking partial updates, Balrog needs to learn this trick if it's needed
* Firefox 2.0 up to 2.0.0.4 fail because of the redirect from aus2.m.o to aus4.m.o, and the latter uses an intermediate CA which is unknown to the browser
* Firefox 2.0.0.5 to 2.0.0.7 allow the CA, download the complete mar but complain the "integrity of the update could not be verified". The console has "onStopRequest: http://download.mozilla.org/?product=firefox-12.0-complete&os=win&lang=en-US&force=1, status = 2147549183"
* Firefox 2.0.0.8, 2.0.0.10, 20.0.0.20, and 3.0.19 all work OK. On WinXP I updated 3.0.19 -> 12.0 -> 43.0.1 -> 47.0.2, and verified the three 2.0.0.x mentioned get to 12.0. Didn't check 3.5.x or 3.6.x, covered in comment #30

--> We should adjust our Balrog rule which blocks updates for 'version < 2.0' to 'version < 2.0.0.8', or 2.0.0.5 if we can figure out the integrity issue.

--> We could ask SoftVision to do some testing, but would need to define a test plan for them.

Example urls:
https://aus2.mozilla.org/update/2/Firefox/2.0/2006101023/WINNT_x86-msvc/en-US/release-cdntest/Windows_NT%205.1/update.xml
https://aus2.mozilla.org/update/3/Firefox/3.0.19/2010031422/WINNT_x86-msvc/en-US/release/Windows_NT%205.1/default/default/update.xml?force=1
https://aus2.mozilla.org/update/3/Firefox/3.5.10/20100504085753/Darwin_Universal-gcc3/en-US/release-cdntest/Darwin%2014.3.0/default/default/update.xml?force=1
https://aus4.mozilla.org/update/3/Firefox/3.6.12/20101026210630/WINNT_x86-msvc/en-US/release-cdntest/Windows_NT%206.2/default/default/update.xml
https://aus4.mozilla.org/update/3/Firefox/3.6.28/20120306064154/Darwin_x86-gcc3-u-ppc-i386/en-US/release-cdntest/Darwin%2014.3.0/default/default/update.xml

Logging notes:
Set the following prefs to True - app.update.log.all, app.update.log.Checker, app.update.log - and restart Firefox.
Assignee: nobody → nthomas
Flags: needinfo?(robert.strong.bugs)
Other work has been keeping me busy and I'll be evaluating our options in the near future.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2333] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2333] [fce-active]
Bulk change of QA Contact to :jlund, per https://bugzilla.mozilla.org/show_bug.cgi?id=1428483
QA Contact: rail → jlund
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2333] [fce-active] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2333] [fce-active-legacy]
rstrong, last call for this bug ?
We've decided to WONTFIX this given the age of 2.0/3.5/3.6.
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(robert.strong.bugs)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.