602275 - serve custom snippet to broken windows users on trunk

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

The Windows users on buildids 20101002041357 and 20101003041324 on mozilla-central are unable to update. We'd like to serve them a custom update snippet which will force a dialog to be shown that will give them directions on how to fix themselves manually.

We need to do this without serving this snippet to unaffected users.

Not sure how to do this, since the nightly update channels force people to the latest nightly update.

Comment 1

[Mike Beltzner [:beltzner, not reading bugmail]]

This is pretty high-priority work; we're missing about half of the nightly user base because of this issue.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> The Windows users on buildids 20101002041357 and 20101003041324 on
> mozilla-central are unable to update. We'd like to serve them a custom update
> snippet which will force a dialog to be shown that will give them directions on
> how to fix themselves manually.
> 
> We need to do this without serving this snippet to unaffected users.
> 
> Not sure how to do this, since the nightly update channels force people to the
> latest nightly update.

I am looking at the code to see if there's a way to override (maybe Mike knows already).

Worst-case scenario, could we build the update.xml files by hand and circumvent AUS, using an Apache rewrite (for example)?

Comment 3

[Robert Helmer [:rhelmer]]

(In reply to comment #0)
> The Windows users on buildids 20101002041357 and 20101003041324 on
> mozilla-central are unable to update. We'd like to serve them a custom update
> snippet which will force a dialog to be shown that will give them directions on
> how to fix themselves manually.

I think the URLs the clients are trying should be something like:

https://aus2.mozilla.org/update/1/Firefox/4.0b7pre/20101002041357/WINNT_x86-msvc/en-US/nightly/update.xml

https://aus2.mozilla.org/update/1/Firefox/4.0b7pre/20101003041324/WINNT_x86-msvc/en-US/nightly/update.xml

Can we confirm exactly what these are? Preferably with the actual clients.

Comment 4

[Robert Strong (they/them - no direct email)]

I created an update snippet with the new parameters to use as a template to accomplish this.
http://exchangecode.com/robert/work/snippets/please_reinstall.xml

as well as a template for the billboard with the new dimensions and the billboard attribute
http://exchangecode.com/robert/work/snippets/please_reinstall.html

I'll get the urls for the affected builds.

Comment 5

[Robert Strong (they/them - no direct email)]

Using http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ for the builds the following builds are affected. All urls have the force parameter since I manually checked. Though I didn't replace the non-l10n builds locale param with <all locales> it is probably best to do all locales for them as well.

firefox-4.0b7pre.en-US.win64-x86_64 2010-10-02-01
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-02-01-mozilla-central/
https://aus3.mozilla.org/update/3/Firefox/4.0b7pre/20101002015448/WINNT_x86_64-msvc/en-US/nightly/<all windows versions>/default/default/update.xml?force=1

firefox-4.0b7pre.en-US.win32 2010-10-02-04
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-02-04-mozilla-central/
and
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-02-04-mozilla-central-l10n/
https://aus3.mozilla.org/update/3/Firefox/4.0b7pre/20101002041357/WINNT_x86-msvc/<all locales>/nightly/<all windows versions>/default/default/update.xml?force=1

firefox-4.0b7pre.en-US.win64-x86_64 2010-10-03-03
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-03-03-mozilla-central/
https://aus3.mozilla.org/update/3/Firefox/4.0b7pre/20101003030544/WINNT_x86_64-msvc/en-US/nightly/<all windows versions>/default/default/update.xml?force=1

firefox-4.0b7pre.en-US.win32 2010-10-03-04
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-03-04-mozilla-central/
and
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010-10-03-04-mozilla-central-l10n/
https://aus3.mozilla.org/update/3/Firefox/4.0b7pre/20101003041324/WINNT_x86-msvc/<all locales>/nightly/<all windows versions>/default/default/update.xml?force=1

Comment 6

[Robert Helmer [:rhelmer]]

Thanks for getting all that info together!

On the AUS server side, as far as I can tell we can't easily override individual nightly snippets with AUS configuration alone, this would be a new feature we'd have to add. Also comment 4 uses parameters that AUS does not support yet (bug 459972), billboardURL and showPrompt.

I think the safest and most expedient way forward would be to rewrite requests looking for these patterns (UNTESTED) to a single update.xml based on the "please_reinstall.xml" in comment 4:

^/update/3/Firefox/4.0b7pre/20101002015448/WINNT_x86_64-msvc/en-US/nightly/(.*)/default/default/update.xml$
^/update/3/Firefox/4.0b7pre/20101002041357/WINNT_x86-msvc/(.*)/nightly/(.*)/default/default/update.xml$
^/update/3/Firefox/4.0b7pre/20101003030544/WINNT_x86_64-msvc/en-US/nightly/(.*)/default/default/update.xml$
^/update/3/Firefox/4.0b7pre/20101003041324/WINNT_x86-msvc/(.*)/nightly/(.*)/default/default/update.xml$
To make it a little more readable, we could probably safely match any URLs that begin with (also UNTESTED):

/update/3/Firefox/4.0b7pre/20101002015448/WINNT_x86_64-msvc/en-US/nightly/
/update/3/Firefox/4.0b7pre/20101002041357/WINNT_x86-msvc/
/update/3/Firefox/4.0b7pre/20101003030544/WINNT_x86_64-msvc/en-US/nightly/
/update/3/Firefox/4.0b7pre/20101003041324/WINNT_x86-msvc/

Let's get some advice from IT, if no one objects or has alternative suggestions.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

The only other way I know to do this is a variant of 
  http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88
That would have the advantage of keeping all the custom changes in one place for transparency, at the cost of being more intrusive. Possibly affects downstream users too.

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to comment #7)
> The only other way I know to do this is a variant of 
>   http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88
> That would have the advantage of keeping all the custom changes in one place
> for transparency, at the cost of being more intrusive. Possibly affects
> downstream users too.

True, although we would have to fix bug 459972 also, since we need to return a valid update.xml in this case (as opposed to the easy-out that example gets from printing the empty update.xml snippet)

In the case of this bug, we *could* just have the snippet inline in the source, and print it out:

if (bad) {
  header('Content-type: text/xml;');
  print <<< inlineHack
<?xml version="1.0" encoding="UTF-8"?>
<updates>
 <update type="minor" version="4.0b7pre" extensionVersion="4.0b7pre" buildID="20101003" detailsURL="http://exchangecode.com/robert/work/snippets/details.html" billboardURL="http://exchangecode.com/robert/work/snippets/please_reinstall.html" showPrompt="true">
  <patch type="complete" URL="http://exchangecode.com/robert/work/snippets/empty.mar" hashFunction="MD5" hashValue="6232cd43a1c77e30191c53a329a3f99d" size="775"/>
 </update>
</updates>
inlineHack
  exit;
}

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to comment #8)
> (In reply to comment #7)
> > The only other way I know to do this is a variant of 
> >   http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88
> > That would have the advantage of keeping all the custom changes in one place
> > for transparency, at the cost of being more intrusive. Possibly affects
> > downstream users too.
> 
> True, although we would have to fix bug 459972 also, since we need to return a
> valid update.xml in this case (as opposed to the easy-out that example gets
> from printing the empty update.xml snippet)
> 
> In the case of this bug, we *could* just have the snippet inline in the source,
> and print it out:

Just to be clear, we wouldn't really link to exchangecode.com of course; we could also of course respond with an HTTP redirect, or read a file from disk and write it to the network instead of having the XML inline in the source like that... my main point is that involving all of the {partial,complete}.txt -> XML machinery probably isn't going to gain anything.

Doing this in AUS is pretty much equivalent to doing it in Apache via a rewrite, so I am fine with either way. Having the update.xml be inline in the source versus on the disk is fine with me, whatever releng people feel is more palatable/readable would be my choice. 

I think there is a definite need to be able to override individual update paths with custom update.xml responses, so perhaps we should look into making hacks like http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/index.php#88 something that can go into the config file, instead. Would be happy to follow that up as a separate bug, to not hold this one up though.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to comment #9)
> Doing this in AUS is pretty much equivalent to doing it in Apache via a
> rewrite, so I am fine with either way. Having the update.xml be inline in the
> source versus on the disk is fine with me, whatever releng people feel is more
> palatable/readable would be my choice. 

Considering that this is a one off and fairly temporary I'm in favour of whatever you think is better.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

In comment #7 I was just throwing another idea out, but it's clear that the disadvantages outweigh the advantage. Redirect is fine by me too, especially if we can do it in CVS like webtools/aus/xml/htaccess.dist does. Assuming that works for IT of course.

Comment 12

[Robert Helmer [:rhelmer]]

(In reply to comment #11)
> In comment #7 I was just throwing another idea out, but it's clear that the
> disadvantages outweigh the advantage. Redirect is fine by me too, especially if
> we can do it in CVS like webtools/aus/xml/htaccess.dist does. Assuming that
> works for IT of course.

We already use rewrites here so it would be a good place to do it (we know Apache is set up to let us override etc):
http://mxr.mozilla.org/mozilla/source/webtools/aus/xml/htaccess.dist

Let's confirm with IT that they use this and not Apache config, the RewriteBase in there is obviously wrong and we don't usually ask them to merge this file, so we'll need to take extra care here.

We do need a place to host the update.xml, either FTP or somewhere in the docroot on the AUS servers would be fine (could be checked into CVS for example). I like the idea of checking it into CVS, any opinions on that?

Comment 13

[Robert Helmer [:rhelmer]]

(In reply to comment #11)
> In comment #7 I was just throwing another idea out, but it's clear that the
> disadvantages outweigh the advantage. Redirect is fine by me too, especially if
> we can do it in CVS like webtools/aus/xml/htaccess.dist does. Assuming that
> works for IT of course.

.htaccess is not used in production, and overrides are not allowed. The rewrites needed are in the httpd.conf instead.

After chatting with oremj, I think we should leave AUS code alone, and test the rewrite rules in staging and roll them out to production after testing (httpd.conf is puppet-managed). 

Exactly where the update.xml should live is still an open question, I think we should probably name it something other than update.xml ("./bug602275/please_reinstall.xml") roll it out to the docroot via puppet, or put it in AUS CVS.

The worst case scenario I can see with this plan (if we made a mistake in the rules) is that unintended users would see the "please reinstall" message. The update is empty so should not actually affect their install. I think this is very unlikely though, since we'll be testing it first and then rolling it out in an automated way.

Comment 14

[Mike Beltzner [:beltzner, not reading bugmail]]

When you need help with the strings for the billboard, ping me.

Comment 15

[Bruno 'Aqualon' Escherl]

(In reply to comment #1)
> This is pretty high-priority work; we're missing about half of the nightly user
> base because of this issue.
Just as a thought from someone who once learned, that using nightlies is at your own risk and may break at any time. Do you really think that people who can't manage to manually update a broken nightly without given a hint by the update mechanism are suitable nightly testers? If that happened for a release version I could understand your efforts here, but I'm not sure it's the correct thing to do for nightlies, which may have much bigger problems than just a broken update mechanism.

Comment 16

[Mike Beltzner [:beltzner, not reading bugmail]]

(In reply to comment #15)
> thing to do for nightlies, which may have much bigger problems than just a
> broken update mechanism.

Nightly testers have - rightfully - gotten used to their browser updating them to the latest nightly. We should take steps to inform them that this isn't happening, and that they need to take a manual action to rectify.

If this had happened to release channel users, I'd be mortified and the insist on a single update that would apply and fix the issue. That I'm happy to use the MU billboard to present a URL for update is really a concession.

Stats indicate that we've still got about 40% of the Windows nightly audience stuck on the old versions. I don't think that's an implication that our testers are ineffective, just that they depend on this functionality working.

Comment 17

[Robert Helmer [:rhelmer]]

Filed bug 602692 (blocking this one) to get the rewrite rules from comment 6 onto aus2-staging for testing.

Comment 18

[Robert Helmer [:rhelmer]]

We've loaded Rob Strong's test update.xml (with everything pointing to his examples on exchangecode.com) onto aus2-staging, so we can start testing that the rewrite rules are doing the right thing. Anything in comment #5 should be getting rewritten, and nothing else. Here is an example (note that AUS accepts anything into some of the parts of the path, like "all_windows_versions"):

http://aus2-staging.mozilla.org/update/3/Firefox/4.0b7pre/20101002015448/WINNT_x86_64-msvc/en-US/nightly/all_windows_versions/default/default/update.xml?force=1

Things left to do:

1) decide where detailsURL and billboardURL will point to
2) host empty.mar somewhere on FTP
3) check in CVS:mozilla/webtools/aus/xml/bug602275/please_reinstall.xml (need results of #1 and #2 for this)
4) push AUS CVS to aus2-staging and test
5) push AUS CVS and Apache rewrite rules to production and test

(In reply to comment #14)
> When you need help with the strings for the billboard, ping me.

This can happen in parallel to the above, so whenever you are ready.

I am signing off for the day, Nick is going to push this forward in the meantime. Thanks Nick!

Comment 19

[Robert Helmer [:rhelmer]]

(In reply to comment #18)
> http://aus2-staging.mozilla.org/update/3/Firefox/4.0b7pre/20101002015448/WINNT_x86_64-msvc/en-US/nightly/all_windows_versions/default/default/update.xml?force=1

Apparently you need to be on the VPN or you get an outage page... Maybe we can get that opened up a bit?

Comment 20

[Robert Strong (they/them - no direct email)]

Since it will still be possible to download the update and then try to apply it I think we should go with an invalid hash on the update so the user is immediately notified of the error if they didn't read / chose to ignore the billboard. If this isn't done they will see the error after restarting instead.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

1,2) created
 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/bug602275/
owned by cltbld, so that it doesn't get cleaned up in 30 days. Pulled rs's mar and two html files down there.

3) Modified rhelmer's snippet and landed it
Checking in please_reinstall.xml;
/cvsroot/mozilla/webtools/aus/xml/bug602275/please_reinstall.xml,v  <--  please_reinstall.xml
initial revision: 1.1
done
This has a bogus hash so that we don't try to install the empty mar (and fail).

4) aus2-staging is updated to tip in CVS, which is currently AUS2_PRODUCTION + xml/bug602275/please_reinstall.xml. rhelmer's file moved to /opt/aus2/app/bug602275.bak

Ready for a wordsmith, so I'm assigning this to beltzner. After that it can go back to someone like bhearsum to publish on ftp, then get some testing to complete 4).

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #21)
> 1,2) created
>  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/bug602275/
> owned by cltbld, so that it doesn't get cleaned up in 30 days. Pulled rs's mar
> and two html files down there.

Actually, cltbld ownership will see it cleaned up in 90 days. If people haven't updated by then it's their own problem.

Comment 23

[Robert Strong (they/them - no direct email)]

I verified that if the billboard is ignored and the update with the invalid hash is downloaded the error message with the link to ttp://www.mozilla.org/products/firefox/ will be shown immediately (bug 601830 is to make the link specific to the build e.g. nightly, release, and perhaps beta). The experience with a valid hash would be the download would complete and the error apply update failure message would be displayed on the next startup. Also, an update check can be performed again immediately following this message and clicking "Apply..." in the about window will display the billboard. I've verified that the "No Thanks" button to ignore the update is not available and that after a background check the Billboard is displayed after the idle timer timeout which is 60 seconds.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

All but please_reinstall.html had been removed from the ftp dir. Redid copying the files down to ftp.m.o, with updated file modifications times to prevent find removing them again.

Comment 25

[Mike Beltzner [:beltzner, not reading bugmail]]

Working on a new billboard HTML file, should have it in 15 minutes.

Comment 26

[Mike Beltzner [:beltzner, not reading bugmail]]

Attachment: Click to update Firefox manually billboard

Let's use this for the billboard

Comment 27

[Mike Beltzner [:beltzner, not reading bugmail]]

Attachment: Click here to update Minefield

Put the word "here" back in to clarify where the person should click.

Comment 28

[[e-mail] (:bhearsum)]

(In reply to comment #27)
> Created attachment 482545 [details]
> Click here to update Minefield
> 
> Put the word "here" back in to clarify where the person should click.

Updated http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/bug602275/please_reinstall.html to this.

Comment 29

[[e-mail] (:bhearsum)]

I did a final test on aus2-staging, and filed bug 603669 to get production updated with the same rewrite rules.

Comment 30

[[e-mail] (:bhearsum)]

AUS rules are updated in production, and I manually tested with the 20101002041357 build, which popped the billboard when checking for updates. I believe we're all done here.

Comment 31

[Mike Beltzner [:beltzner, not reading bugmail]]

(In reply to comment #30)
> AUS rules are updated in production, and I manually tested with the
> 20101002041357 build, which popped the billboard when checking for updates. I
> believe we're all done here.

Thanks, Ben. Rob will keep an eye on the ADUs and let us know if we need further action or if this gets most people to move over.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Made a slight tweak, s/more recently nightly/more recent nightly/. Nice work everyone.