1108898 - Provide ability to download latest stub installer from an old stub installer

Status: NEW

(Firefox :: Installer, enhancement, P5)

Description

[Robert Strong (they/them - no direct email)]

I’m trying to track down a sudden increase in stub installer download retries failing due to reaching the maximum number of retries that started on 12/1 UTC. I checked for changes in the stub for Firefox 34 and all changes were landed for Firefox 33 so I don’t think it is due to a change in the stub installer. 

The following is only for the release channel.

Date UTC  Too Many Retries
11/13/14    1.67%
11/14/14    1.69%
11/15/14    1.76%
11/16/14    1.76%
11/17/14    1.71%
11/18/14    1.66%
11/19/14    1.63%
11/20/14    1.64%
11/21/14    1.71%
11/22/14    1.72%
11/23/14    1.75%
11/24/14    1.57%
11/25/14    1.57%
11/26/14    1.65%
11/27/14    1.64%
11/28/14    1.64%
11/29/14    1.69%
11/30/14    1.70%
12/01/14    2.50%
12/02/14    4.17%
12/03/14    4.11%
12/04/14    4.05%
12/05/14    3.97%
12/06/14    4.00%
12/07/14    3.87%

I’m going to continue to analyze the data to look for clues (e.g. locales, regions, Windows version, etc.).

Comment 1

[Robert Strong (they/them - no direct email)]

Whatever caused this, it has also increased cancellations

Date UTC   Cancel
11/13/14   8.72%
11/14/14   8.62%
11/15/14   8.80%
11/16/14   8.54%
11/17/14   8.52%
11/18/14   8.76%
11/19/14   8.83%
11/20/14   8.92%
11/21/14   8.91%
11/22/14   8.98%
11/23/14   8.73%
11/24/14   8.83%
11/25/14   8.91%
11/26/14   9.04%
11/27/14   8.90%
11/28/14   8.78%
11/29/14   8.68%
11/30/14   8.56%
12/01/14   9.61%
12/02/14   11.98%
12/03/14   11.56%
12/04/14   11.60%
12/05/14   11.62%
12/06/14   11.55%
12/07/14   11.09%

Comment 2

[Robert Strong (they/them - no direct email)]

It looks like this happened on the Beta channel and moved to the Release channel with the merge so it appears to be code related.

Date UTC    Cancel   Too Many Retries
08/01/14    14.36%    2.69%
08/02/14    12.60%    2.53%
08/03/14    15.16%    2.70%
08/04/14    14.18%    2.80%
08/05/14    12.95%    2.62%
08/06/14    13.86%    2.79%
08/07/14    13.08%    2.36%
08/08/14    12.94%    1.98%
08/09/14    13.52%    2.42%
08/10/14    13.66%    2.17%
08/11/14    13.55%    2.48%
08/12/14    13.91%    2.63%
08/13/14    14.93%    2.40%
08/14/14    14.20%    2.73%
08/15/14    13.79%    2.64%
08/16/14    12.49%    2.66%
08/17/14    14.62%    2.24%
08/18/14    14.09%    2.49%
08/19/14    12.37%    3.20%
08/20/14    12.68%    2.36%
08/21/14    13.68%    2.71%
08/22/14    14.44%    2.49%
08/23/14    12.35%    2.29%
08/24/14    13.81%    2.58%
08/25/14    13.07%    2.55%
08/26/14    11.80%    2.11%
08/27/14    11.41%    1.88%
08/28/14    13.11%    2.50%
08/29/14    13.91%    2.68%
08/30/14    13.82%    2.09%
08/31/14    14.01%    2.45%
09/01/14    12.61%    2.97%
09/02/14    13.82%    2.55%
09/03/14    14.04%    2.77%
09/04/14    11.86%    2.58%
09/05/14    12.61%    2.24%
09/06/14    15.82%    3.43%
09/07/14    15.68%    4.13%
09/08/14    16.62%    4.50%
09/09/14    14.52%    4.88%
09/10/14    15.32%    3.14%
09/11/14    14.91%    3.58%
09/12/14    15.68%    3.45%
09/13/14    15.03%    3.39%
09/14/14    14.87%    3.69%
09/15/14    15.05%    3.50%
09/16/14    14.34%    3.32%
09/17/14    15.10%    3.22%
09/18/14    15.13%    3.10%
09/19/14    15.64%    3.75%
09/20/14    15.06%    2.77%
09/21/14    13.89%    2.73%
09/22/14    11.01%    3.56%
09/23/14    12.54%    2.26%
09/24/14    11.98%    2.48%
09/25/14    12.18%    2.47%
09/26/14    12.79%    2.68%
09/27/14    12.90%    2.33%
09/28/14    12.58%    2.57%
09/29/14    12.11%    1.89%
09/30/14    11.91%    2.10%
10/01/14    12.91%    1.76%
10/02/14    11.74%    1.68%
10/03/14    12.08%    2.31%
10/04/14    11.87%    2.42%
10/05/14    10.58%    2.06%
10/06/14    12.02%    2.42%
10/07/14    13.76%    2.12%
10/08/14    11.80%    2.20%
10/09/14    12.97%    1.87%
10/10/14    12.82%    2.33%
10/11/14    12.99%    1.44%
10/12/14    11.78%    2.10%
10/13/14    12.08%    2.38%
10/14/14    13.16%    2.12%
10/15/14    11.91%    2.02%
10/16/14    12.50%    2.21%
10/17/14    16.10%    7.18%
10/18/14    17.84%    7.66%
10/19/14    17.29%    8.43%
10/20/14    15.02%    7.95%
10/21/14    18.54%    8.73%
10/22/14    17.42%    9.87%
10/23/14    18.28%    9.69%
10/24/14    16.96%    8.74%
10/25/14    18.67%    9.04%
10/26/14    17.02%    8.26%
10/27/14    17.73%    8.07%
10/28/14    18.83%    10.29%
10/29/14    18.14%    9.48%
10/30/14    18.40%    10.79%
10/31/14    18.98%    10.01%
11/01/14    17.52%    8.44%
11/02/14    17.34%    8.01%
11/03/14    17.96%    8.89%
11/04/14    17.60%    9.21%
11/05/14    17.42%    7.54%
11/06/14    18.36%    8.62%
11/07/14    18.18%    9.40%
11/08/14    18.27%    8.86%
11/09/14    17.16%    8.18%
11/10/14    14.59%    8.03%
11/11/14    14.51%    7.57%
11/12/14    18.25%    8.18%
11/13/14    16.49%    8.97%
11/14/14    17.01%    8.44%
11/15/14    16.94%    7.10%
11/16/14    16.82%    7.66%
11/17/14    15.52%    6.95%
11/18/14    15.00%    5.86%
11/19/14    13.88%    5.85%
11/20/14    14.56%    7.05%
11/21/14    13.64%    6.19%
11/22/14    14.94%    6.16%
11/23/14    14.53%    6.23%
11/24/14    14.75%    6.03%
11/25/14    15.03%    6.27%
11/26/14    14.78%    5.19%
11/27/14    14.44%    4.47%
11/28/14    14.61%    5.61%
11/29/14    14.26%    6.13%
11/30/14    13.09%    5.54%
12/01/14    13.23%    5.69%
12/02/14    12.67%    5.31%
12/03/14    14.74%    5.51%
12/04/14    13.64%    5.13%
12/05/14    14.04%    5.37%
12/06/14    13.91%    4.36%
12/07/14    13.52%    5.14%

Comment 3

[Robert Strong (they/them - no direct email)]

This appears to be related to people using an old stub installer similar to bug 995684. I'll ping bouncer people again to see if it can return an error if the stub sent a version that isn't current when requesting the download url.

Comment 4

[Robert Strong (they/them - no direct email)]

Proposed way to deal with this

Add an optional version number to the url requesting the full installer download on bouncer Currently, the url to download the full installer is:
http://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US

What I propose is the following where the stub installer will send the ver param:
http://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US&ver=#
where "#" is the version of the stub installer. It would be optional so our web pages that use that url can just continue serving the stub installer without the version check.

When the stub installer changes in a manner that requires the user to use the latest stub installer the value of the ver param would be increased to the new stub version in the stub and bouncer. Bouncer would then send an http error or similar for clients with a lesser value and the stub installer would direct them to download the latest stub installer.

Comment 5

[Robert Strong (they/them - no direct email)]

Filed bug 1109342 for the changes needed to bouncer

Comment 6

[Daniel Veditz [:dveditz]]

Downloading a new stub and executing it should be OK under the following conditions:
 * new stub downloaded over TLS (of course)
 * we check that the signature is valid
 * we check that the cert Subject is "Mozilla Corporate (US)" or
   maybe more generically that it match the subject of the old stub.

Since broken pinned certs seems to be part of the problem we're trying to solve we don't want to get too deep, but we do need some kind of sanity check.

This will not protect a user if there is an active MITM and the attacker has a forged Mozilla authenticode cert that windows will accept. Under those same conditions the user loses if they manually downloaded Firefox that day instead of running the old stub so that should be acceptable. Once a user manages to get a valid copy of Firefox our own update system performs stronger checks.

Comment 7

[Robert Strong (they/them - no direct email)]

Latest numbers on the release channel

The reduction of the Cert Attributes % is due to these failures caused by using a really old stub installer moving over to Cancel or Retry failures. The other increase failures is caused by using a less old stub installer that has the correct certificate information but doesn't have other more recent code changes.

Date        Cancel %   Retries %  Cert Attributes %
11/24/14      8.83%      1.57%      0.26%
11/25/14      8.91%      1.57%      0.25%
11/26/14      9.04%      1.65%      0.25%
11/27/14      8.90%      1.64%      0.24%
11/28/14      8.78%      1.64%      0.23%
11/29/14      8.68%      1.69%      0.23%
11/30/14      8.56%      1.70%      0.25%
12/01/14      9.61%      2.50%      0.14%
12/02/14     11.98%      4.17%      0.00%
12/03/14     11.56%      4.11%      0.00%
12/04/14     11.60%      4.05%      0.00%
12/05/14     11.62%      3.97%      0.00%
12/06/14     11.55%      4.00%      0.00%
12/07/14     11.09%      3.87%      0.00%
12/08/14     11.28%      3.78%      0.00%
12/09/14     11.19%      3.70%      0.00%
12/10/14     10.67%      3.51%      0.00%
12/11/14      9.96%      2.89%      0.00%
12/12/14      9.94%      2.58%      0.00%
12/13/14     10.10%      2.87%      0.00%
12/14/14     10.27%      3.01%      0.00%
12/15/14     10.71%      3.01%      0.00%
12/16/14     11.50%      3.07%      0.00%
12/17/14     11.48%      3.21%      0.00%
12/18/14     11.46%      3.20%      0.00%
12/19/14     11.61%      3.28%      0.00%
12/20/14     11.75%      3.33%      0.00%
12/21/14     11.66%      3.39%      0.00%
12/22/14     11.44%      3.19%      0.00%
12/23/14     11.12%      3.22%      0.00%
12/24/14     11.14%      3.23%      0.00%
12/25/14     10.43%      3.09%      0.00%
12/26/14     10.96%      3.25%      0.00%
12/27/14     11.02%      3.25%      0.00%
12/28/14     10.89%      3.22%      0.00%
12/29/14     10.78%      3.05%      0.00%