770594 - Check for Updates not working when proxy provider's CA certificate is trusted in FF Authorities list

Status: RESOLVED WONTFIX

(Toolkit :: Application Update, defect)

Description

[wsiegler]

Attachment: Zscaler.crt

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0
Build ID: 20120420145725

Steps to reproduce:

My update options are set to the following:
  "Check for updates, but let me choose whether to install them"
  Unchecked "Use a background service to install updates"

Using Zscaler cloud-based proxy service with SSL decryption.  This uses Zscaler's own certificate as issuer (trusted man in the middle).  I have trusted Zscaler's certificate under Authorities in Firefox [can identify websites, software makers] and it works fine on other secure websites visited (including this Bugzilla site).  So I'm led to believe that the check for updates is not trusting the "Zscaler-issued" version of the aus3.mozilla.org certificate when it should.  I've attached the Zscaler CA certificate.

Here's the HTTP call and partial response (SSL handshake data is encrypted):

CONNECT aus3.mozilla.org:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0
Proxy-Connection: keep-alive
Host: aus3.mozilla.org

HTTP/1.1 200 Connection Established
Proxy-Agent: Zscaler/3.6
...



Actual results:

When I click Help menu, About Firefox and click the "Check for Updates" button, FF incorrectly reports "Firefox is up to date"

Another user using the background service to update is getting the following error: "Something is trying to trick Firefox into accepting an insecure update. Please contact your network provider and seek help."

I believe both of these are due to the same issue.


Expected results:

Firefox should have referenced its trusted Authorities and have allowed the secure connection to the update server.

Comment 1

[Matthias Versen [:Matti]]

You are right, a software update is very security sensitive and only certificates issued by 2 CA are allowed. Certificates from other issuers are rejected and this is intended.

I guess this is a wontfix

Comment 2

[Matthias Versen [:Matti]]

found it - bug 544442

Comment 3

[Robert Strong (they/them - no direct email)]

dveditz, is there anything else we can do to work around this security requirement for users beyond the current hidden pref (app.update.cert.requireBuiltIn) to disable it?

Comment 4

[Daniel Veditz [:dveditz]]

These types of MITM security boxes generally happen in companies, and we had thought that just as companies roll out the cert they could roll out the pref changes to handle it. Apparently not, though. Could this explain part of the numbers of people who don't upgrade?

For a fixed desktop machine you might as well just disable the cert checking. If our updates get compromised with a fraudulently issued cert the Zscaler or other MITM box will just re-write it to use it's own cert. Adding the Zscaler cert info the update check doesn't protect you in this case.

For something mobile like a laptop that's sometimes used outside the corporate network there would be value in trying to add the Zscaler cert to the pin list rather than simply turning the checks off -- but that's well beyond the ability of the average user. I think (but am not sure!) you would create the prefs

app.update.certs.3.commonName
    aus3.mozilla.org

app.update.certs.3.issuerName
    CN=Zscaler,OU=Zscaler Inc.,O=www.zscaler.com,L=Santa Clara,ST=CA,C=US

I may have the order backward, it might be
    C=US,ST=CA,L=Santa Clara,O=www.zscaler.com,OU=Zscaler Inc.,CN=Zscaler

My uncertainty comes from not looking at the code, and I've got two different tools that give me two different orders.

There's also the pref app.update.cert.checkAttributes. From the name I would have expected that to be the one to use, but looking at the code disabling requireBuiltIns renders that one moot.

There's a serious usability issue here, do we have any telemetry metrics on how many users might be in this situation? Would it make any sense to make the pref change an easy override from the warning dialog? Terrible from a security POV (exactly the wrong thing to do if you _are_ under attack) but if false positives are blocking updates that's a bigger practical security problem than the small chance of a MITM (small unless you live in a country like Iran, anyway).

Comment 5

[Robert Strong (they/them - no direct email)]

It definitely could account for some "part" of people not upgrading and this has been the case since way back when it was implemented.

Prior to the "you might be under attack" warning UI the requirement for a builtin cert (which has been around longer than I have owned this component) would just silently fail for background update checks. As I see it, the requirement in general is a serious useability issue in that we either fail silently or we fail with a message that is vague as it is now though I do think where we are now is better than failing silently as it behaved previously.

We don't have telemetry data on this and we should add this to the plethora of other data we are gathering for app update.

Even without the telemtry data, I have to wonder whether we really need all certs to be builtin and not just the last one along with the cert attribute checks.

Comment 6

[Matthias Versen [:Matti]]

Could we accept Certificates from manual installed CA but reject Certificates that are in our CA root store except the whitelisted ones ?

Comment 7

[Robert Strong (they/them - no direct email)]

(In reply to Matthias Versen (Matti) from comment #6)
> Could we accept Certificates from manual installed CA but reject
> Certificates that are in our CA root store except the whitelisted ones ?
Possibly though I'd still prefer only placing the requirements on the last cert since *I think* that should be more than adequate.

Comment 9

[Daniel Veditz [:dveditz]]

The (weak) cert-pinning requirement was added because our update security depended solely on the reliability of 650+ certificate issuers, a terrible situation. The correct solution is to cryptographically sign the update /content/ and not the update /channel/ -- which was in fact the original request from the security team. This has been done now for Windows. The updates are signed on Linux and Mac but not yet verified (iirc). Once we have secured the update content through other means we can afford to be less paranoid about the security of the channel and consider dropping the CA pin.

Comment 10

[Lucas Adamski [:ladamski]]

These MITM boxes usually have the ability to whitelist certain hosts that rely on pinning behavior, such as update services.  They should use that.

Comment 11

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I agree with Dan in Comment 9.

I'd go further and say that on any platforms where we are using signed MARs, we should NOT do the pinning check at all.

Comment 12

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

cviecco, issues like this one are why I want to hold off on doing any of the cert key pinning stuff for AUS.

In general, before we land the key pinning patch, we need to make sure we understand its ramifications for HTTPS proxies. I know there are some limitations in our support for HTTPS + proxies now.

Comment 13

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I forgot to mention: cviecco is implementing an improved form of key pinning that any/every website will be able to use, in bug 744204. However, exactly because of the this kind of issue with trusted man-in-the-middle proxies, we only enforce the pinning for built-in CAs; if the certificate we encounter is issued by a user-added certificate authority, we will not enforce the pinning. In other words, it's basically the opposite logic from what the AUS-specific CA pinning is doing.

Comment 14

[Ben Bucksch (:BenB)]

> Another user using the background service to update is getting the following error:
> "Something is trying to trick Firefox into accepting an insecure update.
> Please contact your network provider and seek help."

This is the correct behaviour.

I strongly suggest WONTFIX. MITM SSL proxies is something we as project work against, it's not a legitimate usecase we actively support. This has been discussed at length on the SSL policy newsgroups.

Corporates can, technically, simply change the preferences that pin a certain CA or certificate for the update service. See https://developer.mozilla.org/en-US/docs/Mozilla/Setting_up_an_update_server . Thus, this bug is not even a valid feature request.

Comment 15

[Robert Strong (they/them - no direct email)]

Now that we have mar files signed on Windows the certificate checks are no longer performed on Windows for Firefox 27 in bug 928489 so this bug only applies to Mac and Linux. Once we have mar signing on Mac and Linux it will be disabled there as well in bug 920750. Resolving as wontfix.