688383 - XPI installation from a https-source with an installed CA fails through a "Certificate issuer is not built-in"-Error

Status: VERIFIED INVALID

(Toolkit :: Add-ons Manager, defect)

Description

[Markus Liebschner]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Build ID: 20110413222027

Steps to reproduce:

I tryed to install (Header Content-Type: application/x-xpinstall .xpi) an XPI by downloading it over https. 
The Certificate is valid and was installed manually by me.


Actual results:

I got a Connection-Error Popup-Message.

The ErrorConsol shows:
Warnung: WARN addons.xpi: Download failed: [Exception... "Certificate issuer is not built-in."  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 92"  data: no]
Quelldatei: resource://gre/modules/CertUtils.jsm
Zeile: 92

By the Way: A change of app.update.cert.requireBuiltIn do not change this behaviour.


Expected results:

It should work as like the installation over http (without a change of any config).

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Is this a self-signed certificate? Do you have an URL we could use for testing?  The pref you have specified above is not the one which gets checked for installing an extension from a HTTPs resource if the cert is not correctly signed. Instead it will be 'extensions.install.requireBuiltInCerts'.

Comment 2

[Markus Liebschner]

Yes it's self signed, but obviously it's valid because i don't get any warning when i request web-resources using this cert.
After adding the pref extensions.install.requireBuiltInCerts=false it works.
So the default-value seems to be true by default. I guess this is the problem, or why it is OK if i install over http?

Comment 3

[Dave Townsend [:mossop]]

We hold installation of add-ons over SSL to a higher standard than regular web pages since add-ons can take over your entire computer whereas webpages (unless they find a security hole) cannot.

The idea is that if the webpage requests that we install the add-on securely then we make sure to do so, if the webpage does not care and just uses http then we still respect that for now.

This seems to be working as expected.

Comment 4

[Tobias Gunkel]

But if the user consciously imports the Certificate Authority which was used to sign the SSL Certificate of the XPI source - why would you not respect that?

With importing the CA you explicitly agree to trust the identity of websites, E-Mail users and software developers.

I don't see any security problems with this, so why bypassing the users preferences?

Comment 5

[Markus Liebschner]

Yes this is what i mean.

Not the webpage requests that you install an addon securely - the client does.
With valid https the client can check whether a addon comes from a trusted orign or not.

I agree with you if you mean, that it might be not enough if the client only temporarily accept a certificate. But if he import the CA and mark it as trusted it should be handled like this. 
What are the alternatives? To set the global property extensions.install.requireBuiltInCerts=false or to switch to http? This is much more insecure.

Comment 6

[Dave Townsend [:mossop]]

The UI we have at the moment still allows for allowing rogue certificates to access potentially compromised websites. Allowing such for website access is one thing, then trusting that as a source of extensions which can compromise your entire machine very easily is another and not something we want to allow at this time.

If you require secure XPI installation you can use http with a verification hash.

Comment 7

[Petr Vobornik]

Hello, 

Would you please reconsider allowing installation of XPIs signed by custom certificate authority (ie. corporate CA) when:
 * the CA cert is imported and trusted
 * the certificate of website which serves the extension is also signed by the same CA

We would like to use HTTP Strict Transport Security in intranet application, but we can't because we are forced to use http just for the extension installation. It makes the whole setup less secure.

Thank You

Comment 8

[Rebecca Menessec]

My employer--and I'm absolutely certain I'm not alone--forces all SSL / TLS traffic through an "SSL proxy". Obviously, this means accepting some certs that aren't part of the public PKI... unless you prefer not to use HTTPS, ever.

This hard stance on refusing to allow non-public certificates means I can't get updates to my extensions. Ever. That, or I can simply not use Firefox, if I prefer to use extensions.

I'm not oblivious to the dangers of using an "SSL proxy". This one in particular advertises support for ciphers all the way down to RC2 and MACs all the way down to MD4. It's vulnerable to BEAST, and it appears to be incapable of using TLS > 1.0.

However, the flip side is that I can't get security (or any) updates to extensions. Apparently, I can't even get signature updates for Ghostery. (Fails silently.)

Addendum: This is quite likely to increase demand for "SSL proxies" that *do* use certificates signed by the public PKI; something Mozilla has also taken a hard stance against. Then there's that HTTP/2.0 proposal for "trusted SSL proxies".

Comment 9

[Daniel Veditz [:dveditz]]

(In reply to Rebecca Menessec from comment #8)
> This hard stance on refusing to allow non-public certificates means I can't
> get updates to my extensions. Ever. That, or I can simply not use Firefox,
> if I prefer to use extensions.

The hard stance makes the vast majority of users who are not in your special circumstances safer. For users in corporate environments where you know and trust your MITM there's the pref extensions.install.requireBuiltInCerts mentioned in comment 1.

Ultimately we are moving to requiring signed add-ons, and if the update content is signed then the SSL communication channel doesn't matter as much.

Comment 10

[ngyikp]

I stumbled upon this report after figuring out why some of my users complained that they can't install my  extension through HTTPS, but works just fine if I give them a HTTP link.

Turns out that their antivirus (BitDefender or Avast!) installed their own CA and performing a SSL MITM, this pretty much means that *every* Firefox extension that wants to be installed through HTTPS can't be installed.

So this is a conundrum, I want to offer better security for extension downloads, but some user setups make this more painful :(

Comment 11

[ngyikp]

I finally set up a Windows machine with Avast in order to see how they are doing the HTTPS interception, turns out this option is turned on by default, and they intentionally do not intercept *.mozilla.org

That's most likely why there's no huge number of complaints from users unable to install extensions, only extension authors (that don't wish to host on the Mozilla Add-ons site, and serving the XPI file through HTTPS) are having headaches.

Comment 12

[Dan Stillman]

We (Zotero) have been getting daily reports from users either unable to install the Zotero extension or running outdated versions, and it's becoming a significant support burden. In our case, as for ngyikp, it's almost always people running security software. (Yes, generally Avast and Bitdefender, but I believe others too, and while I think they should quit this, it seems inevitable that AV vendors will increasingly add SSL scanning capabilities so they can continue to claim they provide protection as the web turns to HTTPS.)

First, I should say that don't really understand the policy to begin with. Maybe things have changed since 2011 when mossop said that "The UI we have at the moment still allows for allowing rogue certificates to access potentially compromised websites", but I'm unclear how something that managed to install a custom root certificate into the user's browser couldn't do whatever else it wanted on the user's system -- including toggling the prefs to allow updates later -- without waiting for an installation attempt.

But even if we're going to take the policy as a given, there are a few serious issues with the current implementation:

1) ngyikp said that Avast isn't intercepting *.mozilla.org, but as far as I can tell, AMO is actually treated differently here by Firefox itself. If I run my connection through mitmproxy with a custom root cert, I can still install from AMO but nowhere else, even sites that I've allowed add-on installations from previously. It seems possible that Avast and others are actually just toggling the app.update.cert.requireBuiltIn pref but aren't toggling the extensions ones because they're not aware of them, since they (erroneously?) don't apply to AMO. So if this policy is deemed useful, it obviously should apply to AMO as well, and the security vendors should be contacted to let them know they should toggle the extensions pref when enabling SSL scanning and installing a root certificate.

2) The error message -- "The add-on could not be downloaded because of a connection failure on www.zotero.org." -- is quite unfortunate, since it makes it sound like this is a problem with the website in question and gives no indication of how to address the problem.

3) Serving updates over HTTP with a verification hash, as suggested in comment 6, is not a workaround. First, that doesn't help with initial installations, but also, as far as I can tell, HTTP updates are in fact ignored if the RDF file is served from a third-party HTTPS site when a custom root certificate is involved, which is why we have users on out-of-date versions without any indication in the add-ons manager that updates are available.

So if the policy is going to stay, I'd appreciate if the above issues could be addressed, and if there's agreement I can open up separate tickets for those. I understand that the upcoming signing requirements will supersede these checks, but that's still many months away, and this is affecting a significant number of our users. As it is, we can only hope that people make their way to our forums so that we can point them to our KB article. That often doesn't happen, judging by the number of people with outdated versions of Zotero who come to our forums to report problems that we fixed months ago (often compatibility updates for more recent versions of Firefox, which they do have installed).

Comment 13

[Dan Stillman]

(So I guess comment 6 was probably referring to McCoy-signed updates manifests that are served over HTTP as well. Still, that doesn't help with the initial installation unless the XPIs are signed too, which is a significant burden, and it doesn't help existing users on old versions pointing to an HTTPS update manifest.)

Comment 14

[ngyikp]

I think I finally solved the mystery why installing addons from addons.mozilla.org works on Avast HTTPS interception.



1.
It is due to how Avast handles EV certificates. Avast is trying to detect websites with EV certificates from going through their HTTPS proxy, Avast will hijack on the first connection attempt, but after a reload, the website's EV status shows properly, this means Avast decided not to intercept that website.

Since addons.mozilla.org DOES use an EV certificate, it's not intercepted.



2.
I dug through XPIProvider.jsm: https://lxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm#5540

If a hash string is provided on installation, it will not attempt to verify the certificate.



Testing my theories:

If you look at the source code for the Download button at https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/ , it is like this:

<a class="button  prominent  add installer" data-hash="sha256:9b550e2e30a2f4b702f853fab399c2b32f12254954ae27061c526ec29fb583fd" href="/firefox/downloads/latest/1865/addon-1865-latest.xpi?src=hp-dl-mostpopular"><b></b><span>Add to Firefox</span></a>

If you remove the "data-hash" attribute, then it will still work because addons.mozilla.org has a EV certificate. (Another note is that the confirm installation dialog shows "addons.mozilla.org" as the download URL origin, not "addons.cdn.mozilla.net", this is *probably* a Firefox bug)

If you change the "href" to the direct link of "https://addons.cdn.mozilla.net/user-media/addons/1865/adblock_plus-2.6.7-sm+tb+fx+an.xpi", it will still work because there will be a hash and that skips the certificate validation that I pointed out in XPIProvider.jsm

However, if you do *both* of these, then BOOM! "The add-on cannot be installed due to a connection failure on addons.mozilla.org"



To extension authors, you should use InstallTrigger to install your extension instead of pointing directly to the XPI file.
InstallTrigger is documented at https://developer.mozilla.org/en/docs/Installing_Extensions_and_Themes_From_Web_Pages

I added a hash to my InstallTrigger code, and now my addon installs perfect for me! :D

Example code:

InstallTrigger.install({
	'YOUR ADDON NAME': {
		URL: 'YOUR URL'
		,IconURL: 'YOUR ICON'
		,Hash: 'sha256:YOUR_SHA256_GENERATED_HASH'
	}
});

Next to research: Figure out if updates are being affected

Comment 15

[Dan Stillman]

ngyikp: You're right — it's the hash that was allowing add-ons on AMO via mitmproxy (which doesn't avoid EV certs), not some sort of whitelist. I can't say that this makes any more sense to me as a security policy, since it seems like an attacker who has installed a rogue cert (but somehow doesn't have access to the rest of the user's system...) then just needs to add some JS and a hash to the download page, but I'm happy for the workaround.

Updates are definitely affected (hence the extensions.update.requireBuiltIn pref), but I believe the solution is to sign them via McCoy [1]. (I haven't tested that yet, so not sure whether sending update.rdf via HTTP is necessary to avoid the built-in cert check, but it'd be safe to do so.)

Won't help existing users on old versions, but if they manage to reinstall once more manually they'd be set. This is good enough to get us through to the signing change. Thanks for digging into this.

[1] https://developer.mozilla.org/en-US/docs/McCoy

Comment 16

[ngyikp]

About the update part: (The more I research this, the more damning conclusions I'm finding out...)

I mentioned that Avast has a lovely behavior that if the website has a EV certificate, it will not be intercepted on the second connection. Well, it turns out that it also excludes subdomains that do not use a EV certificate such as https://versioncheck.addons.mozilla.org/



How I tested:
I installed an old version of Adblock Plus at https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/versions/ and an old version of my own self-hosted extension.

With Avast's shields disabled, I go to Add-ons and click "Check for Updates". Both extensions are offered updates.

I restarted Firefox to clear the update results, and then with Avast enabled, I check for updates. Adblock Plus is offered an update, while my extension does not.

I restarted Firefox, and use Fiddler that will intercept any HTTPS traffic regardless of the certificate status, and configured Firefox to trust the Fiddler CA. (Think of some corporate firewall) I check for updates, and none of the extensions are offered updates.



Worst part is that it fails *silently*, there is NO warning in the console, Browser Toolbox or anywhere that a bad root CA is causing addon updates to not work.

"updateHash" in update.rdf does *not* work.

I have not fully verified it yet, but McCoy does *not* work, if you analyze the code for AddonUpdateChecker.jsm, you'll see that Firefox tries to verify the certificate for the update manifest itself before the "updateKey" is even checked: https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/AddonUpdateChecker.jsm#434



Conclusions for extension authors that want to use HTTPS (and don't wish to use Mozilla Add-ons):
- During install, make sure you provide a hash on install
- Build your own update checker, treat the browser's addon update checker as a bonus but don't fully rely on it as users may disable it
- For updates, you may be forced to either serve the update manifest over HTTP with an updateKey... 
- Consider forcing the "extensions.install.requireBuiltInCerts" and "extensions.update.requireBuiltInCerts" configs to "false" when installing to save yourself headaches and tears in the future. (well, this is slightly naughty ;p but there are far more sensitive security settings than this...)

Comment 17

[Fred]

Hi, I've been having this problem for ages and only finally just got annoyed with it enough and tracked it down to this issue.

As I was able to INSTALL add-ons, I'd just go to AOM and install new versions of an addon. Which was fine when I only had a few addons installed (noscript, CS Lite, RequestPolicy/Policeman, ABP, Ghostery), but recently I've been installing more addons for general usability (tree style tab and such like), so the number of addons has blown out, which makes the "just visit AMO and download updated addons" impractical.

I'm using a corporate AV solution, a MIM proxy. It is this MIM proxy signing cert's that I believe is causing the issues. INSTALLING updates works fine, but on checking version, it doesn't like it on versioncheck as per the error console:

____________________________________________________________________
Expected certificate attribute 'issuerName' value incorrect, expected: 'CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US', got: <cut out> self-hosted:210:0

Expected certificate attribute 'issuerName' value incorrect, expected: 'CN=Thawte SSL CA,O="Thawte, Inc.",C=US', got: <cut out> self-hosted:210:0

Certificate checks failed. See previous errors for details.1 CertUtils.jsm:112:0

Toolkit.GMP	ERROR	GMPInstallManager.onLoadXML could not load xml: [Exception... "Certificate checks failed. See previous errors for details."  nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  location: "JS frame :: resource://gre/modules/CertUtils.jsm :: validateCert :: line 113"  data: no]1 Log.jsm:749:0

Toolkit.GMP	ERROR	GMPWrapper(gmp-gmpopenh264) findUpdates() - updateTask for gmp-gmpopenh264 threw: {"target":{},"status":200,"message":"[Exception... \"Certificate checks failed. See previous errors for details.\"  nsresult: \"0x80070057 (NS_ERROR_ILLEGAL_VALUE)\"  location: \"JS frame :: resource://gre/modules/CertUtils.jsm :: validateCert :: line 113\"  data: no]"}1 Log.jsm:749:0

addons.update-checker	WARN	Request failed: https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=jid1-XpkTj50ngv9GBQ@jetpack&version=2.0&maxAppVersion=30.0&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=41.0&appOS=WINNT&appABI=x86-msvc&locale=en-US&currentAppVersion=41.0&updateType=97&compatMode=normal - [Exception... "Certificate issuer is not built-in."  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 171"  data: no]

<and a heap more of these addons.update-checker messages, 1 for each extension I assume>
____________________________________________________________________


It appears that extensions.install.requireBuiltInCerts is no longer present in 41.0, so can't use that to get around this issue, although there is a media.gmp-manager.cert.requireBuiltIn, but when I toggle this to false, while I lose the first 5 errors above, I still get the addons.update-checker WARN, and no updates.

Comment 18

[Dave Townsend [:mossop]]

(In reply to Fred from comment #17)
> It appears that extensions.install.requireBuiltInCerts is no longer present
> in 41.0, so can't use that to get around this issue.

This preference is still used, you just have to create it in about:config.

Comment 19

[Ben McCann]

You can also use "extensions.update.requireBuiltInCerts" for testing the update mechanism