Closed Bug 1424478 Opened 5 years ago Closed 5 years ago

Extensions update check fails due to "Certificate issuer is not built-in"

Categories

(Toolkit :: Add-ons Manager, defect)

57 Branch
defect
Not set
normal

Tracking

()

RESOLVED INVALID

People

(Reporter: rainer.klute, Unassigned)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
Build ID: 20171128222554

Steps to reproduce:

Since some time already, Firefox doesn't detect any extensions updates anymore. Manually triggering "Check for Updates" on the about:addons page always results in "No updates found". However, actually there ARE updates available for some of the addons I have installed. For example, take the "Update Scanner" (sic!) addon. I have installed in version 4.0.0 in November, while today version 4.2.0 is available.

So I dug a little deeper …


Actual results:

When I check for an update of the "Update Scanner" addon, the browser console displays the following messages:

Public-Key-Pins: The certificate used by the site was not issued by a certificate in the default root certificate store. To prevent accidental breakage, the specified header was ignored.[Learn More]  VersionCheck.php
1512816157167	addons.update-checker	WARN	Request failed: https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id={c07d1a49-9894-49ff-a594-38960ede8fb9}&version=4.0.0&maxAppVersion=null&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=57.0.1&appOS=Linux&appABI=x86_64-gcc3&locale=en-US-u-va-posix&currentAppVersion=57.0.1&updateType=97&compatMode=normal - [Exception... "Certificate issuer is not built-in."  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 169"  data: no]

An equivalent messsage is issued for each addon when doing the global update check.

I disabled security.cert_pinning, but that didn't help. The Public-Key-Pins error message disappeared; however, the "Certificate issuer is not built-in" warning remained. And of course the available addon update wasn't detected.

My next step was to access the URL given in the warning message, i.e. https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id={c07d1a49-9894-49ff-a594-38960ede8fb9}&version=4.0.0&maxAppVersion=null&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=57.0.1&appOS=Linux&appABI=x86_64-gcc3&locale=en-US-u-va-posix&currentAppVersion=57.0.1&updateType=97&compatMode=normal, in a browser window. In this case, no certificate error occurred. In fact, the server response looks quite sensible and contains the proper information regarding the current 4.2.0:

<RDF:RDF><RDF:Description about="urn:mozilla:extension:{c07d1a49-9894-49ff-a594-38960ede8fb9}"><em:updates><RDF:Seq><RDF:li resource="urn:mozilla:extension:{c07d1a49-9894-49ff-a594-38960ede8fb9}:4.2.0"/></RDF:Seq></em:updates></RDF:Description><RDF:Description about="urn:mozilla:extension:{c07d1a49-9894-49ff-a594-38960ede8fb9}:4.2.0"><em:version>4.2.0</em:version><em:targetApplication><RDF:Description><em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id><em:minVersion>57.0a2</em:minVersion><em:maxVersion>*</em:maxVersion><em:updateLink>https://addons.cdn.mozilla.net/user-media/addons/3362/update_scanner-4.2.0-an+fx.xpi?filehash=sha256%3A274e3582f078633417a86362c9778e7ffaa3e79f4e60a5656c89ed6d50a1174e</em:updateLink><em:updateInfoURL>https://addons.mozilla.org/versions/updateInfo/2257257/%APP_LOCALE%/</em:updateInfoURL><em:updateHash>sha256:274e3582f078633417a86362c9778e7ffaa3e79f4e60a5656c89ed6d50a1174e</em:updateHash></RDF:Description></em:targetApplication></RDF:Description></RDF:RDF>

Now I am stuck. Thanks for any help!

By the way, I also tried all this in a test profile, but to no avail. :-(


Expected results:

Firefox should detect available updates of installed addon.
Component: Untriaged → Add-ons Manager
Product: Firefox → Toolkit
When you load the versioncheck url manually, can you click the lock icon in the location bar and click through to get the certificate details and paste them here?  I suspect you have some some sort of TLS man-in-the-middle software or device, which would make this essentially a duplicate of bug 1403075 (that one is about installs, not updates, but its the same issue)
Flags: needinfo?(rainer.klute)
I browsed bug 1403075, however, I don't have any anti-virus or other MITM software installed that I'd be aware of. As far as I can see there's nothing suspicious. The certificate chain is (unfortunately I cannot copy & paste):

DigiCert Global Root CA → DigiCert SHA2 Secure Servcer CA → versioncheck.addons.mozilla.org

Here's the actual certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:41:86:3b:81:e6:c5:ba:b0:60:b2:03:1d:1d:11:1e
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA
        Validity
            Not Before: Jan 27 00:00:00 2017 GMT
            Not After : Feb 21 12:00:00 2020 GMT
        Subject: C=US, ST=California, L=Mountain View, O=Mozilla Foundation, OU=Cloud Services, CN=versioncheck.addons.mozilla.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a6:0c:c9:b9:ba:0e:1d:3a:eb:f6:c8:83:c7:93:
                    d9:ec:63:bf:cc:63:2c:56:41:0e:0c:a9:bc:ec:88:
                    95:f3:da:22:91:79:04:40:bc:b8:14:a5:2e:c6:32:
                    53:90:ef:06:94:cc:51:12:3d:67:cf:1a:07:64:2d:
                    d6:77:89:61:ab:b0:e7:de:8a:cd:6d:ce:d3:42:f3:
                    04:3c:37:8e:11:eb:d3:06:de:b5:ac:72:b5:a3:be:
                    50:44:ab:9e:95:77:10:ee:3a:f7:c5:28:a7:e1:23:
                    99:f1:06:64:53:83:27:d3:68:25:86:3e:c3:47:e7:
                    cd:d2:08:d1:27:96:fe:a8:9f:ed:41:3e:ec:a0:4c:
                    c1:6b:99:5e:39:83:cf:35:44:d3:70:ce:54:fa:ef:
                    0b:73:26:82:dc:52:4c:38:5c:b5:37:1f:5d:df:b9:
                    01:59:44:a5:d1:d4:5b:dc:c2:0c:48:5b:45:b1:c8:
                    6c:e3:f2:cc:b4:eb:04:30:c2:bb:d8:57:2d:b1:c2:
                    38:d5:e4:f4:b5:5e:5e:75:96:c6:c1:df:9b:b1:7f:
                    25:62:26:80:fb:04:9b:62:b2:fd:0b:fc:95:20:2d:
                    63:40:a6:20:b4:6b:34:bb:24:09:10:d6:2b:5c:e5:
                    26:5a:88:9d:fb:9b:93:19:92:9e:aa:43:61:a6:fe:
                    c3:87
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:0F:80:61:1C:82:31:61:D5:2F:28:E7:8D:46:38:B4:2C:E1:C6:D9:E2

            X509v3 Subject Key Identifier: 
                8E:5F:54:4A:FA:52:B3:67:9E:A4:2A:3E:CE:DF:FC:AE:A1:6D:46:3A
            X509v3 Subject Alternative Name: 
                DNS:versioncheck.addons.mozilla.org, DNS:versioncheck-bg.addons.mozilla.org, DNS:blocklist.addons.mozilla.org
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl3.digicert.com/ssca-sha2-g5.crl

                Full Name:
                  URI:http://crl4.digicert.com/ssca-sha2-g5.crl

            X509v3 Certificate Policies: 
                Policy: 2.16.840.1.114412.1.1
                  CPS: https://www.digicert.com/CPS
                Policy: 2.23.140.1.2.2

            Authority Information Access: 
                OCSP - URI:http://ocsp.digicert.com
                CA Issuers - URI:http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt

            X509v3 Basic Constraints: critical
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         0d:a9:3d:58:a7:21:37:19:4b:89:42:97:46:f3:14:fc:1e:ac:
         b7:47:a3:46:e9:d3:a7:28:d6:da:67:7b:1f:5b:5d:2e:78:ee:
         14:37:0c:8a:de:17:8b:cc:80:36:06:53:0c:fe:4c:17:5a:d3:
         b0:2e:89:2f:70:20:f2:09:95:9b:4c:2c:9c:42:3b:8b:59:e5:
         7e:66:cc:91:e0:31:fd:15:32:81:f4:80:26:aa:83:65:04:0d:
         19:b6:15:e3:ee:ad:24:6a:89:b8:4c:43:b7:16:0d:d1:0f:35:
         2b:e6:ee:0b:35:67:b5:3a:82:76:a0:ab:ee:cd:c1:ba:4c:6c:
         b6:4d:0e:53:58:f2:f3:b1:78:ee:7a:d4:bd:ac:01:78:4c:2f:
         d6:4f:44:29:15:bc:e3:bf:ec:23:1a:b9:67:ff:37:31:a6:cd:
         f8:3b:16:6c:e9:f2:7f:ba:ab:b9:bb:50:1f:c6:4e:5f:4e:c1:
         89:b7:12:72:e1:9f:52:98:55:7a:06:71:be:c2:03:ab:94:ae:
         7d:b9:f3:d7:39:21:2a:a3:0b:2b:d3:2c:a2:ab:06:db:b4:b7:
         dc:2a:cf:40:48:5c:51:62:69:18:be:cb:ce:0e:73:32:fa:3f:
         01:8d:37:b4:5e:ad:29:3e:2b:07:5b:2c:5e:a8:33:42:d5:2f:
         ca:0a:7c:f2
-----BEGIN CERTIFICATE-----
MIIFnDCCBISgAwIBAgIQBEGGO4HmxbqwYLIDHR0RHjANBgkqhkiG9w0BAQsFADBN
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMScwJQYDVQQDEx5E
aWdpQ2VydCBTSEEyIFNlY3VyZSBTZXJ2ZXIgQ0EwHhcNMTcwMTI3MDAwMDAwWhcN
MjAwMjIxMTIwMDAwWjCBmjELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3Ju
aWExFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxGzAZBgNVBAoTEk1vemlsbGEgRm91
bmRhdGlvbjEXMBUGA1UECxMOQ2xvdWQgU2VydmljZXMxKDAmBgNVBAMTH3ZlcnNp
b25jaGVjay5hZGRvbnMubW96aWxsYS5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCmDMm5ug4dOuv2yIPHk9nsY7/MYyxWQQ4MqbzsiJXz2iKReQRA
vLgUpS7GMlOQ7waUzFESPWfPGgdkLdZ3iWGrsOfeis1tztNC8wQ8N44R69MG3rWs
crWjvlBEq56VdxDuOvfFKKfhI5nxBmRTgyfTaCWGPsNH583SCNEnlv6on+1BPuyg
TMFrmV45g881RNNwzlT67wtzJoLcUkw4XLU3H13fuQFZRKXR1FvcwgxIW0WxyGzj
8sy06wQwwrvYVy2xwjjV5PS1Xl51lsbB35uxfyViJoD7BJtisv0L/JUgLWNApiC0
azS7JAkQ1itc5SZaiJ37m5MZkp6qQ2Gm/sOHAgMBAAGjggIoMIICJDAfBgNVHSME
GDAWgBQPgGEcgjFh1S8o541GOLQs4cbZ4jAdBgNVHQ4EFgQUjl9USvpSs2eepCo+
zt/8rqFtRjowbAYDVR0RBGUwY4IfdmVyc2lvbmNoZWNrLmFkZG9ucy5tb3ppbGxh
Lm9yZ4IidmVyc2lvbmNoZWNrLWJnLmFkZG9ucy5tb3ppbGxhLm9yZ4IcYmxvY2ts
aXN0LmFkZG9ucy5tb3ppbGxhLm9yZzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMGsGA1UdHwRkMGIwL6AtoCuGKWh0dHA6Ly9j
cmwzLmRpZ2ljZXJ0LmNvbS9zc2NhLXNoYTItZzUuY3JsMC+gLaArhilodHRwOi8v
Y3JsNC5kaWdpY2VydC5jb20vc3NjYS1zaGEyLWc1LmNybDBMBgNVHSAERTBDMDcG
CWCGSAGG/WwBATAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy5kaWdpY2VydC5j
b20vQ1BTMAgGBmeBDAECAjB8BggrBgEFBQcBAQRwMG4wJAYIKwYBBQUHMAGGGGh0
dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBGBggrBgEFBQcwAoY6aHR0cDovL2NhY2Vy
dHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMlNlY3VyZVNlcnZlckNBLmNydDAM
BgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQANqT1YpyE3GUuJQpdG8xT8
Hqy3R6NG6dOnKNbaZ3sfW10ueO4UNwyK3heLzIA2BlMM/kwXWtOwLokvcCDyCZWb
TCycQjuLWeV+ZsyR4DH9FTKB9IAmqoNlBA0ZthXj7q0kaom4TEO3Fg3RDzUr5u4L
NWe1OoJ2oKvuzcG6TGy2TQ5TWPLzsXjuetS9rAF4TC/WT0QpFbzjv+wjGrln/zcx
ps34OxZs6fJ/uqu5u1Afxk5fTsGJtxJy4Z9SmFV6BnG+wgOrlK59ufPXOSEqowsr
0yyiqwbbtLfcKs9ASFxRYmkYvsvODnMy+j8BjTe0Xq0pPisHWyxeqDNC1S/KCnzy
-----END CERTIFICATE-----
Flags: needinfo?(rainer.klute)
I just defined the extensions.install.requireBuiltInCerts preference option and set it to false, but that didn't change anything.
Sorry I still haven't had a chance to look into this more closely, but the preference from comment 3 doesn't apply to updates, the preference that does is extensions.updates.requireBuiltInCerts
One other question, are you using an official Firefox build?
Flags: needinfo?(rainer.klute)
Thanks for the hint regarding extensions.updates.requireBuiltInCerts! I changed that to extensions.update.requireBuiltInCerts, i.e. removed the 's', and now it works!

Of course there's still a bug somewhere, but this observation might help to spot it.

By the way, I am using the Firefox build from openSUSE, currently version 57.0.1-1.3, build on Mon Dec 4 05:52:51 2017. My OS version is openSUSE 42.3.
Flags: needinfo?(rainer.klute)
(In reply to Rainer Klute from comment #6)
> By the way, I am using the Firefox build from openSUSE, currently version
> 57.0.1-1.3, build on Mon Dec 4 05:52:51 2017. My OS version is openSUSE 42.3.

Can you try reproducing this with a Mozilla build?  From all the comments above, things seem to be working properly, I wonder if the openSUSE build is not including the built-in certificates...
Flags: needinfo?(rainer.klute)
I just downloaded and tried Firefox 57.0.2 from https://www.mozilla.org/en-US/firefox/: looks good, very good!

So it's definitely an openSUSE issue. I'll file a bug report with them.

Thanks for your help, Andrew!
Flags: needinfo?(rainer.klute)
Thanks for doing all that leg work and for the prompt responses.
Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID
(In reply to Andrew Swan [:aswan] from comment #7)
> Can you try reproducing this with a Mozilla build?  From all the comments
> above, things seem to be working properly, I wonder if the openSUSE build is
> not including the built-in certificates...

So Andrew, can you please point me (as a packager for that build) what might be wrong in the build process?

The only real difference I know is that we use NSPR and NSS from system libs instead of the built-in libraries. I'm not aware of other "built-in certificates". At least I also do not explicitely disable anything.
Status: RESOLVED → REOPENED
Ever confirmed: true
Flags: needinfo?(aswan)
Resolution: INVALID → ---
I'm not familiar with how the built-in certificates are handled in the build process, hopefully keeler can help.
Flags: needinfo?(aswan) → needinfo?(dkeeler)
To differentiate Mozilla-built-in roots from others, we check for the attribute CKA_NSS_MOZILLA_CA_POLICY on the PKCS#11 object for the certificate. This changed fairly recently, so if your profile is using an old built-in roots module, the attribute will never be present and no root will be considered built-in. If you go to about:preferences, search for "Security Devices", click on the highlighted button, and then "Builtin Roots Module", you can see what file is being loaded as your roots module. If you attach that to this bug, we'll be able to see if that's the problem.
Flags: needinfo?(dkeeler) → needinfo?(rainer.klute)
While discussing this issue in the openSUSE-specific bug report https://bugzilla.opensuse.org/show_bug.cgi?id=1073146, it turned out that openSUSE a) distributes Firefox over several packages and b) does not force matching versions of these packages in all cases – at least that's my understanding. Once I resolved the package mismatch on my machine, checking the certificates and updating the extensions went fine.
Flags: needinfo?(rainer.klute)
Closing, see the linked openSUSE bug for further details
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.