1005364 - Nightly only: loading https://services.addons.mozilla.org returns sec_error_application_callback_error

Status: VERIFIED FIXED

(Core :: Security: PSM, defect)

Description

[Edwin Wong [:edwong]]

Attachment: Screen Shot 2014-05-02 at 4.00.35 PM.png

doesn't happen on release or cert checks, but only when i'm on nightly.  I don't have any self-signed certs on that browser

1. goto about:addons or https://services.addons.mozilla.org

result:
An error occurred during a connection to services.addons.mozilla.org. The certificate was rejected by extra checks in the application. (Error code: sec_error_application_callback_error)

Comment 1

[Edwin Wong [:edwong]]

this looks like mac only. Reproduce on another mac but No repro on Windows Nightly

Comment 2

[Edwin Wong [:edwong]]

disreguard comment #1 - user updated nightly on windows and now reproduces the  sec_error_application_callback_error

moving to firefox as it's nightly specific

Comment 3

[Tim Taubert [:ttaubert] (inactive)]

This also breaks the "Get Add-ons" pane in the add-on manager.

Comment 4

[Tim Taubert [:ttaubert] (inactive)]

Manually switching back to the old pkix verifier enabled by bug 915930 doesn't seem to help.

Comment 5

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b227a707080f&tochange=e2e1b19fcffc

Comment 6

[Tim Taubert [:ttaubert] (inactive)]

Caused by bug 744204. Backing out affd460bc3d7 fixes it for me.

Comment 7

[[:mmc] Monica Chew (no longer reading bugmail)]

Camilo, looks like we are missing some Verisign certs in the mozilla pinset.

Comment 8

[[:mmc] Monica Chew (no longer reading bugmail)]

David, Camilo: I suggest we rollback include_subdomains=true on addons.mozilla.org for now, audit the certs from the CT database for mozilla properties, and try again. What do you think?

Comment 9

[Camilo Viecco (:cviecco)]

No, we should fix by changeing the built-inlist. Patching sometime today...

Comment 10

[Camilo Viecco (:cviecco)]

Workaround: set (hidden)integer pref security.cert_pinning.enforcement_level to 0.

Comment 11

[Camilo Viecco (:cviecco)]

Attachment: move-all-p1-to-cdn-mozilla

Made addons.mozilla.org be for the cdn pinning list not just the mozilla one.

Comment 12

[Camilo Viecco (:cviecco)]

Comment on attachment 8416963 [details] [diff] [review]
move-all-p1-to-cdn-mozilla

Review of attachment 8416963 [details] [diff] [review]:
-----------------------------------------------------------------

Something happended with my local repo (PreloadedHPKPins.json) is on the incorrect location, but this should do it.

Comment 13

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8416963 [details] [diff] [review]
move-all-p1-to-cdn-mozilla

Review of attachment 8416963 [details] [diff] [review]:
-----------------------------------------------------------------

Looks like you need to sync, the preload file is in security/manager/tools. Also, did you test going to https://services.addons.mozilla.org with this change?

::: security/manager/boot/src/PreloadedHPKPins.json
@@ -93,5 @@
>    ],
>  
>    "entries": [
> -    { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },
> -    { "name": "addons.mozilla.net", "include_subdomains": true, "pins": "mozilla" },

Are we not going to use the mozilla pinset at all? If so, we should remove it from the list so no one uses it by accident in the future.

Comment 14

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8416963 [details] [diff] [review]
move-all-p1-to-cdn-mozilla

Review of attachment 8416963 [details] [diff] [review]:
-----------------------------------------------------------------

Right now, we do not have a robust process by which we determine what Mozilla properties should be pinned by what key sets. This needs to be fixed before we can have confidence that changes we make to this preloaded data will be correct. Thus, I think the best approach is to disable pinning for now, develop a process to determine this data (which requires strong involvement from ops), and then turn it back on again.

Comment 15

[Camilo Viecco (:cviecco)]

(In reply to David Keeler (:keeler) from comment #14)
> Comment on attachment 8416963 [details] [diff] [review]
> move-all-p1-to-cdn-mozilla
> 
> Review of attachment 8416963 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Right now, we do not have a robust process by which we determine what
> Mozilla properties should be pinned by what key sets. This needs to be fixed
> before we can have confidence that changes we make to this preloaded data
> will be correct. Thus, I think the best approach is to disable pinning for
> now, develop a process to determine this data (which requires strong
> involvement from ops), and then turn it back on again.

Since we are so far away from the merge day I disagree. Yes we need a better process
to determine this list, but the reason why we discovered this issue is precisely because
the pin list is on (and the process issue a bug and forget failed). If you desire we
can go with Monica's idea of disabling subdomains for addons.mozilla.org, but I strongly oppose at disabling pinning.

With this in mind what are your thoughts?

Comment 16

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I agree with David that it is better to disable the pinning for mozilla.org until there's clear documentation and verification that the pins are correct and not DoSing importing services. Pinning is defense-in-depth and we've lived without it forever so disabling the pins for a few days more isn't likely to hurt anybody. DoSing services.addons.mozilla.org is a potentially-serious security issue since it seems to be the server used for addon (security) updates.

Comment 17

[[:mmc] Monica Chew (no longer reading bugmail)]

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #16)
> I agree with David that it is better to disable the pinning for mozilla.org

Just for clarification for anyone reading along, pinning is not enabled for all of *.mozilla.org. The full list is at http://mxr.mozilla.org/mozilla-central/source/security/manager/tools/PreloadedHPKPins.json

  "entries": [
96     { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },
97     { "name": "addons.mozilla.net", "include_subdomains": true, "pins": "mozilla" },
98     { "name": "cdn.mozilla.net", "include_subdomains": true, "pins": "mozilla_cdn" },
99     { "name": "cdn.mozilla.org", "include_subdomains": true, "pins": "mozilla_cdn" },
100     { "name": "media.mozilla.com", "include_subdomains": true, "pins": "mozilla_cdn" },
101     { "name": "include-subdomains.pinning.example.com", "include_subdomains": true, "pins": "mozilla_test" },
102     { "name": "exclude-subdomains.pinning.example.com", "include_subdomains": false, "pins": "mozilla_test" }

> until there's clear documentation and verification that the pins are correct
> and not DoSing importing services. Pinning is defense-in-depth and we've
> lived without it forever so disabling the pins for a few days more isn't
> likely to hurt anybody. DoSing services.addons.mozilla.org is a
> potentially-serious security issue since it seems to be the server used for
> addon (security) updates.

Visiting addons.mozilla.org itself doesn't break with pinning on for me, and turning off include-subdomains lets us do an incremental rollout on important subdomains https://bugzilla.mozilla.org/show_bug.cgi?id=1005430. I agree with David we need to audit cert usage before turning on subdomains for anything: https://bugzilla.mozilla.org/show_bug.cgi?id=1005653

Either way, we need to push a fix today so people who update tomorrow are fixed.

Comment 20

[[:mmc] Monica Chew (no longer reading bugmail)]

I found this cert in the CT database. This is not on any of the pinsets, so we need to disable until we understand what's going on. There are also some Digicerts CAs that we might need that aren't on the list.

CN:addons.mozilla.org
ISSUER:UTN-USERFirst-Hardware
DNS:addons.mozilla.org
DNS:www.addons.mozilla.org
CRL:http://crl.comodoca.com/UTN-USERFirst-Hardware.crl
CRL:http://crl.comodo.net/UTN-USERFirst-Hardware.crl
OCSP:http://ocsp.comodoca.com
-----BEGIN CERTIFICATE-----
MIIF+DCCBOCgAwIBAgIRAJI51TSPQNFpWnRUcOHyP0MwDQYJKoZIhvcNAQEFBQAw
gZcxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtl
IENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMY
aHR0cDovL3d3dy51c2VydHJ1c3QuY29tMR8wHQYDVQQDExZVVE4tVVNFUkZpcnN0
LUhhcmR3YXJlMB4XDTExMDMxNTAwMDAwMFoXDTE0MDMxNDIzNTk1OVowgeIxCzAJ
BgNVBAYTAlVTMQ4wDAYDVQQREwUzODQ3NzEQMA4GA1UECBMHRmxvcmlkYTEQMA4G
A1UEBxMHRW5nbGlzaDEXMBUGA1UECRMOU2VhIFZpbGxhZ2UgMTAxFDASBgNVBAoT
C0dvb2dsZSBMdGQuMRMwEQYDVQQLEwpUZWNoIERlcHQuMSgwJgYDVQQLEx9Ib3N0
ZWQgYnkgR1RJIEdyb3VwIENvcnBvcmF0aW9uMRQwEgYDVQQLEwtQbGF0aW51bVNT
TDEbMBkGA1UEAxMSYWRkb25zLm1vemlsbGEub3JnMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAq8ZtNvMVc3iDc850hdWu7LLw4CQfE4O4IKy7mv6Iu6uh
HQsfRQCqSbc1Nwxq70dMudG+41cSBI2Sx7bsAby22seBOCCtcoXmDvyBbAetaHY4
xUTXzMZKxZc+ZPRR5vB+suxW9yWCTUmYyxaY3SPxiZHRF5dAmSbW4qIrXt+9ifIb
GlMtzFBBetA9KgxVcBQB6VhJEHoLk4KL4R7tOoAQgs6WijTwzNfTubRQh1VUCbid
QihVAOWMNVS/3SWRRrcN5V2DqOWL+4TkPK522sRDK1t0C/i+XWjxeFu1zn3xXZlA
2sruOIFQvpihbLgkrfOvjA/XESgshBhMfbXZjzC1GwIDAQABo4IB8DCCAewwHwYD
VR0jBBgwFoAUoXJfJhsomEOVXQc31YWWnUvSw0UwHQYDVR0OBBYEFN2A0lQ990xw
yqOw3TR6MuToO1o7MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1Ud
JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjBGBgNVHSAEPzA9MDsGDCsGAQQBsjEB
AgEDBDArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8uY29tL0NQ
UzB7BgNVHR8EdDByMDigNqA0hjJodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9VVE4t
VVNFUkZpcnN0LUhhcmR3YXJlLmNybDA2oDSgMoYwaHR0cDovL2NybC5jb21vZG8u
bmV0L1VUTi1VU0VSRmlyc3QtSGFyZHdhcmUuY3JsMHEGCCsGAQUFBwEBBGUwYzA7
BggrBgEFBQcwAoYvaHR0cDovL2NydC5jb21vZG9jYS5jb20vVVROQWRkVHJ1c3RT
ZXJ2ZXJDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNv
bTA1BgNVHREELjAsghJhZGRvbnMubW96aWxsYS5vcmeCFnd3dy5hZGRvbnMubW96
aWxsYS5vcmcwDQYJKoZIhvcNAQEFBQADggEBADM7YxX8sewULJPddZTegVrZTpm+
+0qkOVVNoUB63hMqh6k3z+jV+63Re21vjCCHglTmV0m8ICiEzdYB2ZOLF24jZuWE
yIA/xqFwgOTsTR35/JFac2IpmvcgHGHgizmfyrx+jd282bHjn57fFVORIVIL2Roj
D2Y226yTlkqjpSLPKfeimaj2ttlArtl+tvZYLpusNspkj2VS3IacgqtuUEvaX/oF
AIgwDt6NVr+BR409BuKyYpJnj57ImrLlBrhwJLh3fCMKOMN5CNixUZ2slRHHQBee
oxyP8hGnaCfaSQWEGHxYLQFnXOWfoSm7SjlFL78Rqnmi7bTUtWVDt5NGitM=
-----END CERTIFICATE-----

Comment 21

[[:mmc] Monica Chew (no longer reading bugmail)]

Attachment: Disable pinning for all mozilla properties (

Comment 22

[[:mmc] Monica Chew (no longer reading bugmail)]

Comment on attachment 8417052 [details] [diff] [review]
Disable pinning for all mozilla properties (

Review of attachment 8417052 [details] [diff] [review]:
-----------------------------------------------------------------

I'm leaving soon, so I may not be able to check in myself.

Comment 23

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Comment on attachment 8417052 [details] [diff] [review]
Disable pinning for all mozilla properties (

Review of attachment 8417052 [details] [diff] [review]:
-----------------------------------------------------------------

r=me. I don't have my private key on this machine, so I can't check this in either this weekend :/
I'll mark checkin-needed and hopefully a sheriff will get to it.

::: security/manager/tools/PreloadedHPKPins.json
@@ +93,5 @@
>    ],
>  
>    "entries": [
> +    // Disable until bug 1005653 is fixed.
> +    // { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },

I'm not sure there's much value in leaving these in commented-out, but since this is sufficient and time-sensitive, I think it's fine.

@@ +97,5 @@
> +    // { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },
> +    // { "name": "addons.mozilla.net", "include_subdomains": true, "pins": "mozilla" },
> +    // { "name": "cdn.mozilla.net", "include_subdomains": true, "pins": "mozilla_cdn" },
> +    // { "name": "cdn.mozilla.org", "include_subdomains": true, "pins": "mozilla_cdn" },
> +    //{ "name": "media.mozilla.com", "include_subdomains": true, "pins": "mozilla_cdn" },

nit: space after "//"

Comment 24

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(In reply to [:mmc] Monica Chew (please use needinfo) from comment #20)
> I found this cert in the CT database. This is not on any of the pinsets, so
> we need to disable until we understand what's going on. There are also some
> Digicerts CAs that we might need that aren't on the list.
> 
> CN:addons.mozilla.org
> ISSUER:UTN-USERFirst-Hardware
> DNS:addons.mozilla.org
> DNS:www.addons.mozilla.org

This one isn't legit, and we've already blacklisted it:
https://www.eff.org/deeplinks/2011/03/iranian-hackers-obtain-fraudulent-https
https://www.comodo.com/Comodo-Fraud-Incident-2011-03-23.html
https://mxr.mozilla.org/mozilla-central/source/security/nss/lib/ckfw/builtins/certdata.txt#19970

Comment 25

[William Schlitz]

Reproduced on 2011 Macbook Air on the latest version of Nightly (2014-05-04)

Screnshot here: http://i.imgur.com/YvS5PUY.png


Build info:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:32.0) Gecko/20100101 Firefox/32.0

20140504030205

Comment 26

[[:mmc] Monica Chew (no longer reading bugmail)]

 https://hg.mozilla.org/integration/mozilla-inbound/rev/607eb10d17ba

Comment 29

[[:mmc] Monica Chew (no longer reading bugmail)]

https://hg.mozilla.org/mozilla-central/rev/e07b3f07953c

Hopefully it's not too late to make it into nightly for tomorrow's update.

Comment 30

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/607eb10d17ba

Comment 31

[[:mmc] Monica Chew (no longer reading bugmail)]

This appears to be fixed in the 5/5 Nightly update.

Comment 32

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Verified based on comment 31.

Comment 33

[f0rhum]

Just upgraded to FF35.0.1 ubuntu 12.04 32bits and I get this issue. Unchecking "Query OCSP responder to check the validity of your certicates" allows me to show the about:addons's Catalog page.
Re-enabling the responder still allows the page to be shown... until FF restart, then it fails agin;
A look in about:config with OCSP shows keys:
security.OCSP.GET.enabled default bool false
security.OCSP.enabled default integer 1
security.OCSP.require user defined bool true
security.ssl.enable_ocsp_stapling default bool true
services.sync.prefs.sync.security.OCSP.enabled default bool true
services.sync.prefs.sync.security.OCSP.require default bool true

Comment 34

[f0rhum]

PS: I have SSL Version Control 0.4 installed

Comment 35

[f0rhum]

PS: same in Windows XP : sec_error_revoked_certificate

Comment 36

[f0rhum]

PS: I was redirected here by dup https://bugzilla.mozilla.org/show_bug.cgi?id=1005655

Comment 37

[Dana Keeler (she/her) (use needinfo) [:keeler]]

f0rhum, if you're using 35, you're encountering a different issue. Please file a new bug here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM
Thanks!

Comment 38

[f0rhum]

Ok for me  in ff36