Closed Bug 1005364 Opened 10 years ago Closed 10 years ago

Nightly only: loading https://services.addons.mozilla.org returns sec_error_application_callback_error

Categories

(Core :: Security: PSM, defect)

Product:
Core
Component:
Security: PSM
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla32
Tracking Flags:
Tracking Status
firefox32 --- verified

People

(Reporter: edwong, Assigned: mmc)

References

Details

(Keywords: regression)

Attachments

(2 files, 1 obsolete file)

Screen Shot 2014-05-02 at 4.00.35 PM.png
498.03 KB, image/png
Details
Disable pinning for all mozilla properties (
2.88 KB, patch
keeler
: review+
Details | Diff | Splinter Review 
doesn't happen on release or cert checks, but only when i'm on nightly.  I don't have any self-signed certs on that browser

1. goto about:addons or https://services.addons.mozilla.org

result:
An error occurred during a connection to services.addons.mozilla.org. The certificate was rejected by extra checks in the application. (Error code: sec_error_application_callback_error)
this looks like mac only. Reproduce on another mac but No repro on Windows Nightly
disreguard comment #1 - user updated nightly on windows and now reproduces the  sec_error_application_callback_error

moving to firefox as it's nightly specific
Assignee: server-ops-webops → nobody
Component: WebOps: SSL and Domain Names → General
Product: Infrastructure & Operations → Firefox
QA Contact: nmaul
Version: other → Trunk
This also breaks the "Get Add-ons" pane in the add-on manager.
Severity: normal → major
OS: Mac OS X → All
Hardware: x86 → All
Manually switching back to the old pkix verifier enabled by bug 915930 doesn't seem to help.
Caused by bug 744204. Backing out affd460bc3d7 fixes it for me.
Blocks: 744204
Component: General → Security: PSM
Product: Firefox → Core
Camilo, looks like we are missing some Verisign certs in the mozilla pinset.
Flags: needinfo?(cviecco)
Assignee: nobody → cviecco
David, Camilo: I suggest we rollback include_subdomains=true on addons.mozilla.org for now, audit the certs from the CT database for mozilla properties, and try again. What do you think?
Flags: needinfo?(dkeeler)
No, we should fix by changeing the built-inlist. Patching sometime today...
Flags: needinfo?(cviecco)
Workaround: set (hidden)integer pref security.cert_pinning.enforcement_level to 0.
Attached patch move-all-p1-to-cdn-mozilla (obsolete) — — Splinter Review 
Made addons.mozilla.org be for the cdn pinning list not just the mozilla one.
Comment on attachment 8416963 [details] [diff] [review]
move-all-p1-to-cdn-mozilla

Review of attachment 8416963 [details] [diff] [review]:
-----------------------------------------------------------------

Something happended with my local repo (PreloadedHPKPins.json) is on the incorrect location, but this should do it.
Attachment #8416963 - Flags: review?(dkeeler)
Comment on attachment 8416963 [details] [diff] [review]
move-all-p1-to-cdn-mozilla

Review of attachment 8416963 [details] [diff] [review]:
-----------------------------------------------------------------

Looks like you need to sync, the preload file is in security/manager/tools. Also, did you test going to https://services.addons.mozilla.org with this change?

::: security/manager/boot/src/PreloadedHPKPins.json
@@ -93,5 @@
>    ],
>  
>    "entries": [
> -    { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },
> -    { "name": "addons.mozilla.net", "include_subdomains": true, "pins": "mozilla" },

Are we not going to use the mozilla pinset at all? If so, we should remove it from the list so no one uses it by accident in the future.
Comment on attachment 8416963 [details] [diff] [review]
move-all-p1-to-cdn-mozilla

Review of attachment 8416963 [details] [diff] [review]:
-----------------------------------------------------------------

Right now, we do not have a robust process by which we determine what Mozilla properties should be pinned by what key sets. This needs to be fixed before we can have confidence that changes we make to this preloaded data will be correct. Thus, I think the best approach is to disable pinning for now, develop a process to determine this data (which requires strong involvement from ops), and then turn it back on again.
Attachment #8416963 - Flags: review?(dkeeler) → review-
Flags: needinfo?(dkeeler)
(In reply to David Keeler (:keeler) from comment #14)
> Comment on attachment 8416963 [details] [diff] [review]
> move-all-p1-to-cdn-mozilla
> 
> Review of attachment 8416963 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Right now, we do not have a robust process by which we determine what
> Mozilla properties should be pinned by what key sets. This needs to be fixed
> before we can have confidence that changes we make to this preloaded data
> will be correct. Thus, I think the best approach is to disable pinning for
> now, develop a process to determine this data (which requires strong
> involvement from ops), and then turn it back on again.

Since we are so far away from the merge day I disagree. Yes we need a better process
to determine this list, but the reason why we discovered this issue is precisely because
the pin list is on (and the process issue a bug and forget failed). If you desire we
can go with Monica's idea of disabling subdomains for addons.mozilla.org, but I strongly oppose at disabling pinning.

With this in mind what are your thoughts?
Flags: needinfo?(dkeeler)
I agree with David that it is better to disable the pinning for mozilla.org until there's clear documentation and verification that the pins are correct and not DoSing importing services. Pinning is defense-in-depth and we've lived without it forever so disabling the pins for a few days more isn't likely to hurt anybody. DoSing services.addons.mozilla.org is a potentially-serious security issue since it seems to be the server used for addon (security) updates.
(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #16)
> I agree with David that it is better to disable the pinning for mozilla.org

Just for clarification for anyone reading along, pinning is not enabled for all of *.mozilla.org. The full list is at http://mxr.mozilla.org/mozilla-central/source/security/manager/tools/PreloadedHPKPins.json

  "entries": [
96     { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },
97     { "name": "addons.mozilla.net", "include_subdomains": true, "pins": "mozilla" },
98     { "name": "cdn.mozilla.net", "include_subdomains": true, "pins": "mozilla_cdn" },
99     { "name": "cdn.mozilla.org", "include_subdomains": true, "pins": "mozilla_cdn" },
100     { "name": "media.mozilla.com", "include_subdomains": true, "pins": "mozilla_cdn" },
101     { "name": "include-subdomains.pinning.example.com", "include_subdomains": true, "pins": "mozilla_test" },
102     { "name": "exclude-subdomains.pinning.example.com", "include_subdomains": false, "pins": "mozilla_test" }

> until there's clear documentation and verification that the pins are correct
> and not DoSing importing services. Pinning is defense-in-depth and we've
> lived without it forever so disabling the pins for a few days more isn't
> likely to hurt anybody. DoSing services.addons.mozilla.org is a
> potentially-serious security issue since it seems to be the server used for
> addon (security) updates.

Visiting addons.mozilla.org itself doesn't break with pinning on for me, and turning off include-subdomains lets us do an incremental rollout on important subdomains https://bugzilla.mozilla.org/show_bug.cgi?id=1005430. I agree with David we need to audit cert usage before turning on subdomains for anything: https://bugzilla.mozilla.org/show_bug.cgi?id=1005653

Either way, we need to push a fix today so people who update tomorrow are fixed.
I found this cert in the CT database. This is not on any of the pinsets, so we need to disable until we understand what's going on. There are also some Digicerts CAs that we might need that aren't on the list.

CN:addons.mozilla.org
ISSUER:UTN-USERFirst-Hardware
DNS:addons.mozilla.org
DNS:www.addons.mozilla.org
CRL:http://crl.comodoca.com/UTN-USERFirst-Hardware.crl
CRL:http://crl.comodo.net/UTN-USERFirst-Hardware.crl
OCSP:http://ocsp.comodoca.com
-----BEGIN CERTIFICATE-----
MIIF+DCCBOCgAwIBAgIRAJI51TSPQNFpWnRUcOHyP0MwDQYJKoZIhvcNAQEFBQAw
gZcxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtl
IENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMY
aHR0cDovL3d3dy51c2VydHJ1c3QuY29tMR8wHQYDVQQDExZVVE4tVVNFUkZpcnN0
LUhhcmR3YXJlMB4XDTExMDMxNTAwMDAwMFoXDTE0MDMxNDIzNTk1OVowgeIxCzAJ
BgNVBAYTAlVTMQ4wDAYDVQQREwUzODQ3NzEQMA4GA1UECBMHRmxvcmlkYTEQMA4G
A1UEBxMHRW5nbGlzaDEXMBUGA1UECRMOU2VhIFZpbGxhZ2UgMTAxFDASBgNVBAoT
C0dvb2dsZSBMdGQuMRMwEQYDVQQLEwpUZWNoIERlcHQuMSgwJgYDVQQLEx9Ib3N0
ZWQgYnkgR1RJIEdyb3VwIENvcnBvcmF0aW9uMRQwEgYDVQQLEwtQbGF0aW51bVNT
TDEbMBkGA1UEAxMSYWRkb25zLm1vemlsbGEub3JnMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAq8ZtNvMVc3iDc850hdWu7LLw4CQfE4O4IKy7mv6Iu6uh
HQsfRQCqSbc1Nwxq70dMudG+41cSBI2Sx7bsAby22seBOCCtcoXmDvyBbAetaHY4
xUTXzMZKxZc+ZPRR5vB+suxW9yWCTUmYyxaY3SPxiZHRF5dAmSbW4qIrXt+9ifIb
GlMtzFBBetA9KgxVcBQB6VhJEHoLk4KL4R7tOoAQgs6WijTwzNfTubRQh1VUCbid
QihVAOWMNVS/3SWRRrcN5V2DqOWL+4TkPK522sRDK1t0C/i+XWjxeFu1zn3xXZlA
2sruOIFQvpihbLgkrfOvjA/XESgshBhMfbXZjzC1GwIDAQABo4IB8DCCAewwHwYD
VR0jBBgwFoAUoXJfJhsomEOVXQc31YWWnUvSw0UwHQYDVR0OBBYEFN2A0lQ990xw
yqOw3TR6MuToO1o7MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1Ud
JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjBGBgNVHSAEPzA9MDsGDCsGAQQBsjEB
AgEDBDArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8uY29tL0NQ
UzB7BgNVHR8EdDByMDigNqA0hjJodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9VVE4t
VVNFUkZpcnN0LUhhcmR3YXJlLmNybDA2oDSgMoYwaHR0cDovL2NybC5jb21vZG8u
bmV0L1VUTi1VU0VSRmlyc3QtSGFyZHdhcmUuY3JsMHEGCCsGAQUFBwEBBGUwYzA7
BggrBgEFBQcwAoYvaHR0cDovL2NydC5jb21vZG9jYS5jb20vVVROQWRkVHJ1c3RT
ZXJ2ZXJDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNv
bTA1BgNVHREELjAsghJhZGRvbnMubW96aWxsYS5vcmeCFnd3dy5hZGRvbnMubW96
aWxsYS5vcmcwDQYJKoZIhvcNAQEFBQADggEBADM7YxX8sewULJPddZTegVrZTpm+
+0qkOVVNoUB63hMqh6k3z+jV+63Re21vjCCHglTmV0m8ICiEzdYB2ZOLF24jZuWE
yIA/xqFwgOTsTR35/JFac2IpmvcgHGHgizmfyrx+jd282bHjn57fFVORIVIL2Roj
D2Y226yTlkqjpSLPKfeimaj2ttlArtl+tvZYLpusNspkj2VS3IacgqtuUEvaX/oF
AIgwDt6NVr+BR409BuKyYpJnj57ImrLlBrhwJLh3fCMKOMN5CNixUZ2slRHHQBee
oxyP8hGnaCfaSQWEGHxYLQFnXOWfoSm7SjlFL78Rqnmi7bTUtWVDt5NGitM=
-----END CERTIFICATE-----
Assignee: cviecco → mmc
Status: NEW → ASSIGNED
Comment on attachment 8417052 [details] [diff] [review]
Disable pinning for all mozilla properties (

Review of attachment 8417052 [details] [diff] [review]:
-----------------------------------------------------------------

I'm leaving soon, so I may not be able to check in myself.
Attachment #8417052 - Flags: review?(dkeeler)
Comment on attachment 8417052 [details] [diff] [review]
Disable pinning for all mozilla properties (

Review of attachment 8417052 [details] [diff] [review]:
-----------------------------------------------------------------

r=me. I don't have my private key on this machine, so I can't check this in either this weekend :/
I'll mark checkin-needed and hopefully a sheriff will get to it.

::: security/manager/tools/PreloadedHPKPins.json
@@ +93,5 @@
>    ],
>  
>    "entries": [
> +    // Disable until bug 1005653 is fixed.
> +    // { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },

I'm not sure there's much value in leaving these in commented-out, but since this is sufficient and time-sensitive, I think it's fine.

@@ +97,5 @@
> +    // { "name": "addons.mozilla.org", "include_subdomains": true, "pins": "mozilla" },
> +    // { "name": "addons.mozilla.net", "include_subdomains": true, "pins": "mozilla" },
> +    // { "name": "cdn.mozilla.net", "include_subdomains": true, "pins": "mozilla_cdn" },
> +    // { "name": "cdn.mozilla.org", "include_subdomains": true, "pins": "mozilla_cdn" },
> +    //{ "name": "media.mozilla.com", "include_subdomains": true, "pins": "mozilla_cdn" },

nit: space after "//"
Attachment #8417052 - Flags: review?(dkeeler) → review+
Attachment #8416963 - Attachment is obsolete: true
Flags: needinfo?(dkeeler)
(In reply to [:mmc] Monica Chew (please use needinfo) from comment #20)
> I found this cert in the CT database. This is not on any of the pinsets, so
> we need to disable until we understand what's going on. There are also some
> Digicerts CAs that we might need that aren't on the list.
> 
> CN:addons.mozilla.org
> ISSUER:UTN-USERFirst-Hardware
> DNS:addons.mozilla.org
> DNS:www.addons.mozilla.org

This one isn't legit, and we've already blacklisted it:
https://www.eff.org/deeplinks/2011/03/iranian-hackers-obtain-fraudulent-https
https://www.comodo.com/Comodo-Fraud-Incident-2011-03-23.html
https://mxr.mozilla.org/mozilla-central/source/security/nss/lib/ckfw/builtins/certdata.txt#19970
Reproduced on 2011 Macbook Air on the latest version of Nightly (2014-05-04)

Screnshot here: http://i.imgur.com/YvS5PUY.png


Build info:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:32.0) Gecko/20100101 Firefox/32.0

20140504030205
https://hg.mozilla.org/mozilla-central/rev/e07b3f07953c

Hopefully it's not too late to make it into nightly for tomorrow's update.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
This appears to be fixed in the 5/5 Nightly update.
Target Milestone: --- → mozilla32
Verified based on comment 31.
Status: RESOLVED → VERIFIED
status-firefox32: --- → verified
Just upgraded to FF35.0.1 ubuntu 12.04 32bits and I get this issue. Unchecking "Query OCSP responder to check the validity of your certicates" allows me to show the about:addons's Catalog page.
Re-enabling the responder still allows the page to be shown... until FF restart, then it fails agin;
A look in about:config with OCSP shows keys:
security.OCSP.GET.enabled default bool false
security.OCSP.enabled default integer 1
security.OCSP.require user defined bool true
security.ssl.enable_ocsp_stapling default bool true
services.sync.prefs.sync.security.OCSP.enabled default bool true
services.sync.prefs.sync.security.OCSP.require default bool true
PS: I have SSL Version Control 0.4 installed
PS: same in Windows XP : sec_error_revoked_certificate
PS: I was redirected here by dup https://bugzilla.mozilla.org/show_bug.cgi?id=1005655
f0rhum, if you're using 35, you're encountering a different issue. Please file a new bug here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM
Thanks!
Flags: needinfo?(f0rhum)
Flags: needinfo?(f0rhum)
Ok for me  in ff36
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: