Closed Bug 1141048 Opened 9 years ago Closed 8 years ago

[Stingray] cannot load resource from *.cdn.mozilla.net

Categories

(Core :: Security: PSM, defect)

Product:
Core
Component:
Security: PSM
Platform:
Other
Other
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: ktoiyama.moz, Unassigned)

References

Details

(Whiteboard: [stingray])

Attachments

(5 files)

Screenshot from 2015-03-09 15_40_01.png
232.24 KB, image/png
Details
Issue explanation document : screen capture
487.25 KB, application/pdf
Details
[snapshot] mozilla-pkix-error-key-pinning-failure
174.80 KB, image/png
Details
[fail log for TV] pinning_tv.log
642.68 KB, text/x-log
Details
[success log for desktop (gecko 34)]pinning_b2g34.log
5.17 MB, text/x-log
Details 
When mozilla.org is accessed from stingray device browser, the layout is not loaded correctly. Pages which should be shown correctly (as minimum) are 

    1) https://www.mozilla.org/en-US/
    2) https://marketplace.firefox.com/privacy-policy
It seems the CSS and JS files from 'https://mozorg.cdn.mozilla.net/' are failed to load.
From the same device, I have tried to access several non-mozilla https sites and all displays without any problems.  I don't have a list of sites I have tested but it was mainly banking site and telephone operator login page and etc.
Hi Toiyama-san, could you help capture the NSPR log for us to analyze the behavior of HTTP? Please turn on  NSPR_LOG_MODULES=nsHttp:5. We can check why HTTP transaction stop for mozorg.cdn.mozilla.net.
Hi SC.  I just sent it to you.  Could you please take a look at it?
An NSS error page will be displayed if we directly make XMLHttpRequest to resources on https://mozorg.cdn.mozilla.net.

Full error description:
An error occurred during a connection to mozorg.cdn.mozilla.net. The Server uses key pining (HPKP) but no trusted certificate chain could be constructed that matches the pinset. Key pinning violations cannot be overriden. (Error code: mozilla_pkix_error_key_pining_failure)
Thank you.  We are checking which certificate are causing problem.
According to this wiki [1], *.cdn.mozilla.net is one of the pinned site since Firefox 32. It's weird to see this error cause because stingray is based on Gecko 34.

[1] https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning
In our code base, returning false in |PublicKeyPinningService::ChainHasValidPins| [1] is the only case that can generate this error. You can turn on NSPR_LOG_MODULES=PublicKeyPinningService:5 to see detailed debug information.

[1] https://dxr.mozilla.org/mozilla-central/source/security/manager/boot/src/PublicKeyPinningService.cpp#357
Hi SC.  Thank you so much for all your help.  After investigation on our side, it seems the browser was using the unintended certificate for mozilla.org access. The certificate is needed for particular service partner for an app and can not remove.

We changed security.cert_pinning.enforcement_level from 2 to 1 and the problem seems to be resolved.  I hope this solution is OK.
I don't think I can make the judgement because I'm not familiar with our security module. @mmc should be more capable of answering your question.

@mmc, partner installs additional certificate and it breaks the *.cdn.mozilla.org from loading correctly. They found changing security.cert_pinning.enforcement_level from 2 to 1 will solve this issue. Could you help comment if this solution is safe to go? Thanks.
Flags: needinfo?(mmc)
Component: Networking: HTTP → Security: PSM
Summary: Mozilla.org site can not be displayed correctly on stingray device. → [Stingray] cannot load resource from *.cdn.mozilla.net
Whiteboard: [stingray]
Blocks: 1059202
Flags: needinfo?(mmc)
Hi SC,

The default security.cert_pinning.enforcement_level in FF desktop is set to 1 for this very reason. It looks like in b2g it was changed to 2 in bug 1059202 (http://hg.mozilla.org/mozilla-central/diff/07cdb131dccb/b2g/app/b2g.js). I would not have recommended changing the default on b2g.

There is an additional problem, which is that I thought that pinning was not enforced at all on b2g because we can't control the update cycle, and pin lifetimes should only be 8 weeks from stable release date. In fact I restricted CA pinning to desktop in bug 1012882 (and expanded it to fennec in bug 1019255).

Zoran, could you tell me how b2g came to enforce pinning at all?

FYI rbarnes and keeler.

Thanks,
Monica
Flags: needinfo?(zoran.jovanovic)
Flags: needinfo?(rlb)
Flags: needinfo?(dkeeler)
Monica, for what it's worth, it's fine to turn b2g back to default since no one is using THA.
Yeah, I agree it is fine to make b2g less strict about pinning. I'm only questioning why we're enforcing pinning on b2g in the first place!
If I recall correctly, a partner had a requirement that they be able to specify pins for hosts (or maybe we required it of them to use trusted hosted apps?). The change in comment 10 is a reasonable solution if that requirement is still in play.

Another question is why are pins failing at all? Is some sort of https-proxying going on? Or is that pinset incorrect? (in which case we should definitely disable and/or fix pinning on b2g for the time being)
Flags: needinfo?(dkeeler)
I just chatted to Gregor and explained that pins are supposed to live for only 8 weeks after hitting release. He agreed that it makes no sense to enforce pins on b2g. If that's the case then we just rollback this change: http://hg.mozilla.org/mozilla-central/diff/07cdb131dccb/b2g/app/b2g.js
to default the enforcement level to 0 (don't enforce pins).

ni'ing Gregor to make sure.
Flags: needinfo?(anygregor)
Jonas reviewed bug 1059202. Do you see any reason why we can't go back to default?
Flags: needinfo?(anygregor) → needinfo?(jonas)
(In reply to Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz) from comment #11)
> @mmc, partner installs additional certificate and it breaks the
> *.cdn.mozilla.org from loading correctly. They found changing
> security.cert_pinning.enforcement_level from 2 to 1 will solve this issue.
> Could you help comment if this solution is safe to go? Thanks.

Simply installing an additional root certificate wouldn't cause connections to *.cdn.mozilla.org to use a different certificate.

Shih-Chiang, can you verify with the partner what changes they are actually making that is causing this problem?
Flags: needinfo?(jonas)
Hi Jonas, partner is now at the very last minute to have the final firmware, if there's no big concern setting the level back to 1 and partner confirmed the issue can be solved, can we just go with the solution at this moment?

But we definitely should continue to find out the root cause with partner and put the fix (if any) in the next version.
Flags: needinfo?(jonas)
(In reply to Jonas Sicking (:sicking) from comment #18)
> (In reply to Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz) from comment
> #11)
> > @mmc, partner installs additional certificate and it breaks the
> > *.cdn.mozilla.org from loading correctly. They found changing
> > security.cert_pinning.enforcement_level from 2 to 1 will solve this issue.
> > Could you help comment if this solution is safe to go? Thanks.
> 
> Simply installing an additional root certificate wouldn't cause connections
> to *.cdn.mozilla.org to use a different certificate.

The pinset for cdn.mozilla.org is correct, I was able to fetch https://mozorg.cdn.mozilla.net/media/css/tabzilla-min.1f0fba45b96e.css in Nightly (and I double-checked that its cert is in http://mxr.mozilla.org/mozilla-central/source/security/manager/boot/src/StaticHPKPins.h), so there must be some proxying going on using to cause a pinning violation.

Could Howie or ktoiyama explain if this device is proxying HTTPS traffic deliberately and why? Comment 10 is troubling because it implies that the proxying may be unintended from a partner app. We don't want another superfish incident.
Hi Monica, do you have any suggested NSPR log set for debugging this? I can setup the test environment locally so we can gather information more quickly.
Flags: needinfo?(mmc)
PublicKeyPinningService:5,certverifier:5

nsHttp:5 might be helpful as well.
Flags: needinfo?(mmc)
Hi SC

The problem is caused by having DigiCert High Assurance CA-3 in our system.  It can be downloaded from link below.

http://cacerts.digicert.com/DigiCertHighAssuranceCA-3.crt

Could you try if you can reproduce this on your device as well?
No, finding out what's causing this problem is absolutely a blocker issue.

The fix might be to simply revert the pref change, but we need to understand what's causing this problem in order to tell.
Flags: needinfo?(jonas)
(In reply to [:mmc] Monica Chew (please use needinfo) from comment #12)
> Hi SC,
> 
> The default security.cert_pinning.enforcement_level in FF desktop is set to
> 1 for this very reason. It looks like in b2g it was changed to 2 in bug
> 1059202
> (http://hg.mozilla.org/mozilla-central/diff/07cdb131dccb/b2g/app/b2g.js). I
> would not have recommended changing the default on b2g.
> 
> There is an additional problem, which is that I thought that pinning was not
> enforced at all on b2g because we can't control the update cycle, and pin
> lifetimes should only be 8 weeks from stable release date. In fact I
> restricted CA pinning to desktop in bug 1012882 (and expanded it to fennec
> in bug 1019255).
> 
> Zoran, could you tell me how b2g came to enforce pinning at all?
> 
> FYI rbarnes and keeler.
> 
> Thanks,
> Monica

Trusted Hosted Apps required that all traffic with THA is funneled through domains with preloaded pins (on FFOS device this would be controlled in factory and OTA). As Fabrice said in Comment 13, since no one is using THA it's probably OK to relax the preference, but then also Jonas is right in Comment 24 that it's first important to understand why this particular pin didn't work.
Flags: needinfo?(zoran.jovanovic)
Here is the relevant log for checking key pinning on cdn.mozilla.net. Monica, does this means we need to pre-install more certificates?

606861296[241d2880]: Top of VerifyCert
1361  606861296[241d2880]: verifycert: Inside the Callback
1362  606861296[241d2880]: BuiltInRoot? subject=CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US token=Software Security Device
1363  606861296[241d2880]: pkpin: Querying pinsets for host: 'mozorg.cdn.mozilla.net'
1364  606861296[241d2880]: pkpin: Didn't find pinset for host: 'mozorg.cdn.mozilla.net'
1365  606861296[241d2880]: pkpin: Querying pinsets for host: 'cdn.mozilla.net'
1366  606861296[241d2880]: pkpin: Found pinset for host: 'cdn.mozilla.net'
1367  606861296[241d2880]: pkpin: certArray subject: 'CN=*.cdn.mozilla.net,OU=Technology,O=Mozilla Corporation,L=Mountain View,ST=California,C=US'
1368  606861296[241d2880]: pkpin: certArray common_name: 'DigiCert High Assurance CA-3'
1369  606861296[241d2880]: pkpin: certArray subject: 'CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US'
1370  606861296[241d2880]: pkpin: certArray common_name: 'DigiCert High Assurance EV Root CA'
1371  606861296[241d2880]: pkpin: no matches found
1372  606861296[241d2880]: pkpin: Pin check failed for mozilla host 'mozorg.cdn.mozilla.net' (mode=production)
Flags: needinfo?(mmc)
I also found that MOZ_NATIVE_NSS is turned on by partner, which means this device might not have all the built-in CA as in Firefox.
Thanks for the logs, SC. I chatted with keeler about this. We looked at the logs together and found that it does *not* look like any traffic is being proxied. Rather, what is happening is that Digicert CA-3 is normally an intermediate, so when it is installed as a trust anchor the verification checks stop there (even though it is signed by a built-in, Digicert High Assurance EV) rather than continuing up the chain. This is an unsupported configuration for pinning.

David was also able to reproduce the error on desktop by manually adding trust bits to the Digicert CA-3 cert (which shows up as a cached intermediate in the cert manager UI), so we're pretty sure that's what ths problem is.

I still don't understand the partner requirement for installing this intermediate cert as a root (since it's signed by a built-in root already) from comment 10, but if this is required I recommend the following:

1) Remove the trust bit from Digicert CA 3. If that fixes the problem and the partner app still works, you should do this. If you need help removing the trust bit, keeler can help you.

2) Disable pinning by setting the security.cert_pinning.enforcement_level to 0. In reality, users will never see the error from comment 0 because the pin lifetimes will time out before devices ever reach them. So there's no point in trying to enforce pins on b2g.

Please try 1) before doing 2), so we can verify we understand the root cause.

Thanks,
Monica
Flags: needinfo?(mmc)
I'm glad we are on the way to tracking this down, but MOZ_NATIVE_NSS is a problem as well. It may not be possible to fix this before the partner ships v1, but it needs fixing. Our partner agreements should have tight controls on changes to the root store; what the partner has done here is wholesale throw out our root store and replace it with their own!

Gerv
Toiyama-san, our analysis result is pretty close to yours. Please try the proposal in comment #30. Thanks!
Flags: needinfo?(ktoiyama.moz)
The modification is being tested in UK because the partner who requested is uk broadcaster. Please wait untill we hear the test result from uk.
Flags: needinfo?(ktoiyama.moz)
Flags: needinfo?(rlb)
Is this still an issue?
Flags: needinfo?(ktoiyama.moz)
https://marketplace.firefox.com/privacy-policy gives error but the issue is not the same as originally reported.  Closing this bug.
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(ktoiyama.moz)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: