1473573 - Enterprise roots mode should also import Intermediate CAs on Windows

Status: RESOLVED FIXED

(Core :: Security: PSM, enhancement, P1)

Description

[Christoph Pilgersdorfer]

Attachment: certificates.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36 OPR/53.0.2907.106

Steps to reproduce:

Installed Firefox 60.1.0 ESR. Open any website (www.google.com).
"pref("security.enterprise_roots.enabled", true);" is set in the mozilla.cfg file.
I create a new GPO and enroll the necessary certificate in the "Trusted Root Certification Authority"


Actual results:

The message "the connection is not secure" appears - because of the missing certificate.
The necessary certificate will be enroll automatically from the RootCA and imported in the "Intermediate Certification Authorities" store.
If the certificate is in the "Trusted Root Certificate Authorties" there is no message "the connection is not secure".



Expected results:

Firefox should import from all certificate stores not only from Trusted Root Certification Authorities.
Maybe we can get a option in the about:config to set the certificate store.

Comment 1

[Peter Magyari (Desktop QA)]

Hello Christoph, 

I want to verify this issue but I need more details about GPO,could you please add all the steps that would be necessary to create a new GPO so I can try and reproduce this issue?

Thank you.

Comment 2

[Christoph Pilgersdorfer]

Hello Peter,

thank you for your answer.

We have a development GPO linked to a OU in Active Directory.
I edit the GPO and add the certificates: Computer Configuration - Policies - Windows Settings - Security Settings - Public Key Policies - Trusted Root Certification Authorities. 
Rightclick on "Trusted Root Certification Authorities" choose "Import" - Local Machine - take the certificate.

(attachment)

All our enterprise certificates are in the "Intermediate Certification Authorities" but Firefox doesn't import.

Comment 3

[Christoph Pilgersdorfer]

Attachment: gpo.png

Comment 4

[Peter Magyari (Desktop QA)]

Thank you for the quick reply, I'm sorry but I have never worked with GPO and I don't know much about it. 
Is there a program I need to install in order to create a new GPO or how does that work? It would be very helpful if you could provide me with as many details as you can regarding the issue.

Thank you.

Comment 5

[Christoph Pilgersdorfer]

You need a Active Directory and use the program "Group Policy Management" to create a new GPO.
When you open the progam there should be a "Default Domain Policy"
If you don't want create a new one, you can use this policy. I write the steps again to add the certificates:

Computer Configuration - Policies - Windows Settings - Security Settings - Public Key Policies - Trusted Root Certification Authorities. 
Rightclick on "Trusted Root Certification Authorities" choose "Import" - Local Machine - get the certificate.

Comment 6

[Christoph Pilgersdorfer]

(In reply to Peter from comment #4)
> Thank you for the quick reply, I'm sorry but I have never worked with GPO
> and I don't know much about it. 
> Is there a program I need to install in order to create a new GPO or how
> does that work? It would be very helpful if you could provide me with as
> many details as you can regarding the issue.
> 
> Thank you.

I have new informations.
You don't need a active directory. 
Only a certificate which you import in the "Trusted Certification Authorities" Store.

But when i test the problem, the certificate don't appears in the certificates (preferences - privacy&security).
It seems like firefox not importing the certificates only using the windows certificate store, that would be great, BUT i want define which certificate store.

Comment 7

[Peter Magyari (Desktop QA)]

Thanks Christoph, I understand the STR, but unfortunately, we do not have access to our AD and creating or playing around with GPOs is not possible.
Even though we cannot reproduce it due to the above reasons, the issue logged makes sense to me, therefore let's assign the proper component and have the specialized people look at it in more detail.

Meanwhile, could you please test and report the Nightly Fx63 behavior(https://download.mozilla.org/?product=firefox-nightly-stub&os=win&lang=en-US) related to this issue?

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

If you set the environment variable MOZ_LOG to pipnss:4 and MOZ_LOG_FILE to e.g. %TEMP%/moz.log and run firefox, we might have a better idea of what's going on.

Comment 9

[Christoph Pilgersdorfer]

(In reply to Peter from comment #7)
> Thanks Christoph, I understand the STR, but unfortunately, we do not have
> access to our AD and creating or playing around with GPOs is not possible.
> Even though we cannot reproduce it due to the above reasons, the issue
> logged makes sense to me, therefore let's assign the proper component and
> have the specialized people look at it in more detail.
> 
> Meanwhile, could you please test and report the Nightly Fx63
> behavior(https://download.mozilla.org/?product=firefox-nightly-
> stub&os=win&lang=en-US) related to this issue?

Thank you!
I will test the Nightly Fx63 version and report this as bug.

Comment 10

[Christoph Pilgersdorfer]

(In reply to [:keeler] (use needinfo) from comment #8)
> If you set the environment variable MOZ_LOG to pipnss:4 and MOZ_LOG_FILE to
> e.g. %TEMP%/moz.log and run firefox, we might have a better idea of what's
> going on.

Hello!
How can i do this?

Comment 11

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Try this:
* Click the windows menu
* Type "cmd"
* Click "Command Prompt"
* In the resulting window, type "set MOZ_LOG=pipnss:4" and hit enter
* Type "set MOZ_LOG_FILE=%TEMP%\moz.log" and hit enter
* Type 'cd "C:\Program Files\Mozilla Firefox"' and hit enter
* Type "firefox.exe" and hit enter

Comment 12

[Christoph Pilgersdorfer]

(In reply to [:keeler] (use needinfo) from comment #11)
> Try this:
> * Click the windows menu
> * Type "cmd"
> * Click "Command Prompt"
> * In the resulting window, type "set MOZ_LOG=pipnss:4" and hit enter
> * Type "set MOZ_LOG_FILE=%TEMP%\moz.log" and hit enter
> * Type 'cd "C:\Program Files\Mozilla Firefox"' and hit enter
> * Type "firefox.exe" and hit enter

Thank you!
I attached the logfiles.

Comment 13

[Christoph Pilgersdorfer]

Attachment: moz_no_cert.log

Comment 14

[Christoph Pilgersdorfer]

Attachment: moz_with_cert.log

Comment 15

[Christoph Pilgersdorfer]

(In reply to Peter from comment #7)
> Thanks Christoph, I understand the STR, but unfortunately, we do not have
> access to our AD and creating or playing around with GPOs is not possible.
> Even though we cannot reproduce it due to the above reasons, the issue
> logged makes sense to me, therefore let's assign the proper component and
> have the specialized people look at it in more detail.
> 
> Meanwhile, could you please test and report the Nightly Fx63
> behavior(https://download.mozilla.org/?product=firefox-nightly-
> stub&os=win&lang=en-US) related to this issue?

The Firefox 63.0a1 has the same problem. I will report this related to the issue too.

Comment 16

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Hmmm - I guess I misunderstood the issue. Is the problem that your PKI hierarchy involves an intermediate that is both not sent by the server and not imported by Firefox when you enable the enterprise root preference?

Comment 17

[Christoph Pilgersdorfer]

(In reply to [:keeler] (use needinfo) from comment #16)
> Hmmm - I guess I misunderstood the issue. Is the problem that your PKI
> hierarchy involves an intermediate that is both not sent by the server and
> not imported by Firefox when you enable the enterprise root preference?

I think you are on the right way. I try to explain the problem again.
I have set the enterprise root preference to "true".
Our PKI enroll the certificates in the "Intermediate Certification Authorities" store.

But Firefox not look in this certificate store.
When i import the missing certificate via GPO in the "Trusted Root Certification Authorities" store, Firefox using the certificate. The certificate not listed under "Preferences - Privacy & Security - Vie Certificates... - Authorities", but it's working.
If i import the missing certificate manually in the "Trusted Root Certification Authorities" store, Firefox using the certificate and will be listed under "Preferences - Privacy & Security - Vie Certificates... - Authorities" and all works fine.

Maybe this is a bug, or a security feature?
Possible we can get a setting via about:config to set which certificate store Firefox should use?

Comment 18

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Firefox is expecting the server to include the necessary intermediate certificates, so it only looks for and imports root certificates.
(Also note that there is currently no UI that shows the imported enterprise certificates, so it not showing up in the certificate manager is the expected behavior.)
I think I'm going to wontfix this for now since this is the intended behavior.

Comment 19

[Mike Kaply [:mkaply]]

I'm going to reopen this because we continue to get this coming up.

https://github.com/mozilla/policy-templates/issues/291

Is there something simple we can do to resolve this?

Comment 20

[ccc]

Hello,

PLEASE can this be fixed? This issue is holding back Firefox in the enterprise environment and the overlords here are thinking about ripping out Firefox and replacing with Chrome in our image.

Issue here:
https://github.com/mozilla/policy-templates/issues/291

Comment 21

[ccc]

(In reply to Dana Keeler [:keeler] (she/her) (use needinfo) from comment #18)
> Firefox is expecting the server to include the necessary intermediate
> certificates, so it only looks for and imports root certificates.
> (Also note that there is currently no UI that shows the imported enterprise
> certificates, so it not showing up in the certificate manager is the
> expected behavior.)
> I think I'm going to wontfix this for now since this is the intended
> behavior.

Firefox is expecting the wrong behavior. Windows has two different storage locations for root certs and intermediates.
Firefox should look at both stores just like other browsers are!

Comment 22

[J.C. Jones [:jcj] (he/him)]

Just to be clear, is this the situation?

1) There is a self-signed root CA in the Trusted Root Certification Authorities store
2) There is an intermediate CA in the Intermediate Certification Authorities store
3) The web server(s) do not provide the intermediate CA certificate in their handshake

Or is it that:

1) There is an intermediate CA in the Intermediate Certification Authorities store
2) The web server(s) do provide the intermediate CA in the handshake

The difference being that in the second scenario, we would need to treat everything in the Intermediate Certification Authorities store as a root CA, while in the first scenario we'd need to pre-load everything in the Intermediate Certification Authorities store as a known certificate.

Comment 23

[ccc]

I believe we are caught in scenario 1 BUT the cert is not self signed. Its signed by our internal PKI. 

Firefox happily imports root certificates from "Trusted Root Certification Authorities" but NOT "Intermediate Certification Authorities" 

Screenshot here for clarification: https://user-images.githubusercontent.com/39263203/48569503-d736e580-e8f9-11e8-9e00-b2ebb2b78805.png


Without the intermediate certificate imported Firefox will not load the page.

Comment 24

[J.C. Jones [:jcj] (he/him)]

I am a little confused, and I'm afraid it matters to determine which remedy we need to do to enable this case for you.

In reply to ccc from comment #23)
> I believe we are caught in scenario 1 BUT the cert is not self signed. Its
> signed by our internal PKI. 

Is the "internal PKI" certificate in the Trusted Root Certification Authorities store?

> Firefox happily imports root certificates from "Trusted Root Certification
> Authorities" but NOT "Intermediate Certification Authorities" 

Right. But if the CA cert here is signed by a Root CA which _is_ in the Trusted Root Certification Authorities store, then a fast remedy is for the web server to provide the intermediate certificate in the handshake. (Which is technically mandatory anyway.)

However, if the situation is that you do not have the internal PKI certificate provided in the Windows stores, only the intermediate, and itself only in the Intermediate Certification Authorities store, then the only Firefox-side solution is to rework how the enterprise code functions. Right now it assumes that it is loading root trust anchors, not handling the side-effects of loading intermediates too.

Comment 25

[ccc]

Hello again,

Yes our trusted root cert is in the root Windows Trusted Root Store and our intermediate cert is in the Windows Intermediates Store.

If I import the intermediate cert into Firefox, everything works fine. 

Can the same import Firefox applies to the Trusted Root Store be applied to the Intermediates store?

This setup works perfectly well in all other major browsers. This isn't a shortcoming in our certificate setup.

Comment 26

[ccc]

(In reply to Dana Keeler [:keeler] (she/her) (use needinfo) from comment #16)
> Is the problem that your PKI
> "hierarchy involves an intermediate that is both not sent by the server and
> not imported by Firefox when you enable the enterprise root preference"

This is a great explanation of the issue my organisation and others are facing. 
This does work fine in other browsers because they are looking at the intermediates store.

Comment 27

[J.C. Jones [:jcj] (he/him)]

OK, in the mean time, if the server provides the intermediate in the handshake, that will solve your problem too. 

Marking this for the Q1 backlog.

Comment 28

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

For future reference, here's a quick outline of how someone might go about fixing this:

nsNSSComponent handles the process by calling GatherEnterpriseRoots. So
item 1 would be to maybe change that to "GatherEnterpriseCertificates"
or something and have the implementation look in more registry locations:

https://searchfox.org/mozilla-central/rev/647b9eb1099e373e079e16f147f74f3d1d65eed3/security/manager/ssl/EnterpriseRoots.cpp#187

Also the isRoot restriction would have to be lifted here:

https://searchfox.org/mozilla-central/rev/647b9eb1099e373e079e16f147f74f3d1d65eed3/security/manager/ssl/EnterpriseRoots.cpp#65

After gathering the certificates, nsNSSComponent changes their trust in
TrustLoaded3rdPartyRoots:

https://searchfox.org/mozilla-central/rev/647b9eb1099e373e079e16f147f74f3d1d65eed3/security/manager/ssl/nsNSSComponent.cpp#688

Item 2 would be to add a call to CERT_IsCACert and only set the trust
bits for certificates that are actually self-signed CAs (see also bug 1478826).

Also at that point "enterprise roots" wouldn't really be accurate, so it
might be good to rename those instances (e.g.
nsIX509CertDB.getEnterpriseRoots()) to refer to "certificates" instead
of "roots". That mostly only affects tests, though, so it's only
ourselves we're confusing if we don't address that.

Comment 30

[ccc]

Did anything happen here? I am trying the GPO on Firefox 65. No change in the issue.

Comment 31

[J.C. Jones [:jcj] (he/him)]

(In reply to ccc from comment #30)

Did anything happen here? I am trying the GPO on Firefox 65. No change in the issue.

No; you'll see this bug get resolved fixed and a version number. I'm afraid we don't have a target date yet, but we're working on one. Feel free to ping me outside the bug if you need more information.

Comment 32

[ghost.orchid2001]

This is my testing with Firefox ESR 60.5.0
I only activated 1 setting in GPO which is Import Enterprise Roots = Enabled

My server certificate is internally sign with the following hierarchy
root1 (root cert) - ca1 (intermediate cert) - server cert

root1 is imported to Trusted Root Certification Authorities Store
ca1 is imported to Intermediate Certificate Authorities Store
server cert is the signed server certificate, appended with the ca1 cert (intermediate)

(In reply to J.C. Jones [:jcj] (he/him) from comment #27)

OK, in the mean time, if the server provides the intermediate in the
handshake, that will solve your problem too.

Marking this for the Q1 backlog.

Even when the server provides the intermediate in the handshake, the problem still presist

the following is my log file
you can see that the root1 certificate is correctly imported

[2392:Main Thread]: D/pipnss nsNSSComponent::ctor
[2392:Main Thread]: D/pipnss Beginning NSS initialization
[2392:Main Thread]: D/pipnss nsNSSComponent::InitializeNSS
[2392:Main Thread]: D/pipnss NSS Initialization beginning
[2392:Main Thread]: D/pipnss NSS profile at 'C:\Users\Hendry.Leo\AppData\Roaming\Mozilla\Firefox\Profiles\J1H24C1.DEF'
[2392:Main Thread]: D/pipnss inSafeMode: 0
[2392:Main Thread]: D/certverifier InitializeNSS(sql:C:\Users\Hendry.Leo\AppData\Roaming\Mozilla\Firefox\Profiles\J1H24C1.DEF, 0, 1)
[2392:Main Thread]: D/pipnss initialized NSS in r/w mode
[2392:Main Thread]: D/pipnss AccountHasFamilySafetyEnabled?
[2392:Main Thread]: D/pipnss Users subkey not present - Parental Controls not enabled
[2392:Main Thread]: D/pipnss certificate is trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss Imported 'Microsoft Root Certificate Authority'
[2392:Main Thread]: D/pipnss certificate not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss skipping cert not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss certificate is trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss Imported 'Microsoft Root Authority'
[2392:Main Thread]: D/pipnss certificate is trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss Imported 'Microsoft Root Certificate Authority 2011'
[2392:Main Thread]: D/pipnss certificate not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss skipping cert not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss certificate is trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss Imported 'DigiCert High Assurance EV Root CA'
[2392:Main Thread]: D/pipnss certificate is trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss Imported 'Microsoft Root Certificate Authority 2010'
[2392:Main Thread]: D/pipnss certificate not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss skipping cert not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss certificate not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss skipping cert not trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss imported 5 roots
[2392:Main Thread]: D/pipnss certificate is trust anchor for TLS server auth
[2392:Main Thread]: D/pipnss Imported 'root1<removed local domain name>'*
[2392:Main Thread]: D/pipnss imported 1 roots
[2392:Main Thread]: D/pipnss imported 0 roots
[2392:Main Thread]: D/pipnss NSS Initialization done
[2392:Main Thread]: D/pipnss nsNSSComponent: adding observers
[2392:LoadRoots]: D/pipnss loaded CKBI from c:\PROGRA2\MOZILL1
[2392:Socket Thread]: D/pipnss [1F6E9FE0] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [1F6E9FE0] Socket set up
[2392:Socket Thread]: D/pipnss [1F6E9FE0] connecting SSL socket
[2392:Socket Thread]: E/pipnss [1F6E9FE0] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [1FFFD0C0] starting AuthCertificateHook
[2392:SSL Cert #1]: D/pipnss [0CF4FAB0] SSLServerCertVerificationJob::Run
[2392:SSL Cert #1]: D/certverifier Top of VerifyCert
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: IsChainValid
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #1]: D/certverifier OCSPCache::Get(2147F234,"") not in cache
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: no stapled OCSP response
[2392:SSL Cert #1]: D/certverifier OCSPCache::Get(2147F524,"") not in cache
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #1]: D/certverifier Setting up OCSP request: pre all path =/ pathlen=-1
[2392:SSL Cert #1]: D/pipnss nsNSSHttpRequestSession::trySendAndReceiveFcn to http://ocsp.digicert.com:80/
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: caching OCSP response
[2392:SSL Cert #1]: D/certverifier OCSPCache::Put(2147F524, "") added to cache
[2392:SSL Cert #1]: D/certverifier NSSCertDBTrustDomain: returning after VerifyEncodedOCSPResponse
[2392:SSL Cert #1]: D/pipnss AuthCertificate setting NEW cert 20B85C60
[2392:Socket Thread]: D/pipnss [1F6E9FE0] nsNSSSocketInfo::NoteTimeUntilReady
[2392:Socket Thread]: D/pipnss CanFalseStartCallback [1FFFD0C0] ok
[2392:Socket Thread]: D/pipnss [1FFFD0C0] HandshakeCallback: succeeded using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss HandshakeCallback KEEPING existing cert
[2392:Socket Thread]: D/pipnss [1F6E9FE0] nsNSSSocketInfo::SetHandshakeCompleted
[2392:Socket Thread]: D/pipnss [0A7A2060] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [0A7A2060] Socket set up
[2392:Socket Thread]: D/pipnss [0A7A2060] connecting SSL socket
[2392:Socket Thread]: E/pipnss [0A7A2060] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [0A745240] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [0A745240] Socket set up
[2392:Socket Thread]: D/pipnss [0A745240] connecting SSL socket
[2392:Socket Thread]: E/pipnss [0A745240] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [0A745BE0] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [0A745BE0] Socket set up
[2392:Socket Thread]: D/pipnss [0A745BE0] connecting SSL socket
[2392:Socket Thread]: E/pipnss [0A745BE0] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [0A745BE0] Shutting down socket
[2392:Socket Thread]: D/pipnss [0A745980] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [0A745980] Socket set up
[2392:Socket Thread]: D/pipnss [0A745980] connecting SSL socket
[2392:Socket Thread]: E/pipnss [0A745980] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [0A7A22C0] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [0A7A22C0] Socket set up
[2392:Socket Thread]: D/pipnss [0A7A22C0] connecting SSL socket
[2392:Socket Thread]: E/pipnss [0A7A22C0] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [0A7A2100] starting AuthCertificateHook
[2392:SSL Cert #2]: D/pipnss [0CF4FEF0] SSLServerCertVerificationJob::Run
[2392:SSL Cert #2]: D/certverifier Top of VerifyCert
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: IsChainValid
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #2]: D/certverifier OCSPCache::Get(224FF09C,"") not in cache
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: no stapled OCSP response
[2392:SSL Cert #2]: D/certverifier OCSPCache::Get(224FF38C,"") not in cache
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #2]: D/certverifier Setting up OCSP request: pre all path =/GTSGIAG3 pathlen=9
[2392:SSL Cert #2]: D/pipnss nsNSSHttpRequestSession::trySendAndReceiveFcn to http://ocsp.pki.goog:80/GTSGIAG3
[2392:Socket Thread]: D/pipnss [0A745520] starting AuthCertificateHook
[2392:SSL Cert #3]: D/pipnss [0A7E1230] SSLServerCertVerificationJob::Run
[2392:SSL Cert #3]: D/certverifier Top of VerifyCert
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: IsChainValid
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #3]: D/certverifier OCSPCache::Get(225FEECC,"") not in cache
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: no stapled OCSP response
[2392:SSL Cert #3]: D/certverifier OCSPCache::Get(225FF1BC,"") not in cache
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #3]: D/certverifier Setting up OCSP request: pre all path =/GTSGIAG3 pathlen=9
[2392:SSL Cert #3]: D/pipnss nsNSSHttpRequestSession::trySendAndReceiveFcn to http://ocsp.pki.goog:80/GTSGIAG3
[2392:Socket Thread]: D/pipnss [0A745B00] starting AuthCertificateHook
[2392:SSL Cert #4]: D/pipnss [0A7E1450] SSLServerCertVerificationJob::Run
[2392:SSL Cert #4]: D/certverifier Top of VerifyCert
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: IsChainValid
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #4]: D/certverifier OCSPCache::Get(226FF30C,"") not in cache
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: no stapled OCSP response
[2392:SSL Cert #4]: D/certverifier OCSPCache::Get(226FF5FC,"") not in cache
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #4]: D/certverifier Setting up OCSP request: pre all path =/GTSGIAG3 pathlen=9
[2392:SSL Cert #4]: D/pipnss nsNSSHttpRequestSession::trySendAndReceiveFcn to http://ocsp.pki.goog:80/GTSGIAG3
[2392:Socket Thread]: D/pipnss [0A7A2660] starting AuthCertificateHook
[2392:SSL Cert #5]: D/pipnss [0A7E1670] SSLServerCertVerificationJob::Run
[2392:SSL Cert #5]: D/certverifier Top of VerifyCert
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: IsChainValid
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #5]: D/certverifier OCSPCache::Get(227FF1CC,"") not in cache
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: Top of CheckRevocation
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: no stapled OCSP response
[2392:SSL Cert #5]: D/certverifier OCSPCache::Get(227FF4BC,"") not in cache
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: no cached OCSP response
[2392:SSL Cert #5]: D/certverifier Setting up OCSP request: pre all path =/GTSGIAG3 pathlen=9
[2392:SSL Cert #5]: D/pipnss nsNSSHttpRequestSession::trySendAndReceiveFcn to http://ocsp.pki.goog:80/GTSGIAG3
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: caching OCSP response
[2392:SSL Cert #2]: D/certverifier OCSPCache::Put(224FF38C, "") added to cache
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: returning after VerifyEncodedOCSPResponse
[2392:SSL Cert #2]: D/pipnss AuthCertificate setting NEW cert 0A7A0980
[2392:Socket Thread]: D/pipnss [0A7A2100] HandshakeCallback: succeeded using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss HandshakeCallback KEEPING existing cert
[2392:Socket Thread]: D/pipnss [0A7A2060] nsNSSSocketInfo::NoteTimeUntilReady
[2392:Socket Thread]: D/pipnss [0A7A2060] nsNSSSocketInfo::SetHandshakeCompleted
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: caching OCSP response
[2392:SSL Cert #3]: D/certverifier OCSPCache::Put(225FF1BC, "") already in cache - replacing
[2392:SSL Cert #3]: D/certverifier NSSCertDBTrustDomain: returning after VerifyEncodedOCSPResponse
[2392:SSL Cert #3]: D/pipnss AuthCertificate setting NEW cert 07656480
[2392:Socket Thread]: D/pipnss [0A745520] HandshakeCallback: succeeded using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss HandshakeCallback KEEPING existing cert
[2392:Socket Thread]: D/pipnss [0A745240] nsNSSSocketInfo::NoteTimeUntilReady
[2392:Socket Thread]: D/pipnss [0A745240] nsNSSSocketInfo::SetHandshakeCompleted
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: caching OCSP response
[2392:SSL Cert #4]: D/certverifier OCSPCache::Put(226FF5FC, "") already in cache - replacing
[2392:SSL Cert #4]: D/certverifier NSSCertDBTrustDomain: returning after VerifyEncodedOCSPResponse
[2392:SSL Cert #4]: D/pipnss AuthCertificate setting NEW cert 0A7D67C0
[2392:Socket Thread]: D/pipnss [0A745B00] HandshakeCallback: succeeded using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss HandshakeCallback KEEPING existing cert
[2392:Socket Thread]: D/pipnss [0A745980] nsNSSSocketInfo::NoteTimeUntilReady
[2392:Socket Thread]: D/pipnss [0A745980] nsNSSSocketInfo::SetHandshakeCompleted
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: caching OCSP response
[2392:SSL Cert #5]: D/certverifier OCSPCache::Put(227FF4BC, "") already in cache - replacing
[2392:SSL Cert #5]: D/certverifier NSSCertDBTrustDomain: returning after VerifyEncodedOCSPResponse
[2392:SSL Cert #5]: D/pipnss AuthCertificate setting NEW cert 0A745960
[2392:Socket Thread]: D/pipnss [0A7A2660] HandshakeCallback: succeeded using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss HandshakeCallback KEEPING existing cert
[2392:Socket Thread]: D/pipnss [0A7A22C0] nsNSSSocketInfo::NoteTimeUntilReady
[2392:Socket Thread]: D/pipnss [0A7A22C0] nsNSSSocketInfo::SetHandshakeCompleted
[2392:Socket Thread]: D/pipnss [0A745240] Shutting down socket
[2392:Socket Thread]: D/pipnss [0A745980] Shutting down socket
[2392:Socket Thread]: D/pipnss [0A7A22C0] Shutting down socket
[2392:Socket Thread]: D/pipnss [0A857160] nsSSLIOLayerSetOptions: using TLS version range (0x0301,0x0304)
[2392:Socket Thread]: D/pipnss [0A857160] Socket set up
[2392:Socket Thread]: D/pipnss [0A857160] connecting SSL socket
[2392:Socket Thread]: E/pipnss [0A857160] Lower layer connect error: -5934
[2392:Socket Thread]: D/pipnss [0A857EE0] starting AuthCertificateHook
[2392:SSL Cert #2]: D/pipnss [0A885450] SSLServerCertVerificationJob::Run
[2392:SSL Cert #2]: D/certverifier Top of VerifyCert
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #2]: D/certverifier NSSCertDBTrustDomain: CheckSignatureDigestAlgorithm
[2392:SSL Cert #2]: D/pipnss [0A857EE0][0A858460] Before dispatching CertErrorRunnable
[2392:Main Thread]: D/pipnss [0A857EE0][0A858460] top of CheckCertOverrides
[2392:Main Thread]: D/pipnss [0A857EE0][0A858460] no HSTS or HPKP - overrides allowed
[2392:Main Thread]: D/pipnss [0A857EE0][0A858460] Certificate error was not overridden
[2392:Socket Thread]: D/pipnss [0A857160] polling SSL socket right after certificate verification failed or NSS shutdown or SDR logout 6
[2392:Socket Thread]: D/pipnss [0A857160] Shutting down socket
[2392:Socket Thread]: D/pipnss [0A7A2060] Shutting down socket
[2392:Socket Thread]: D/pipnss [1F6E9FE0] Shutting down socket
[2392:Main Thread]: D/pipnss receiving profile change or XPCOM shutdown notification
[2392:Main Thread]: D/pipnss nsNSSComponent::ShutdownNSS
[2392:Main Thread]: D/certverifier OCSPCache::Clear: clearing cache
[2392:Main Thread]: D/pipnss receiving profile change or XPCOM shutdown notification
[2392:Main Thread]: D/pipnss nsNSSComponent::ShutdownNSS
[2392:Main Thread]: D/pipnss nsNSSComponent::dtor
[2392:Main Thread]: D/pipnss nsNSSComponent::ShutdownNSS
[2392:Main Thread]: D/pipnss nsNSSComponent::dtor finished

Comment 33

[ghost.orchid2001]

Attachment: server cert with intermediate.png

Comment 34

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to ghost.orchid2001 from comment #33)

Created attachment 9040312 [details]
server cert with intermediate.png

Can you file a new bug with as much detail as you can include? (Looking at what you've said so far, the quickest way for me to help you is if I can look at the actual certificates you're using.)

https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM

Thanks.

Comment 35

[ghost.orchid2001]

I have open new bug

https://bugzilla.mozilla.org/show_bug.cgi?id=1524903

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #34)

(In reply to ghost.orchid2001 from comment #33)

Created attachment 9040312 [details]
server cert with intermediate.png

Can you file a new bug with as much detail as you can include? (Looking at what you've said so far, the quickest way for me to help you is if I can look at the actual certificates you're using.)

https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM

Thanks.

Comment 36

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1473573 - import intermediate certificates as well as roots r?jcj

Comment 37

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

This is complicated enough that I'm going to break it up into Windows first and then MacOS.

Comment 38

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e93614576dfa110405cd830904d8873e3fdc95f0

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c8e523ac7349
import intermediate certificates as well as roots r=jcj

Comment 40

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/c8e523ac7349

Comment 41

[Mike Kaply [:mkaply]]

Is this upliftable to ESR and/or beta?

Comment 42

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I wouldn't be comfortable with that - there are a number of bugs we would have to uplift as well, and there's already been one regression.

Comment 43

[scott.driver42]

Enterprises need to leverage ESR for supportability and maintability. Not being able to centrally manage certificates is a major gap. Frankly this should be table stakes. I'd strongly ask you to reconsider bringing this back to ESR.

Comment 44

[J.C. Jones [:jcj] (he/him)]

(In reply to scott.driver42 from comment #43)

Enterprises need to leverage ESR for supportability and maintability. Not being able to centrally manage certificates is a major gap. Frankly this should be table stakes. I'd strongly ask you to reconsider bringing this back to ESR.

This would be declining to move a large number of patches back to ESR 60 (from here in 67). ESR 68 is the next train, starting mid-month, and this (and everything else since 60) would be included. See https://wiki.mozilla.org/Release_Management/Calendar

Comment 46

[Ken Kirchner]

I just tested FF 67.0b8 in my lab and can confirm that it works correctly, loading the intermediate and root certs from the Windows OS store. I have the enterprise_roots.enable set to true in both a mozilla.cfg file and via the Firefox ADMX file. Both are pushed via GPO.

Comment 48

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.