EV/OV status of website is not displayed when using publicproxy1.scl3.mozilla.com due to hard limitation to mozqa.com

RESOLVED FIXED

Status

Infrastructure & Operations
Infrastructure: Other
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: whimboo, Assigned: lerxst)

Tracking

other
Bug Flags:
sec-review -

Details

(Whiteboard: [qa-automation-blocked])

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
Created attachment 8529769 [details]
screenshot.png

With bug 917204 we now have a public proxy, which we can use for testing Mozilla web properties as hosted on mozqa.com. By testing the site https://ssl-ev.mozqa.com/ I have seen that the EV certificate details are not shown in the UI. That also means that we do not show a green label!

Steps to reproduce:
1. Open https://ssl-ev.mozqa.com/ to verify you see the EV status
2. Configure the proxy publicproxy1.scl3.mozilla.com on port 3128
3. Clear all the recent history including the cache
4. Open https://ssl-ev.mozqa.com/ again

With step 4 you will not see that the page offers an EV certificate. Also the one who runs this site is marked as unknown.

This is happening across all platforms and any supported version of Firefox. To ensure that only Firefox is affected, I tested with Internet Explorer on Windows, which is working as expected.

Please also have a look at the attached screenshot. As you can see the certificate viewer also shows the EV status of the certificate. So we have a perfect mismatch in our UI.

[Tracking Requested - why for this release]:
This bug affects how people can trust the website (bank, etc.) in use. Not sure for which versions drivers would want to block on. So keeping all flags set for now.
I don't think we should block on this for 34. Is this a regression? How far back does this issue go?
on 33 this is pretty much wfm.

My guess is that you configured firefox to use the proxy for all requests and it will not serve the OCSP request for ssl-ev.mozqa.com (because it only serves mozilla resources) as that would require contacting the CA if it wasn't stapled.. we'll still do https without ocsp succeeding, but will downgrade EV to DV in that case.

The ocsp handling has a cache - so funny things can happen when testing without restarts.

interestingly on 36 I don't see ev for ssl-ev.mozqa.com at all - even direct. I don't know why.. you might want to check with dkeeler

I'm not sure what you are pointing out in the screen shot.. the words "extended validation" do appear - but they're just in the name of the CA.
(In reply to Patrick McManus [:mcmanus] from comment #2)
> 
> 
> interestingly on 36 I don't see ev for ssl-ev.mozqa.com at all - even
> direct. I don't know why.. you might want to check with dkeeler
> 

and now it does show ev. can't explain it. (I had restarted before to confirm as well)
(Reporter)

Comment 4

3 years ago
(In reply to Patrick McManus [:mcmanus] from comment #2)
> My guess is that you configured firefox to use the proxy for all requests
> and it will not serve the OCSP request for ssl-ev.mozqa.com (because it only
> serves mozilla resources) as that would require contacting the CA if it
> wasn't stapled.. we'll still do https without ocsp succeeding, but will
> downgrade EV to DV in that case.

Oh! That would be a good explanation for it. So it might be a ACL configuration problem for the proxy because it only allows connections to mozqa.com? Does it mean we would also have to add the domains like digicert, to allow the OCSP request to succeed? Which protocol is used for it? I assume this also applies to OV?

So all this might indeed not be a bug in Firefox then. We can move this to the correct component once I got the feedback to my question above. Thanks.
Flags: needinfo?(mcmanus)
you could consider it a proxy config problem, or you could consider it a firefox config problem (for routing stuff to the proxy that it isn't configured for).

the ocsp url is definied in the cert used by mozqa.com - I don't think it surfaces in the UI though I might be wrong about that. It is generally plaintext http to avoid a boostrapping problem (the ocsp response itself is signed).

I believe you're right about OV, but I'm not positive. keeler would know - or just try it out.
Flags: needinfo?(mcmanus)
I'm going to mark tracking- as my understanding is that this issue is with mozqa.com and not something that we're going to fix in Firefox. Can this bug be resolved as invalid?
status-firefox33: wontfix → unaffected
status-firefox34: affected → unaffected
status-firefox35: affected → unaffected
status-firefox36: affected → unaffected
status-firefox-esr31: affected → unaffected
tracking-firefox34: ? → -
tracking-firefox35: ? → -
tracking-firefox36: ? → -
tracking-firefox-esr31: ? → -
(Reporter)

Comment 7

3 years ago
(In reply to Patrick McManus [:mcmanus] from comment #5)
> you could consider it a proxy config problem, or you could consider it a
> firefox config problem (for routing stuff to the proxy that it isn't
> configured for).

It works at least for the Internet Explorer when the proxy is set. So given that this works, how should it be a proxy problem? Or maybe IE doesn't obey the proxy settings and tries via a direct connection?

> the ocsp url is definied in the cert used by mozqa.com - I don't think it
> surfaces in the UI though I might be wrong about that. It is generally
> plaintext http to avoid a boostrapping problem (the ocsp response itself is
> signed).
> 
> I believe you're right about OV, but I'm not positive. keeler would know -
> or just try it out.

David, would you mind to have a read-through on this bug, and give us your opinion? I'm a bit puzzled and don't really know where to continue here. Thanks.
Flags: needinfo?(dkeeler)
Are there instructions for configuring that proxy? It seems to be refusing connections for me (even after connecting to the VPN).
Flags: needinfo?(dkeeler) → needinfo?(hskupin)
(Reporter)

Comment 9

3 years ago
David, you don't have to be on VPN. It's a public proxy. All what you have to do for configuration is listed in comment 0. The proxy is the same for all protocols. Please be aware that you cannot test any other website beside *.mozqa.com. Maybe you were seeing this problem?
Flags: needinfo?(hskupin)
Oh, that's probably what was going on. Thanks!
Anyway, comment 2 is a good explanation of what's happening here. Not being able to connect to hosts other than *.mozqa.com means no OCSP, which means no EV. We could certainly improve the UX of this. One idea would be to expose OCSP requests in the network monitor.
(Reporter)

Comment 11

3 years ago
That sounds good then. Thanks David! But one thing is remaining for me here... how do I easily find out the OSCP domains and ports we have to white-list to allow successful connections to the following to domains?

https://ssl-ev.mozqa.com/
https://ssl-ov.mozqa.com/

Given that both are from DigiCert, will we have to allow for www.digicert.com:80?
Assignee: nobody → infra
Blocks: 917204
Component: Security: UI → Infrastructure: Other
Flags: needinfo?(dkeeler)
Product: Core → Infrastructure & Operations
QA Contact: jdow
Summary: EV status of website is not displayed if going through a proxy → EV/OV status of website is not displayed when using publicproxy1.scl3.mozilla.com due to hard limitation to mozqa.com
Whiteboard: [qa-automation-blocked]
Version: Trunk → other
This is paging the onduty staff. I'm going to see if I can get someone to look at this, but bringing the status down to stop the SMS's
Severity: major → normal
(Reporter)

Comment 13

3 years ago
Sorry Ryan, totally missed to lower the severity for this move to infra.
Looks like the host/port you'll have to allow is ocsp.digicert.com:80 (in general, in the certificate viewer, if you scroll down to the "Authority Information Access" extension in the details pane, the OCSP URI has the information).
Flags: needinfo?(dkeeler)
(Reporter)

Comment 15

3 years ago
(In reply to David Keeler [:keeler] (use needinfo?) from comment #14)
> Looks like the host/port you'll have to allow is ocsp.digicert.com:80 (in

Great information! Thanks for letting me know where this can be found.

Dan, can you help us to get this additional network setting setup? We would appreciate that! Thanks
Flags: needinfo?(dparsons)
This is our current OCSP list from the production Squid cluster:

  cdp1.public-trust.com
  ocsp.omniroot.com
  ocsp.msocsp.com
  www.msftncsi.com
  msftncsi.com
  ipv6.msftncsi.com
  ocsp.geotrust.com
  crl.geotrust.com
  gtssl-ocsp.geotrust.com
  gtssl-crl.geotrust.com

:ulfr, these are in production's 'global_allow_domains.txt.erb'. Should we enable global worldwide proxying to the known OCSP endpoints?
Flags: sec-review?(jvehent)
(Assignee)

Comment 17

3 years ago
:whimboo, let's see what :ulfr says, and I'll add everything suggested.
Flags: needinfo?(dparsons)
All our certs come from digicert, so we should just open their OCSP resolver: http://ocsp.digicert.com

That being said, all of our web properties hosted on ZLBs support OCSP Stapling, and thus Firefox should obtain the OCSP response from our load balancers, and not connect to digicert. We shouldn't have to open anything unless the target site is not hosted on ZLB and thus does not provide OCSP Stapling (AMO, Marketplace, everything in AWS).

I would recommend to verify that the offending site does not support OCSP Stapling, and then only open ocsp.digicert.com.

Updated

3 years ago
Flags: sec-review?(jvehent) → sec-review-
another alternative is to configure the brower, via PAC, to go DIRECT for ocsp.digicert.com
(Reporter)

Comment 20

3 years ago
(In reply to Patrick McManus [:mcmanus] from comment #19)
> another alternative is to configure the brower, via PAC, to go DIRECT for
> ocsp.digicert.com

This proxy is mostly used for testing, specifically with Mozmill and Marionette. So using PAC would not work because it would break when the test is using system settings or the manual configuration.

(In reply to Julien Vehent [:ulfr] (use needinfo) from comment #18)
> I would recommend to verify that the offending site does not support OCSP
> Stapling, and then only open ocsp.digicert.com.

All what I personally have is the information as listed on:
https://mana.mozilla.org/wiki/display/websites/mozqa.com

Dan, I think that you will be a better fit to give an update here. Thanks.
Flags: needinfo?(dparsons)
I tested the sites listed on that mana page and while they are all hosted on the ZLB, not all of them have OCSP Stapling. That might be expected for sslv2.mozqa.com, and just be a missing parameter for the others. Can you confirm that you're issue the error on a site that does have OCSP Stapling enabled?

$ for t in ssl-dv-mozqa-zlb.vips.scl3.mozilla.com ssl-md5-mozqa-zlb.vips.scl3.mozilla.com ssl-ov-mozqa-zlb.vips.scl3.mozilla.com ssl-selfsigned-mozqa-zlb.vips.scl3.mozilla.com ssl-unknownissuer-mozqa-zlb.vips.scl3.mozilla.com ssl-ev-mozqa-zlb.vips.scl3.mozilla.com sslv2.mozqa.com sslv3.mozqa.com tlsv1.mozqa.com tlsv1-1.mozqa.com summitbookmozillaorg-mozqa-zlb.vips.scl3.mozilla.com; do echo -n "$t: "; openssl s_client -connect $t:443 -status 2>&1 <<< 'Q' |grep 'OCSP Response Status'||echo No OCSP Stapling; done|column -t -o " "

ssl-dv-mozqa-zlb.vips.scl3.mozilla.com:               OCSP Response Status:  successful (0x0)
ssl-md5-mozqa-zlb.vips.scl3.mozilla.com:              No   OCSP     Stapling
ssl-ov-mozqa-zlb.vips.scl3.mozilla.com:               OCSP Response Status:  successful (0x0)
ssl-selfsigned-mozqa-zlb.vips.scl3.mozilla.com:       No   OCSP     Stapling
ssl-unknownissuer-mozqa-zlb.vips.scl3.mozilla.com:    No   OCSP     Stapling
ssl-ev-mozqa-zlb.vips.scl3.mozilla.com:               OCSP Response Status:  successful (0x0)
sslv2.mozqa.com:                                      No   OCSP     Stapling
sslv3.mozqa.com:                                      OCSP Response Status:  successful (0x0)
tlsv1.mozqa.com:                                      OCSP Response Status:  successful (0x0)
tlsv1-1.mozqa.com:                                    OCSP Response Status:  successful (0x0)
summitbookmozillaorg-mozqa-zlb.vips.scl3.mozilla.com: No   OCSP     Stapling
Unfortunately, OCSP stapling alone won't fix this. To verify the end-entity certificate for EV, we must also have fresh revocation information for the intermediate certificate. Since OCSP multi-stapling isn't a thing yet, this means making an OCSP request for the intermediate.
That makes sense. In this case, opening ocsp.digicert.com in the proxy is the only option.
(Assignee)

Comment 24

3 years ago
OK, I've added ocsp.digicert.com to the allowed domains. Please test and let me know if everything is working correctly.
Assignee: infra → dparsons
Flags: needinfo?(dparsons)
(Reporter)

Comment 25

3 years ago
Dan, don't we need an update of the network ACLs we have set? I think adding it alone to the allowed domains does not fix it. I still get an access denied.
(Assignee)

Comment 26

3 years ago
Please try again and let me know if it's working now?
(Reporter)

Comment 27

3 years ago
That works great now! Thanks a lot for fixing it that quickly. Dan, do we have documentation about this proxy somewhere?
Status: NEW → ASSIGNED
(Assignee)

Comment 28

3 years ago
This proxy isn't that different from any of the other squid installs we have, and all the relevant information for the service is contained in its puppet node definition.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.