Closed Bug 474606 Opened 11 years ago Closed 10 years ago

EV problem with missing CRLs - AMO EV cert treated as DV (ie GlobalSign)

Categories

(Core :: Security: PSM, defect, critical)

defect
Not set
critical

Tracking

()

RESOLVED FIXED
Tracking Status
status1.9.1 --- ?

People

(Reporter: jmjjeffery, Assigned: KaiE)

References

Details

(Keywords: regression)

Attachments

(1 file)

Visiting AMO the EV ID (Green) is not showing in latest build. 

Regress: 

======================================================================
1/20/09 2253hrs build  Bad
changeset: http://hg.mozilla.org/mozilla-central/rev/44d8c5d92a44

1/20/09 1916hrs build  OK
changeset: http://hg.mozilla.org/mozilla-central/rev/cbb494ae43cd
The only change in the window is Kai's landing of the new NSS_3_12_3_BETA2 tag.

Kai - anything in that landing that seems likely to have demoted addons.mozilla.org's certificate?  Paypal is apparently still getting EV recognition (right Jim?)
Blocks: 473837
Summary: AMO EV ID is not working - Blue instead of Green → NSS_3_12_3_BETA2 landing breaks AMO EV cert - Treated as DV
Yes, Paypal still renders the EV OK
(In reply to comment #0)
> Visiting AMO the EV ID (Green) is not showing in latest build. 

When looking closer at the certificate for addons.mozilla.org, what is actually surprising is that previous versions of Firefox 3.x *do show* the EV UI for this certificate - because it doesn't include any OCSP responder URI, so NSS can't retrieve fresh revocation info and PSM should therefore treat it as non-EV.

I assume that one of the fixes which went into NSS_3_12_3_BETA2 now has the (correct, IMO) side effect that the AMO cert is only treated as a standard SSL cert. See also bug 405139 for further background.
bug 405139 describes our behaviour when an OCSP responder IS supplied, but our attempt to retrieve an OCSP response fails.  It doesn't cover the case where OCSP isn't supplied.  (See also, e.g. www.networksolutions.com).
(In reply to comment #4)
> bug 405139 describes our behaviour when an OCSP responder IS supplied, but our
> attempt to retrieve an OCSP response fails.

More exactly, it's dealing with the case where OCSP checking has been disabled by the user.

From a cert validation standpoint, however, this distinction is irrelevant - as long as PSM/NSS does not have fresh revocation info, it must not show the EV UI for a certificate (otherwise Mozilla may run into liability issues, see bug 405139 comment 12).
When we wrote the specification for the new cert verification API for 
libPKIX in NSS 3.12, we gave it a rather large number of option flags 
that the caller may set to very precisely specify the desired revocation 
behavior.  These flags control behavior for such things as the presence
or absence of extensions, presence or absence of "fresh" CRLs or OCSP 
responses, order of checking CRLs vs OCSP, etc. etc.  In NSS 3.12.0 
NSS did not honor all of those flags.  Some were still unimplemented.

In NSS 3.12.3 (of which FF is now testing a Beta), those flags are 
(more) fully implemented.  NSS is now giving the browser the behavior 
that the browser has been requesting all along.  And it turns out that 
this may not be the behavior that the browser actually wants.  So the
question before the browser folks is "What behavior do we really want 
for this case?". 

I believe Kai is writing up some discussion of this subject.
I should clarify/correct one thing I wrote.  NSS still does not implement
fetching of CRLs from URLs found in CrlDP extensions in certs.  That is 
the one remaining thing it does not do that the browser asks it to do in
those option flags.
PSM code requests stricter checking than old NSS was able to perform.
In order to restore the old behavior in conjunction with the new NSS landing, a change to PSM will be required, that explicitly allows missing OCSP source information.
To recap: prior to the change from bug 473837, Firefox 3.x was using NSS 3.12.x, where the new revocation API (bug 294531) has already been available.

And nsNSSCertificate::hasValidEVOidTag is using the following method-independent flags (http://mxr.mozilla.org/mozilla/ident?i=revMethodIndependentFlags):

  PRUint64 revMethodIndependentFlags =
    CERT_REV_MI_TEST_ALL_LOCAL_INFORMATION_FIRST
    | CERT_REV_MI_REQUIRE_SOME_FRESH_INFO_AVAILABLE;

which is supposed to have the following semantics:

 * REQUIRE_SOME_FRESH_INFO_AVAILABLE means:
 *     After the individual tests have been executed, we must have
 *     been able to find fresh information using at least one method
 *     If we were unable to find fresh info, it's a failure.

... and which makes perfect sense (for EV certs, you really want REQUIRE_SOME_FRESH_INFO_AVAILABLE). Most likely there was a deficiency in the revocation checking in NSS 3.12.0 through 3.12.2 - which has now been fixed, so CERT_REV_MI_REQUIRE_SOME_FRESH_INFO_AVAILABLE is working as expected.

Bug 413997 is about a similar case, and I assume it's also addressed by this NSS upgrade.
(In reply to comment #6)
> In NSS 3.12.3 (of which FF is now testing a Beta), those flags are 
> (more) fully implemented.  NSS is now giving the browser the behavior 
> that the browser has been requesting all along.  And it turns out that 
> this may not be the behavior that the browser actually wants.  So the
> question before the browser folks is "What behavior do we really want 
> for this case?". 

According to the EV Cert Guidelines, certs must support revocation, and certs issued starting 2011 MUST support OSCP. Prior to that CRL is allowed instead. The AMO EV cert has a CrlDP extension in it so it should still be a valid EV cert.

Given your comment 7 is it really the browser that's wrong here?

Going forward we have some choices to make, and which one we do depends on both the Firefox and NSS schedules.

1. NSS implements CrlDP lookup
2. Firefox stops requiring fresh info
3. Leave it as-is (AMO and many other EV certs lose EV-ness)

1 is obviously best, is there a timeframe for that? If CrlDP lookup can make Firefox 3.1 final then we can pick the least-bad of 2 or 3 for the upcoming beta and it probably doesn't matter much (I lean toward 3 rather than change client code and then risk forgetting to change it back, but the site owners and CA's affected probably prefer 2).

If CRL lookup isn't going to make Firefox 3.1 final then we've got a hard choice to make, and at this point the Firefox leads don't even realize they need to make one.
Flags: wanted1.9.1?
Flags: blocking1.9.1?
I know very little about this security stuff.  But option 3 above seems to me it should not even be an option.  A site with EV-certs and we show it as a downgraded normal SSL site could get Mozilla in heaps of trouble should someone get taken in by not realizing its wrong when he 'thought' he was secure. 

Maybe Option 4:
Back out the NSS beta ?
(In reply to comment #10)

> If CRL lookup isn't going to make Firefox 3.1 final then we've got a hard
> choice to make, and at this point the Firefox leads don't even realize they
> need to make one.

The Firefox leads are working through options as we speak, sorry for not commenting here.

(In reply to comment #11)
> I know very little about this security stuff.  But option 3 above seems to me
> it should not even be an option.  A site with EV-certs and we show it as a
> downgraded normal SSL site could get Mozilla in heaps of trouble should someone
> get taken in by not realizing its wrong when he 'thought' he was secure. 
> 
> Maybe Option 4:
> Back out the NSS beta ?

Heaps of trouble aside, this has only landed on trunk right now, which is not facing imminent ship and hence has a little more time for things like Dan's option 1 to happen, or at least for us to get notice out to the impacted CAs.  The immediate question is what to do on branch, where this hasn't landed yet.
Dan, what's the current target delivery date for 3.1 Final?
(In reply to comment #13)
> Dan, what's the current target delivery date for 3.1 Final?

I know it was addressed to Dan, but FF3.1 is targeting first quarter of this year; beta 3 code freeze is this Sunday, after which we move to release candidates.  When we have a release candidate with no blocking bugs, it becomes the release, effectively.
(In reply to comment #10)
> 3. Leave it as-is (AMO and many other EV certs lose EV-ness)

Looking at the twelve CAs currently enabled for EV in nsIdentityChecking.cpp, there are three who do not operate an OCSP reponder, currently:

GlobalSign
Network Solutions
Trustwave

For the other eight, option 3 would not have any adverse effect, I assume, as they have OCSP support:

Comodo
Diginotar
Entrust
Go Daddy
Geotrust
QuoVadis
Thawte
Verisign

[Of course the bare number of CAs is less relevant than their effective market share, but maybe someone else wants to make a statement about the latter.]

> but the site owners and CA's affected probably prefer 2).

Site owners maybe, but not the CAs, I think... if an EV certificate has been revoked, they really want that to see reflected in the browser (and option 2 would currently fail to do so for CRL-only CAs).
OS: Windows Vista → All
Hardware: x86 → All
(In reply to comment #12)
> (In reply to comment #10)
> 
> > If CRL lookup isn't going to make Firefox 3.1 final then we've got a hard
> > choice to make, and at this point the Firefox leads don't even realize they
> > need to make one.
> 
> The Firefox leads are working through options as we speak, sorry for not
> commenting here.

This is not resolved yet.

On the one hand, work is being done on bug 321755 to implement a baseline form of CRLDP support.  The 3 CRL-using CAs employ full CRLs, not incrementals, which means a scaled back version of CRLDP support that handled full CRLs would be enough here.  People interested in that work should follow that bug.

In the event that that code is not production ready in time, the fallback approach would be get bug 371522 fixed so that CRLs can be auto-fetched properly, and then set Firefox 3.1 up to pre-fetch those.  This is obviously less likable for a couple reasons, but it is also more well-contained.
Blocking, potentially to become WONTFIX pending a decision! Johnath is to be the driving force for a decision, to co-ordinate with Shaver and others.
Flags: blocking1.9.1? → blocking1.9.1+
Depends on: 477244
I hate doing this but I don't understand the whole certificate things.  Is this bug the same thing as...

1) Go to https://www.usaa.com/inet/ent_logon/Logon
2) Watch it load and see the green certificate thing show up (sometimes it disappears when the page is done loading)
3) If certificate still shows when page is done loading, hit refresh, certificate still shows
4) Hit ctrl+f5 for a hard refresh and the certificate doesn't show up and says the site does not supply identity information.

If this isn't the same bug, could someone please point me to the correct bug?
(In reply to comment #18)
> I hate doing this but I don't understand the whole certificate things.  Is this
> bug the same thing as...
> 
> 1) Go to https://www.usaa.com/inet/ent_logon/Logon
> 2) Watch it load and see the green certificate thing show up (sometimes it
> disappears when the page is done loading)
> 3) If certificate still shows when page is done loading, hit refresh,
> certificate still shows
> 4) Hit ctrl+f5 for a hard refresh and the certificate doesn't show up and says
> the site does not supply identity information.
> 
> If this isn't the same bug, could someone please point me to the correct bug?

Hi Kurt,

You're describing a site that's going into mixed mode, serving unsecured content in an otherwise secured page. It's most likely caused by the site, sourcing a javascript file or an image or something from http instead of https (maybe an ad banner, or an analytics script?)  There is, of course, some possibility that you're seeing bug 455367 as well, but it is not this bug, either way.
(In reply to comment #15)
> ...For the other eight, option 3 would not have any adverse effect, I assume, as
> they have OCSP support:
> Comodo

Comodo only started to include our OCSP Responder URL in EV Certificates approximately 1 year ago.  A number of our customers are still using EV Certificates without OCSP Responder URLs that we issued before then (for example, https://www.gadgetshop.com).  We don't want these customers' EV sites to lose the EV UI in any future FF versions, so...

Johnathan, please keep us (rob@comodo.com, robin@comodo.com) informed about your plans for changing the EV revocation behaviour.  Thanks.
Clarifying subject, this bug is about CRLs.
Summary: NSS_3_12_3_BETA2 landing breaks AMO EV cert - Treated as DV → EV problem with missing CRLs - AMO EV cert treated as DV
Adding bug 321755 to the dependency list. I'm aware that it's unlikely to be fixed in time for ff3.1, however if it were fixed, it would resolve this bug and as such it represents a dependency which someone is welcome to try to fix properly.
Depends on: CRLDP
Blocks: 481968
Depends on: 483168
There is a requirement that all EV certificates issued after 31 December 2010 specify OSCP responder URLs.  So, the current behavior will definitely be correct for certificates issued from 2011 and beyond.

I don't like the idea, but I kind of suspect that the answer is that for EV certificates issued before that date, we can not demote them for lack of providing an OSCP URL.
In reply to comment #23:
> There is a requirement that all EV certificates issued after 31 December 2010
> specify OSCP responder URLs.

Bill, that's not actually true, IMHO.  See bug #477244 comment #94.

Now, you might ask why a CA which operates an OCSP Responder would deliberately choose to omit its OCSP Responder URL from the EV certificates it issues.  Look no further than bug #477244 comments #67 and #90 for one real-world example!
I recently discovered that IE seems to have changed to
display EV status without checking revocation in two
situations:

1. if the user tells IE to not check revocation, by
turning off "Check for server certificate revocation"
in Internet Options > Advanced > Settings > Security;

or

2. if network problems prevent IE from downloading CRLs
or talking to OCSP responders.

This new finding is thought-provoking.
Wan-Teh, In reply to comment 25, 

Q) what version(s) of IE and/or Windows behave that way?  I ask this because
at one time, it was necessary to explicitly enable revocation checking to get
EV indicators.

thought: Maybe you should report this to the CAB Forum.
(In reply to comment #26)
> Wan-Teh, In reply to comment 25, 
> 
> Q) what version(s) of IE and/or Windows behave that way?  I ask this because
> at one time, it was necessary to explicitly enable revocation checking to get
> EV indicators.
> 
> thought: Maybe you should report this to the CAB Forum.

In order to show the green bar for an EV certificate IE7 currently requires either that Check for server certificate revocation is enabled, or that The Phishing Filter is enabled.

From what I remember, that is the way it has always worked.
In reply to comment #25:
> I recently discovered that IE seems to have changed to display EV status
> without checking revocation in two situations:
<snip>
> 2. if network problems prevent IE from downloading CRLs or talking to OCSP
> responders.

I quizzed our contact at Microsoft about this a few months ago.  He said:
'To show EV, we require that the client have revocation enabled, and we require the revocation check to not return "Revoked." We had originally planned to make revocation check failure (e.g. down rev-server) a semi-fatal error for all cert types (such that the address bar would go yellow, wiping out the EV green) but pulled this at the last moment because it turned out that a HUGE number of cert revocation checks failed at that time.  We could revisit this and reassess, however I doubt doing so would be in time to make changes in the next release of IE8.'
Re: comment 27

Bill: you're right.  I didn't notice that I had the
"Turn on automatic website checking" radio button
selected.  Thanks for correcting me.

So please ignore #1 in my comment 27.  That would
be more thought-provoking than #2 if it were true.
Sorry about the confusion.

Re: comment 28

Rob, thanks for sharing the info regarding #2 that
you got from Microsoft.
Unblocking on this as it's actually the as-designed behaviour.
Flags: blocking1.9.1+ → blocking1.9.1-
(In reply to comment #30)
> Unblocking on this as it's actually the as-designed behaviour.

I requested this blocking1.9.1- because I think this bug is describing a behaviour that makes us sad, but is more correct. Moreover, I don't think there's a resolution to this bug that should block the release.

Instead, I think we:

a) take the new NSS 3_12_3 tag for beta 4 - bug 481968
b) take the default OCSP responder fix for beta 4 - bug 485052
c) contact the remaining EV CAs using CRLs and invite them to supply responders (which can happen post-beta-4).  That is Globalsign (AMO's CA), Trustwave, and SECOM.  Verizon/Cybertrust is currently undergoing public discussion phase, and I have commented there on this issue.
d) continue to work on CRLDP support so that we can eliminate the need for such workarounds - bug 321755, not likely to make FF3.5, as far as I can tell.
Since most of the people interested in CRL fetching from CrlDP URLs are 
CC'ed on this bug (:-) I'm using this bug to appeal for your help.
I'll also post this to dev-tech-crypto.

I'm in the process of trying to wrap up CRL DP.  Alexei did 99.5% of the 
job and I'm trying to polish up a few loose ends. 

I have a build of NSS 3.12.4 Beta (post RC0), which, when dropped on top 
of the NSS shipped with Firefox 3.0.8, does CRL fetching from the CRL DP 
URLs in Comodo's EV certs, causing FF 3.0.8 to show EV UI on the basis of 
an actual CRL check, and neither ignoring the missing CRL, nor fetching 
an OCSP surrogate response.

Now, I need to test this heavily.  It would help me enormously to get a 
list of https URLs for sites with EV certs from as many different issuers
as possible. I'd like to have URLs for sites with EV certs with 
- CrlDP but no AIA/OCSP
- AIA/OCSP but no CrlDP
- Both
- Neither                                              (<- just kidding :)

Since each issuer is presumed to have just one CRL (at present), having 
many URLs of sites with EV certs from the same issuer doesn't help nearly 
as much as having many URLs of sites with EV certs from diverse issuers.  
So, please send me URLs, either as email, or postings to dev-tech-crypto
or even as brief comments added to thsi bug.  

Thanks
(In reply to comment #32)
> having many URLs of sites with EV certs from diverse issuers.  
> So, please send me URLs, either as email, or postings to dev-tech-crypto
> or even as brief comments added to thsi bug.

Trying to be brief - here's a more or less arbitrary selection of 32 hosts configured with an EV cert issued by one of 17 distinct EV ICAs:

app.quickblogcast.com diskunion.net join.wickedpictures.com minkabu.jp mixi.jp secure.echomusic.com secureecn40.websitecomplete.com www.GoDaddy.com www.at-link.ad.jp www.facebook.com www.fedex.com www.gilt.com www.heberjahiz.com www.hostmysite.com www.hostpoint.ch www.idealworld.tv www.igourmet.com www.ilike.com www.janpara.co.jp www.loopylove.com www.mediafire.com www.meebo.com www.nedbank.co.za www.newgrounds.com www.oisix.com www.overstock.com www.pixellogo.com www.rescue-center.jp www.softlayer.com www.switch.ch www.toysrus.co.uk www.xe.com

Hope this helps.
I tested all the sites in Kaspar's preceding comment with NSS's CRL DP code.
All the pages worked correctly.  Pages that had OCSP used it and not CRLs,
even if the CRLs were also available (as intended).  Pages that had only 
CRLDPs fetched the CRLs as needed, and retained them thereafter for the 
process lifetime, so that they did not need to be fetched again and again 
during the browser session.  

Some notes on the sites.
a) More than a few sites seemed to go through all the EV validation, and 
even download some additional material over EV connections, only to end up
redirecting the browser to an http page.  In some cases, changing the URL 
to point to a non-existent page caused those sites to return an EV 404 page.
So, I changed most of the URLs in the attached page to do that. 

Two of the sites have EV certs that chain up to Verisign's old roots, and
do not show EV UI in Firefox because those old roots have not been marked 
as authorized for those EV policy OIDs.  See bug 420760.
Oh, any many of those sites did not show EV UI, despite their certs passing 
all the EV UI tests, because the sites showed "mixed content", that is, 
images or frames from non-EV or non-SSL sites.
(In reply to comment #35)
> Oh, any many of those sites did not show EV UI, despite their certs passing 
> all the EV UI tests, because the sites showed "mixed content", that is, 
> images or frames from non-EV or non-SSL sites.

Nelson, in many cases it helps to use /robots.txt as the path name, which also has the nice side effect of not having to retrieve tons of images, JS files, iframes and other unnecessary stuff (when testing EV certs in the browser). Sometimes it even "prevents" redirects to http.
Good suggestion, Kaspar.  I also considered using favicons.
I also get that error the EV sign is showing shortly and then it disappears ...
In reply to comment #35:
> ..."mixed content", that is, images or frames from non-EV or non-SSL sites.

Nelson, are you sure that "non-EV" counts as "mixed content" on an otherwise EV page?

I see the EV UI (and it doesn't disappear once the page has finished loading) on an EV https page that includes an external Javascript file from a non-EV https URL.  I get this result with FF 3.0.x CVS + NSS HEAD, mozilla-1.9.1 and mozilla-central.
In reply to comment 39,  perhaps "mixed content" is the wrong term to use.
I should probably have written "non-EV content".

Some EV pages display as ordinary SSL pages, full lock icon but no EV 
indicators, despite having an EV cert.  I believe this is due to the 
presence of content (images) from non-EV https sites.  

Example: https://minkabu.jp/xxx displays as ordinary https.  I believe this
is due to the presence of an image from https://masstune.112.2o7.net/ which
has an ordinary (non-EV) SSL server cert.
Hmm. I'm now less certain than I was about my "non-EV content" theory. 
https://minkabu.jp/robots.txt contains no images or ancillary content.
It's a straight text page from a site with a cert that claims to be EV,
but the site does not get EV treatment.  Maybe Firefox hasn't yet approved
that root for EV.
Testing with IE7 (fully patched) on WinXP, there is no sign of EV showing as the URL bar does not turn 'green' like say on Paypal, only the standard 'lock w/blue background'.
In reply to comment #41:
> https://minkabu.jp/robots.txt
> ...Maybe Firefox hasn't yet approved that root for EV.

This Root belongs to Verizon/Cybertrust.  It's not currently EV-enabled in Firefox, but a request to EV-enable it is currently in public discussion on mozilla.dev.security.policy.
Not sure what may have changed, but I noticed today that the EV SiteID for AMO is now 'Green' again... 

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2a1pre) Gecko/20090506 Minefield/3.6a1pre Firefox/3.0.7 ID:20090506045618

maybe ? https://bugzilla.mozilla.org/show_bug.cgi?id=490864 fixed it ?
Duplicate of this bug: 497268
Flags: wanted1.9.1.x?
Updating summary for easier searching.
Summary: EV problem with missing CRLs - AMO EV cert treated as DV → EV problem with missing CRLs - AMO EV cert treated as DV (ie GlobalSign)
Duplicate of this bug: 501683
While a current Minefield build displays the EV button for AMO the recent candidate build for Firefox 3.5.1 only presents the DV button.

Are there any differences between those two branches?
> Are there any differences between those two branches?

Yeah, LOTS.  NSS 3.12.3 vs NSS 3.12.4 RC0 IIRC.
Blocks: 500495
Henrik, you might want to look at Bug 497172.  It's marked checkin-needed.  It will restore the EV UI for the AMO site.
status1.9.1: --- → ?
Flags: wanted1.9.1?
Flags: wanted1.9.1.x?
Bug #497172 was fixed in FF3.5.2 and also back-ported to FF3.0.13.  Bug #497172 comment #31 confirmed that this restored the EV UI for the AMO site in FF3.5.2.

However, when I visit https://addons.mozilla.org today, I find that it has reverted to using an old DV certificate for *.mozilla.org.  Does anybody know why this has happened?
Ah, I've found the answer.  Bug #508408.
I think that this can be closed now?
(In reply to comment #53)
> I think that this can be closed now?

Indeed, sir!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.