650355 - Stop accepting MD5 as a hash algorithm in signatures (toggle security.enable_md5_signatures to false)

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

+++ This bug was initially created as a clone of Bug #590364 +++

This is the bug about whatever PSM changes are needed to support the June 30th, 2011 cutoff for Mozilla products' support of MD5 as a hash algorithm in digital signatures. The NSS-level changes are being tracked in bug 590364.

Outside of certificate signature validation, we need to find other uses of MD5 (e.g. digest authentication) and file new bugs for each.

Comment 1

[Steven Medin]

To be explicit and match the documented policy, please ensure that this refers to intermediate CA and EE certificates.  It excludes roots by nature of the fact that root self-signatures are not verified during a TLS handshake.  Reference is https://wiki.mozilla.org/CA:MD5and1024. 

A nod to my statement reassures those I have not yet migrated.

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I understand that Steven. If a certificate is a trust anchor then won't exclude it just because it uses an MD5-based signature.

Comment 3

[Eddy Nigg (StartCom)]

Are there still any such MD based roots in NSS? I believe most have been removed already, no?

Comment 4

[Kathleen Wilson]

(In reply to comment #3)
> Are there still any such MD based roots in NSS? 

There are still MD5 based roots in NSS. You can see the details in the BuiltInCAs spreadsheet that I maintain here:
http://spreadsheets.google.com/pub?key=ttwCVzDVuWzZYaDosdU6e3w&single=true&gid=0&output=html
There is a column called "Signature Algorithm". You can search for MD5 and find all of the relevant roots.
There are 3 from Entrust (disabled -- trust bits are turned off by default), 2 from Geotrust, 1 from GTE CyberTrust/Verizon, 3 from NetLock, 2 from TC TrustCenter (expired), 4 from Thawte.
 

As per https://wiki.mozilla.org/CA:MD5and1024
"The MD5 root certificates don’t necessarily need to be removed from NSS, because the signatures of root certificates are not validated (roots are self-signed)."

Comment 5

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to comment #3)
> Are there still any such MD based roots in NSS? I believe most have been
> removed already, no?

Regardless, we would still allow MD5-based roots and self-signed certs (?) that are in the user's certificate database to work.

Anyway, I tried to get a rough idea how many sites would be affected. It seems there are relatively few MD5-based certs in the SSL observatory data from December 2010. The vast majority of them are Equifax:

    * Equifax Secure eBusiness CA-1: 12
    * Equifax Secure Global eBusiness CA-1: 8,378

Every other root listed in the SSL observatory data has 3 or fewer that expire after June 30th, except Verisign, which has 7. It would be a good idea to alert whoever controls the Equifax root to the fact that thousands of its customers' websites are about to stop working.

Comment 6

[Eddy Nigg (StartCom)]

Equifax = GeoTrust = Verisign

:-)

Comment 7

[Kathleen Wilson]

(In reply to comment #5)
> Anyway, I tried to get a rough idea how many sites would be affected. It seems
> there are relatively few MD5-based certs in the SSL observatory data from
> December 2010. The vast majority of them are Equifax:
> 
>     * Equifax Secure eBusiness CA-1: 12
>     * Equifax Secure Global eBusiness CA-1: 8,378
> 
> ...It would be a good idea to alert
> whoever controls the Equifax root to the fact that thousands of its customers'
> websites are about to stop working.

I talked with the appropriate folks from VeriSign and GeoTrust about this many times over the past year, and they are well aware of the June 30 date.

Comment 8

[Varga Viktor]

Netlock has MD5 roots, and they will rollovered at the SHA256 deadline.
No still valid MD5 intermediate or end certificate was issued at the Netlock.

Comment 9

[Kai Engert [:KaiE:]]

We want to have a pref at the Mozilla-application-evel that can toggle the acceptance of MD5-signatures.

Bug 732390 implements that pref (still accepting MD5-sigantures) Once we have some testing, we can use this bug to toggle the pref to reject-by-default.

Comment 10

[Kai Engert [:KaiE:]]

I'm removing the June 30, 2011 date from the summary, because we have passed that date.

Comment 11

[Kai Engert [:KaiE:]]

Attachment: Patch v1

I'll not ask a specific person for review.

Once we agree that the time has come to flip the pref, please mark r+.

Comment 12

[Kai Engert [:KaiE:]]

Started a test build with MD5-signatures disabled:

https://tbpl.mozilla.org/?tree=Try&rev=7537d48293b0

Comment 13

[Johnathan Nightingale [:johnath]]

Comment on attachment 604683 [details] [diff] [review]
Patch v1

r=me for landing AFTER this week's migration. That is, let's get this early in the FF14 mozilla-central cycle.

From my note to r-d:

As a final nod to compatibility, I propose to flip the switch after migration, so that it rides a full set of trains on FF14. This is later than I'd like, but it gives site operators maximum notice that they're busted. The CAs in our root program have already had plenty of notice, but in-house CAs like the US DoD won't, and flipping the pref today means it hits Aurora tomorrow, cutting their window of time to react by 6 weeks.

I'll mark the bug right now to ask for a post-migration landing. If someone disagrees and wants to get it landed on FF13 as well, we can process that through regular aurora triage. Fair?

J

Comment 14

[Kai Engert [:KaiE:]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dd6e4d39c56d

Comment 15

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

https://hg.mozilla.org/mozilla-central/rev/dd6e4d39c56d

Comment 16

[Kent Fredric]

I just discovered one site affected by this issue ( nightly user ). Fortunately, the people running that site were reasonably competent in analysing the situation and quickly resolved to take the right approach: getting a new cert made.

( Were using CACert who still had a 3rd party RSA-MD5 cert )

However, given the error message one receives when encountering this, its reasonably unhelpful if neither the client or host understand the problem in advance.

An "Invalid Signature" with   (Error code: sec_error_bad_signature)  message and no obvious explanation as to why or what is going on is incredibly unhelpful.

The result will inevitably be: early testers hitting websites with "bad" certs that are broken, reporting "Durr, your website is broken for me, I don't know why, I get this message", and the provider responding "Weird, it works for everyone else, even IE users, don't understand the problem either, must be something on your end broken". 

Seeing its not an /error/  as such with the certificate, but its a security *decision* to mark that signing key as invalid, the resulting error should /really/ convey this much more clearly.

This may quantify as "a new bug", but I just think its a defect in the present implementation of this bugs solution. But I'm loathe to file a bug on this, as there are already a few bugs vaguely covering this problem in varying degrees of non-specificity and confusion : https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20sec_error_bad_signature;list_id=2628634

Comment 17

[Kai Engert [:KaiE:]]

(In reply to Kent Fredric from comment #16)
> 
> However, given the error message one receives when encountering this, its
> reasonably unhelpful if neither the client or host understand the problem in
> advance.
> 
> An "Invalid Signature" with   (Error code: sec_error_bad_signature)  message
> and no obvious explanation as to why or what is going on is incredibly
> unhelpful.


We cannot distinguish "bad signature" from "deliberately rejecting good signature".

Why not?

Because we don't know if it's a "real good signature" or a "good looking but rogue signature created using a weakness".

We're rejecting a certificate because it has a bad signature.


> The result will inevitably be: early testers hitting websites with "bad"
> certs that are broken, reporting "Durr, your website is broken for me, I
> don't know why, I get this message", and the provider responding "Weird, it
> works for everyone else, even IE users, don't understand the problem either,
> must be something on your end broken".


Any provider should contact their CA when their customers experience problems with the certificate the customer has received, and the CA should be well aware of what's going on.


> Seeing its not an /error/  as such with the certificate, but its a security
> *decision* to mark that signing key as invalid, the resulting error should
> /really/ convey this much more clearly.


Who has time to write a code enhancement to NSS that will clearly mark all deliberately untrusted signatures with a different error code?

And even if we got this code contribution, what would we say instead?

  "We reject this certificate, because the certificate presented by that site 
   uses an older mechanism which is considered outdated and insecure."

Would that be any better?

If you believe it's worthy to continue this thought, and if you think it's likely that someone will be able to find programmer resources to actually get this done, then please file a new bug.

Comment 18

[Wan-Teh Chang]

I just attached a patch to bug 590364 to add a new error code for
a certificate signed with a signature algorithm that is disabled
by policy.

Comment 19

[Varga Viktor]

(In reply to Kai Engert (:kaie) from comment #12)
> Started a test build with MD5-signatures disabled:
> 
> https://tbpl.mozilla.org/?tree=Try&rev=7537d48293b0

Dear Kai,
How can I get this test build to try it? 
I am at the Netlock, and I will be sure, that the old MD5 roots will be not disabled after when this feature will be included into the Firefox.

Regards, Viktor

Comment 20

[Robert Relyea]

NOTE: this patch would also allow override of MD-2 and MD-4 signatures. The latter is moot, though, because NSS currently does not support MD-4 hashes at all.

bob

Comment 21

[Kai Engert [:KaiE:]]

(In reply to Wan-Teh Chang from comment #18)
> I just attached a patch to bug 590364 to add a new error code for
> a certificate signed with a signature algorithm that is disabled
> by policy.

I've moved your patch to new bug 738454, and I've commented in that bug.

Changes to PSM will be necessary. I've filed bug 738457 to get a better error code shown on the Firefox error page.

Comment 22

[Kai Engert [:KaiE:]]

(In reply to Varga Viktor from comment #19)
> 
> Dear Kai,
> How can I get this test build to try it? 

By now you can use a "nightly" build, get one from here:
  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/

(email me in private if you need help picking the right file,
 which platform do you want to use for testing?)
 

To verify you indeed got a build with the feature disabled by default, go to
    about:config
type "md5", you should the preference security.enable_md5_signatures,
and you should see it's set to "false".


> I am at the Netlock, and I will be sure, that the old MD5 roots will be not
> disabled after when this feature will be included into the Firefox.

Thanks in advance for testing and for feedback.

Comment 23

[Varga Viktor]

(In reply to Kai Engert (:kaie) from comment #22)
> (In reply to Varga Viktor from comment #19)
> > 
> > Dear Kai,
> > How can I get this test build to try it? 
> 
> By now you can use a "nightly" build, get one from here:
>   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
> 
> (email me in private if you need help picking the right file,
>  which platform do you want to use for testing?)
>  
> 
> To verify you indeed got a build with the feature disabled by default, go to
>     about:config
> type "md5", you should the preference security.enable_md5_signatures,
> and you should see it's set to "false".

Dear Kai, 
Thank you, I found it. 
I checked the mentioned settings up here, and I went to check one thing quickly. 
1. MD5 root - SHA1 EE certificate - working as expected.
Another case I should check if the chain has somewhere an MD5, not on the top, but others are SHA1. (Tomorrow I will check it.)

Comment 24

[Daniel Veditz [:dveditz]]

> (In reply to Varga Viktor from comment #19)
> > I am at the Netlock, and I will be sure, that the old MD5 roots will be not
> > disabled after when this feature will be included into the Firefox.

Roots are trusted because they are in the trusted list, their signature hash algorithm is irrelevant. Intermediates with MD5 will get blocked, though, same as EE certs.

Comment 25

[Varga Viktor]

(In reply to Daniel Veditz [:dveditz] from comment #24)

> Roots are trusted because they are in the trusted list, their signature hash
> algorithm is irrelevant. Intermediates with MD5 will get blocked, though,
> same as EE certs.

I think so, but:
Everywhere can appear a bug, so it's better to test it. :)

(There was a certificate related bug in the Outlook Express, and I though, it will be corrected. Now I know, that the Windows Mail has the codebase of the OE, becuse the same bug is still in. :) )

Comment 26

[Alex Keybl [:akeybl]]

We need to back this out of FF14 and FF15, and only re-land once:

* Strings and error code changes in bug 738457 and bug 738454
* A blog post explaining the upcoming change and how to identify the error
* An email to the enterprise notifying them of the change (and the fact that ESR will not be affected till 17), since we expect their self-signed internal apps will be most affected
* Telemetry around the number of times this error was hit to know whether the metric is acceptable before it hits the beta channel

Comment 27

[Kai Engert [:KaiE:]]

Attachment: backout patch

does this backout patch need review?

(It's simply the reverse of the original patch.)

Comment 28

[Kai Engert [:KaiE:]]

Reopening, because drivers proposed to postpone this change to some later date.

Comment 29

[Wan-Teh Chang]

Comment on attachment 620008 [details] [diff] [review]
backout patch

r=wtc.

Comment 30

[Alex Keybl [:akeybl]]

Comment on attachment 620008 [details] [diff] [review]
backout patch

[Triage Comment]
Approved for FF14 Aurora - we'll try to complete blocking tasks to make this happen for FF15.

Comment 31

[Kai Engert [:KaiE:]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d9525fcfdd8b

Comment 32

[Kai Engert [:KaiE:]]

previous comment was mozilla-inbound BACKOUT

aurora BACKOUT:
https://hg.mozilla.org/releases/mozilla-aurora/rev/e0dd3b0fbda2

Comment 33

[Ed Morley [:emorley]]

Backout merged:

https://hg.mozilla.org/mozilla-central/rev/d9525fcfdd8b

Comment 34

[Kai Engert [:KaiE:]]

In order to allow for early testing of this proposed change, I've made binaries available at http://flowerbeetle.org and http://flowerduck.org - in addition the binaries include support for fallback error prompts in the e-mail client. This could allow users who are unsure whether their connection failures are caused by MD5 certificates to know for sure (by testing with these experimental binaries and checking the error messages.)

Comment 35

[Alex Keybl [:akeybl]]

We're OK with landing this in FF15 now.

(In reply to Alex Keybl [:akeybl] from comment #26)
> We need to back this out of FF14 and FF15, and only re-land once:
> 
> * Strings and error code changes in bug 738457 and bug 738454

This is now done.

> * A blog post explaining the upcoming change and how to identify the error

dveditz let me know 

> * An email to the enterprise notifying them of the change (and the fact that
> ESR will not be affected till 17), since we expect their self-signed
> internal apps will be most affected

I'll handle this.

> * Telemetry around the number of times this error was hit to know whether
> the metric is acceptable before it hits the beta channel

We've decided to forgo this in favor of being reactive since we're not sure Telemetry will give us a clear answer one way or the other. We'll instead change security.enable_md5_signatures back to true if we find out it's disruptive prior to release, or through an add-on hotfix post-release if necessary.

Comment 36

[Kai Engert [:KaiE:]]

(In reply to Alex Keybl [:akeybl] from comment #35)
> 
> We'll instead
> change security.enable_md5_signatures back to true if we find out it's
> disruptive prior to release, or through an add-on hotfix post-release if
> necessary.


I don't understand this part, it seems to be based on a false assumptions.

We already did that.

As of now,
security.enable_md5_signatures has value "true" everywhere,
in other words,
the value is "still accept" everywhere.

See comment 31, comment 32 and comment 33.

Comment 37

[Kai Engert [:KaiE:]]

(In reply to Alex Keybl [:akeybl] from comment #35)
> We're OK with landing this in FF15 now.

Ok, sorry, I missed this part.

Comment 38

[Alex Keybl [:akeybl]]

As Kathleen implies through the bug relation, we should wait until bug 758314 lands before resolving this.

Comment 39

[Kathleen Wilson]

There are several bugs shown in the "Depends On" list. 
Are they all truly blocking this bug? 
Or is bug #758314 the only one that truly blocks this bug?

Comment 40

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

It is important to have an automated test for this, but I cannot get that test done in time for the next merge. Instead, let's use Kai's testcase as a litmus test.

I filed bug 773477 to add automated tests.

I added qawanted because that's the best way I can think of to indicate to QA that I want them to create a litmus test. (QA: Please educate me as to the proper way of doing this.)

> Test case:
> 
> - create a separate test profile, because having Kai's test CA
>   certificate installed is not safe for general use.
>
> - set security.enable_md5_signatures = false
> 
> - install Kai's test CA from http://kuix.de/ca/nss-test-ca.php
>   (check the boxes, OK)
> 
> - go to https://kuix.de:9450
> 
> - you should see an error page
> 
> - you should be able to add an override and connect to the page

Comment 41

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1dd81bbdb0e0

Comment 42

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/1dd81bbdb0e0

Comment 43

[u279076]

(In reply to Brian Smith (:bsmith) from comment #40)
> I added qawanted because that's the best way I can think of to indicate to
> QA that I want them to create a litmus test. (QA: Please educate me as to
> the proper way of doing this.)

The "right" process is to simply set the in-moztrap flag to ?. Feel free to use me as the requestee in the future. I'll review the testcase and decide whether this needs a manual test (MozTrap) or an automated test (Mozmill).

Thanks

Comment 44

[u279076]

I've added a manual test in MozTrap:
https://moztrap.mozilla.org/manage/cases/_detail/6367/

Brian, please review it and let me know if anything needs changing. I'm working in parallel with the QA Automation Development team to determine if this is something we can automate with Mozmill.

Comment 45

[u279076]

CCing Henrik to speak to the automation of the MozTrap testcase in comment 44. We talked briefly on email and he thought it would be doable as a restart test.

Comment 46

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #44)
> I've added a manual test in MozTrap:
> https://moztrap.mozilla.org/manage/cases/_detail/6367/
> 
> Brian, please review it and let me know if anything needs changing. I'm

Brian or Kai, can one of you please check this testcase? If all the steps and expected results are valid I would like to get started with a Mozmill test. Thanks.

Comment 47

[Kai Engert [:KaiE:]]

Henrik, thanks, sounds good to me.

In case it was accidental, there are spaces inside this string:
  security.enable_ md5 _signatures

Comment 48

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Kai. Anthony, would you mind to file a bug for the Mozmill test implementation? Thanks.

Comment 49

[u279076]

(In reply to Kai Engert (:kaie) from comment #47)
> Henrik, thanks, sounds good to me.
> 
> In case it was accidental, there are spaces inside this string:
>   security.enable_ md5 _signatures

Yeah, this wasn't accidental. MozTrap interprets enable_md5_signatures as enablemd5signatures, where md5 is bold. I suppose that could be a bug in MozTrap.   

(In reply to Henrik Skupin (:whimboo) from comment #48)
> Thanks Kai. Anthony, would you mind to file a bug for the Mozmill test

Sure, Henrik, I'll file this bug.
> implementation? Thanks.

Comment 50

[u279076]

(In reply to Henrik Skupin (:whimboo) from comment #48)
> Thanks Kai. Anthony, would you mind to file a bug for the Mozmill test
> implementation? Thanks.

See bug 795398.

Comment 51

[u279076]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #49)
> (In reply to Kai Engert (:kaie) from comment #47)
> > Henrik, thanks, sounds good to me.
> > 
> > In case it was accidental, there are spaces inside this string:
> >   security.enable_ md5 _signatures
> 
> Yeah, this wasn't accidental. MozTrap interprets enable_md5_signatures as
> enablemd5signatures, where md5 is bold. I suppose that could be a bug in
> MozTrap.   

Nevermind, figured this out -- the pref should now display correctly.

Comment 52

[dynamis (Tomoya ASAI)]

Hi Alex and all,

(In reply to Alex Keybl [:akeybl] from comment #35)
> > * A blog post explaining the upcoming change and how to identify the error
> 
> dveditz let me know 
> 
> > * An email to the enterprise notifying them of the change (and the fact that
> > ESR will not be affected till 17), since we expect their self-signed
> > internal apps will be most affected
> 
> I'll handle this.

Firefox 16 with this fix will be released soon. I couldn't find any blog post nor notification mail to enterprise ML. You finally decided not to write?
If you've already wrote blog/mail, could you tell me? I want to answer to Japanese cert provider who asked us about this.

Comment 53

[Onno]

In FF 16, the about:config says: security.ssl3.rsa_rc4_128_md5;true
Is this correct?

Comment 54

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Coyote from comment #53)
> In FF 16, the about:config says: security.ssl3.rsa_rc4_128_md5;true
> Is this correct?

The SSL cipher suites that use MD5 actually use HMAC-MD5 and/or MD5 in conjunction with SHA-1. Consequently, they are considered safe, unlike the plain MD5 hashes used in signatures.

Comment 58

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The mozmill test has been landed in our default branch for Nightly and will move with the trains via bug 795398.