Closed Bug 650355 Opened 13 years ago Closed 12 years ago

Stop accepting MD5 as a hash algorithm in signatures (toggle security.enable_md5_signatures to false)

Categories

(Core :: Security: PSM, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla16
Tracking Status
firefox14 - ---
firefox15 - ---
firefox16 + fixed

People

(Reporter: briansmith, Assigned: KaiE)

References

Details

(Keywords: compat, relnote)

Attachments

(2 files)

+++ This bug was initially created as a clone of Bug #590364 +++

This is the bug about whatever PSM changes are needed to support the June 30th, 2011 cutoff for Mozilla products' support of MD5 as a hash algorithm in digital signatures. The NSS-level changes are being tracked in bug 590364.

Outside of certificate signature validation, we need to find other uses of MD5 (e.g. digest authentication) and file new bugs for each.
To be explicit and match the documented policy, please ensure that this refers to intermediate CA and EE certificates.  It excludes roots by nature of the fact that root self-signatures are not verified during a TLS handshake.  Reference is https://wiki.mozilla.org/CA:MD5and1024. 

A nod to my statement reassures those I have not yet migrated.
I understand that Steven. If a certificate is a trust anchor then won't exclude it just because it uses an MD5-based signature.
Are there still any such MD based roots in NSS? I believe most have been removed already, no?
(In reply to comment #3)
> Are there still any such MD based roots in NSS? 

There are still MD5 based roots in NSS. You can see the details in the BuiltInCAs spreadsheet that I maintain here:
http://spreadsheets.google.com/pub?key=ttwCVzDVuWzZYaDosdU6e3w&single=true&gid=0&output=html
There is a column called "Signature Algorithm". You can search for MD5 and find all of the relevant roots.
There are 3 from Entrust (disabled -- trust bits are turned off by default), 2 from Geotrust, 1 from GTE CyberTrust/Verizon, 3 from NetLock, 2 from TC TrustCenter (expired), 4 from Thawte.
 

As per https://wiki.mozilla.org/CA:MD5and1024
"The MD5 root certificates don’t necessarily need to be removed from NSS, because the signatures of root certificates are not validated (roots are self-signed)."
(In reply to comment #3)
> Are there still any such MD based roots in NSS? I believe most have been
> removed already, no?

Regardless, we would still allow MD5-based roots and self-signed certs (?) that are in the user's certificate database to work.

Anyway, I tried to get a rough idea how many sites would be affected. It seems there are relatively few MD5-based certs in the SSL observatory data from December 2010. The vast majority of them are Equifax:

    * Equifax Secure eBusiness CA-1: 12
    * Equifax Secure Global eBusiness CA-1: 8,378

Every other root listed in the SSL observatory data has 3 or fewer that expire after June 30th, except Verisign, which has 7. It would be a good idea to alert whoever controls the Equifax root to the fact that thousands of its customers' websites are about to stop working.
Equifax = GeoTrust = Verisign

:-)
(In reply to comment #5)
> Anyway, I tried to get a rough idea how many sites would be affected. It seems
> there are relatively few MD5-based certs in the SSL observatory data from
> December 2010. The vast majority of them are Equifax:
> 
>     * Equifax Secure eBusiness CA-1: 12
>     * Equifax Secure Global eBusiness CA-1: 8,378
> 
> ...It would be a good idea to alert
> whoever controls the Equifax root to the fact that thousands of its customers'
> websites are about to stop working.

I talked with the appropriate folks from VeriSign and GeoTrust about this many times over the past year, and they are well aware of the June 30 date.
Netlock has MD5 roots, and they will rollovered at the SHA256 deadline.
No still valid MD5 intermediate or end certificate was issued at the Netlock.
Assignee: bsmith → nobody
We want to have a pref at the Mozilla-application-evel that can toggle the acceptance of MD5-signatures.

Bug 732390 implements that pref (still accepting MD5-sigantures) Once we have some testing, we can use this bug to toggle the pref to reject-by-default.
Depends on: 732390
I'm removing the June 30, 2011 date from the summary, because we have passed that date.
Summary: Stop accepting MD5 as a hash algorithm in PSM on June 30, 2011 → Stop accepting MD5 as a hash algorithm in PSM
Keywords: compat, relnote
Summary: Stop accepting MD5 as a hash algorithm in PSM → Stop accepting MD5 as a hash algorithm in PSM (toggle security.enable_md5_signatures to false)
Summary: Stop accepting MD5 as a hash algorithm in PSM (toggle security.enable_md5_signatures to false) → Stop accepting MD5 as a hash algorithm in certificate signatures (toggle security.enable_md5_signatures to false)
Attached patch Patch v1Splinter Review
I'll not ask a specific person for review.

Once we agree that the time has come to flip the pref, please mark r+.
Attachment #604683 - Flags: review?
Assignee: nobody → kaie
Started a test build with MD5-signatures disabled:

https://tbpl.mozilla.org/?tree=Try&rev=7537d48293b0
Comment on attachment 604683 [details] [diff] [review]
Patch v1

r=me for landing AFTER this week's migration. That is, let's get this early in the FF14 mozilla-central cycle.

From my note to r-d:

As a final nod to compatibility, I propose to flip the switch after migration, so that it rides a full set of trains on FF14. This is later than I'd like, but it gives site operators maximum notice that they're busted. The CAs in our root program have already had plenty of notice, but in-house CAs like the US DoD won't, and flipping the pref today means it hits Aurora tomorrow, cutting their window of time to react by 6 weeks.

I'll mark the bug right now to ask for a post-migration landing. If someone disagrees and wants to get it landed on FF13 as well, we can process that through regular aurora triage. Fair?

J
Attachment #604683 - Flags: review? → review+
Target Milestone: --- → mozilla14
https://hg.mozilla.org/mozilla-central/rev/dd6e4d39c56d
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
I just discovered one site affected by this issue ( nightly user ). Fortunately, the people running that site were reasonably competent in analysing the situation and quickly resolved to take the right approach: getting a new cert made.

( Were using CACert who still had a 3rd party RSA-MD5 cert )

However, given the error message one receives when encountering this, its reasonably unhelpful if neither the client or host understand the problem in advance.

An "Invalid Signature" with   (Error code: sec_error_bad_signature)  message and no obvious explanation as to why or what is going on is incredibly unhelpful.

The result will inevitably be: early testers hitting websites with "bad" certs that are broken, reporting "Durr, your website is broken for me, I don't know why, I get this message", and the provider responding "Weird, it works for everyone else, even IE users, don't understand the problem either, must be something on your end broken". 

Seeing its not an /error/  as such with the certificate, but its a security *decision* to mark that signing key as invalid, the resulting error should /really/ convey this much more clearly.

This may quantify as "a new bug", but I just think its a defect in the present implementation of this bugs solution. But I'm loathe to file a bug on this, as there are already a few bugs vaguely covering this problem in varying degrees of non-specificity and confusion : https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20sec_error_bad_signature;list_id=2628634
(In reply to Kent Fredric from comment #16)
> 
> However, given the error message one receives when encountering this, its
> reasonably unhelpful if neither the client or host understand the problem in
> advance.
> 
> An "Invalid Signature" with   (Error code: sec_error_bad_signature)  message
> and no obvious explanation as to why or what is going on is incredibly
> unhelpful.


We cannot distinguish "bad signature" from "deliberately rejecting good signature".

Why not?

Because we don't know if it's a "real good signature" or a "good looking but rogue signature created using a weakness".

We're rejecting a certificate because it has a bad signature.


> The result will inevitably be: early testers hitting websites with "bad"
> certs that are broken, reporting "Durr, your website is broken for me, I
> don't know why, I get this message", and the provider responding "Weird, it
> works for everyone else, even IE users, don't understand the problem either,
> must be something on your end broken".


Any provider should contact their CA when their customers experience problems with the certificate the customer has received, and the CA should be well aware of what's going on.


> Seeing its not an /error/  as such with the certificate, but its a security
> *decision* to mark that signing key as invalid, the resulting error should
> /really/ convey this much more clearly.


Who has time to write a code enhancement to NSS that will clearly mark all deliberately untrusted signatures with a different error code?

And even if we got this code contribution, what would we say instead?

  "We reject this certificate, because the certificate presented by that site 
   uses an older mechanism which is considered outdated and insecure."

Would that be any better?

If you believe it's worthy to continue this thought, and if you think it's likely that someone will be able to find programmer resources to actually get this done, then please file a new bug.
I just attached a patch to bug 590364 to add a new error code for
a certificate signed with a signature algorithm that is disabled
by policy.
(In reply to Kai Engert (:kaie) from comment #12)
> Started a test build with MD5-signatures disabled:
> 
> https://tbpl.mozilla.org/?tree=Try&rev=7537d48293b0

Dear Kai,
How can I get this test build to try it? 
I am at the Netlock, and I will be sure, that the old MD5 roots will be not disabled after when this feature will be included into the Firefox.

Regards, Viktor
NOTE: this patch would also allow override of MD-2 and MD-4 signatures. The latter is moot, though, because NSS currently does not support MD-4 hashes at all.

bob
(In reply to Wan-Teh Chang from comment #18)
> I just attached a patch to bug 590364 to add a new error code for
> a certificate signed with a signature algorithm that is disabled
> by policy.

I've moved your patch to new bug 738454, and I've commented in that bug.

Changes to PSM will be necessary. I've filed bug 738457 to get a better error code shown on the Firefox error page.
(In reply to Varga Viktor from comment #19)
> 
> Dear Kai,
> How can I get this test build to try it? 

By now you can use a "nightly" build, get one from here:
  http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/

(email me in private if you need help picking the right file,
 which platform do you want to use for testing?)
 

To verify you indeed got a build with the feature disabled by default, go to
    about:config
type "md5", you should the preference security.enable_md5_signatures,
and you should see it's set to "false".


> I am at the Netlock, and I will be sure, that the old MD5 roots will be not
> disabled after when this feature will be included into the Firefox.

Thanks in advance for testing and for feedback.
(In reply to Kai Engert (:kaie) from comment #22)
> (In reply to Varga Viktor from comment #19)
> > 
> > Dear Kai,
> > How can I get this test build to try it? 
> 
> By now you can use a "nightly" build, get one from here:
>   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/
> 
> (email me in private if you need help picking the right file,
>  which platform do you want to use for testing?)
>  
> 
> To verify you indeed got a build with the feature disabled by default, go to
>     about:config
> type "md5", you should the preference security.enable_md5_signatures,
> and you should see it's set to "false".

Dear Kai, 
Thank you, I found it. 
I checked the mentioned settings up here, and I went to check one thing quickly. 
1. MD5 root - SHA1 EE certificate - working as expected.
Another case I should check if the chain has somewhere an MD5, not on the top, but others are SHA1. (Tomorrow I will check it.)
> (In reply to Varga Viktor from comment #19)
> > I am at the Netlock, and I will be sure, that the old MD5 roots will be not
> > disabled after when this feature will be included into the Firefox.

Roots are trusted because they are in the trusted list, their signature hash algorithm is irrelevant. Intermediates with MD5 will get blocked, though, same as EE certs.
(In reply to Daniel Veditz [:dveditz] from comment #24)

> Roots are trusted because they are in the trusted list, their signature hash
> algorithm is irrelevant. Intermediates with MD5 will get blocked, though,
> same as EE certs.

I think so, but:
Everywhere can appear a bug, so it's better to test it. :)

(There was a certificate related bug in the Outlook Express, and I though, it will be corrected. Now I know, that the Windows Mail has the codebase of the OE, becuse the same bug is still in. :) )
Blocks: 739558
Summary: Stop accepting MD5 as a hash algorithm in certificate signatures (toggle security.enable_md5_signatures to false) → Stop accepting MD5 as a hash algorithm in signatures (toggle security.enable_md5_signatures to false)
We need to back this out of FF14 and FF15, and only re-land once:

* Strings and error code changes in bug 738457 and bug 738454
* A blog post explaining the upcoming change and how to identify the error
* An email to the enterprise notifying them of the change (and the fact that ESR will not be affected till 17), since we expect their self-signed internal apps will be most affected
* Telemetry around the number of times this error was hit to know whether the metric is acceptable before it hits the beta channel
Attached patch backout patchSplinter Review
does this backout patch need review?

(It's simply the reverse of the original patch.)
Attachment #620008 - Flags: review?
Attachment #620008 - Flags: approval-mozilla-aurora?
Reopening, because drivers proposed to postpone this change to some later date.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 620008 [details] [diff] [review]
backout patch

r=wtc.
Attachment #620008 - Flags: review? → review+
Comment on attachment 620008 [details] [diff] [review]
backout patch

[Triage Comment]
Approved for FF14 Aurora - we'll try to complete blocking tasks to make this happen for FF15.
Attachment #620008 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
https://hg.mozilla.org/integration/mozilla-inbound/rev/d9525fcfdd8b
Whiteboard: [keep open after merge]
Target Milestone: mozilla14 → mozilla15
previous comment was mozilla-inbound BACKOUT

aurora BACKOUT:
https://hg.mozilla.org/releases/mozilla-aurora/rev/e0dd3b0fbda2
Backout merged:

https://hg.mozilla.org/mozilla-central/rev/d9525fcfdd8b
Target Milestone: mozilla15 → ---
In order to allow for early testing of this proposed change, I've made binaries available at http://flowerbeetle.org and http://flowerduck.org - in addition the binaries include support for fallback error prompts in the e-mail client. This could allow users who are unsure whether their connection failures are caused by MD5 certificates to know for sure (by testing with these experimental binaries and checking the error messages.)
We're OK with landing this in FF15 now.

(In reply to Alex Keybl [:akeybl] from comment #26)
> We need to back this out of FF14 and FF15, and only re-land once:
> 
> * Strings and error code changes in bug 738457 and bug 738454

This is now done.

> * A blog post explaining the upcoming change and how to identify the error

dveditz let me know 

> * An email to the enterprise notifying them of the change (and the fact that
> ESR will not be affected till 17), since we expect their self-signed
> internal apps will be most affected

I'll handle this.

> * Telemetry around the number of times this error was hit to know whether
> the metric is acceptable before it hits the beta channel

We've decided to forgo this in favor of being reactive since we're not sure Telemetry will give us a clear answer one way or the other. We'll instead change security.enable_md5_signatures back to true if we find out it's disruptive prior to release, or through an add-on hotfix post-release if necessary.
(In reply to Alex Keybl [:akeybl] from comment #35)
> 
> We'll instead
> change security.enable_md5_signatures back to true if we find out it's
> disruptive prior to release, or through an add-on hotfix post-release if
> necessary.


I don't understand this part, it seems to be based on a false assumptions.

We already did that.

As of now,
security.enable_md5_signatures has value "true" everywhere,
in other words,
the value is "still accept" everywhere.

See comment 31, comment 32 and comment 33.
(In reply to Alex Keybl [:akeybl] from comment #35)
> We're OK with landing this in FF15 now.

Ok, sorry, I missed this part.
Depends on: 758314
As Kathleen implies through the bug relation, we should wait until bug 758314 lands before resolving this.
There are several bugs shown in the "Depends On" list. 
Are they all truly blocking this bug? 
Or is bug #758314 the only one that truly blocks this bug?
No longer depends on: 549460, 590364, 663313, 739563
See Also: → 549460, 590364, 663313, 739563
It is important to have an automated test for this, but I cannot get that test done in time for the next merge. Instead, let's use Kai's testcase as a litmus test.

I filed bug 773477 to add automated tests.

I added qawanted because that's the best way I can think of to indicate to QA that I want them to create a litmus test. (QA: Please educate me as to the proper way of doing this.)

> Test case:
> 
> - create a separate test profile, because having Kai's test CA
>   certificate installed is not safe for general use.
>
> - set security.enable_md5_signatures = false
> 
> - install Kai's test CA from http://kuix.de/ca/nss-test-ca.php
>   (check the boxes, OK)
> 
> - go to https://kuix.de:9450
> 
> - you should see an error page
> 
> - you should be able to add an override and connect to the page
Keywords: qawanted
Whiteboard: [keep open after merge]
https://hg.mozilla.org/mozilla-central/rev/1dd81bbdb0e0
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
(In reply to Brian Smith (:bsmith) from comment #40)
> I added qawanted because that's the best way I can think of to indicate to
> QA that I want them to create a litmus test. (QA: Please educate me as to
> the proper way of doing this.)

The "right" process is to simply set the in-moztrap flag to ?. Feel free to use me as the requestee in the future. I'll review the testcase and decide whether this needs a manual test (MozTrap) or an automated test (Mozmill).

Thanks
Flags: in-qa-testsuite?
Flags: in-moztrap?(anthony.s.hughes)
Keywords: qawanted
I've added a manual test in MozTrap:
https://moztrap.mozilla.org/manage/cases/_detail/6367/

Brian, please review it and let me know if anything needs changing. I'm working in parallel with the QA Automation Development team to determine if this is something we can automate with Mozmill.
Flags: in-moztrap?(anthony.s.hughes) → in-moztrap+
CCing Henrik to speak to the automation of the MozTrap testcase in comment 44. We talked briefly on email and he thought it would be doable as a restart test.
Flags: in-qa-testsuite? → in-qa-testsuite?(hskupin)
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #44)
> I've added a manual test in MozTrap:
> https://moztrap.mozilla.org/manage/cases/_detail/6367/
> 
> Brian, please review it and let me know if anything needs changing. I'm

Brian or Kai, can one of you please check this testcase? If all the steps and expected results are valid I would like to get started with a Mozmill test. Thanks.
Henrik, thanks, sounds good to me.

In case it was accidental, there are spaces inside this string:
  security.enable_ md5 _signatures
Thanks Kai. Anthony, would you mind to file a bug for the Mozmill test implementation? Thanks.
(In reply to Kai Engert (:kaie) from comment #47)
> Henrik, thanks, sounds good to me.
> 
> In case it was accidental, there are spaces inside this string:
>   security.enable_ md5 _signatures

Yeah, this wasn't accidental. MozTrap interprets enable_md5_signatures as enablemd5signatures, where md5 is bold. I suppose that could be a bug in MozTrap.   

(In reply to Henrik Skupin (:whimboo) from comment #48)
> Thanks Kai. Anthony, would you mind to file a bug for the Mozmill test

Sure, Henrik, I'll file this bug.
> implementation? Thanks.
(In reply to Henrik Skupin (:whimboo) from comment #48)
> Thanks Kai. Anthony, would you mind to file a bug for the Mozmill test
> implementation? Thanks.

See bug 795398.
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #49)
> (In reply to Kai Engert (:kaie) from comment #47)
> > Henrik, thanks, sounds good to me.
> > 
> > In case it was accidental, there are spaces inside this string:
> >   security.enable_ md5 _signatures
> 
> Yeah, this wasn't accidental. MozTrap interprets enable_md5_signatures as
> enablemd5signatures, where md5 is bold. I suppose that could be a bug in
> MozTrap.   

Nevermind, figured this out -- the pref should now display correctly.
Hi Alex and all,

(In reply to Alex Keybl [:akeybl] from comment #35)
> > * A blog post explaining the upcoming change and how to identify the error
> 
> dveditz let me know 
> 
> > * An email to the enterprise notifying them of the change (and the fact that
> > ESR will not be affected till 17), since we expect their self-signed
> > internal apps will be most affected
> 
> I'll handle this.

Firefox 16 with this fix will be released soon. I couldn't find any blog post nor notification mail to enterprise ML. You finally decided not to write?
If you've already wrote blog/mail, could you tell me? I want to answer to Japanese cert provider who asked us about this.
In FF 16, the about:config says: security.ssl3.rsa_rc4_128_md5;true
Is this correct?
(In reply to Coyote from comment #53)
> In FF 16, the about:config says: security.ssl3.rsa_rc4_128_md5;true
> Is this correct?

The SSL cipher suites that use MD5 actually use HMAC-MD5 and/or MD5 in conjunction with SHA-1. Consequently, they are considered safe, unlike the plain MD5 hashes used in signatures.
Depends on: 816495
The mozmill test has been landed in our default branch for Nightly and will move with the trains via bug 795398.
Flags: in-qa-testsuite?(hskupin) → in-qa-testsuite+
You need to log in before you can comment on or make changes to this bug.