Closed Bug 1109613 Opened 9 years ago Closed 8 years ago

Test failure 'expertContentHeading.getNode(...) is null' in /testSecurity/testMD5HashSignature.js

Categories

(Infrastructure & Operations :: SSL Certificates, task, P1)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: cosmin-malutan, Assigned: cliang)

References

()

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/2151] [mozmill-test-failure])

Attachments

(1 file)

Test:      testMD5HashSignature    
Failure:   'expertContentHeading.getNode(...) is null'
Branches:  mozilla-central
Platforms: All

This start failing today on nightly. I could reproduce it when I ran the test alone , but it reproduced on a full testrun, this also should be skipped.
Attached patch skipSplinter Review
Here is the skip patch.
Attachment #8534862 - Flags: review?(andreea.matei)
Comment on attachment 8534862 [details] [diff] [review]
skip

Review of attachment 8534862 [details] [diff] [review]:
-----------------------------------------------------------------

I get Secure connection Failed page when running this locally or manually opening the page. Something must have changed, we need to find what. 
Skipping until then.
Attachment #8534862 - Flags: review?(andreea.matei) → review+
Secure Connection Failed

An error occurred during a connection to ssl-md5.mozqa.com. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap) 
Disabled:
http://hg.mozilla.org/qa/mozmill-tests/rev/87e20bae4726 (default)

Cosmin, please check what's wrong here and file a dependent bug if needed.
This is not a Firefox regression, Firefox simply doesn't support SSL v3.0
https://blog.mozilla.org/security/2014/10/14/the-poodle-attack-and-the-end-of-ssl-3-0/

The preference that turned that down was "security.tls.version.fallback-limit" from commit bellow:
https://hg.mozilla.org/integration/mozilla-inbound/rev/35f5ec149ad5


To summarize what test does:
1 Downloads and adds the certificate from http://mozqa.com/data/firefox/security/certificates/md5/importSSL.php
2 Adds all options as trust: "trustSSL", "trustEmail", "trustObjSign"
3 Navigates to: https://ssl-md5.mozqa.com
Expected: "This Connection is Untrusted" page opens and we should add the exception
Actual Result: "Problems loading page" page opens with the following content:
>Secure Connection Failed
> 
>An error occurred during a connection to ssl-md5.mozqa.com. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap)
> 
>    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
>    Please contact the website owners to inform them of this problem.


This certificate was generated in bug 882632 by Brandon but his account is disabled.
Masatoshi Kimura, do you know why changing this preference causes this and what should we to keep this test for MD5 security.
Flags: needinfo?(VYV03354)
ssl-md5.mozqa.com is TLS intolerant:
https://www.ssllabs.com/ssltest/analyze.html?d=ssl-md5.mozqa.com
The server software should be fixed.
Flags: needinfo?(VYV03354)
Blocks: 1084025
So what's next here so we can get this test re-enabled? What has to be fixed server-side exactly? Can this be done by adding TLS v1.2 to the list of supported protocols? Thing is that in this specific SSL test we only want to test the MD5 cert, but don't want to fallback to TLS 1.2. Masatoshi, can you give us some more details please? Thanks.
(In reply to Henrik Skupin (:whimboo) from comment #6)
> Can this be done by adding TLS v1.2 to the list of
> supported protocols?

The server does not have to support TLS 1.2. All it has to do is to negotiate with the known protocol version (e.g. TLS 1.1) instead of terminating the connection abruptly even if the client offered a higher protocol version.

> Thing is that in this specific SSL test we only want to
> test the MD5 cert, but don't want to fallback to TLS 1.2.

Sorry, I couldn't parse this sentence. What does "fallback to TLS 1.2" mean?
(In reply to Henrik Skupin (:whimboo) from comment #6)
> So what's next here so we can get this test re-enabled? What has to be fixed
> server-side exactly? Can this be done by adding TLS v1.2 to the list of
> supported protocols? Thing is that in this specific SSL test we only want to
> test the MD5 cert, but don't want to fallback to TLS 1.2. Masatoshi, can you
> give us some more details please? Thanks.

If all you want to test is the use of the MD5 cert, then you should definitely enable TLS 1.2, because that's going to avoid hitting a lot of other issues.

However, the actual issue is that the server has a bug. Firefox is saying "Hey, I support TLS 1.2 and maybe some earlier versions." The server is supposed to reply "Let's talk using TLS 1.0" but instead it barfs. This kind of bug is usually a sign that the server is using old, outdated, and probably non-secure software.

What TLS/HTTP software is the server running?

Note that Gecko's xpcshell test suite already has some tests that verify that MD5 certificates are not accepted and that MD5 certificate errors can be overridden.
Flags: needinfo?(hskupin)
(In reply to Brian Smith (:briansmith, :bsmith, use NEEDINFO?) from comment #8)
> However, the actual issue is that the server has a bug. Firefox is saying
> "Hey, I support TLS 1.2 and maybe some earlier versions." The server is
> supposed to reply "Let's talk using TLS 1.0" but instead it barfs. This kind
> of bug is usually a sign that the server is using old, outdated, and
> probably non-secure software.
> 
> What TLS/HTTP software is the server running?

Philippe was managing all the SSL related requests lately. So he will be able to give a better and more in-depth reply to you.

> Note that Gecko's xpcshell test suite already has some tests that verify
> that MD5 certificates are not accepted and that MD5 certificate errors can
> be overridden.

Requesting additional information from Kai given that this test request came in from him. Maybe the tests you are referring here have been added recently to xpcshell?
Flags: needinfo?(kaie)
Flags: needinfo?(hskupin)
Flags: needinfo?(gozer)
(In reply to Brian Smith (:briansmith, :bsmith, use NEEDINFO?) from comment #8)
> (In reply to Henrik Skupin (:whimboo) from comment #6)
> > So what's next here so we can get this test re-enabled? What has to be fixed
> > server-side exactly? Can this be done by adding TLS v1.2 to the list of
> > supported protocols? Thing is that in this specific SSL test we only want to
> > test the MD5 cert, but don't want to fallback to TLS 1.2. Masatoshi, can you
> > give us some more details please? Thanks.
> 
> If all you want to test is the use of the MD5 cert, then you should
> definitely enable TLS 1.2, because that's going to avoid hitting a lot of
> other issues.

According to Zeus (our SSL terminating load-balancer) all TLS versions are enabled and supported.

> However, the actual issue is that the server has a bug. Firefox is saying
> "Hey, I support TLS 1.2 and maybe some earlier versions." The server is
> supposed to reply "Let's talk using TLS 1.0" but instead it barfs. This kind
> of bug is usually a sign that the server is using old, outdated, and
> probably non-secure software.

Would be nice to see an easy to reproduce openssl invocation (or something similar)

> What TLS/HTTP software is the server running?

Zeus.
 
> Note that Gecko's xpcshell test suite already has some tests that verify
> that MD5 certificates are not accepted and that MD5 certificate errors can
> be overridden.

$> ./cipherscan ssl-md5.mozqa.com
......
Target: ssl-md5.mozqa.com:443

prio  ciphersuite         protocols      pfs_keysize
1     DHE-RSA-AES128-SHA  TLSv1,TLSv1.1  DH,1024bits
2     DHE-RSA-AES256-SHA  TLSv1,TLSv1.1  DH,1024bits
3     AES128-SHA          TLSv1,TLSv1.1
4     AES256-SHA          TLSv1,TLSv1.1
5     DES-CBC3-SHA        TLSv1,TLSv1.1

Is it possible this is hapenning because the server can't perform TLS1.2 with an MD5 signature?

I could try and just disable TLS1.2 on that virtual server to see what happens then.

Otherwise, open to suggestions.
Flags: needinfo?(gozer)
(In reply to Philippe M. Chiasson (:gozer) from comment #10)
> Is it possible this is hapenning because the server can't perform TLS1.2
> with an MD5 signature?

Yes, it is possible. In the ClientHello, Firefox gives a list of signature algorithms that are supported, and no MD5-based signature algorithms are in that list. It is possible that the server recognizes that the MD5-based certificate won't be accepted by the browser (because the browser told it so), and thus the server effectively rejects the certificate itself before Firefox even has a chance to.

> I could try and just disable TLS1.2 on that virtual server to see what
> happens then.

That might work, because it might cause the server to ignore the signature_algorithms extension when TLS 1.2 is not used. However, it also might not work, if the server (wrongly?) processes the signature_algorithms extension for TLS < 1.2.
(In reply to Brian Smith (:briansmith, :bsmith, use NEEDINFO?) from comment #11)
> > I could try and just disable TLS1.2 on that virtual server to see what
> > happens then.
> 
> That might work, because it might cause the server to ignore the
> signature_algorithms extension when TLS 1.2 is not used. However, it also
> might not work, if the server (wrongly?) processes the signature_algorithms
> extension for TLS < 1.2.

Lets get this tested at least. Maybe we have to disable all TLS and SSL versions?
As long as this is not fixed we also have to skip it for esr31:
https://hg.mozilla.org/qa/mozmill-tests/rev/c931c2307e55

Philippe, can we get comment 11 tested? Looks like we missed to follow-up on that comment in all the last months.
Flags: needinfo?(kaie) → needinfo?(gozer)
This needs to bounce into the webops queue...
Assignee: nobody → server-ops-webops
Flags: needinfo?(gozer)
Can someone from webops please check comment 11 so we can make progress on this bug? Thanks.
QA Contact: mozmill-tests → hskupin
Nearly 3 months passed by here without a reply from webops. Philippe do you know whom from webop we could ask?
Flags: needinfo?(gozer)
Let's see if Atoll can pick this one up!
Flags: needinfo?(gozer) → needinfo?(rsoderberg)
(In reply to Henrik Skupin (:whimboo) from comment #16)
> Nearly 3 months passed by here without a reply from webops.

I apologize that you waited 3 months unnecessarily; please do consider reaching out through non-Bugzilla means in the future, to ensure that we're aware of your request. Three months ago, the Assignee or QA Contact of this bug was altered to a non-monitored Bugzilla placeholder account. We do not use that particular approach in Bugzilla. You can bring things to our awareness by any/all of:

- file a dependent bug in a WebOps queue
- cc: a Webops team member
- reach out to our manager on IRC
- email infra-webops@mozilla.com.

(In reply to Cosmin Malutan, [:cosmin-malutan] from comment #4)
> To summarize what test does:
> 1 Downloads and adds the certificate from
> http://mozqa.com/data/firefox/security/certificates/md5/importSSL.php
> 2 Adds all options as trust: "trustSSL", "trustEmail", "trustObjSign"
> 3 Navigates to: https://ssl-md5.mozqa.com
> Expected: "This Connection is Untrusted" page opens and we should add the
> exception
> Actual Result: "Problems loading page" page opens with the following content:
> >Secure Connection Failed
> > 
> >An error occurred during a connection to ssl-md5.mozqa.com. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap)
> > 
> >    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
> >    Please contact the website owners to inform them of this problem.

This is a very helpful summary, thank you. We will use it to try and reconfigure the VIP to behave correctly once more.

> This certificate was generated in bug 882632 by Brandon but his account is
> disabled.

Brandon is formerly of Webops, so we'll pick up from here.

No other bug has been filed about this issue, so I'm claiming this bug into the Webops queue so that we can track it and respond appropriately.

We are under a mandatory change freeze due to the Firefox release this week, and will not be able to make alterations to repair this issue until November 11th (six days from now). We are permitted to override this for emergency changes that interfere with the launch week or with our ability to do business as a company. If you need such an override, please let us know; otherwise, sadly, we'll need to wait until the 11th to help you with this.
Component: Mozmill Tests → WebOps: SSL and Domain Names
Flags: needinfo?(rsoderberg)
Product: Mozilla QA → Infrastructure & Operations
QA Contact: hskupin → smani
Version: unspecified → other
(In reply to Richard Soderberg [:atoll] from comment #18)

> I apologize that you waited 3 months unnecessarily; please do consider
> reaching out through non-Bugzilla means in the future, to ensure that we're
> aware of your request.

Yes, please. You can always email the team at infra-webops at mozilla dot com.
Richard no problems at all. It can surely wait 6 more days given that it was stuck that long. I didn't reach out because I thought the updated QA contact would work. Anyway, I will remember and good to see some progress will be made soon. Thanks in advance.
Whiteboard: [mozmill-test-failure] → [kanban:https://webops.kanbanize.com/ctrl_board/2/2151] [mozmill-test-failure]
I've explicitly disabled TLS 1.2 and explicitly enabled SSL v2 and SSL v3 (for fallback).  Can you please test again and let me know what the results are?
Flags: needinfo?(hskupin)
Flags: needinfo?(brian)
Assignee: server-ops-webops → cliang
I don't work on this stuff any more.
Flags: needinfo?(brian)
Sorry for the slow reply. I wanted to have a look at the changes before I left for Christmas holidays but wasn't able to get around them. I will get it tested shortly.
So when I run the Mozmill test for the MD5 hash signature it now works with a release build of Firefox 43.0. So it looks like that those changes made it work.

But I wonder if we should only keep secure versions enabled as fallback, which would include TLS 1.0 and 1.1 but not any SSL version.
Flags: needinfo?(hskupin) → needinfo?(cliang)
I don't believe we can offer you MD5 and also TLS at the same time. I *think* that Zeus restricts that to the SSL3 protocol only. So I'd rather not make any further changes to the "ssl-md5" VIP, which is now SSL (3.0) and MD5 as originally intended, and if you want to set up a *new* endpoint for new tests, we're happy to help - just file a new bug in WebOps: SSL with a precise definition of which ciphers/protocols/etc. you want enabled and we'll see if we can make that happen.

RESO FIXE per comment 24, but please feel free to reopen if the MD5 hash signature tests break again.
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(cliang)
Resolution: --- → FIXED
Alright. Will do. Thanks for the changes then!

FYI the test will still not work starting from current Firefox Beta 44.0 most likely because of the security center changes. I don't think that anything here should be server-side related.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: