1099123 - [en-US] Test failures in /testSecurity/testSSLDisabledErrorPage.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Mihaela Velimiroviciu (:mihaelav)]

Module:    /testSecurity/testSSLDisabledErrorPage.js
Test:      testDisableSSL
Failures:   
* The SSL error message contains disabled information - 'Firefox cannot guarantee the safety of your data on mozqa.com because it uses SSLv3, a broken security protocol.' should contain 'ssl_error_no_cypher_overlap'
* The correct SSL error title is shown - 'Unable to Connect Securely' should equal 'Secure Connection Failed'
Branches:  Beta (34.0)
Platforms: All

Reports: http://mozmill-release.blargon7.com/#/remote/failure?app=Firefox&branch=All&platform=All&from=2014-11-07&to=2014-11-14&test=%2FtestSecurity%2FtestSSLDisabledErrorPage.js&func=testDisableSSL

Started to fail today

Comment 1

[Andrei Eftimie]

Probably caused by bug 1096695
http://hg.mozilla.org/releases/mozilla-beta/rev/d585e4e50468

Comment 2

[Andrei Eftimie]

This is hardcoded for en-US and 34.

Comment 3

[Andrei Eftimie]

So the only way to have this not fail our tests, is for us to also hardcode the same strings for en-US && 34.

Check http://hg.mozilla.org/releases/mozilla-beta/rev/d585e4e50468 for what we need changing.

Henrik, would this be fine with you?
I'm not that comfortable with it, but we'll only need this for 34, so this hack will only live for 6 weeks.

Comment 4

[:Gijs (he/him)]

(In reply to Andrei Eftimie from comment #3)
> So the only way to have this not fail our tests, is for us to also hardcode
> the same strings for en-US && 34.
> 
> Check http://hg.mozilla.org/releases/mozilla-beta/rev/d585e4e50468 for what
> we need changing.
> 
> Henrik, would this be fine with you?
> I'm not that comfortable with it, but we'll only need this for 34, so this
> hack will only live for 6 weeks.

34 *and 35* which just landed, too.

Comment 5

[Andrei Eftimie]

Attachment: hack.patch

This is for 34. I haven't yet tested it enough. Works fine on en-US, but I'd like to test other locales first. (Should be fine, but still).

Today's build for 35 is not affected yet. With Gijs info, this will fail in the following days on 35 en-US builds as well.

We can land this on Monday morning when we can also get the fix for 35.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrei Eftimie from comment #3)
> So the only way to have this not fail our tests, is for us to also hardcode
> the same strings for en-US && 34.
> 
> Check http://hg.mozilla.org/releases/mozilla-beta/rev/d585e4e50468 for what
> we need changing.

FYI I will add Axel and Francesco, so they are aware of that change.

> Henrik, would this be fine with you?
> I'm not that comfortable with it, but we'll only need this for 34, so this
> hack will only live for 6 weeks.

It would be around for more than 12 weeks. But if this is only temporary, it should be fine. But maybe Axel and Flod have an opinion here too.

Comment 7

[Francesco Lodolo [:flod]]

We're aware of that change: it's enabled only for en-US and hard-coded in Fx 34 and Fx 35, it should be fully localizable in Fx 36 (bug 1098371). I guess a temporary fix to the test is OK.

Comment 8

[Andrei Eftimie]

For 36 this test is still passing, we'll monitor bug 1098371, and fix it appropriately when this changes.

attachment 8523002 [details] [diff] [review] is fixes the problem for both 34 and 35. I've tested various locales, and all appears to be working nicely.

We do have another, unrelated, problem with this test, as something regressed over the weekend on the server / certificate side on mozqa.com . There's bug 1100236 for that, so even if we fix this issue, the test is currently still failing because of that.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8523002 [details] [diff] [review]
hack.patch

Review of attachment 8523002 [details] [diff] [review]:
-----------------------------------------------------------------

::: firefox/tests/remote/testSecurity/testSSLDisabledErrorPage.js
@@ +54,5 @@
> +  // Bug 1096695
> +  // 34 en-US has this title hardcoded
> +  if (utils.appInfo.locale === "en-US") {
> +    nssFailure2title = "Unable to Connect Securely";
> +  }

If it is hard-coded for en-US, it should also be for l10n builds. So this test should fail at the moment for localized builds of the 35 and 34 branch.

Comment 10

[Francesco Lodolo [:flod]]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> If it is hard-coded for en-US, it should also be for l10n builds. So this
> test should fail at the moment for localized builds of the 35 and 34 branch.

It's actually disabled for l10n (they display the old page, not English text).

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8523002 [details] [diff] [review]
hack.patch

Review of attachment 8523002 [details] [diff] [review]:
-----------------------------------------------------------------

I see, so please update the comment so it lists Firefox 34.0 and 35.0. With that r=me.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This patch should really have been updated and landed yesterday. We get all sorts of failures now. Cosmin please quickly fix it so we can land for 35 and 34.

Comment 13

[Cosmin Malutan, [:cosmin-malutan]]

Sure, whimboo, but I just ran the latest beta en-US and ro locales with and without the patch and it fails every single time with the failure from bug 972770. My opinion is that we should back-port the skip patch there.

Comment 14

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: bug1099123.patch

I updated the comment but it still fails for me with the failure in bug 972770.

Comment 15

[Andrei Eftimie]

Comment on attachment 8524477 [details] [diff] [review]
bug1099123.patch

Review of attachment 8524477 [details] [diff] [review]:
-----------------------------------------------------------------

Please also change the test to use a subdomain of mozqa.com where we can see a the Security Error page, this should work. I can't properly test this ATM.

Comment 16

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8524477 [details] [diff] [review]
bug1099123.patch

Review of attachment 8524477 [details] [diff] [review]:
-----------------------------------------------------------------

That fix is in bug 972770, where it belongs for that failure fix. With both fixes this passes now, so I'll land this first and then the other which also enables the test.

http://hg.mozilla.org/qa/mozmill-tests/rev/0b1e03963fe9 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/3f7a4d9e4248 (beta)

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrei Eftimie from comment #15)
> Please also change the test to use a subdomain of mozqa.com where we can see
> a the Security Error page, this should work. I can't properly test this ATM.

That will be done by the other bug 972770.

Comment 18

[Mihaela Velimiroviciu (:mihaelav)]

Failed again with Nightly across all platforms:
http://mozmill-daily.blargon7.com/#/remote/failure?app=Firefox&branch=All&platform=All&from=2015-01-01&to=2015-01-12&test=%2FtestSecurity%2FtestSSLDisabledErrorPage.js&func=testDisableSSL

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Please do not reopen bugs which have been marked as fixed months ago. File always a new one. Those are cheap enough. Thanks.