1535666 - Perma browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | That message should appear - Got false, expected true when Gecko 67 merges to Beta on 2019-03-15

Status: RESOLVED FIXED

(Firefox :: General, defect, P1)

Description

[Arthur Iakab [arthur_iakab]]

[Tracking Requested - why for this release]:

Central as beta simulation :https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=41345e0f6b586cb2e26095254d68e32b83ee92c7&selectedJob=234155120

Log https://treeherder.mozilla.org/logviewer.html#?job_id=234155120&repo=try

Part of that log
[task 2019-03-15T15:17:30.581Z] 15:17:30 INFO - TEST-START | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js
[task 2019-03-15T15:17:30.921Z] 15:17:30 INFO - TEST-INFO | started process screentopng
[task 2019-03-15T15:17:31.269Z] 15:17:31 INFO - TEST-INFO | screentopng: exit 0
[task 2019-03-15T15:17:31.270Z] 15:17:31 INFO - Buffered messages logged at 15:17:30
[task 2019-03-15T15:17:31.270Z] 15:17:31 INFO - Entering test bound checkUntrustedCertError
[task 2019-03-15T15:17:31.271Z] 15:17:31 INFO - Loading https://no-subject-alt-name.example.com:443 which does not have a subject specified in the certificate
[task 2019-03-15T15:17:31.271Z] 15:17:31 INFO - Loading and waiting for the cert error
[task 2019-03-15T15:17:31.271Z] 15:17:31 INFO - Clicking the exceptionDialogButton in advanced panel
[task 2019-03-15T15:17:31.271Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | advancedButton found - {} == true -
[task 2019-03-15T15:17:31.271Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | advancedButton should be clickable - false == false -
[task 2019-03-15T15:17:31.272Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | badCertAdvancedPanel found - {} == true -
[task 2019-03-15T15:17:31.274Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | badCertTechnicalInfo found - {} == true -
[task 2019-03-15T15:17:31.274Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | "SSL_ERROR_BAD_CERT_DOMAIN" == "SSL_ERROR_BAD_CERT_DOMAIN" -
[task 2019-03-15T15:17:31.275Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | viewCertificate found - {} == true -
[task 2019-03-15T15:17:31.276Z] 15:17:31 INFO - Buffered messages finished
[task 2019-03-15T15:17:31.276Z] 15:17:31 INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | That message should appear - Got false, expected true
[task 2019-03-15T15:17:31.277Z] 15:17:31 INFO - Stack trace:
[task 2019-03-15T15:17:31.278Z] 15:17:31 INFO - chrome://mochikit/content/browser-test.js:test_is:1315
[task 2019-03-15T15:17:31.278Z] 15:17:31 INFO - chrome://mochitests/content/browser/browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js:checkCorrectMessages:36
[task 2019-03-15T15:17:31.279Z] 15:17:31 INFO - chrome://mochitests/content/browser/browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js:checkUntrustedCertError:49
[task 2019-03-15T15:17:31.280Z] 15:17:31 INFO - TEST-PASS | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | That message shouldn't appear -
[task 2019-03-15T15:17:31.280Z] 15:17:31 INFO - Leaving test bound checkUntrustedCertError
[task 2019-03-15T15:17:31.281Z] 15:17:31 INFO - GECKO(1071) | MEMORY STAT | vsize 639MB | residentFast 273MB | heapAllocated 97MB
[task 2019-03-15T15:17:31.282Z] 15:17:31 INFO - TEST-OK | browser/base/content/test/about/browser_aboutCertError_noSubjectAltName.js | took 399ms
[task 2019-03-15T15:17:31.282Z] 15:17:31 INFO - checking window state

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Attachment: Bug 1535666 - Explicitly turn on new cert error pages in browser_aboutCertError_noSubjectAltName.js to let it pass on beta

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b12dda048dd5
Explicitly turn on new cert error pages in browser_aboutCertError_noSubjectAltName.js to let it pass on beta r=Gijs

Comment 3

[Andreea Pavel [:apavel]]

Unfortunately, the patch here does not solve the problem. I imported it in the latest beta sims and the failure still occurs:

https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&revision=1c5b6c8feb9874db9c532be6a7252b78358a4cff&selectedJob=234339474

Comment 4

[Andreea Pavel [:apavel]]

Backed out from autoland

Backout: https://hg.mozilla.org/integration/autoland/rev/55935c0cdcb403767b529b865be953b7bc529157

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified the new version works with Firefox Developer Edition.

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Gijs, can you review this, please? The new version is completely different from the old one but Phabricator still remembers the review from the previous version. Thank you.

Comment 7

[:Gijs (away/busy until Mar 10)]

r=me.

Johann, can you file a follow-up to fix these tests to Not Do This?

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/995b45bd8313
Explicitly turn on new cert error pages in browser_aboutCertError_noSubjectAltName.js to let it pass on beta r=Gijs

Comment 9

[Johann Hofmann [:johannh]]

I'm confused, not to do what? Mention Nightly? Didn't you just fix that?

Comment 10

[:Gijs (away/busy until Mar 10)]

(In reply to Johann Hofmann [:johannh] from comment #9)

I'm confused, not to do what? Mention Nightly? Didn't you just fix that?

Not to test for strings shown at all, really...

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/d15b22ef4b88a6be3e0adaf3163022e61fe3e275

Comment 12

[Intermittent Failures Robot]

19 failures in 413 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 19

Platform breakdown:

• macosx64-devedition: 2
• macosx64-nightly: 2
• linux64-nightly: 2
• windows10-64-devedition: 2
• windows10-64-nightly: 2
• windows7-32-devedition: 2
• linux64-devedition: 2
• linux32-devedition: 2
• windows7-32-nightly: 2
• linux32-nightly: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1535666&startday=2019-03-18&endday=2019-03-18&tree=all

Comment 13

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/995b45bd8313

Comment 14

[Johann Hofmann [:johannh]]

Thank you for fixing this up and sorry for the hassle :)

As discussed on IRC, not having Fluent (or another way to dynamically get ahold of these strings) means we would effectively get rid of a lot of helpful assertions by doing this. I think that this bug was quite a special case and that folks who are updating the english in-tree copy can be bothered to update tests (speaking as the person who has done or reviewed most of the recent copy updates here).

If these sorts of incidents repeat I'm happy to reconsider.

Comment 15

[Intermittent Failures Robot]

19 failures in 2561 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 19

Platform breakdown:

• macosx64-devedition: 2
• macosx64-nightly: 2
• linux64-nightly: 2
• windows10-64-devedition: 2
• windows10-64-nightly: 2
• windows7-32-devedition: 2
• linux64-devedition: 2
• linux32-nightly: 1
• linux32-devedition: 2
• windows7-32-nightly: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1535666&startday=2019-03-18&endday=2019-03-24&tree=all