1146314 - Test failure 'sec_error_expired_certificate != sec_error_unknown_issuer' in testSecurityNotification.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This test failure started to happen over the weekend and totally breaks our tests across Nightly, and Aurora so far. I haven't tested with beta and release yet.

ERROR | Test Failure | {
  "fail": {
    "message": "The error code is a SEC Expired certificate error - 'summitbook.mozilla.org uses an invalid security certificate.\n\nThe certificate is not trusted because the issuer certificate is unknown.\nThe server might not be sending the appropriate intermediate certificates.\nAn additional root certificate may need to be imported.\nThe certificate expired on 7/8/2011 5:41 AM. The current time is 3/22/2015 6:57 AM.\n\n(Error code: sec_error_unknown_issuer)\n' should contain 'sec_error_expired_certificate'", 
    "fileName": "file:///c:/jenkins/workspace/mozilla-central_remote/data/mozmill-tests/firefox/tests/remote/testSecurity/testSecurityNotification.js", 
    "name": "testSecNotification", 
    "lineNumber": 70
  }

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Definitely a regression in our tests or in Firefox as started yesterday March 22nd.

Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b8e628af0b5c&tochange=82ae3b4e2215

It's worth investigating because it's our P1 test area for conversation to Marionette.

I will skip this test on default and aurora for now.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: skip

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Skip patch landed:

https://hg.mozilla.org/qa/mozmill-tests/rev/2c26c90e52db (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/b1d615a2c8b9 (aurora)

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As it looks like an expired certificate is no longer listed as such in the certificate exception page. Instead it is marked as unknown issuer. The change in behavior came in via this merge from m-i:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2a404169de2d&tochange=f949be6cd23e

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This is clearly a regression in NSPR and started with the version bump to 3.18 via bug 1137470.

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=2da0cd9e724a&tochange=289fa2204f9f

I will attach two screenshots for pre and post behavior. I assume that Kai will be able to tell us if something unusual is happening here or if we have to update our test.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Screenshot correct error code.png

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Screenshot wrong error code.png

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Steps:

1. Take latest Nightly or Aurora
2. Open https://summitbook.mozilla.org/
3. Expand the "Technical Details" section

As you will notice the error code has been changed: the sec_error_expired_certificate => sec_error_unknown_issuer

Kaie, if that is a problem I will get a new bug filed for NSPR. Please let me know. Thanks.

Comment 9

[Kai Engert (:KaiE:)]

Henrik, the change in behaviour is expected.

The upgrade to NSS included bug 986019. We removed an old Equifax root CA certificate from the NSS and Firefox trust store, because we are phasing out CA certificates that use a weaker key with only 1024 bit.

The server certificate used by summitbook.mozilla.org was issued by the CA that we removed.

Issuer: "OU=Equifax Secure Certificate Authority,O=Equifax,C=US"

Subject: "CN=summitbook.mozilla.org,O=Mozilla Corporation,L=Mountain View,ST=California,C=US,serialNumber=-csW8OPdwDOZBFy1pK2SKOEzx7o9IqlO"

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I see! Thanks a lot for this information Kai. As it looks like we are badly blocked on running this test until IT is able to give us a new expired certificate via bug 1106077. :(

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: security_notification v1

Similar to the test for Marionette lets skip the code part which checks for the expired certificate only. Would be good to keep the other tests running.

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8581680 [details] [diff] [review]
security_notification v1

Review of attachment 8581680 [details] [diff] [review]:
-----------------------------------------------------------------

This patch trades one mode of skipping a test for another that skips less of the test. Is review for changes like these really necessary?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Maybe not. So in the future we should only ask for review if code changes are involved. But thanks for doing the review.

Landed the changes as:

https://hg.mozilla.org/qa/mozmill-tests/rev/b1589c37920a (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/27944d3cca56 (aurora)

I will remove the skipped whiteboard entry given that only parts of the test are skipped now. The bug will stay open until we can make use of the code again.

Syd, maybe you will have to re-enable it once the dependent bug has been fixed.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Bug 1148182 finally fixed that issue.