1067806 - Test failure "Identity is verified - 'verifiedIdentity' should equal 'verifiedDomain'" in testDVCertificate.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Andreea Matei [:AndreeaMatei]]

Started to fail today with the beta build, on OSX and Linux so far (I imagine windows will run soon). Might be a regression, Daniel will look into this, we need a skip patch too.

http://mozmill-release.blargon7.com/#/remote/report/ee6ae3df35a164c4abc5cec1a3a1c724

Affected line:
http://hg.mozilla.org/qa/mozmill-tests/file/mozilla-beta/firefox/tests/remote/testSecurity/testDVCertificate.js#l52

Comment 1

[Andrei Eftimie]

We're using https://ssl-dv.mozqa.com and we've got some updates regarding ceritifcates on this machine yesterday in bug 891288. 

I don't see that any obvious change in regards to the ssl-dv subdomain should have taken place...
we'll need to investigate a bit.

Comment 2

[Andrei Eftimie]

Actually the changes from bug 891288 went live on the 12th, so this issue is probably something else... ugh.

Using one of the new subdomains made available (like sslv3.mozqa.com) makes this test pass again since that is a Self-Signed cert. 

(I am by far not an expert on certificates, so all this might be wrong).
It seems that https://ssl-dv.mozqa.com is not using an external signed (verified) cert where we previously had a self-signed one on this domain specific for tests like this one.

Comment 3

[Daniel Gherasim]

Attachment: skip.patch

Applies cleanly on all branches.

Comment 4

[Andrei Eftimie]

Seems this way, the certificate shown on https://ssl-dv.mozqa.com has been generated on 15/09/14. So indeed someone has changed our certificate yesterday.

Lets disable the test for now until we figure this out.

This is worrisome as we didn't request this, and didn't receive any warning that one of our certificates used for automation would be changed!

Philippe (I am asking you since you did work on mozqa.com certs recently, and you might point us in the right direction),
do you know of this change, why it has been done?

This domain should have hosted a self-signed cert AFAIK.

Comment 5

[Andrei Eftimie]

Comment on attachment 8489873 [details] [diff] [review]
skip.patch

Review of attachment 8489873 [details] [diff] [review]:
-----------------------------------------------------------------

Disabled:
https://hg.mozilla.org/qa/mozmill-tests/rev/c53a38a762af (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/bab73a7f5695 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/f27573388647 (mozilla-beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/53e68e0619bd (mozilla-release)
https://hg.mozilla.org/qa/mozmill-tests/rev/1557e32abbfb (mozilla-esr31)

Comment 6

[Andrei Eftimie]

As a fix we can switch to one of the new domains made available in bug 891288. I'm currently checking them and all our dependencies, we might switch multiple test domains once everything is fine.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No, as the description of the failure says, we explicitly check for a "verifiedDomain". So using a self-signed cert will not be helpful here. It's indeed worrisome that it has been changed and that the test is causing problems now, but as I see we have a duration of a year, so maybe it had to be extended. But even then the test should not fail.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So the problem here is that we do no longer have a DV but an EV cert:

DigiCert SHA2 Extended Validation Server CA

It may be that the wrong certificate has been chosen for this virtual host. Lets get this fixed on the other bug.

Comment 9

[Philippe M. Chiasson (:gozer)]

Yes, mea culpa, that was me. The previous cert was expiring soon, so I renewed it.

I didn't realize this was used by automation, but thought it was merely for one of the mozqa websites.

And, of course, renewing it as an EV very instead of a DV one on top of it.

I will re-issue a new one and fix this today.

Apologies and in the future, all mozqa certs will get renewed with notice with QA first so you know what's happening.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Philippe, if the costs are not that much for EV, I would say we keep that, and create a ssl-ev.mozqa.com subdomain for this type. Initially we haven't created such one given we were told it's too expensive. But if that is not the case we really want to have that! So lets benefit from this accident!

Comment 11

[Philippe M. Chiasson (:gozer)]

(In reply to Henrik Skupin (:whimboo) from comment #10)
> Philippe, if the costs are not that much for EV, I would say we keep that,
> and create a ssl-ev.mozqa.com subdomain for this type. Initially we haven't
> created such one given we were told it's too expensive. But if that is not
> the case we really want to have that! So lets benefit from this accident!

That's easily done, as I can just change the name of the new certificate to ssl-ev.mozqa.com. I'll get that done and setup a separate virtual server for it.

For the DV certificate, things are more interesting, as our current SSL provider (DigiCert) doesn't issue DV certificates (https://www.digicert.com/dv-ssl-certificate.htm)

I'll need to renew the old cert with another SSL provider, and that will be a little bit longer.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, lets continue with that on the other bug then. Thanks!

Comment 13

[Andreea Matei [:AndreeaMatei]]

We need to backport to esr24 as well, the patch is not applying. Daniel please check this asap. Thanks!

Comment 14

[Daniel Gherasim]

Attachment: skip-esr24.patch

Comment 15

[Andrei Eftimie]

Comment on attachment 8490654 [details] [diff] [review]
skip-esr24.patch

Review of attachment 8490654 [details] [diff] [review]:
-----------------------------------------------------------------

Disabled:
https://hg.mozilla.org/qa/mozmill-tests/rev/0b65fd51847a (mozilla-esr24)

Comment 16

[Philippe M. Chiasson (:gozer)]

Certificate renewal issued with Thawte, awaiting approval

Comment 17

[Philippe M. Chiasson (:gozer)]

Certificate on ssl-dv.mozqa.com updated to a Domain Verified one from Thawte:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            19:6b:f5:28:10:98:75:fa:33:82:a0:93:4e:10:07:3a
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=thawte, Inc., OU=Domain Validated SSL, CN=thawte DV SSL CA - G2
        Validity
            Not Before: Sep 17 00:00:00 2014 GMT
            Not After : Sep 16 23:59:59 2016 GMT
        Subject: CN=ssl-dv.mozqa.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b5:6e:b1:96:47:15:4a:d4:cf:c7:03:68:17:d2:
                    fe:31:c6:79:b8:4a:de:8d:c6:d0:48:14:b3:53:c8:
                    1e:1f:3f:62:dc:9a:6a:b7:9f:4e:11:4f:f7:17:b2:
                    49:62:55:0d:56:da:40:6d:d3:b2:eb:2a:25:07:9e:
                    22:ea:c7:c7:cf:c0:4e:56:ec:af:a2:37:8a:ab:f7:
                    54:be:41:3a:72:a3:1c:c1:72:9d:30:cb:68:56:9e:
                    24:e2:cf:ae:39:56:bb:6d:ce:f2:89:53:e0:65:e8:
                    c2:bd:27:5e:f2:62:da:1d:22:c4:2f:20:2b:29:ff:
                    67:c1:13:f2:81:d1:05:47:de:a4:86:c1:3e:57:ba:
                    2a:31:6b:68:dc:e9:ac:81:8d:5b:51:43:8d:00:ae:
                    8d:63:8a:b9:da:f4:29:d5:2a:93:06:af:ee:e6:cc:
                    be:82:3e:f2:97:ea:c1:0e:c9:8d:8f:d6:9b:9d:b2:
                    d2:46:0d:dd:91:65:89:f8:67:17:0c:bd:77:00:7f:
                    d9:43:cb:16:b3:f3:fa:01:5e:d1:3d:a6:ed:a4:ff:
                    6b:f5:9a:d6:b8:31:00:c4:80:41:2a:72:42:2c:29:
                    e9:4f:0c:7d:7a:3e:15:60:da:40:d5:b3:36:fd:5a:
                    bb:1c:52:e9:a1:dc:c2:41:64:61:b9:c4:c6:76:64:
                    a9:bb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:ssl-dv.mozqa.com
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://tn.symcb.com/tn.crl

            X509v3 Certificate Policies: 
                Policy: 2.16.840.1.113733.1.7.54
                  CPS: https://www.thawte.com/cps
                  User Notice:
                    Explicit Text: https://www.thawte.com/repository

            X509v3 Authority Key Identifier: 
                keyid:9F:B8:C1:A9:6C:F2:F5:C0:22:2A:94:ED:5C:99:AC:D4:EC:D7:C6:07

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            Authority Information Access: 
                OCSP - URI:http://tn.symcd.com
                CA Issuers - URI:http://tn.symcb.com/tn.crt

    Signature Algorithm: sha256WithRSAEncryption
         c9:27:3f:8e:1c:f5:ac:42:71:7c:80:2d:b8:3b:fe:ba:4f:79:
         b4:62:26:50:9e:40:1e:fd:37:b4:13:96:96:9c:37:d7:74:4e:
         98:01:e1:00:71:cd:bc:5b:4d:7e:d9:07:a9:2d:ff:a9:29:a7:
         d1:09:c3:64:79:77:f1:ee:9f:d6:82:a7:ea:36:98:b1:e7:31:
         0f:da:7b:05:ee:2a:d6:25:b2:3d:52:9e:05:79:79:e5:55:c5:
         f2:09:96:b0:8e:2f:a5:5b:c0:38:f1:03:22:02:d7:07:50:03:
         32:4f:2b:27:bd:e3:3d:69:ea:78:47:1d:6d:79:31:72:70:f5:
         5e:1f:db:c9:57:20:fe:78:44:55:0e:7d:2a:6d:5d:5b:41:97:
         0f:f6:01:f6:76:bb:82:00:d0:bb:ca:4f:af:ea:77:c5:02:18:
         f8:ee:2a:20:14:5b:37:8c:4e:42:c1:7d:3f:0c:6f:b6:64:8e:
         6d:23:60:43:8d:9b:a9:cd:62:e2:05:67:1e:8e:f2:75:f4:02:
         20:1f:c6:9b:dd:8c:80:45:8a:11:81:5c:3c:89:dd:3c:46:36:
         24:e0:81:9f:00:68:9f:f6:9f:ef:d6:9f:63:c6:51:c4:ff:18:
         82:89:24:4b:69:c3:40:58:42:b3:af:fc:c3:5a:2f:86:02:2f:
         9b:27:2a:03
-----BEGIN CERTIFICATE-----
MIIEgDCCA2igAwIBAgIQGWv1KBCYdfozgqCTThAHOjANBgkqhkiG9w0BAQsFADBj
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMdGhhd3RlLCBJbmMuMR0wGwYDVQQLExRE
b21haW4gVmFsaWRhdGVkIFNTTDEeMBwGA1UEAxMVdGhhd3RlIERWIFNTTCBDQSAt
IEcyMB4XDTE0MDkxNzAwMDAwMFoXDTE2MDkxNjIzNTk1OVowGzEZMBcGA1UEAwwQ
c3NsLWR2Lm1venFhLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALVusZZHFUrUz8cDaBfS/jHGebhK3o3G0EgUs1PIHh8/YtyaarefThFP9xeySWJV
DVbaQG3TsusqJQeeIurHx8/ATlbsr6I3iqv3VL5BOnKjHMFynTDLaFaeJOLPrjlW
u23O8olT4GXowr0nXvJi2h0ixC8gKyn/Z8ET8oHRBUfepIbBPle6KjFraNzprIGN
W1FDjQCujWOKudr0KdUqkwav7ubMvoI+8pfqwQ7JjY/Wm52y0kYN3ZFlifhnFwy9
dwB/2UPLFrPz+gFe0T2m7aT/a/Wa1rgxAMSAQSpyQiwp6U8MfXo+FWDaQNWzNv1a
uxxS6aHcwkFkYbnExnZkqbsCAwEAAaOCAXYwggFyMBsGA1UdEQQUMBKCEHNzbC1k
di5tb3pxYS5jb20wCQYDVR0TBAIwADArBgNVHR8EJDAiMCCgHqAchhpodHRwOi8v
dG4uc3ltY2IuY29tL3RuLmNybDByBgNVHSAEazBpMGcGCmCGSAGG+EUBBzYwWTAm
BggrBgEFBQcCARYaaHR0cHM6Ly93d3cudGhhd3RlLmNvbS9jcHMwLwYIKwYBBQUH
AgIwIwwhaHR0cHM6Ly93d3cudGhhd3RlLmNvbS9yZXBvc2l0b3J5MB8GA1UdIwQY
MBaAFJ+4wals8vXAIiqU7VyZrNTs18YHMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUE
FjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwVwYIKwYBBQUHAQEESzBJMB8GCCsGAQUF
BzABhhNodHRwOi8vdG4uc3ltY2QuY29tMCYGCCsGAQUFBzAChhpodHRwOi8vdG4u
c3ltY2IuY29tL3RuLmNydDANBgkqhkiG9w0BAQsFAAOCAQEAySc/jhz1rEJxfIAt
uDv+uk95tGImUJ5AHv03tBOWlpw313ROmAHhAHHNvFtNftkHqS3/qSmn0QnDZHl3
8e6f1oKn6jaYsecxD9p7Be4q1iWyPVKeBXl55VXF8gmWsI4vpVvAOPEDIgLXB1AD
Mk8rJ73jPWnqeEcdbXkxcnD1Xh/byVcg/nhEVQ59Km1dW0GXD/YB9na7ggDQu8pP
r+p3xQIY+O4qIBRbN4xOQsF9PwxvtmSObSNgQ42bqc1i4gVnHo7ydfQCIB/Gm92M
gEWKEYFcPIndPEY2JOCBnwBon/af79afY8ZRxP8YgokkS2nDQFhCs6/8w1ovhgIv
mycqAw==
-----END CERTIFICATE-----

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Not fixed yet. We still have to re-enable our test across all the branches.

Daniel, please check if we have to change something in the test given that this new cert is from Thawte now.

Comment 19

[Andrei Eftimie]

Test is passing fine now: http://mozmill-crowd.blargon7.com/#/remote/report/2f982f72826307fed840a3b11c482ce0

Thanks Philippe for the help with the certs here!

Backed out skip:
https://hg.mozilla.org/qa/mozmill-tests/rev/1f4f24b57b0f (default)

Comment 20

[Andrei Eftimie]

Backout transplanted:
https://hg.mozilla.org/qa/mozmill-tests/rev/d5e70e946177 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/c4d338fb4c78 (mozilla-beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/bfd19703382d (mozilla-release)
https://hg.mozilla.org/qa/mozmill-tests/rev/1fad361f3502 (mozilla-esr31)