Closed Bug 768358 Opened 12 years ago Closed 12 years ago

OV certificate for mozqa.com has expired

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: whimboo, Assigned: bburton)

References

(
URL
)

Details

We want to use mozqa.com for SSL tests and there is already a subdomain up with an EV SSL certificate. Sadly this one has been expired:

The certificate expired on 6/24/12 11:28 PM. The current time is 6/26/12 10:21 AM.

We need a new one as soon as possible to continue moving our SSL tests to that machine.
Blocks: 661121
So how would I go about getting a new SSL certificate?
(In reply to Jason Smith [:jsmith] from comment #1)
> So how would I go about getting a new SSL certificate?

Lets CC Al so he can give instructions. Also setting the user doc needed keyword so we can get this documented on MDN for later usage.
Keywords: user-doc-needed
Hm, that reminds me that Al is away the next time. So I have checked bug 639932 where we have requested the DV cert and it looks like IT would be in charge here.

mrz, could you help us so that we can get this service up again? Do we have to move it to another product/component? Thanks.
Seeing no action here we probably have to move it into the IT section.
Assignee: nobody → server-ops-infra
Component: Infrastructure → Server Operations: Infrastructure
Product: Mozilla QA → mozilla.org
QA Contact: jdow
Version: unspecified → other
Assignee: server-ops-infra → server-ops-webops
Component: Server Operations: Infrastructure → Server Operations: Web Operations
QA Contact: jdow → cshields
Assignee: server-ops-webops → bburton
-----BEGIN CERTIFICATE REQUEST-----
MIIC2zCCAcMCAQAwgZUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRwwGgYDVQQKExNNb3ppbGxhIENvcnBv
cmF0aW9uMRQwEgYDVQQDFAsqLm1venFhLmNvbTElMCMGCSqGSIb3DQEJARYWaG9z
dG1hc3RlckBtb3ppbGxhLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAMZs4Ejqb9b4AcGpnj4lPbg1KpPSMPsG1k1kjf5QAlhk1AYEi2jv9J80T4jy
GHOoR+8q9MGFa9el4QpCE+0YfXr4MtAdxdDoOXo5PTEDFtVCZQ8ls3zx/SPeoYqF
DRiOyaOYg+09CUKqmCVbgqGoBSk2e4xP/zJQp/5f5zDppp+waUtWiBwS0phoCbsS
sDljPENnET1NZM1X90zY9a9dcTqzdhj0iQ+2UJACJiNtLeLDzLs0Ht3qNDRAf5YB
4BUdQHpXcTXlsh7aOxUqnaO1l8pgybg3x5PNbH6ufziotSycOp6Ur8DopgfeM9mJ
PD0vLTBSilad7b8B+q/9VHKKOLcCAwEAAaAAMA0GCSqGSIb3DQEBBQUAA4IBAQBG
i1v+mmtzE/JOSQ46CQJviiv/wNYCdIr4chRpU3vrv24IOSn+Hlg6zeNLOAwiH7AG
mlG4dLDY2fumYtFY28eHUhrzYniaujYctXOk1SGmz1BFbKG7m6A5BU04StpQbtAw
9a9g2BVBIDeWKJ3/9H/Lf12q08e4qx6sFPdOC6SUAc3w9aSOr3Sxr5austP2lT8V
pPJCfGYupzxR6xOqKmy1eIwGbHhTJlJWDgpouOW+dtmOtk8J6SeHaJXZ3GoxY8fO
Rgit3/ROWzeibY4mxKDMSjxzExQ1eZJA2AcRcF7k4JgnHcVtJXiYgK5QVekJ2GQc
DNU91Dj78jna8Lg9Azp3
-----END CERTIFICATE REQUEST-----
Please note :

1) If this is a public facing website, we need to change whois information from Mozilla Corporation to Mozilla Foundation (so the EV cert will say Mozilla Foundation (US) in accordance with what engagement wants with public facing websites)

2) If 1 above doesn't hold, then we don't care :)
Jason, can you please take care of it? Brandon shall we move this bug into Mozilla QA / Infrastructure again, given that your work has been done?
(In reply to Henrik Skupin (:whimboo) from comment #7)
> Jason, can you please take care of it? Brandon shall we move this bug into
> Mozilla QA / Infrastructure again, given that your work has been done?

This has not been ordered yet, I will be working on that today and it may take up to 10 business days for Geotrust to do the verification process
Status: NEW → ASSIGNED
So I guess there was a misunderstanding on my part about this certificate and what kinds of certificate combinations are valid.

The previous certificate was a wildcard certificate only.

Per http://www.networksolutions.com/support/why-can-t-i-get-a-wildcard-extended-validation-ev-ssl-certificate/ 

"Extended Validation (EV) SSL Certificates provide a higher level of assurance than older and existing types of SSL Certificates.  In order to ensure that EV SSL Certificates are not issued fraudulently or misused after issuance, the regulatory body governing the issuance of EV SSL Certificates known as the CA/B Forum decided to require that issuing CA's validate the legitimacy of each and every Web address to which an EV SSL Certificate is assigned.  Therefore, the issuance of "Wildcard" EV SSL Certificates for Web addresses such as "*.networksolutions.com" is not permitted.  If you would like to purchase multiple EV SSL Certificates to replace your Organizationally Validated Wildcard SSL Certificate, contact Customer Support to inquire about our volume discounts."

So we are not going to be able to issue you a certificate that is both a wildcard for *.mozqa.com and it have the EV level of verification.

If you want the certificate to be an EV level, then you'll need to provide a list of names, up to 10 names total, and I'll get the order going.

Or, if you want a wildcard for mozqa.com, let me know and I'll proceed with an order for it.

Thanks!
Brandon, then how did we get an EV cert a year ago?
(In reply to Al Billings [:abillings] from comment #10)
> Brandon, then how did we get an EV cert a year ago?

It had to have been for a single name or a limited set of names, do you have details on what URL(s) you had listed as being EV enabled?
Whiteboard: [pending decision on ssl cert type]
Oh, wait. See bug 639937. Looks like we decided to not use an EV cert on our side given the amount of costs. In that case we should stop any process and get the given subdomain removed from mozqa.com.
(In reply to Henrik Skupin (:whimboo) from comment #12)
> Oh, wait. See bug 639937. Looks like we decided to not use an EV cert on our
> side given the amount of costs. In that case we should stop any process and
> get the given subdomain removed from mozqa.com.

Ok, so per bug 639937 an EV certificate was never purchased for a mozqa.com DNS name.

But, https://ssl-ev.mozqa.com/data/ was/is falling under the non-EV wildcard for mozqa.com

Do you want me to proceed with a renewal of the non-EV wildcard for mozqa.com?
Whiteboard: [pending decision on ssl cert type] → [pending decision on wildcard renewal]
See Also: → 639937
So given the IP address (67.23.44.24) of this subdomain it falls back to ssl-ov.mozqa.com as given by the Apache configuration. That means it's not the EV certificate which has been expired but the OV certificate (bug 639936).

So there should be two things to fix here:

1. Do a renewal of the OV cert

2. Fix the Apache config so we do not fallback to OV certs for HTTPS connections (https://werwerwe.mozqa.com/). Jason can you fix that please?
(In reply to Henrik Skupin (:whimboo) [away 07/27 - 08/05] from comment #14)
> So given the IP address (67.23.44.24) of this subdomain it falls back to
> ssl-ov.mozqa.com as given by the Apache configuration. That means it's not
> the EV certificate which has been expired but the OV certificate (bug
> 639936).
> 
> So there should be two things to fix here:
> 
> 1. Do a renewal of the OV cert
> 

Sorry, I missed your comment here, when you say OV, do you mean a wildcard for *.mozqa.com ?

Thanks


> 2. Fix the Apache config so we do not fallback to OV certs for HTTPS
> connections (https://werwerwe.mozqa.com/). Jason can you fix that please?
(In reply to Brandon Burton [:solarce] from comment #15)
> Sorry, I missed your comment here, when you say OV, do you mean a wildcard
> for *.mozqa.com ?

Yes, please check the cert yourself on the updated URL.
URL: https://ssl-ev.mozqa.com/data/ → https://ssl-ov.mozqa.com/data/
Summary: EV certificate for mozqa.com has expired → OV certificate for mozqa.com has expired
Blocks: 725486
Brandon, can we get an update on that renewal? Thanks.
No longer blocks: 725486
Sorry for the delay, I've been on PTO

True BusinessID Wildcard Enrollment

  	
Thank you for your True BusinessID Wildcard certificate request
 
  	Your order number is: 8913651 

Processing time is 7-10 business days, but we can usually get them to do it in 2-3 days, I'll update the bug as I get updates from them.
Web Server CERTIFICATE
-----------------

-----BEGIN CERTIFICATE-----
MIIE+zCCA+OgAwIBAgIDAZnTMA0GCSqGSIb3DQEBBQUAMEAxCzAJBgNVBAYTAlVT
MRcwFQYDVQQKEw5HZW9UcnVzdCwgSW5jLjEYMBYGA1UEAxMPR2VvVHJ1c3QgU1NM
IENBMB4XDTEyMDgxMDE3NTcyMloXDTE0MDgxMzE2MTkwNVowgZkxKTAnBgNVBAUT
IFlKYUF2eHU4NXNhelNHbU16YjlWMlQxa093RjRoMnRJMQswCQYDVQQGEwJVUzET
MBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEcMBoG
A1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEUMBIGA1UEAwwLKi5tb3pxYS5jb20w
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDGbOBI6m/W+AHBqZ4+JT24
NSqT0jD7BtZNZI3+UAJYZNQGBIto7/SfNE+I8hhzqEfvKvTBhWvXpeEKQhPtGH16
+DLQHcXQ6Dl6OT0xAxbVQmUPJbN88f0j3qGKhQ0YjsmjmIPtPQlCqpglW4KhqAUp
NnuMT/8yUKf+X+cw6aafsGlLVogcEtKYaAm7ErA5YzxDZxE9TWTNV/dM2PWvXXE6
s3YY9IkPtlCQAiYjbS3iw8y7NB7d6jQ0QH+WAeAVHUB6V3E15bIe2jsVKp2jtZfK
YMm4N8eTzWx+rn84qLUsnDqelK/A6KYH3jPZiTw9Ly0wUopWne2/Afqv/VRyiji3
AgMBAAGjggGiMIIBnjAfBgNVHSMEGDAWgBRCeVQbYc1VKz5j1TxIV/Wf+0XOSjAO
BgNVHQ8BAf8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCEG
A1UdEQQaMBiCCyoubW96cWEuY29tggltb3pxYS5jb20wPQYDVR0fBDYwNDAyoDCg
LoYsaHR0cDovL2d0c3NsLWNybC5nZW90cnVzdC5jb20vY3Jscy9ndHNzbC5jcmww
HQYDVR0OBBYEFDqKteZ6qGqS0mLaBVHIb5Vp0cqAMAwGA1UdEwEB/wQCMAAwbwYI
KwYBBQUHAQEEYzBhMCoGCCsGAQUFBzABhh5odHRwOi8vZ3Rzc2wtb2NzcC5nZW90
cnVzdC5jb20wMwYIKwYBBQUHMAKGJ2h0dHA6Ly9ndHNzbC1haWEuZ2VvdHJ1c3Qu
Y29tL2d0c3NsLmNydDBMBgNVHSAERTBDMEEGCmCGSAGG+EUBBzYwMzAxBggrBgEF
BQcCARYlaHR0cDovL3d3dy5nZW90cnVzdC5jb20vcmVzb3VyY2VzL2NwczANBgkq
hkiG9w0BAQUFAAOCAQEAiohRxcKZ3BKgr2TO8LlWBpNrwyzBnDHH8uhFqOb+5zAf
dX2JxkbuiWPDyt5YLvcJxVunwpHXsNnCu6v5VWRO1JH1cHyGyDsVd/ZFIcb1HS4t
G8EyYfTU4HEl+L1I1fCXm8e6/QNP0wY2YCF77vIlXFn9KmrR9G9PH3iJZnNIuQdp
wE8eb6yL+GCdV+JLfgT5NT2KsrI4uxjsxIxVEz4n4XBc/vBjc/wSvdPbtm07Mo3S
Ujfnoc7wlnOZfMmKReQsB96pb27KmzqKqOMSp+yB8EUKp3KvAmEQwydfqpxnf6J4
cIS8ABM8IZTzdiVv+8NbbImYdh7IbyJ1cD3BKAwv3A==
-----END CERTIFICATE-----


INTERMEDIATE CA:
---------------------------------------

-----BEGIN CERTIFICATE-----
MIID2TCCAsGgAwIBAgIDAjbQMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTAwMjE5MjIzOTI2WhcNMjAwMjE4MjIzOTI2WjBAMQswCQYDVQQG
EwJVUzEXMBUGA1UEChMOR2VvVHJ1c3QsIEluYy4xGDAWBgNVBAMTD0dlb1RydXN0
IFNTTCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJCzgMHk5Uat
cGA9uuUU3Z6KXot1WubKbUGlI+g5hSZ6p1V3mkihkn46HhrxJ6ujTDnMyz1Hr4Gu
FmpcN+9FQf37mpc8oEOdxt8XIdGKolbCA0mEEoE+yQpUYGa5jFTk+eb5lPHgX3UR
8im55IaisYmtph6DKWOy8FQchQt65+EuDa+kvc3nsVrXjAVaDktzKIt1XTTYdwvh
dGLicTBi2LyKBeUxY0pUiWozeKdOVSQdl+8a5BLGDzAYtDRN4dgjOyFbLTAZJQ50
96QhS6CkIMlszZhWwPKoXz4mdaAN+DaIiixafWcwqQ/RmXAueOFRJq9VeiS+jDkN
d53eAsMMvR8CAwEAAaOB2TCB1jAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFEJ5
VBthzVUrPmPVPEhX9Z/7Rc5KMB8GA1UdIwQYMBaAFMB6mGiNifurBWQMEX2qfWW4
ysxOMBIGA1UdEwEB/wQIMAYBAf8CAQAwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDov
L2NybC5nZW90cnVzdC5jb20vY3Jscy9ndGdsb2JhbC5jcmwwNAYIKwYBBQUHAQEE
KDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5nZW90cnVzdC5jb20wDQYJKoZI
hvcNAQEFBQADggEBANTvU4ToGr2hiwTAqfVfoRB4RV2yV2pOJMtlTjGXkZrUJPji
J2ZwMZzBYlQG55cdOprApClICq8kx6jEmlTBfEx4TCtoLF0XplR4TEbigMMfOHES
0tdT41SFULgCy+5jOvhWiU1Vuy7AyBh3hjELC3DwfjWDpCoTZFZnNF0WX3OsewYk
2k9QbSqr0E1TQcKOu3EDSSmGGM8hQkx0YlEVxW+o78Qn5Rsz3VqI138S0adhJR/V
4NwdzxoQ2KDLX4z6DOW/cf/lXUQdpj6HR/oaToODEj+IZpWYeZqF6wJHzSXj8gYE
TpnKXKBuervdo5AaRTPvvz7SBMS24CqFZUE+ENQ=
-----END CERTIFICATE-----
Thanks Brandon! Jason, can you please update the cert on mozqa.com accordingly?
Ack. Something got screwed up here, and mozqa.com is down now. I did the following:

1. Updated the crt file for the SSLCertificateFile to use the above Web Server Certificate
2. Updated the intermediate crt file (SSLCACertificateFile) to use the above Intermediate CA

Digging into this - Looks like the private key's modulus is different than the above certificate's modulus, which likely implies that either the private key is wrong or the above certificate is wrong. As a result, I can't get the server back up and running right now.

Brandon - Are you sure that's the right cert?
Also restarted the server in the above steps in comment 21.
I've temporarily commented out the apache config part for the above URL and restarted the apache server to get mozqa.com running.
Hm, this is bad. Jason, have you made a backup before replacing the content of the crt files? I hope so and I wish we can revert this change until we found the solution.
Whiteboard: [pending decision on wildcard renewal]
Sorry, I was not expecting you to try to update it, I was planning on doing this myself, but I am in London for a work week and was tied up yesterday.

Yes, we did generate a new private key, I'll replace the private key on your web server and get the config working, shortly
I did not realize mozqa.com is hosted on a rackspace VM. I don't see that we have a root login listed for this box.

Can you add my ssh pubkey to root *or* I can email the private key via a GPG encrypted email?

Thanks

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDYyyo1TF85qmiBg40izzLHoNAmjdvm6i0WHJnJDDm2kIiKZBFd9Va9X9QHaYeyeSqbnpYhicHslHczRjdezGvF4DIDprZc67URA15wohKGoorW1YTBjhGGJ7/1J8ZLUI+Hb5PjcESI+6GVP96+4XhqVjqGHaet8GGZ8nJqMt0jsK27Skh3veCvwAO/D5eBgcJR5xbGjco6xoNlhvkVHxIWRK1NGZSLsBBJ0azKFQVi4Wen2UJEvQEg/8n8vmemB/LMwqMo0YFVEOL9i2eTRC91EDDlS5ndzTUy58/lUTVTyvRuXw/kbyVoFzq43n/CnScIhtKXr+LjioRmOVPnkmrO+rSw72MrxXAB9PN7tjxQPmS2FSwvgAJ8iaIB0Cwh0agWcj93Y7vY517VJBseRzawe0OnKh/r61k5Lc3wV9vcVuO0AVeRBucxlJr8g/BU6ZkDTOQtKbxpqQi7Y+CYZFKaLrW67Wyb5IByNDn67yZK19fX7BNtEmHbTOOis3Ay8yy6SPKdE/CT1hNTgiQv1nVL5d8qmTFOXeXE6zxvknmxNm2jpPWRFPEPfjKwXecAFiF+X9KBym7gk+L4RB+SmSNWZIz/DqOHZhcS4qACRPVOmJyI90MZ/HHFk9MTEIUH0jmP4CRQhjhlU2dTft0T3XtmyjM+KkDf4WaWcvsGLhdMFaJw8zmTOd/VTztPL/Ng28rJL15nsbnJsZ/HFpWA8mCiUQ4swcZ/l2pJbQLHVk4UFggtdZW7RilKh6sF/PLOs2YZ3KWKBKN+ZcvlIffDEWXthuS9OD0kg0Aw0+59Ypoda16t/CRgH9DMaXsSt+WgLYriXGu6eNgpgJ5n1TdxQDvdOEoEnzn1WMoffvhWq7P5KnoX2z1NWPBiDpfuoPhgC7dJscAfZmG2S9PKZhAgEkb2I85wQow94bozGEqVI2FbVC51Sx05HK2PFwpfL5w5s8vbayGzQYEPjagPONXR63fUCLw783khsrXgWahPFanVFkHmUl3xC9OWJpG567g/pLfX5bKmGkdZ7H9rRR0jJxDVNcYrMh13Y1N5SuLgNOY0+QA0cw8YxpH2aUEOSCRoQhb6kzisggPgdfM058Kqt88h8Mb7Al+TMuNXokgD7O+sLYwvMu2MtGqbeRdU1DvjoFdm4DuwPMqsXGzMCriGY0p4geztbRvJIA6nZ4TiV38V5IGSfl5C7vR4+O097CahDMZ3QxE+FqDG5f2sCq5lfmx8rMHa9EZTzDzQksA873ubDxw73qLqvVbSs8oDnaQZ2o1tw1itC+p3tWnXurP0tVN7FxmksVy6uFRpK6FhpMkPq29fd4V3HJ7TUCBjjKGQ4ZTq7a7xxCCNtbSnyvmOWjDr bburton@mozilla.com
Email the private key via a GPG encrypted email.
(In reply to Jason Smith [:jsmith] from comment #27)
> Email the private key via a GPG encrypted email.

Can you email what your PGP key ID is so I can retrieve it from our key server and email your the private key?

Thanks!
Jason, can you please make sure we can get this bug fixed in the next days?
(In reply to Henrik Skupin (:whimboo) from comment #29)
> Jason, can you please make sure we can get this bug fixed in the next days?

Sorry, I was on vacation from 8/23 - 8/27. I'll look into this on 8/28.
(In reply to Brandon Burton [:solarce] from comment #28)
> (In reply to Jason Smith [:jsmith] from comment #27)
> > Email the private key via a GPG encrypted email.
> 
> Can you email what your PGP key ID is so I can retrieve it from our key
> server and email your the private key?
> 
> Thanks!

ping. I sent you my public key, but haven't heard back from you yet.
(In reply to Jason Smith [:jsmith] from comment #31)
> (In reply to Brandon Burton [:solarce] from comment #28)
> > (In reply to Jason Smith [:jsmith] from comment #27)
> > > Email the private key via a GPG encrypted email.
> > 
> > Can you email what your PGP key ID is so I can retrieve it from our key
> > server and email your the private key?
> > 
> > Thanks!
> 
> ping. I sent you my public key, but haven't heard back from you yet.

I am unable to find your PGP key on public key servers, please submit it with gpg --send-keys KEYID so that it's available to search

Thanks!
Brandon, I tried adding the private key you specified with the public key above, but I'm still getting an error:

[error] SSL Library Error: 185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch

And now the server is down again. I'll get that fixed shortly.

I think the problem is likely since that the private key given here only applies to the public key stated above, but doesn't apply to the other public keys being used on the server.
And we're back up, but the issue in comment 33 still stands.
Change comment 33 - the general case with the private key doesn't work.
Whoops, somewhere along the way a mozqa.com csr and key were generated, but the ones I used were named wildcard.mozqa.com, so tab complete fail on my part

I just emailed you what should be the correct key
(In reply to Brandon Burton [:solarce] from comment #36)
> Whoops, somewhere along the way a mozqa.com csr and key were generated, but
> the ones I used were named wildcard.mozqa.com, so tab complete fail on my
> part
> 
> I just emailed you what should be the correct key

That worked, although I realized there's two caveats that will need to be taken care of as followups:

- We need to update the self-signed mozqa certificate against the new private key for ssl-selfsigned.mozqa.com
- We'll need to update the certificate file and CA certificate file for ssl-dv.mozqa.com

Henrik - Are both of these URLs (ssl-selfsigned.mozqa.com and ssl-dv.mozqa.com) needed to be maintained? If so, I'll file followup bugs for the certificate updates for each one.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Keywords: user-doc-needed
Resolution: --- → FIXED
Depends on: 787234
Follow up action item is tracked in bug 787234.
(In reply to Jason Smith [:jsmith] from comment #38)
> Follow up action item is tracked in bug 787234.

Fixed the self-signed cert. Second action item is tracked in bug 787252.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.