Closed Bug 583678 Opened 14 years ago Closed 14 years ago

Acquire a standby certificate for AUS

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: robert.strong.bugs, Assigned: mrz)

References

Details

Bug 544442 makes it so Firefox checks certificate attribute values have the expected values that are checked before allowing an update. Firefox is able to provide certificate attribute name / value pairs for multiple certificate in case we lose, have to revoke a certificate, etc. and to protect against this a standby certificate should be acquired so its name / value pairs that we check can be added to Firefox in case such an event occurs.
Summary: Acquire a standby certificate for the AUS server. → Acquire a standby certificate for AUS
Assignee: morgamic → server-ops
Component: Systems → Server Operations
Product: AUS → mozilla.org
QA Contact: systems → mrz
Version: 3.0 → other
What's the action?   You want a duplicate aus2.mozilla.org certificate?
Assignee: server-ops → mrz
My understanding is that a new certificate for aus2.mozilla.org is what is needed.

johnath, dveditz, etc., the cert used by aus2 is the Equifax *.mozilla.org cert that is used by quite a few sites and if we do need to revoke this cert someday it would affect a lot of sites. If this cert might need to be revoked someday perhaps it would be a good thing to use certs specifically for aus2.mozilla.org instead to lessen the fallout / work required?
(In reply to comment #2)
> My understanding is that a new certificate for aus2.mozilla.org is what is
> needed.

But not just a new certificate but one from a different CA, correct?  Geotrust/Verisign won't let me issue two different certs for the same CN.
Yeah - so Matt, if you have a different put here, we should talk about it more. As I said to you on the phone the other day - we want to narrow down the attack surface for AUS, and having a second CA in the "trusted for AUS" list felt like a way to give you some operational latitude, but if you find that weird, let's find something less weird?

If you don't find it too weird, and were just asking for clarification, then yes: a second cert from a different CA would give you an ability to swap out certs without busting updates.
A follow up point, to be clear: we need to figure out what we want to do here, soon. It can'tcan'tcan't be that we roll this out in FF4 and then in FF4.0.2 we look up and say "Oh shit, we had to turf that cert, and now we can't update our users." That is catastrophic - far better to roll the patch back than to risk that. So let's figure this out.  :)
Hoping to sync up with Rob IRL and doing some research on which CA is appropriate.
The "top" providers I'd look at are essentially all owned by Verisign (Thawte, GeoTrust).  

Is it sufficient to get one cert for aus2.mozilla.org from Thawte and one from GeoTrust?  Or do I need to go outside of the parent org?
Guidance on certificate expiration?  1yr?  2yr?
I talked to Strong in the hallway.  To make deployment easier, we'll get certs for aus3.mozilla.org.
(In reply to comment #9)
> I talked to Strong in the hallway.  To make deployment easier, we'll get certs
> for aus3.mozilla.org.

aus3.mozilla.org? Typo, or why are we bumping the version number? How is that helping anything?
It's a deployment timing issue.  Helps a good deal.  "3" doesn't have to refer to the version number at all (but it'll coincide nicely with AUS3).
(In reply to comment #11)
> It's a deployment timing issue.  Helps a good deal.  "3" doesn't have to refer
> to the version number at all (but it'll coincide nicely with AUS3).

I said this in bug 586213, but I'll mention it here, too...

Why not just get two new certs for aus2.m.o? The current cert is just the
normal wildcard cert, no? Just not seeing why the change in hostname is needed,
especially since we currently don't do any cert checking on AUS, afaik...

Guess I'm not seeing what deployment timing has to do with this or why it would cause a hostname change...
(In reply to comment #13)
> Replied in bug 586213 comment #7

Yep, thanks! No further questions/comments/concerns.
Thawte: 

-----BEGIN CERTIFICATE-----
MIID8TCCAtmgAwIBAgIQMyKhGjwcSXDfRcL5JE0i9zANBgkqhkiG9w0BAQUFADA8
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMVGhhd3RlLCBJbmMuMRYwFAYDVQQDEw1U
aGF3dGUgU1NMIENBMB4XDTEwMDgxMTAwMDAwMFoXDTExMDgxMTIzNTk1OVowgYkx
CzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHFA1Nb3Vu
dGFpbiBWaWV3MRAwDgYDVQQKFAdNb3ppbGxhMSAwHgYDVQQLFBdBdXRvbWF0aWMg
VXBkYXRlIFN5c3RlbTEZMBcGA1UEAxQQYXVzMy5tb3ppbGxhLm9yZzCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAOxCTrFvoYr4j1z2vhZRmAvAEABGOfgy
fdjNQHxmwZOba4zYcTBaSFvKpbE7Cx1q1owRQOVsfbmnfjvBlri1L4c3j+oavDch
ppk+qAXPOczHEq7v4FaNx/k22UrFmkw4tM8kEmOpwci4hW2mwdNs6kAgQ2YNuleT
BLjiFXc1eSCdbN/LQv7q3Xb+6lx6qLznO43o8Wnxsy6jca8tP0FH+OSK7VGxbJQK
VcT7fZWQWzt0mBMxaRAvmVuT45w3kZnqvMMUY9Ex/WK9o9HBIch80+cKrTNtvFre
ceHXPVM2UKtjuq41sNqvZcbjEmLyi0zOC5Znufx1KMl8LhVdwYBf4ecCAwEAAaOB
oDCBnTAMBgNVHRMBAf8EAjAAMDoGA1UdHwQzMDEwL6AtoCuGKWh0dHA6Ly9zdnIt
b3YtY3JsLnRoYXd0ZS5jb20vVGhhd3RlT1YuY3JsMB0GA1UdJQQWMBQGCCsGAQUF
BwMBBggrBgEFBQcDAjAyBggrBgEFBQcBAQQmMCQwIgYIKwYBBQUHMAGGFmh0dHA6
Ly9vY3NwLnRoYXd0ZS5jb20wDQYJKoZIhvcNAQEFBQADggEBAJPYDukDgx/lGAtp
brMyG77n8lR6ROYILQMFEiA1QoEeoVIu1nu3SeZqRDZZjqi7spubBeYWEsIP+tqi
6oDmKVIXU4JEOc2gZBtiHz6m+CoKnXnVExv7M6jy3OVUBe85l/ECJ5ozHP/sLg/L
UPsPh9o3VCaCQRasd5Ce2Rw+CUTrPUkltWZWniM+xtcZ9GruAeLIx8hTgQd/u4n8
QHmPioMxXgzQBcg+uHphohBfYKWbsyMA/QnLkKM3ZjvvUjEMsjWoGP1hsoCuCcRD
vtvmdFiOaTKqnS6BzImp8qzCUFWFDLQa3nb10S0JqLWkDLjfC3PRoCHPcwdZW+ug
QfXRs8M=
-----END CERTIFICATE-----
Geotrust:

Your Web Server Certificate for aus3.mozilla.org

-----BEGIN CERTIFICATE-----
MIID1zCCA0CgAwIBAgIDFDu0MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTAwODEwMDcwNzE4WhcNMTEwODEzMDE1NTM0
WjCBwDEpMCcGA1UEBRMgYWRWYUxrT2F3U01UT1RGV0pOVmpJbGhxc0FWRkl2dmYx
CzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1Nb3Vu
dGFpbiBWaWV3MRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMSAwHgYDVQQL
ExdBdXRvbWF0aWMgVXBkYXRlIFN5c3RlbTEZMBcGA1UEAxMQYXVzMy5tb3ppbGxh
Lm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOxCTrFvoYr4j1z2
vhZRmAvAEABGOfgyfdjNQHxmwZOba4zYcTBaSFvKpbE7Cx1q1owRQOVsfbmnfjvB
lri1L4c3j+oavDchppk+qAXPOczHEq7v4FaNx/k22UrFmkw4tM8kEmOpwci4hW2m
wdNs6kAgQ2YNuleTBLjiFXc1eSCdbN/LQv7q3Xb+6lx6qLznO43o8Wnxsy6jca8t
P0FH+OSK7VGxbJQKVcT7fZWQWzt0mBMxaRAvmVuT45w3kZnqvMMUY9Ex/WK9o9HB
Ich80+cKrTNtvFreceHXPVM2UKtjuq41sNqvZcbjEmLyi0zOC5Znufx1KMl8LhVd
wYBf4ecCAwEAAaOByzCByDAfBgNVHSMEGDAWgBRI5mj5K9KylddH2CMgEE8zmJCf
1DAOBgNVHQ8BAf8EBAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MBsGA1UdEQQUMBKCEGF1czMubW96aWxsYS5vcmcwOgYDVR0fBDMwMTAvoC2gK4Yp
aHR0cDovL2NybC5nZW90cnVzdC5jb20vY3Jscy9zZWN1cmVjYS5jcmwwHQYDVR0O
BBYEFHw/sCdL7A2CwaObV7FPUiqmb9XSMA0GCSqGSIb3DQEBBQUAA4GBAK/EvZeC
Dzpc+iB8zApukbbfabslyP2iX56ffJxdMB9kdqK5rNNT0pV/1QheHUYzh60EiUtN
IhwazcltGiYU4Fl3Gta8Rs18qmXIxqclsw4T4oefTp08WwNuqzAajhc0OElpdG4A
11v119yGQCpJxy1SsaK/9Ui/mRixd7HAZDmy
-----END CERTIFICATE-----
Over to oremj.

1. Need Apache to grok aus3.mozilla.org as aus2.
2. Setup aus3.mozilla.org
3. Document "how to use standby cert"
Assignee: mrz → jeremy.orem+bugs
oremj - key is mradm01:root-ca/aus3.mozilla.org.key
Just so I understand, we need:

1. An IP that contains the cert for aus3
2. aus2.mozilla.org also needs to respond to aus3.mozilla.org
Yep!
Jeremy, when do you think this will be completed?
Severity: normal → major
https://aus3.mozilla.org/ is set up with the new certificate.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
For my peace of mind, could you activate the standby certificate so I can verify that the certificate checks pass for it. It is just a tad too critical to not take the time to verify it works properly. Thanks
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Jeremy, any word when comment #24 can be done?
What do you mean? The new certificate is already running on aus3.mozilla.org.
I'd prefer testing the standby certificate as well so there is 100% confidence the client side values for the certificate are correct vs. only being 99.99% confident since a failure would be rather catastrophic.
Matthew, where is the secondary cert for aus3.mozilla.org?
Not sure where the thawte key is.
Key's the same - I used the same CSR for both.
(In reply to comment #31)
> Key's the same - I used the same CSR for both.

That sounds dangerous... A completely separate key would have been more secure, imho. General safe policy is to never use the same key for more than one certificate.
"more secure" isn't a useful phrase, generally.  "more robust against $attack" would be helpful in assessing different options, though.

I agree that we shouldn't have reused the key in this scenario, though.  If the key is compromised, we will have to revoke the aus certificate, and we want the standby cert against such a case, per comment 0.  If the same key is used, then I believe that the attacker can also compromise traffic from a server configured with the standby key, meaning that we effectively don't have a standby!  (The current situation *is* more resilient against loss of the key, such as due to the servers containing all our copies being hit by a meteor, but I think we need to be concerned about the case of compromise as well.  It's harder to defend against, and will likely require more timely action in response.)

I recommend that we get the aus3 key re-issued with a new key and CSR, as Reed suggests.
Assignee: jeremy.orem+bugs → mrz
(In reply to comment #24)
> For my peace of mind, could you activate the standby certificate so I can
> verify that the certificate checks pass for it. It is just a tad too critical
> to not take the time to verify it works properly. Thanks

The Thawte cert is live for you to test against.  Let me know when I can revert.
The server doesn't have the issuer chain so I had to use an existing profile that has already see the issuer chain. Can the issuer chain for Thawte be installed on the server?

Besides that, I have verified it.
(In reply to comment #34)
> The Thawte cert is live for you to test against.  Let me know when I can
> revert.

Once you get a replacement cert for the Thawte one, as per previous comments, Robert should test it again, imho.
I assumed it was the replacement cert since it has an activation date of today at 5 PM iirc but I will double check.
Was missing a chain there - Safari's no longer showing me a cert warning.  Can you retest?
What does your code match against?  CN & O?
(In reply to comment #38)
> Was missing a chain there - Safari's no longer showing me a cert warning.  Can
> you retest?
Just retested and everything worked as expected. So, both certs have now been tested. Thanks!

(In reply to comment #39)
> What does your code match against?  CN & O?
issuerName and commonName

So, in the case of the standby cert issuerName is:
CN=Thawte SSL CA,O="Thawte, Inc.",C=US

and commonName is:
aus3.mozilla.org
There's been convo outside the bug.  Standby cert works, working on some backend issues but for the sake of this bug, it's resolved.

I reverted to the "primary" cert.
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.