1363814 - Switch stage/prod to Heroku's automated Let's Encrypt based SSL solution (ACM)

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

The prod SSL cert expires soon (bug 1362568):

"""
$ heroku certs --app treeherder-prod
Name                   Common Name(s)          Expires               Trusted  Type
─────────────────────  ──────────────────────  ────────────────────  ───────  ────
parasaurolophus-99121  treeherder.mozilla.org  2017-06-01 12:00 UTC  True     SNI

$ heroku certs:info --app treeherder-prod
Fetching SSL certificate parasaurolophus-99121 info for treeherder-prod... done
Certificate details:
Common Name(s): treeherder.mozilla.org
Expires At:     2017-06-01 12:00 UTC
Issuer:         /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
Starts At:      2014-05-28 00:00 UTC
Subject:        /C=US/ST=CA/L=Mountain View/O=Mozilla Corporation/CN=treeherder.mozilla.org
SSL certificate is verified by a root authority.
"""

Rather than having to update the certs each time, we should use Heroku's new "Automated Certificate Management" feature, which automates SSL certificate management using Let's Encrypt:
https://devcenter.heroku.com/articles/automated-certificate-management

Once Heroku apps are using the SNI based SSL (which we are, as of bug 1316712), it's a zero-downtime, one CLI command step to switch to ACM:
https://devcenter.heroku.com/articles/automated-certificate-management#migrating-existing-applications

We should test on stage first, then migrate production.

Comment 1

[Ed Morley [:emorley]]

Stage is now switched. It took < 5 minutes start to finish - pretty slick.
I'll wait a bit before doing production (if only so the expiry date is staggered in case of issues in 90 days).


$ heroku certs:auto:enable --app treeherder-stage
Enabling Automatic Certificate Management... done
=== Your certificate will now be managed by Heroku.  Check the status by running heroku certs:auto.


$ heroku certs:auto --app treeherder-stage
=== Automatic Certificate Management is enabled on treeherder-stage

Domain                  Status
──────────────────────  ───────
treeherder.allizom.org  Waiting


(Approx 1-2 mins later...)

$ heroku certs:auto --app treeherder-stage
=== Automatic Certificate Management is enabled on treeherder-stage

Domain                  Status
──────────────────────  ────────────
treeherder.allizom.org  DNS Verified


(Approx 3-4 mins later...)

$ heroku certs:auto --app treeherder-stage
=== Automatic Certificate Management is enabled on treeherder-stage

Certificate details:
Common Name(s): treeherder.allizom.org
Expires At:     2017-08-08 16:24 UTC
Issuer:         /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
Starts At:      2017-05-10 16:24 UTC
Subject:        /CN=treeherder.allizom.org
SSL certificate is verified by a root authority.

Domain                  Status
──────────────────────  ──────
treeherder.allizom.org  OK

Comment 2

[Ed Morley [:emorley]]

Prod is now switched over too, taking < 3 minutes start to finish \o/.

$ heroku certs:info --app treeherder-prod
Fetching SSL certificate parasaurolophus-99121 info for treeherder-prod... done
Certificate details:
Common Name(s): treeherder.mozilla.org
Expires At:     2017-06-01 12:00 UTC
Issuer:         /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
Starts At:      2014-05-28 00:00 UTC
Subject:        /C=US/ST=CA/L=Mountain View/O=Mozilla Corporation/CN=treeherder.mozilla.org
SSL certificate is verified by a root authority.

$ heroku certs:auto:enable --app treeherder-prod
Enabling Automatic Certificate Management... done
=== Your certificate will now be managed by Heroku.  Check the status by running heroku certs:auto.

$ heroku certs:auto --app treeherder-prod
=== Automatic Certificate Management is enabled on treeherder-prod

Domain                  Status
──────────────────────  ───────
treeherder.mozilla.org  Waiting

$ date
Thu, May 11, 2017 10:23:10 PM

...

$ heroku certs:auto --app treeherder-prod
=== Automatic Certificate Management is enabled on treeherder-prod

Certificate details:
Common Name(s): treeherder.mozilla.org
Expires At:     2017-08-09 20:25 UTC
Issuer:         /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
Starts At:      2017-05-11 20:25 UTC
Subject:        /CN=treeherder.mozilla.org
SSL certificate is verified by a root authority.

Domain                  Status
──────────────────────  ──────
treeherder.mozilla.org  OK

$ date
Thu, May 11, 2017 10:26:05 PM

Comment 3

[Ed Morley [:emorley]]

The Heroku docs don't state how long before the certificate expiry the automatic renewal takes place (which we need to know so we can adjust the Nagios alerts appropriately), so I've filed:
https://help.heroku.com/tickets/479074

Comment 4

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #3)
> The Heroku docs don't state how long before the certificate expiry the
> automatic renewal takes place (which we need to know so we can adjust the
> Nagios alerts appropriately), so I've filed:
> https://help.heroku.com/tickets/479074

Their reply:
"""
Essentially how it works is that we start attempting to renew at 60 days into the 90 day valid period. Generally, the renewal will just go through and you'll get a fresh 90 day period. However, if anything goes wrong it will continue to retry until successful. Setting up a warning 10 days before expiry (or 80 days into the 90 day period) should be fine. That provides plenty of space for retries, etc. on our end while still providing you with a 10 day cushion to react if things are going horribly wrong and we haven't solved it yet for some reason.
"""