1380177 - Enable ssl endpoint (non-sni) addon for a handful of releng heroku app

Status: RESOLVED INVALID

(Infrastructure & Operations :: RelOps: General, task)

Description

[Jake Watkins [:dividehex]]

As it turns out we still use  python 2.7.5 on m-c build environment and this prevent using sni compatible ssl endpoints such as the free one being utilized on many heroku apps.  In this case, we need to revert back to using an SSL Endpoint addon and with a imported ssl cert (from digicert).

Steps here are:
* Enable SSL Endpoint via cli
* Generate SSL cert and get have it signed (digicert)
* Import SSL cert to heroku via cli
* Change cname under releng terraform config (aws route53) to match new ssl elb provided by heroku

Comment 1

[Jake Watkins [:dividehex]]

I've migrated both tooltool-prodution and tooltool-staging to the ssl endpoint elbs using the previous cert that was originally generated and signed by digicert.  Terraform has been modified and apply to account for the new elb endpoint cnames.

Although, testing the connection sslv3 is still not supported on these endpoints.  Although it doesn't state it in the endpoint ssl doc[1], it does state in the ssl doc [2], that sslv3 is not supported and heroku "maintain parity with the predefined AWS Elastic Load Balancers SSL security policies."

I have a strong suspicion this applies to the 'endpoint' type elbs just the same.  Which means no SSLv3 support at all.

[1] https://devcenter.heroku.com/articles/ssl-endpoint
[2] https://devcenter.heroku.com/articles/ssl#supported-ssl-protocols

Comment 2

[Jake Watkins [:dividehex]]

┌─[heroku][]
└─▪ nmap --script ssl-enum-ciphers tooltool.staging.mozilla-releng.net

Starting Nmap 7.50 ( https://nmap.org ) at 2017-07-11 17:37 PDT
Nmap scan report for tooltool.staging.mozilla-releng.net (23.23.255.72)
Host is up (0.093s latency).
Other addresses for tooltool.staging.mozilla-releng.net (not scanned): 23.23.134.226 174.129.6.32
rDNS record for 23.23.255.72: ec2-23-23-255-72.compute-1.amazonaws.com
Not shown: 848 closed ports, 150 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.0:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.1:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 16.88 seconds

Comment 3

[Amy Rich [:arr] [:arich]]

:garbas: can you please set the dependencies/blockers for this bug?

The question seems to be, can we use heroku at all because it doesn't report ssl v3. Does puppet 2.7.5 on the windows buildbot platform support TLS?

Comment 4

[Amy Rich [:arr] [:arich]]

Sorry python 2.7.5, not puppet.

Comment 5

[Rok Garbas [:garbas]]

Looks like SSL Endpoint solved the issue in the case of the new tooltool[1]. All errors in treeherder are not related to
I had to fix an authentication bug[2] in the way how we handle SECRET_KEY in the new setup.

:dividehex: Could we also do this for following Heroku apps:
 - releng-production-clobberer
 - releng-production-treestatus


[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ccf2acabb6d95a090b3c56150b5502b946fc3da
[2] https://github.com/mozilla-releng/services/pull/507

Comment 6

[Jake Watkins [:dividehex]]

:garbas,  I've gone ahead and re-enabled the ssl endpoint addons (non-sni) for both:
 - releng-production-clobberer
 - releng-production-treestatus

Do you not need the staging counterparts also moved to non-sni endpoints?

Comment 7

[Jake Watkins [:dividehex]]

Attachment: Move heroku cnames to new endpoints

Comment 8

[Jake Watkins [:dividehex]]

Moving treestatus to non-sni broke treestatus. It has since been reverted back to the sni endpoint.

https://bugzilla.mozilla.org/show_bug.cgi?id=1382522#c2

Comment 9

[Rok Garbas [:garbas]]

:dividehex: could i ask you to switch to "SSL Endpoint" certificate for releng-staging-treestatus[1] heroku app? I will then test to see what/where it breaks.


[1] https://dashboard.heroku.com/apps/releng-staging-treestatus

Comment 10

[Mark Cornmesser [:markco]]

(In reply to Amy Rich [:arr] [:arich] from comment #3)
> :garbas: can you please set the dependencies/blockers for this bug?
> 
> The question seems to be, can we use heroku at all because it doesn't report
> ssl v3. Does puppet 2.7.5 on the windows buildbot platform support TLS?

As is 2.7.5 won't support TLS.

Comment 11

[Amy Rich [:arr] [:arich]]

garbas: it sounds from markco's comment like there's some confusion/doubt that the windows platform is going to support the heroku endpoints even with the plugin. Can you meet with him/jake and talk about the testing methodology and what worked and didn't? We need to verify that for sure for tooltool.

Comment 12

[Jake Watkins [:dividehex]]

Attachment: fix terraform cname resources and change treestatus to ssl endpoint type

Comment 13

[Jake Watkins [:dividehex]]

:garbas, I've migrated the releng-staging-treestatus app to the ssl endpoint (non-sni) type.  I did a little more testing of the app before and after.  It looks like it is working fine after the change over.

Comment 14

[Rok Garbas [:garbas]]

:arr: :markco: I ran a test against new tooltool (on heroku) and after switching to "SSL Endpoint"[1] all tooltool related errors went away[2][3].

I will schedule a meeting with dividehex/markco to come to the bottom why is this all of a sudden working.
 

[1] https://devcenter.heroku.com/articles/ssl-endpoint
[2] https://reviewboard.mozilla.org/r/158628/diff/1#index_header
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b87c442f9d98

Comment 15

[Jake Watkins [:dividehex]]

releng-production-treestatus has been migrated (AGAIN) to a non-sni ssl endpoint.  There were a few reports of treestatus not working for people in #releng but they were probably due to stale dns cache on the system.

<dividehex> this still might be a dns caching error
<dividehex> system dns cache vs chrome
<kats> dividehex: ah you're right
<kats> i flushed my system dns cache and it works now
<dividehex> cool!
<kats> (i used `sudo killall -HUP mDNSResponder`, for reference)

Comment 16

[Jake Watkins [:dividehex]]

This bug is outdated and has been superseded by bug 1487798.  Since python has been updated throughout most of the firefox CI pipeline, we have also migrated the releng Heroku apps from the Heroku ssl endpoint addon to Heroku's ACM.