Enable ssl endpoint (non-sni) addon for a handful of releng heroku app

NEW
Assigned to

Status

a year ago
8 months ago

People

(Reporter: dividehex, Assigned: dividehex)

Tracking

Details

Attachments

(2 attachments)

(Assignee)

Description

a year ago
As it turns out we still use  python 2.7.5 on m-c build environment and this prevent using sni compatible ssl endpoints such as the free one being utilized on many heroku apps.  In this case, we need to revert back to using an SSL Endpoint addon and with a imported ssl cert (from digicert).

Steps here are:
* Enable SSL Endpoint via cli
* Generate SSL cert and get have it signed (digicert)
* Import SSL cert to heroku via cli
* Change cname under releng terraform config (aws route53) to match new ssl elb provided by heroku
(Assignee)

Comment 1

a year ago
I've migrated both tooltool-prodution and tooltool-staging to the ssl endpoint elbs using the previous cert that was originally generated and signed by digicert.  Terraform has been modified and apply to account for the new elb endpoint cnames.

Although, testing the connection sslv3 is still not supported on these endpoints.  Although it doesn't state it in the endpoint ssl doc[1], it does state in the ssl doc [2], that sslv3 is not supported and heroku "maintain parity with the predefined AWS Elastic Load Balancers SSL security policies."

I have a strong suspicion this applies to the 'endpoint' type elbs just the same.  Which means no SSLv3 support at all.

[1] https://devcenter.heroku.com/articles/ssl-endpoint
[2] https://devcenter.heroku.com/articles/ssl#supported-ssl-protocols
(Assignee)

Comment 2

a year ago
┌─[heroku][]
└─▪ nmap --script ssl-enum-ciphers tooltool.staging.mozilla-releng.net

Starting Nmap 7.50 ( https://nmap.org ) at 2017-07-11 17:37 PDT
Nmap scan report for tooltool.staging.mozilla-releng.net (23.23.255.72)
Host is up (0.093s latency).
Other addresses for tooltool.staging.mozilla-releng.net (not scanned): 23.23.134.226 174.129.6.32
rDNS record for 23.23.255.72: ec2-23-23-255-72.compute-1.amazonaws.com
Not shown: 848 closed ports, 150 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.0:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.1:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 16.88 seconds
:garbas: can you please set the dependencies/blockers for this bug?

The question seems to be, can we use heroku at all because it doesn't report ssl v3. Does puppet 2.7.5 on the windows buildbot platform support TLS?
Flags: needinfo?(rgarbas)
Flags: needinfo?(mcornmesser)
Sorry python 2.7.5, not puppet.

Updated

a year ago
Blocks: 1284475
Flags: needinfo?(rgarbas)
Looks like SSL Endpoint solved the issue in the case of the new tooltool[1]. All errors in treeherder are not related to
I had to fix an authentication bug[2] in the way how we handle SECRET_KEY in the new setup.

:dividehex: Could we also do this for following Heroku apps:
 - releng-production-clobberer
 - releng-production-treestatus


[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ccf2acabb6d95a090b3c56150b5502b946fc3da
[2] https://github.com/mozilla-releng/services/pull/507
Flags: needinfo?(jwatkins)
(Assignee)

Updated

a year ago
Flags: needinfo?(jwatkins)
Summary: Enable ssl endpoint (non-sni) addon for tooltool releng heroku app → Enable ssl endpoint (non-sni) addon for a handful of releng heroku app
(Assignee)

Comment 6

a year ago
:garbas,  I've gone ahead and re-enabled the ssl endpoint addons (non-sni) for both:
 - releng-production-clobberer
 - releng-production-treestatus

Do you not need the staging counterparts also moved to non-sni endpoints?
Flags: needinfo?(rgarbas)
(Assignee)

Comment 7

a year ago
Created attachment 8888148 [details] [review]
Move heroku cnames to new endpoints
(Assignee)

Comment 8

a year ago
Moving treestatus to non-sni broke treestatus. It has since been reverted back to the sni endpoint.

https://bugzilla.mozilla.org/show_bug.cgi?id=1382522#c2
:dividehex: could i ask you to switch to "SSL Endpoint" certificate for releng-staging-treestatus[1] heroku app? I will then test to see what/where it breaks.


[1] https://dashboard.heroku.com/apps/releng-staging-treestatus
Flags: needinfo?(rgarbas)

Updated

a year ago
Flags: needinfo?(jwatkins)
(In reply to Amy Rich [:arr] [:arich] from comment #3)
> :garbas: can you please set the dependencies/blockers for this bug?
> 
> The question seems to be, can we use heroku at all because it doesn't report
> ssl v3. Does puppet 2.7.5 on the windows buildbot platform support TLS?

As is 2.7.5 won't support TLS.
Flags: needinfo?(mcornmesser)
garbas: it sounds from markco's comment like there's some confusion/doubt that the windows platform is going to support the heroku endpoints even with the plugin. Can you meet with him/jake and talk about the testing methodology and what worked and didn't? We need to verify that for sure for tooltool.
Flags: needinfo?(rgarbas)
(Assignee)

Comment 12

a year ago
Created attachment 8890590 [details] [review]
fix terraform cname resources and change treestatus to ssl endpoint type
Attachment #8890590 - Flags: review?(rgarbas)
(Assignee)

Comment 13

a year ago
:garbas, I've migrated the releng-staging-treestatus app to the ssl endpoint (non-sni) type.  I did a little more testing of the app before and after.  It looks like it is working fine after the change over.
Flags: needinfo?(jwatkins)
:arr: :markco: I ran a test against new tooltool (on heroku) and after switching to "SSL Endpoint"[1] all tooltool related errors went away[2][3].

I will schedule a meeting with dividehex/markco to come to the bottom why is this all of a sudden working.
 

[1] https://devcenter.heroku.com/articles/ssl-endpoint
[2] https://reviewboard.mozilla.org/r/158628/diff/1#index_header
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b87c442f9d98
Flags: needinfo?(rgarbas)
(Assignee)

Comment 15

a year ago
releng-production-treestatus has been migrated (AGAIN) to a non-sni ssl endpoint.  There were a few reports of treestatus not working for people in #releng but they were probably due to stale dns cache on the system.

<dividehex> this still might be a dns caching error
<dividehex> system dns cache vs chrome
<kats> dividehex: ah you're right
<kats> i flushed my system dns cache and it works now
<dividehex> cool!
<kats> (i used `sudo killall -HUP mDNSResponder`, for reference)

Updated

a year ago
Attachment #8890590 - Flags: review?(rgarbas) → review+
You need to log in before you can comment on or make changes to this bug.