As it turns out we still use python 2.7.5 on m-c build environment and this prevent using sni compatible ssl endpoints such as the free one being utilized on many heroku apps. In this case, we need to revert back to using an SSL Endpoint addon and with a imported ssl cert (from digicert). Steps here are: * Enable SSL Endpoint via cli * Generate SSL cert and get have it signed (digicert) * Import SSL cert to heroku via cli * Change cname under releng terraform config (aws route53) to match new ssl elb provided by heroku
I've migrated both tooltool-prodution and tooltool-staging to the ssl endpoint elbs using the previous cert that was originally generated and signed by digicert. Terraform has been modified and apply to account for the new elb endpoint cnames. Although, testing the connection sslv3 is still not supported on these endpoints. Although it doesn't state it in the endpoint ssl doc, it does state in the ssl doc , that sslv3 is not supported and heroku "maintain parity with the predefined AWS Elastic Load Balancers SSL security policies." I have a strong suspicion this applies to the 'endpoint' type elbs just the same. Which means no SSLv3 support at all.  https://devcenter.heroku.com/articles/ssl-endpoint  https://devcenter.heroku.com/articles/ssl#supported-ssl-protocols
┌─[heroku] └─▪ nmap --script ssl-enum-ciphers tooltool.staging.mozilla-releng.net Starting Nmap 7.50 ( https://nmap.org ) at 2017-07-11 17:37 PDT Nmap scan report for tooltool.staging.mozilla-releng.net (18.104.22.168) Host is up (0.093s latency). Other addresses for tooltool.staging.mozilla-releng.net (not scanned): 22.214.171.124 126.96.36.199 rDNS record for 188.8.131.52: ec2-23-23-255-72.compute-1.amazonaws.com Not shown: 848 closed ports, 150 filtered ports PORT STATE SERVICE 80/tcp open http 443/tcp open https | ssl-enum-ciphers: | TLSv1.0: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: server | TLSv1.1: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: server | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A | TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A | compressors: | NULL | cipher preference: server |_ least strength: A Nmap done: 1 IP address (1 host up) scanned in 16.88 seconds
:garbas: can you please set the dependencies/blockers for this bug? The question seems to be, can we use heroku at all because it doesn't report ssl v3. Does puppet 2.7.5 on the windows buildbot platform support TLS?
Sorry python 2.7.5, not puppet.
Looks like SSL Endpoint solved the issue in the case of the new tooltool. All errors in treeherder are not related to I had to fix an authentication bug in the way how we handle SECRET_KEY in the new setup. :dividehex: Could we also do this for following Heroku apps: - releng-production-clobberer - releng-production-treestatus  https://treeherder.mozilla.org/#/jobs?repo=try&revision=0ccf2acabb6d95a090b3c56150b5502b946fc3da  https://github.com/mozilla-releng/services/pull/507
Summary: Enable ssl endpoint (non-sni) addon for tooltool releng heroku app → Enable ssl endpoint (non-sni) addon for a handful of releng heroku app
:garbas, I've gone ahead and re-enabled the ssl endpoint addons (non-sni) for both: - releng-production-clobberer - releng-production-treestatus Do you not need the staging counterparts also moved to non-sni endpoints?
Moving treestatus to non-sni broke treestatus. It has since been reverted back to the sni endpoint. https://bugzilla.mozilla.org/show_bug.cgi?id=1382522#c2
:dividehex: could i ask you to switch to "SSL Endpoint" certificate for releng-staging-treestatus heroku app? I will then test to see what/where it breaks.  https://dashboard.heroku.com/apps/releng-staging-treestatus
(In reply to Amy Rich [:arr] [:arich] from comment #3) > :garbas: can you please set the dependencies/blockers for this bug? > > The question seems to be, can we use heroku at all because it doesn't report > ssl v3. Does puppet 2.7.5 on the windows buildbot platform support TLS? As is 2.7.5 won't support TLS.
garbas: it sounds from markco's comment like there's some confusion/doubt that the windows platform is going to support the heroku endpoints even with the plugin. Can you meet with him/jake and talk about the testing methodology and what worked and didn't? We need to verify that for sure for tooltool.
Created attachment 8890590 [details] [review] fix terraform cname resources and change treestatus to ssl endpoint type
Attachment #8890590 - Flags: review?(rgarbas)
:garbas, I've migrated the releng-staging-treestatus app to the ssl endpoint (non-sni) type. I did a little more testing of the app before and after. It looks like it is working fine after the change over.
:arr: :markco: I ran a test against new tooltool (on heroku) and after switching to "SSL Endpoint" all tooltool related errors went away. I will schedule a meeting with dividehex/markco to come to the bottom why is this all of a sudden working.  https://devcenter.heroku.com/articles/ssl-endpoint  https://reviewboard.mozilla.org/r/158628/diff/1#index_header  https://treeherder.mozilla.org/#/jobs?repo=try&revision=b87c442f9d98
releng-production-treestatus has been migrated (AGAIN) to a non-sni ssl endpoint. There were a few reports of treestatus not working for people in #releng but they were probably due to stale dns cache on the system. <dividehex> this still might be a dns caching error <dividehex> system dns cache vs chrome <kats> dividehex: ah you're right <kats> i flushed my system dns cache and it works now <dividehex> cool! <kats> (i used `sudo killall -HUP mDNSResponder`, for reference)
This bug is outdated and has been superseded by bug 1487798. Since python has been updated throughout most of the firefox CI pipeline, we have also migrated the releng Heroku apps from the Heroku ssl endpoint addon to Heroku's ACM.
Status: NEW → RESOLVED
Last Resolved: 2 months ago
Resolution: --- → INVALID
See Also: → bug 1487798
You need to log in before you can comment on or make changes to this bug.