1077634 - Enable Intermediate TLS configuration on WebOps operated sites

Status: RESOLVED FIXED

(Infrastructure & Operations :: SSL Certificates, task)

Description

[Julien Vehent]

Attachment: webops sites

Mozilla is pushing an effort to enable and improve HTTPS on all our websites and services, regardless of their size or audience.

We started with a basic review of the sites that are operated by the WebOps group (attached to this bug). The majority of these sites use an old SSL configuration (a handful don't have SSL enabled) that could easily be changed without impacting users.

Unless a site needs backward compatibility with Windows XP, we recommend that all sites implement the Intermediate configuration from this page: https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28default.29

According the Chris More, only www.mozilla.org and start.mozilla.org require backward compatibility. www.mozilla.org is already SSL enabled and must keep SSLv3 and 3DES enabled. start.mozilla.org should be SSL enabled (it is not today) and match the configuration of www.mozilla.org.
If you are aware of other sites that need backward compatibility (maybe in product delivery?), please let me know and I will update the list.

For sites that are in AWS, the script `elb_ciphers.py` will automatically set the proper configuration on an ELB: https://wiki.mozilla.org/Security/Server_Side_TLS#elb_ciphers.py

For sites outside of AWS, disabling SSLv3 and DES-CBC3-SHA should do the trick.

A spreadsheet is used to track all the sites at Mozilla. Filter on the Operator column to see the sites that belong to Infrastructure & Operations::WebOps. If a site needs a different configuration than the Intermediate one, please change the "Required Configuration" column.
https://docs.google.com/a/mozilla.com/spreadsheets/d/1RUXbLXYoRuyZoOkl3ARg_oZlU8eH89Ms-pHS0wY8cwY/edit#gid=0

Thanks!

Comment 1

[Jake Maul [:jakem]]

I have state on this and will take care of the WebOps things. Fits in nicely with a similar bug I already own re: OCSP.

Comment 2

[Jake Maul [:jakem]]

As a start I have updated the global settings for PHX1, SCL3, PEK1, and HCI to match the "old" config recommendation. This just means dropping RC4-SHA and SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA (which is apparently actually EDH-RSA-DES-CBC3-SHA, which requires certificate support, which we don't have, and therefore doesn't ever get chosen anyway).

Next step is to identify the sites and apps that need to remain at the "old" level (primarily www.mozilla.org and download.mozilla.org, but maybe others too) and set those vservers explicitly. Then we can change the global defaults to be "intermediate"... which means disabling SSLv3 and removing DES-CBC3-SHA.

Comment 3

[Julien Vehent]

FYI, I maintain a kibana dashboard with the status of webops sites at http://mozdefqa1.private.scl3.mozilla.com:9090/index.html#/dashboard/elasticsearch/WebOps%20Cipherscan

Comment 4

[Jake Maul [:jakem]]

Cipher suite is good, all identified "old" sites are pinned to have SSLv3 enabled.

All that's left to do is change the global default for PHX1, SCL3, and HCI to be "SSLv3 disabled". We probably want to do one first and watch to make sure Pingdom doesn't go crazy before we do the other two. Supposedly it will "fail up" to TLSv1, but that's odd enough behavior we'd like to verify it.

Comment 5

[Jake Maul [:jakem]]

Global config changed. All seems well, no reports of problems so far.

Leaving this open for now, as there are around 20 or so sites/apps that may need some TLC to get up to par... stuff that isn't SSL-terminated at the ZLB layer. Some of this is fixable by us, some are hosted by 3rd parties and may or may not be fixable.

Comment 6

[Julien Vehent]

76% of IT operated sites have disabled SSLv3. A handful of sites still have SSLv3 enabled. They fall in three categories:

1. Sites that will maintain SSLv3 compatibility for our IE6 users. No further action needed here.
    download-installer.cdn.mozilla.net
    download.mozilla.org
    mozorg.cdn.mozilla.net
    v.mozilla.com
    www.mozilla.org

2. Sites that we host and should remove SSLv3. We are collecting more data on these sites, to confirm that disabling SSLv3 will not impact our users negatively. Further action should be taken in the upcoming days.
    mail.mozilla.com
    bugzilla.mozilla.org

3. Sites that should remove SSLv3 but are hosted by 3rd parties. Disabling SSLv3 will require back and forth with the 3rd parties. These sites do not serve user data, so the impact of the SSLv3 vulnerability is low, and mitigation is not urgent.
    activations.cdn.mozilla.net
    addons.cdn.mozilla.net
    code.cdn.mozilla.net
    dynamicua.cdn.mozilla.net
    fhr.cdn.mozilla.net
    glow.cdn.mozilla.net
    marketplace.cdn.mozilla.net
    marketplace-stage.cdn.mozilla.net
    status.mozilla.com
    videos.cdn.mozilla.net

Comment 7

[Jake Maul [:jakem]]

After filtering out 3 bots (Sogou, Baidu, and Pingdom), here's the breakdown of ciphers used on bugzilla.mozilla.org over a 1 hour period:

      1 ssl:SSL_RSA_WITH_3DES_EDE_CBC_SHA, version=SSLv3, bits=168
     58 ssl:SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=SSLv3, bits=128
     73 ssl:SSL_DHE_RSA_WITH_AES_256_CBC_SHA, version=TLSv1.1, bits=256
    186 ssl:SSL_RSA_WITH_3DES_EDE_CBC_SHA, version=TLSv1, bits=168
    659 ssl:SSL_RSA_WITH_AES_128_CBC_SHA, version=TLSv1.1, bits=128
   1861 ssl:SSL_RSA_WITH_AES_128_CBC_SHA, version=TLSv1, bits=128
   6561 ssl:-
   7864 ssl:SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=TLSv1, bits=128
  42857 ssl:SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=TLSv1.1, bits=128

As you can see SSLv3 is the least common by a wide margin... in total, it amounts to approximately 0.11% of encrypted traffic to bugzilla.mozilla.org.

I've looked over the SSLv3 hits and they break down as follows:

     21 Firefox
     15 Konqueror
     25 Safari
      6 SeaMonkey

Based on my previous experience with TLS SNI fallbacks, I'm fairly confident that a large percentage of the Firefox hits are actually fallbacks from TLS handshakes that timed out... I imagine that's true for SeaMonkey as well. In that case a simple reload of the page will probably fix the issue.

I feel pretty confident we can disable SSLv3 on BMO with very minimal impact.



I have not evaluated mail.mozilla.com yet.

Comment 8

[Jake Maul [:jakem]]

I've looked at mail.mozilla.com a bit...

      2 ssl:SSL_RSA_WITH_3DES_EDE_CBC_SHA, version=TLSv1, bits=168
    106 ssl:SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=SSLv3, bits=128
    388 ssl:SSL_RSA_WITH_3DES_EDE_CBC_SHA, version=SSLv3, bits=168
   3009 ssl:SSL_RSA_WITH_AES_128_CBC_SHA, version=TLSv1, bits=128
   5186 ssl:SSL_RSA_WITH_AES_128_CBC_SHA, version=TLSv1.1, bits=128
  14325 ssl:-
 129307 ssl:SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=TLSv1, bits=128
2419195 ssl:SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=TLSv1.1, bits=128


These SSLv3 requests appear to come largely from Zimbra-ZCO, with much smaller hits from OSX CalendarAgent, and iPhone/Android calendar syncing.

Digging into this further, it appears that this may be a handful of users or Zimbra-ZCO installations that are having trouble, as the majority of Zimbra-ZCO connections use TLS (94.4%). This may be something we want to discuss with Zimbra.

CalendarAgent SSLv3 is much rarer than ZCO already, and the population of hits from it over SSLv3 is miniscule... 0.04%. I think we can safely disregard this.



The only concern I have here is to double-check with Zimbra as to why some hits from ZCO clients would use SSLv3 instead of TLSv1 or 1.1.

Comment 9

[Julien Vehent]

Awesome stats. Thanks Jake!

Let's leave mail.mozilla.com as it is. It won't be used for much longer anyway.

The bugzilla team was waiting on these stats to make the move. Based on your stats, and if :mcote can give us a sign off, I think we're good to disable SSLv3 on bmo.

Comment 10

[Mark Côté [:mcote]]

Sounds good; let's get rid of SSLv3 on bmo.

Comment 11

[Kendall Libby [:fubar] (he/him)]

SSLv3 disabled on bugzilla.mozilla.org, *.bugzilla.mozilla.org, bugzilla.allizom.org in SCL3 and PHX1 (where applicable).

Comment 12

[Jake Maul [:jakem]]

I believe this is as "done" as it's going to get any time soon.

Comment 13

[Daniel Veditz [:dveditz]]

Is there any reason for this bug to remain hidden?

Comment 14

[Julien Vehent]

Nope.