Closed Bug 901035 Opened 11 years ago Closed 9 years ago

[mkt] Reorder ciphersuites on https://marketplace.mozilla.org

Categories

(Cloud Services :: Operations: Marketplace, task, P2)

Product:
Cloud Services
Component:
Operations: Marketplace
Platform:
x86_64
Linux
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jvehent, Assigned: jason)

References

Details

Attachments

(2 files)

cipherscan marketplace.mozilla.org
3.52 KB, text/plain
Details
Netscaler supported ciphersuites
1021 bytes, text/plain
Details
The preferred ciphersuite on https://marketplace.mozilla.org is currently RC4-MD5. $ ./CiphersScan.sh marketplace.mozilla.org:443 prio ciphersuite 1 RC4-MD5 2 RC4-SHA 3 DES-CBC3-SHA 4 AES256-SHA 5 AES128-SHA 6 (NONE) Both RC4 and MD5 are weak algorithms. If available, TLS1.2 algorithms should be preferred. But if the SSL termination doesn't support TLS1.2, then at the very least RC4-SHA should be selected first. See Opsec's recommendation for more information: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456 Side question: what's in charge of terminating the SSL for marketplace ? Netscaler or Nginx?
Assignee: server-ops-amo → oremj
Netscaler is in charge of terminating ssl.
The cipher names are a bit different in netscaler than they are in zeus. Can you recommend a group for me? http://support.citrix.com/proddocs/topic/netscaler-traffic-management-10-1-map/ns-ssl-supported-ciphers-list-ref.html
Yep, I can take care of that. Will add it to the recommendation too. Which type/version of netscaler do we use? That's relevant because not all the ciphers are supported everywhere.
I'm a bit confused, because the doc claims that TLS1.2 is supported, but none of the TLS 1.2 ciphersuites show up in my tests. The documentation doesn't mention anything about AES-GCM or ECDHE either. I'm particularly interested in AESGCM, because it solves BEAST. (see attachment "cipherscan"). Can you reach out to our vendor and ask about this? If TLS1.2 is supported, then why aren't we seeing any of the ciphersuites? On the current setup, the reasonable thing to do is to disable SSL3-RC4-MD5 and promote SSL3-RC4-SHA as the preferred ciphersuite. Since RC4-SHA is supported everywhere, no other ciphersuite will be used. But for the sake of consistency, here's the recommended list of cipher. The order matters: SSL3-RC4-SHA TLS1-DHE-DSS-AES-256-CBC-SHA TLS1-DHE-RSA-AES-256-CBC-SHA TLS1-DHE-DSS-AES-128-CBC-SHA TLS1-DHE-RSA-AES-128-CBC-SHA TLS1-AES-256-CBC-SHA TLS1-AES-128-CBC-SHA
Blocks: tls-everything
Flags: needinfo?(oremj)
We have a few other tickets open with citrix about instability. I'd like to wait until those are closed out before filing a ticket about this, so I don't confuse them.
Flags: needinfo?(oremj)
Assignee: oremj → jthomas
I've opened up citrix ticket #61099103 requesting more information about supported TLS 1.2 cipher suites.
I created 'MozillaDefault' ciphergroup in netscaler and added the cipersuites in comment 4. This is now enabled on all netscaler https virtual servers.
Tested and validated. $ ./CiphersScan.sh marketplace.mozilla.org:443 prio ciphersuite protocol 1 RC4-SHA TLSv1 2 AES256-SHA TLSv1 3 AES128-SHA TLSv1 4 (NONE) This is a really short list of ciphers. It shouldn't cause any problem since RC4-SHA is supported everywhere. Does netscaler have logs to detect handshake failures?
Netscaler should log handshake failures, but I have not seen any so far in the logs. Citrix's response on TLS 1.2 support: TLS 1.2 is support in 10.1 119+ Builds, it will be back ported to 10.0 but that has yet to happen. ECDHE support is on the road map, but we do not have an ETA for this yet. We had to downgrade to version 10.0 due to stability issues (bug 900984) and waiting on firmware 11.0 to be released to include that bug fix. We can configure TLS 1.2 once we have upgraded to 11.0.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Sounds good. Thanks for checking with Citrix, and good to see that it's on the way!
Did you upgrade the netscaler? The ciphersuite has changed, and TLS1.2 negotiates successfully. If yes, I'd like to take another pass at the ciphersuite configuration.
We did not upgrade netscaler however our previous config changes were reverted. I believe I forgot to save the running config and during failover it reverted to a saved config. I reapplied the changes made in Comment 9 and made sure to save the config this time.
This happened again. Can we please fix the ciphersuite for good? It's really embarrassing to advertise RC4-MD5 as our preferred cipher... $ ./CiphersScan.sh marketplace.mozilla.org:443 prio ciphersuite protocol pfs_keysize 1 RC4-MD5 TLSv1 2 RC4-SHA TLSv1 3 DES-CBC3-SHA TLSv1 4 AES256-SHA TLSv1 5 AES128-SHA TLSv1
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
The ciphers have been reordered again and Jason is filing a ticket with Citrix about why the config unexpectedly changed back. CiphersScan.sh marketplace.firefox.com:443 prio ciphersuite protocol pfs_keysize 1 RC4-SHA TLSv1.1 2 AES256-SHA TLSv1.1 3 AES128-SHA TLSv1.1
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
I forgot to mention the revised ordering. After discussing it for a while, we decided to push RC4 way down, and prefer the ciphersuite below. Could you please update the MozillaDefault cipher accordingly? TLS1-DHE-DSS-AES-128-CBC-SHA TLS1-DHE-RSA-AES-128-CBC-SHA TLS1-DHE-DSS-AES-256-CBC-SHA TLS1-DHE-RSA-AES-256-CBC-SHA TLS1-AES-128-CBC-SHA TLS1-AES-256-CBC-SHA SSL3-RC4-SHA Copy/Paste conf is at https://wiki.mozilla.org/Security/Server_Side_TLS#Citrix_Netscaler My apologies for not mentioning it earlier...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Blocks: 937555
The issue described in https://bugzilla.mozilla.org/show_bug.cgi?id=937555 is really serious. :oremj, did the netscaler get upgraded? I see that TLS1.2 is now negotiable, and that issue with TLS1.0 and AES/3DES wasn't there before (or we missed it). Can we please review the configuration today? I'm available to help.
Flags: needinfo?(oremj)
Looks like the config reverted again. $ ./CiphersScan.sh marketplace.mozilla.org:443 prio ciphersuite protocols pfs_keysize 1 RC4-MD5 SSLv3,TLSv1,TLSv1.1,TLSv1.2 2 RC4-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 3 DES-CBC3-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 4 AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 5 AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
I re-applied the config. jason@kenshin:~/src|⇒ ./CiphersScan.sh marketplace.firefox.com:443 .... prio ciphersuite protocols pfs_keysize 1 RC4-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 2 AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 3 AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
Could you also reconfigure the default ciphersuite to match the new ordering from https://wiki.mozilla.org/Security/Server_Side_TLS#Citrix_Netscaler ? Thanks!
Ciphersuite reordered. Thanks a lot Jason ! $ ./CiphersScan.sh marketplace.firefox.com:443 ...... prio ciphersuite protocols pfs_keysize 1 DHE-RSA-AES128-SHA SSLv3,TLSv1,TLSv1.1 DH,1024bits 2 DHE-RSA-AES256-SHA SSLv3,TLSv1,TLSv1.1 DH,1024bits 3 AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 4 AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 5 RC4-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
Flags: needinfo?(oremj)
There is a bug with the Netscaler where DHE ciphersuite and TLS1.2 fail to negociate. This doesn't work (tls1_2): $ openssl s_client -connect marketplace.firefox.com:443 -cipher 'DHE-RSA-AES128-SHA' -tls1_2 This works (tls1_1): $ openssl s_client -connect marketplace.firefox.com:443 -cipher 'DHE-RSA-AES128-SHA' -tls1_1 It only impacts the combination of TLS1.2 and DHE. I tried TLS1.2 with AES and it works fine: $ openssl s_client -connect marketplace.firefox.com:443 -cipher 'AES128-SHA' -tls1_2 This is a vendor issue. Can we file a bug against Citrix for this?
Filed ticket #61166304 with Citrix.
Citrix has responded: "Issue ID 0345883: On the NetScaler appliance, TLS protocol version 1.2 does not support ephemeral Diffie-Hellman cipher suites." This issue affects both Netscaler 10.0 and 10.1 firmwares. The current work around is to disable TLS 1.2 or not to use Diffie-Hellman cipher suites. I am still waiting on a response from Citrix if TLS 1.2 can be disabled on Netscaler 10.0.
Thanks for the update (citrix took 20 days to respond?... wow). Indeed the preference would be to disable TLS1.2, keep TLS1.1, and enable DHE ciphers.
Reordering looks good. $ ./cipherscan marketplace.firefox.com:443 ...... prio ciphersuite protocols pfs_keysize 1 DHE-RSA-AES128-SHA SSLv3,TLSv1,TLSv1.1 DH,1024bits 2 DHE-RSA-AES256-SHA SSLv3,TLSv1,TLSv1.1 DH,1024bits 3 AES128-SHA SSLv3,TLSv1,TLSv1.1 4 AES256-SHA SSLv3,TLSv1,TLSv1.1 5 RC4-SHA SSLv3,TLSv1,TLSv1.1 No TLS1.2 (broken with DHE).
ECDHE support on netscaler devices are still limited to specific models[1]. As per comment 11 I believe the support will eventually make it to our model (MPX-11500). I will reopen this bug when it is available. [1] http://support.citrix.com/proddocs/topic/ns-rn-main-release-10-1-map/ns-rn-enhancements-121-x-con.html
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Reverted change due to Bug 959534.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Are you re-opening a bug with Citrix?
I have opened a new bug with Citrix #61194207. I will update once I have more information.
Any more info from citrix on this?
Priority: -- → P2
4 months have passed since the request with Citrix was opened. What's the status?
Flags: needinfo?(jthomas)
As per our meeting yesterday, we are going to look into alternative options (nginx, stm).
Flags: needinfo?(jthomas)
Corey or Jake: would you be willing to donate the experimental ZLB hardware that we were planning to use to test the migration to 9.5? see comment https://bugzilla.mozilla.org/show_bug.cgi?id=959666#c7 AFAIK, these two servers are sitting idle at the moment, but they could do some good in helping us improve marketplace and AMO.
Flags: needinfo?(nmaul)
Flags: needinfo?(cshields)
Summary: Reorder ciphersuites on https://marketplace.mozilla.org → [mkt] Reorder ciphersuites on https://marketplace.mozilla.org
+1
Flags: needinfo?(cshields)
if they're sitting idle, i see no reason why they couldn't be donated to amo :)
WebOps has no plans for them that I'm aware of, so that's fine with me too. :)
Flags: needinfo?(nmaul)
Depends on: 1024016
Depends on: 1038369
Group: mozilla-employee-confidential
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
Version: other → unspecified
Looks like the latest release fixed the issue. The ciphers are now properly ordered, thanks to :jason. $ ./cipherscan marketplace.firefox.com ...... prio ciphersuite protocols pfs_keysize 1 DHE-RSA-AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,1024bits 2 DHE-RSA-AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,1024bits 3 AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 4 AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 5 DES-CBC3-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
Resolution: --- → FIXED
I'm reopening this to update the ciphersuite to the intermediate level. The current level matches the old configuration from https://wiki.mozilla.org/Security/Server_Side_TLS. $ ./cipherscan marketplace.firefox.com ...... prio ciphersuite protocols pfs_keysize 1 DHE-RSA-AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,1024bits 2 DHE-RSA-AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,1024bits 3 AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 4 AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 5 DES-CBC3-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature TLS ticket lifetime hint: None OCSP stapling: not supported Server side cipher ordering Changes needed to match the intermediate level: * remove cipher DES-CBC3-SHA * disable SSLv3 * consider enabling OCSP Stapling
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
We should definitely move away from 1024-bit DH as well. It might not be critical in this case, because we care mostly about authentication and integrity on these sites. A brute force might be feasible, but it takes time, so it only compromises confidentiality. That's definitely a problem if we have long-lived cookies or other secrets (basic/digest authentication, perhaps). Safest thing to do is upgrade. For addons and marketplace, the only reason we might not use the best possible profile is limited availability of capable hardware.
see also bug 949564 for the client-side complement to this.
marketplace.firefox.com has moved to AWS and now fronted by ELB using security policy "ELBSecurityPolicy-2015-05" ./cipherscan marketplace.firefox.com:443 ..................... Target: marketplace.firefox.com:443 prio ciphersuite protocols pfs curves 1 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1 2 ECDHE-RSA-AES128-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1 3 ECDHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1 4 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1 5 ECDHE-RSA-AES256-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1 6 ECDHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1 7 AES128-GCM-SHA256 TLSv1.2 None None 8 AES128-SHA256 TLSv1.2 None None 9 AES128-SHA TLSv1,TLSv1.1,TLSv1.2 None None 10 AES256-GCM-SHA384 TLSv1.2 None None 11 AES256-SHA256 TLSv1.2 None None 12 AES256-SHA TLSv1,TLSv1.1,TLSv1.2 None None 13 DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 None None Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature TLS ticket lifetime hint: 300 OCSP stapling: not supported Cipher ordering: server Curves ordering: server - fallback: no TLS Tolerance: yes
Status: REOPENED → RESOLVED
Closed: 10 years ago → 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: