[mkt] Reorder ciphersuites on https://marketplace.mozilla.org

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: ulfr, Assigned: jason)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

5 years ago
The preferred ciphersuite on https://marketplace.mozilla.org is currently RC4-MD5.

$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite
1     RC4-MD5
2     RC4-SHA
3     DES-CBC3-SHA
4     AES256-SHA
5     AES128-SHA
6     (NONE)


Both RC4 and MD5 are weak algorithms. If available, TLS1.2 algorithms should be preferred. But if the SSL termination doesn't support TLS1.2, then at the very least RC4-SHA should be selected first.

See Opsec's recommendation for more information: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456

Side question: what's in charge of terminating the SSL for marketplace ? Netscaler or Nginx?
Assignee: server-ops-amo → oremj
Netscaler is in charge of terminating ssl.
The cipher names are a bit different in netscaler than they are in zeus. Can you recommend a group for me?

http://support.citrix.com/proddocs/topic/netscaler-traffic-management-10-1-map/ns-ssl-supported-ciphers-list-ref.html
(Reporter)

Comment 3

5 years ago
Yep, I can take care of that. Will add it to the recommendation too.

Which type/version of netscaler do we use? That's relevant because not all the ciphers are supported everywhere.
(Reporter)

Comment 4

5 years ago
I'm a bit confused, because the doc claims that TLS1.2 is supported, but none of the TLS 1.2 ciphersuites show up in my tests. The documentation doesn't mention anything about AES-GCM or ECDHE either. I'm particularly interested in AESGCM, because it solves BEAST. (see attachment "cipherscan").

Can you reach out to our vendor and ask about this? If TLS1.2 is supported, then why aren't we seeing any of the ciphersuites?


On the current setup, the reasonable thing to do is to disable SSL3-RC4-MD5 and promote SSL3-RC4-SHA as the preferred ciphersuite. Since RC4-SHA is supported everywhere, no other ciphersuite will be used. But for the sake of consistency, here's the recommended list of cipher. The order matters:

SSL3-RC4-SHA
TLS1-DHE-DSS-AES-256-CBC-SHA
TLS1-DHE-RSA-AES-256-CBC-SHA
TLS1-DHE-DSS-AES-128-CBC-SHA
TLS1-DHE-RSA-AES-128-CBC-SHA
TLS1-AES-256-CBC-SHA
TLS1-AES-128-CBC-SHA
(Reporter)

Comment 5

5 years ago
Created attachment 785450 [details]
cipherscan marketplace.mozilla.org
(Reporter)

Updated

5 years ago
Blocks: 901393
(Reporter)

Comment 6

5 years ago
Created attachment 785579 [details]
Netscaler supported ciphersuites
(Reporter)

Updated

5 years ago
Flags: needinfo?(oremj)
We have a few other tickets open with citrix about instability. I'd like to wait until those are closed out before filing a ticket about this, so I don't confuse them.
Flags: needinfo?(oremj)
(Assignee)

Updated

5 years ago
Assignee: oremj → jthomas
(Assignee)

Comment 8

5 years ago
I've opened up citrix ticket #61099103 requesting more information about supported TLS 1.2 cipher suites.
(Assignee)

Comment 9

5 years ago
I created 'MozillaDefault' ciphergroup in netscaler and added the cipersuites in comment 4. This is now enabled on all netscaler https virtual servers.
(Reporter)

Comment 10

5 years ago
Tested and validated.

$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite  protocol
1     RC4-SHA      TLSv1
2     AES256-SHA   TLSv1
3     AES128-SHA   TLSv1
4     (NONE)

This is a really short list of ciphers. It shouldn't cause any problem since RC4-SHA is supported everywhere. Does netscaler have logs to detect handshake failures?
(Assignee)

Comment 11

5 years ago
Netscaler should log handshake failures, but I have not seen any so far in the logs.

Citrix's response on TLS 1.2 support:

TLS 1.2 is support in 10.1 119+ Builds, it will be back ported to 10.0 but that has yet to happen. ECDHE support is on the road map, but we do not have an ETA for this yet.

We had to downgrade to version 10.0 due to stability issues (bug 900984) and waiting on firmware 11.0 to be released to include that bug fix.

We can configure TLS 1.2 once we have upgraded to 11.0.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 12

5 years ago
Sounds good. Thanks for checking with Citrix, and good to see that it's on the way!
(Reporter)

Comment 13

5 years ago
Did you upgrade the netscaler? The ciphersuite has changed, and TLS1.2 negotiates successfully.
If yes, I'd like to take another pass at the ciphersuite configuration.
(Assignee)

Comment 14

5 years ago
We did not upgrade netscaler however our previous config changes were reverted. I believe I forgot to save the running config and during failover it reverted to a saved config. I reapplied the changes made in Comment 9 and made sure to save the config this time.
(Reporter)

Comment 15

5 years ago
This happened again. Can we please fix the ciphersuite for good? It's really embarrassing to advertise RC4-MD5 as our preferred cipher...


$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite   protocol  pfs_keysize
1     RC4-MD5       TLSv1
2     RC4-SHA       TLSv1
3     DES-CBC3-SHA  TLSv1
4     AES256-SHA    TLSv1
5     AES128-SHA    TLSv1
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Someone called this out on twitter: https://twitter.com/7adietri/status/401356916070166529
The ciphers have been reordered again and Jason is filing a ticket with Citrix about why the config unexpectedly changed back.

CiphersScan.sh marketplace.firefox.com:443
prio  ciphersuite  protocol  pfs_keysize
1     RC4-SHA      TLSv1.1
2     AES256-SHA   TLSv1.1
3     AES128-SHA   TLSv1.1
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 18

5 years ago
I forgot to mention the revised ordering. After discussing it for a while, we decided to push RC4 way down, and prefer the ciphersuite below. Could you please update the MozillaDefault cipher accordingly? 

TLS1-DHE-DSS-AES-128-CBC-SHA
TLS1-DHE-RSA-AES-128-CBC-SHA
TLS1-DHE-DSS-AES-256-CBC-SHA
TLS1-DHE-RSA-AES-256-CBC-SHA
TLS1-AES-128-CBC-SHA
TLS1-AES-256-CBC-SHA
SSL3-RC4-SHA

Copy/Paste conf is at https://wiki.mozilla.org/Security/Server_Side_TLS#Citrix_Netscaler

My apologies for not mentioning it earlier...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Blocks: 937555
(Reporter)

Comment 19

5 years ago
The issue described in https://bugzilla.mozilla.org/show_bug.cgi?id=937555 is really serious.

:oremj, did the netscaler get upgraded? I see that TLS1.2 is now negotiable, and that issue with TLS1.0 and AES/3DES wasn't there before (or we missed it).

Can we please review the configuration today? I'm available to help.
Flags: needinfo?(oremj)
(Reporter)

Comment 20

5 years ago
Looks like the config reverted again.

$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite   protocols                    pfs_keysize
1     RC4-MD5       SSLv3,TLSv1,TLSv1.1,TLSv1.2
2     RC4-SHA       SSLv3,TLSv1,TLSv1.1,TLSv1.2
3     DES-CBC3-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2
4     AES256-SHA    SSLv3,TLSv1,TLSv1.1,TLSv1.2
5     AES128-SHA    SSLv3,TLSv1,TLSv1.1,TLSv1.2
(Reporter)

Updated

5 years ago
Depends on: 929110
(Assignee)

Comment 21

5 years ago
I re-applied the config.

jason@kenshin:~/src|⇒  ./CiphersScan.sh marketplace.firefox.com:443
....
prio  ciphersuite  protocols                    pfs_keysize
1     RC4-SHA      SSLv3,TLSv1,TLSv1.1,TLSv1.2
2     AES256-SHA   SSLv3,TLSv1,TLSv1.1,TLSv1.2
3     AES128-SHA   SSLv3,TLSv1,TLSv1.1,TLSv1.2
(Reporter)

Comment 22

5 years ago
Could you also reconfigure the default ciphersuite to match the new ordering from https://wiki.mozilla.org/Security/Server_Side_TLS#Citrix_Netscaler ?

Thanks!
(Assignee)

Updated

5 years ago
Duplicate of this bug: 940831
(Reporter)

Comment 24

5 years ago
Ciphersuite reordered. Thanks a lot Jason !

$ ./CiphersScan.sh marketplace.firefox.com:443
......
prio  ciphersuite         protocols                    pfs_keysize
1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1          DH,1024bits
2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1          DH,1024bits
3     AES128-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
4     AES256-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
5     RC4-SHA             SSLv3,TLSv1,TLSv1.1,TLSv1.2
Flags: needinfo?(oremj)
(Reporter)

Comment 25

5 years ago
There is a bug with the Netscaler where DHE ciphersuite and TLS1.2 fail to negociate. 

This doesn't work (tls1_2):

$ openssl s_client -connect marketplace.firefox.com:443 -cipher 'DHE-RSA-AES128-SHA' -tls1_2

This works (tls1_1):

$ openssl s_client -connect marketplace.firefox.com:443 -cipher 'DHE-RSA-AES128-SHA' -tls1_1

It only impacts the combination of TLS1.2 and DHE. I tried TLS1.2 with AES and it works fine:

$ openssl s_client -connect marketplace.firefox.com:443 -cipher 'AES128-SHA' -tls1_2

This is a vendor issue. Can we file a bug against Citrix for this?
(Assignee)

Comment 26

5 years ago
Filed ticket #61166304 with Citrix.
(Assignee)

Comment 27

5 years ago
Citrix has responded: "Issue ID 0345883: On the NetScaler appliance, TLS protocol version 1.2 does not support ephemeral Diffie-Hellman cipher suites." 

This issue affects both Netscaler 10.0 and 10.1 firmwares.

The current work around is to disable TLS 1.2 or not to use Diffie-Hellman cipher suites. I am still waiting on a response from Citrix if TLS 1.2 can be disabled on Netscaler 10.0.
(Reporter)

Comment 28

5 years ago
Thanks for the update (citrix took 20 days to respond?... wow).
Indeed the preference would be to disable TLS1.2, keep TLS1.1, and enable DHE ciphers.
(Assignee)

Updated

5 years ago
Depends on: 951236
(Reporter)

Comment 29

5 years ago
Reordering looks good.

$ ./cipherscan marketplace.firefox.com:443
......
prio  ciphersuite         protocols            pfs_keysize
1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1  DH,1024bits
2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1  DH,1024bits
3     AES128-SHA          SSLv3,TLSv1,TLSv1.1
4     AES256-SHA          SSLv3,TLSv1,TLSv1.1
5     RC4-SHA             SSLv3,TLSv1,TLSv1.1

No TLS1.2 (broken with DHE).
(Assignee)

Comment 30

5 years ago
ECDHE support on netscaler devices are still limited to specific models[1]. As per comment 11 I believe the support will eventually make it to our model (MPX-11500). I will reopen this bug when it is available.

[1] http://support.citrix.com/proddocs/topic/ns-rn-main-release-10-1-map/ns-rn-enhancements-121-x-con.html
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 31

5 years ago
Reverted change due to Bug 959534.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 32

5 years ago
Are you re-opening a bug with Citrix?
(Assignee)

Comment 33

5 years ago
I have opened a new bug with Citrix #61194207. I will update once I have more information.
Any more info from citrix on this?
Priority: -- → P2
(Reporter)

Comment 35

5 years ago
4 months have passed since the request with Citrix was opened. What's the status?
Flags: needinfo?(jthomas)
(Assignee)

Comment 36

5 years ago
As per our meeting yesterday, we are going to look into alternative options (nginx, stm).
Flags: needinfo?(jthomas)
(Reporter)

Comment 37

5 years ago
Corey or Jake: would you be willing to donate the experimental ZLB hardware that we were planning to use to test the migration to 9.5? see comment https://bugzilla.mozilla.org/show_bug.cgi?id=959666#c7

AFAIK, these two servers are sitting idle at the moment, but they could do some good in helping us improve marketplace and AMO.
Flags: needinfo?(nmaul)
Flags: needinfo?(cshields)
Summary: Reorder ciphersuites on https://marketplace.mozilla.org → [mkt] Reorder ciphersuites on https://marketplace.mozilla.org
+1
Flags: needinfo?(cshields)
if they're sitting idle, i see no reason why they couldn't be donated to amo :)
WebOps has no plans for them that I'm aware of, so that's fine with me too. :)
Flags: needinfo?(nmaul)
(Assignee)

Updated

5 years ago
Depends on: 1018258
(Assignee)

Updated

5 years ago
Depends on: 1024016
(Assignee)

Updated

5 years ago
Depends on: 1038369
Group: mozilla-employee-confidential
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
Version: other → unspecified
(Reporter)

Comment 41

4 years ago
Looks like the latest release fixed the issue. The ciphers are now properly ordered, thanks to :jason.

$ ./cipherscan marketplace.firefox.com
......
prio  ciphersuite         protocols                    pfs_keysize
1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
3     AES128-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
4     AES256-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
5     DES-CBC3-SHA        SSLv3,TLSv1,TLSv1.1,TLSv1.2

Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 42

4 years ago
I'm reopening this to update the ciphersuite to the intermediate level. The current level matches the old configuration from https://wiki.mozilla.org/Security/Server_Side_TLS.

    $ ./cipherscan marketplace.firefox.com
    ......
    prio  ciphersuite         protocols                    pfs_keysize
    1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
    2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
    3     AES128-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
    4     AES256-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
    5     DES-CBC3-SHA        SSLv3,TLSv1,TLSv1.1,TLSv1.2

    Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature
    TLS ticket lifetime hint: None
    OCSP stapling: not supported
    Server side cipher ordering

Changes needed to match the intermediate level:
* remove cipher DES-CBC3-SHA
* disable SSLv3
* consider enabling OCSP Stapling
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
We should definitely move away from 1024-bit DH as well.  It might not be critical in this case, because we care mostly about authentication and integrity on these sites.  A brute force might be feasible, but it takes time, so it only compromises confidentiality.  That's definitely a problem if we have long-lived cookies or other secrets (basic/digest authentication, perhaps).

Safest thing to do is upgrade.  For addons and marketplace, the only reason we might not use the best possible profile is limited availability of capable hardware.
(Assignee)

Updated

4 years ago
Depends on: 1094836
(Reporter)

Updated

4 years ago
Duplicate of this bug: 1116961

Comment 45

4 years ago
see also bug 949564 for the client-side complement to this.
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1200040
(Assignee)

Comment 47

3 years ago
marketplace.firefox.com has moved to AWS and now fronted by ELB using security policy "ELBSecurityPolicy-2015-05"

./cipherscan marketplace.firefox.com:443
.....................
Target: marketplace.firefox.com:443

prio  ciphersuite                  protocols              pfs                 curves
1     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
2     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
3     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
4     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
5     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
6     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
7     AES128-GCM-SHA256            TLSv1.2                None                None
8     AES128-SHA256                TLSv1.2                None                None
9     AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
10    AES256-GCM-SHA384            TLSv1.2                None                None
11    AES256-SHA256                TLSv1.2                None                None
12    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
13    DES-CBC3-SHA                 TLSv1,TLSv1.1,TLSv1.2  None                None

Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature
TLS ticket lifetime hint: 300
OCSP stapling: not supported
Cipher ordering: server
Curves ordering: server - fallback: no
TLS Tolerance: yes
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.