Closed Bug 901035 Opened 11 years ago Closed 9 years ago

[mkt] Reorder ciphersuites on https://marketplace.mozilla.org

Categories

(Cloud Services :: Operations: Marketplace, task, P2)

Product:
Cloud Services
Component:
Operations: Marketplace
Platform:
x86_64
Linux
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jvehent, Assigned: jason)

References

Details

Attachments

(2 files)

cipherscan marketplace.mozilla.org
3.52 KB, text/plain
Details
Netscaler supported ciphersuites
1021 bytes, text/plain
Details 
The preferred ciphersuite on https://marketplace.mozilla.org is currently RC4-MD5.

$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite
1     RC4-MD5
2     RC4-SHA
3     DES-CBC3-SHA
4     AES256-SHA
5     AES128-SHA
6     (NONE)


Both RC4 and MD5 are weak algorithms. If available, TLS1.2 algorithms should be preferred. But if the SSL termination doesn't support TLS1.2, then at the very least RC4-SHA should be selected first.

See Opsec's recommendation for more information: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456

Side question: what's in charge of terminating the SSL for marketplace ? Netscaler or Nginx?
Assignee: server-ops-amo → oremj
Netscaler is in charge of terminating ssl.
The cipher names are a bit different in netscaler than they are in zeus. Can you recommend a group for me?

http://support.citrix.com/proddocs/topic/netscaler-traffic-management-10-1-map/ns-ssl-supported-ciphers-list-ref.html
Yep, I can take care of that. Will add it to the recommendation too.

Which type/version of netscaler do we use? That's relevant because not all the ciphers are supported everywhere.
I'm a bit confused, because the doc claims that TLS1.2 is supported, but none of the TLS 1.2 ciphersuites show up in my tests. The documentation doesn't mention anything about AES-GCM or ECDHE either. I'm particularly interested in AESGCM, because it solves BEAST. (see attachment "cipherscan").

Can you reach out to our vendor and ask about this? If TLS1.2 is supported, then why aren't we seeing any of the ciphersuites?


On the current setup, the reasonable thing to do is to disable SSL3-RC4-MD5 and promote SSL3-RC4-SHA as the preferred ciphersuite. Since RC4-SHA is supported everywhere, no other ciphersuite will be used. But for the sake of consistency, here's the recommended list of cipher. The order matters:

SSL3-RC4-SHA
TLS1-DHE-DSS-AES-256-CBC-SHA
TLS1-DHE-RSA-AES-256-CBC-SHA
TLS1-DHE-DSS-AES-128-CBC-SHA
TLS1-DHE-RSA-AES-128-CBC-SHA
TLS1-AES-256-CBC-SHA
TLS1-AES-128-CBC-SHA

    
  
Blocks: tls-everything

    
    

    
  


  

    
  
Flags: needinfo?(oremj)

    
    

    
  
We have a few other tickets open with citrix about instability. I'd like to wait until those are closed out before filing a ticket about this, so I don't confuse them.
Flags: needinfo?(oremj)

    
  
Assignee: oremj → jthomas

    
    

    
  
I've opened up citrix ticket #61099103 requesting more information about supported TLS 1.2 cipher suites.

    
    

    
  
I created 'MozillaDefault' ciphergroup in netscaler and added the cipersuites in comment 4. This is now enabled on all netscaler https virtual servers.

    
    

    
  
Tested and validated.

$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite  protocol
1     RC4-SHA      TLSv1
2     AES256-SHA   TLSv1
3     AES128-SHA   TLSv1
4     (NONE)

This is a really short list of ciphers. It shouldn't cause any problem since RC4-SHA is supported everywhere. Does netscaler have logs to detect handshake failures?

    
    

    
  
Netscaler should log handshake failures, but I have not seen any so far in the logs.

Citrix's response on TLS 1.2 support:

TLS 1.2 is support in 10.1 119+ Builds, it will be back ported to 10.0 but that has yet to happen. ECDHE support is on the road map, but we do not have an ETA for this yet.

We had to downgrade to version 10.0 due to stability issues (bug 900984) and waiting on firmware 11.0 to be released to include that bug fix.

We can configure TLS 1.2 once we have upgraded to 11.0.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED

    
    

    
  
Sounds good. Thanks for checking with Citrix, and good to see that it's on the way!

    
    

    
  
Did you upgrade the netscaler? The ciphersuite has changed, and TLS1.2 negotiates successfully.
If yes, I'd like to take another pass at the ciphersuite configuration.

    
    

    
  
We did not upgrade netscaler however our previous config changes were reverted. I believe I forgot to save the running config and during failover it reverted to a saved config. I reapplied the changes made in Comment 9 and made sure to save the config this time.

    
    

    
  
This happened again. Can we please fix the ciphersuite for good? It's really embarrassing to advertise RC4-MD5 as our preferred cipher...


$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite   protocol  pfs_keysize
1     RC4-MD5       TLSv1
2     RC4-SHA       TLSv1
3     DES-CBC3-SHA  TLSv1
4     AES256-SHA    TLSv1
5     AES128-SHA    TLSv1
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
The ciphers have been reordered again and Jason is filing a ticket with Citrix about why the config unexpectedly changed back.

CiphersScan.sh marketplace.firefox.com:443
prio  ciphersuite  protocol  pfs_keysize
1     RC4-SHA      TLSv1.1
2     AES256-SHA   TLSv1.1
3     AES128-SHA   TLSv1.1
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED

    
    

    
  
I forgot to mention the revised ordering. After discussing it for a while, we decided to push RC4 way down, and prefer the ciphersuite below. Could you please update the MozillaDefault cipher accordingly? 

TLS1-DHE-DSS-AES-128-CBC-SHA
TLS1-DHE-RSA-AES-128-CBC-SHA
TLS1-DHE-DSS-AES-256-CBC-SHA
TLS1-DHE-RSA-AES-256-CBC-SHA
TLS1-AES-128-CBC-SHA
TLS1-AES-256-CBC-SHA
SSL3-RC4-SHA

Copy/Paste conf is at https://wiki.mozilla.org/Security/Server_Side_TLS#Citrix_Netscaler

My apologies for not mentioning it earlier...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
  
Blocks: 937555

    
    

    
  
The issue described in https://bugzilla.mozilla.org/show_bug.cgi?id=937555 is really serious.

:oremj, did the netscaler get upgraded? I see that TLS1.2 is now negotiable, and that issue with TLS1.0 and AES/3DES wasn't there before (or we missed it).

Can we please review the configuration today? I'm available to help.
Flags: needinfo?(oremj)

    
    

    
  
Looks like the config reverted again.

$ ./CiphersScan.sh marketplace.mozilla.org:443
prio  ciphersuite   protocols                    pfs_keysize
1     RC4-MD5       SSLv3,TLSv1,TLSv1.1,TLSv1.2
2     RC4-SHA       SSLv3,TLSv1,TLSv1.1,TLSv1.2
3     DES-CBC3-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2
4     AES256-SHA    SSLv3,TLSv1,TLSv1.1,TLSv1.2
5     AES128-SHA    SSLv3,TLSv1,TLSv1.1,TLSv1.2

    
    

    
  
I re-applied the config.

jason@kenshin:~/src|⇒  ./CiphersScan.sh marketplace.firefox.com:443
....
prio  ciphersuite  protocols                    pfs_keysize
1     RC4-SHA      SSLv3,TLSv1,TLSv1.1,TLSv1.2
2     AES256-SHA   SSLv3,TLSv1,TLSv1.1,TLSv1.2
3     AES128-SHA   SSLv3,TLSv1,TLSv1.1,TLSv1.2

    
    

    
  
Could you also reconfigure the default ciphersuite to match the new ordering from https://wiki.mozilla.org/Security/Server_Side_TLS#Citrix_Netscaler ?

Thanks!

    
    

    
  
Ciphersuite reordered. Thanks a lot Jason !

$ ./CiphersScan.sh marketplace.firefox.com:443
......
prio  ciphersuite         protocols                    pfs_keysize
1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1          DH,1024bits
2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1          DH,1024bits
3     AES128-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
4     AES256-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
5     RC4-SHA             SSLv3,TLSv1,TLSv1.1,TLSv1.2
Flags: needinfo?(oremj)

    
    

    
  
There is a bug with the Netscaler where DHE ciphersuite and TLS1.2 fail to negociate. 

This doesn't work (tls1_2):

$ openssl s_client -connect marketplace.firefox.com:443 -cipher 'DHE-RSA-AES128-SHA' -tls1_2

This works (tls1_1):

$ openssl s_client -connect marketplace.firefox.com:443 -cipher 'DHE-RSA-AES128-SHA' -tls1_1

It only impacts the combination of TLS1.2 and DHE. I tried TLS1.2 with AES and it works fine:

$ openssl s_client -connect marketplace.firefox.com:443 -cipher 'AES128-SHA' -tls1_2

This is a vendor issue. Can we file a bug against Citrix for this?

    
    

    
  
Filed ticket #61166304 with Citrix.

    
    

    
  
Citrix has responded: "Issue ID 0345883: On the NetScaler appliance, TLS protocol version 1.2 does not support ephemeral Diffie-Hellman cipher suites." 

This issue affects both Netscaler 10.0 and 10.1 firmwares.

The current work around is to disable TLS 1.2 or not to use Diffie-Hellman cipher suites. I am still waiting on a response from Citrix if TLS 1.2 can be disabled on Netscaler 10.0.

    
    

    
  
Thanks for the update (citrix took 20 days to respond?... wow).
Indeed the preference would be to disable TLS1.2, keep TLS1.1, and enable DHE ciphers.

    
    

    
  
Reordering looks good.

$ ./cipherscan marketplace.firefox.com:443
......
prio  ciphersuite         protocols            pfs_keysize
1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1  DH,1024bits
2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1  DH,1024bits
3     AES128-SHA          SSLv3,TLSv1,TLSv1.1
4     AES256-SHA          SSLv3,TLSv1,TLSv1.1
5     RC4-SHA             SSLv3,TLSv1,TLSv1.1

No TLS1.2 (broken with DHE).

    
    

    
  
ECDHE support on netscaler devices are still limited to specific models[1]. As per comment 11 I believe the support will eventually make it to our model (MPX-11500). I will reopen this bug when it is available.

[1] http://support.citrix.com/proddocs/topic/ns-rn-main-release-10-1-map/ns-rn-enhancements-121-x-con.html
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
Resolution: --- → FIXED

    
    

    
  
Reverted change due to Bug 959534.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
Are you re-opening a bug with Citrix?

    
    

    
  
I have opened a new bug with Citrix #61194207. I will update once I have more information.

    
    

    
  
Any more info from citrix on this?

    
  
Priority: -- → P2

    
    

    
  
4 months have passed since the request with Citrix was opened. What's the status?
Flags: needinfo?(jthomas)

    
    

    
  
As per our meeting yesterday, we are going to look into alternative options (nginx, stm).
Flags: needinfo?(jthomas)

    
    

    
  
Corey or Jake: would you be willing to donate the experimental ZLB hardware that we were planning to use to test the migration to 9.5? see comment https://bugzilla.mozilla.org/show_bug.cgi?id=959666#c7

AFAIK, these two servers are sitting idle at the moment, but they could do some good in helping us improve marketplace and AMO.
Flags: needinfo?(nmaul)
Flags: needinfo?(cshields)

    
  
Summary: Reorder ciphersuites on https://marketplace.mozilla.org → [mkt] Reorder ciphersuites on https://marketplace.mozilla.org

    
    

    
  
+1
Flags: needinfo?(cshields)

    
    

    
  
if they're sitting idle, i see no reason why they couldn't be donated to amo :)

    
    

    
  
WebOps has no plans for them that I'm aware of, so that's fine with me too. :)
Flags: needinfo?(nmaul)

    
  
Depends on: 1024016

    
  
Depends on: 1038369

    
  
Group: mozilla-employee-confidential
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
Version: other → unspecified

    
    

    
  
Looks like the latest release fixed the issue. The ciphers are now properly ordered, thanks to :jason.

$ ./cipherscan marketplace.firefox.com
......
prio  ciphersuite         protocols                    pfs_keysize
1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
3     AES128-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
4     AES256-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
5     DES-CBC3-SHA        SSLv3,TLSv1,TLSv1.1,TLSv1.2

Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED

    
    

    
  
I'm reopening this to update the ciphersuite to the intermediate level. The current level matches the old configuration from https://wiki.mozilla.org/Security/Server_Side_TLS.

    $ ./cipherscan marketplace.firefox.com
    ......
    prio  ciphersuite         protocols                    pfs_keysize
    1     DHE-RSA-AES128-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
    2     DHE-RSA-AES256-SHA  SSLv3,TLSv1,TLSv1.1,TLSv1.2  DH,1024bits
    3     AES128-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
    4     AES256-SHA          SSLv3,TLSv1,TLSv1.1,TLSv1.2
    5     DES-CBC3-SHA        SSLv3,TLSv1,TLSv1.1,TLSv1.2

    Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature
    TLS ticket lifetime hint: None
    OCSP stapling: not supported
    Server side cipher ordering

Changes needed to match the intermediate level:
* remove cipher DES-CBC3-SHA
* disable SSLv3
* consider enabling OCSP Stapling
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
We should definitely move away from 1024-bit DH as well.  It might not be critical in this case, because we care mostly about authentication and integrity on these sites.  A brute force might be feasible, but it takes time, so it only compromises confidentiality.  That's definitely a problem if we have long-lived cookies or other secrets (basic/digest authentication, perhaps).

Safest thing to do is upgrade.  For addons and marketplace, the only reason we might not use the best possible profile is limited availability of capable hardware.

    
    

    
  
see also bug 949564 for the client-side complement to this.

    
    

    
  
marketplace.firefox.com has moved to AWS and now fronted by ELB using security policy "ELBSecurityPolicy-2015-05"

./cipherscan marketplace.firefox.com:443
.....................
Target: marketplace.firefox.com:443

prio  ciphersuite                  protocols              pfs                 curves
1     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
2     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
3     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
4     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
5     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
6     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
7     AES128-GCM-SHA256            TLSv1.2                None                None
8     AES128-SHA256                TLSv1.2                None                None
9     AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
10    AES256-GCM-SHA384            TLSv1.2                None                None
11    AES256-SHA256                TLSv1.2                None                None
12    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
13    DES-CBC3-SHA                 TLSv1,TLSv1.1,TLSv1.2  None                None

Certificate: trusted, 2048 bits, sha256WithRSAEncryption signature
TLS ticket lifetime hint: 300
OCSP stapling: not supported
Cipher ordering: server
Curves ordering: server - fallback: no
TLS Tolerance: yes
Status: REOPENED → RESOLVED
Closed: 10 years ago → 9 years ago
Resolution: --- → FIXED

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: