Closed Bug 904077 Opened 11 years ago Closed 10 years ago

Review ciphersuite on login.persona.org

Categories

(Cloud Services :: Server: Identity, defect)

Product:
Cloud Services
Component:
Server: Identity
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: jvehent, Assigned: gene)

References

Details

The ciphersuite returned by login.persona.org prefers ECDHE-RSA-AES256-SHA (which is great), but also supports unwanted ciphers (Camellia, etc...). TLSv1.2 is also not supported. Can we review the ciphersuite and make it match OpSec recommendation? https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456 This might not be possible if SSL is terminated on ELBs. But should be easy if Nginx or Apache are terminating it. $ ./CiphersScan.sh login.persona.org:443 prio ciphersuite protocol 1 ECDHE-RSA-AES256-SHA TLSv1 2 DHE-RSA-AES256-SHA TLSv1 3 DHE-RSA-CAMELLIA256-SHA TLSv1 4 AES256-SHA TLSv1 5 CAMELLIA256-SHA TLSv1 6 ECDHE-RSA-DES-CBC3-SHA TLSv1 7 EDH-RSA-DES-CBC3-SHA TLSv1 8 DES-CBC3-SHA TLSv1 9 ECDHE-RSA-AES128-SHA TLSv1 10 DHE-RSA-AES128-SHA TLSv1 11 DHE-RSA-CAMELLIA128-SHA TLSv1 12 AES128-SHA TLSv1 13 CAMELLIA128-SHA TLSv1 14 ECDHE-RSA-RC4-SHA TLSv1 15 RC4-SHA TLSv1 16 (NONE)
login.persona.org is terminated on ELB. The persona.org and browserid.org redirects to login.persona.org, as well as the old browserid.org/verify are on Apache.
That's interesting, because AWS ELB doc makes no mention of ECDHE algorithms! If they do indeed support ECDHE, we need to update our recommendation. Would it be possible to get a copy of the ELB conf? API dump is fine. Does the API/UI have an option to reorder the ciphers, so we can sort them by preference? (It didn't in my last tests)
Group: mozilla-corporation-confidential
Here's what's possible : SSL Protocols Protocol-SSLv2 Protocol-SSLv3 Protocol-TLSv1 SSL Ciphers ADH-AES128-SHA ADH-AES256-SHA ADH-CAMELLIA128-SHA ADH-CAMELLIA256-SHA ADH-DES-CBC-SHA ADH-DES-CBC3-SHA ADH-RC4-MD5 ADH-SEED-SHA AES128-SHA AES256-SHA CAMELLIA128-SHA CAMELLIA256-SHA DES-CBC-MD5 DES-CBC-SHA DES-CBC3-MD5 DES-CBC3-SHA DHE-DSS-AES128-SHA DHE-DSS-AES256-SHA DHE-DSS-CAMELLIA128-SHA DHE-DSS-CAMELLIA256-SHA DHE-DSS-SEED-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-CAMELLIA128-SHA DHE-RSA-CAMELLIA256-SHA DHE-RSA-SEED-SHA EDH-DSS-DES-CBC-SHA EDH-DSS-DES-CBC3-SHA EDH-RSA-DES-CBC-SHA EDH-RSA-DES-CBC3-SHA EXP-ADH-DES-CBC-SHA EXP-ADH-RC4-MD5 EXP-DES-CBC-SHA EXP-EDH-DSS-DES-CBC-SHA EXP-EDH-RSA-DES-CBC-SHA EXP-KRB5-DES-CBC-MD5 EXP-KRB5-DES-CBC-SHA EXP-KRB5-RC2-CBC-MD5 EXP-KRB5-RC2-CBC-SHA EXP-KRB5-RC4-MD5 EXP-KRB5-RC4-SHA EXP-RC2-CBC-MD5 EXP-RC4-MD5 IDEA-CBC-SHA KRB5-DES-CBC-MD5 KRB5-DES-CBC-SHA KRB5-DES-CBC3-MD5 KRB5-DES-CBC3-SHA KRB5-RC4-MD5 KRB5-RC4-SHA PSK-3DES-EDE-CBC-SHA PSK-AES128-CBC-SHA PSK-AES256-CBC-SHA PSK-RC4-SHA RC2-CBC-MD5 RC4-MD5 RC4-SHA SEED-SHA Here's the default's which we have enabled SSL Protocols Protocol-SSLv3 Protocol-TLSv1 SSL Ciphers AES128-SHA AES256-SHA DES-CBC3-SHA RC4-MD5 RC4-SHA There isn't an option for ordering the ciphers
(In reply to Gene Wood [:gene] from comment #3) > SSL Ciphers > AES128-SHA > AES256-SHA > DES-CBC3-SHA > RC4-MD5 > RC4-SHA > > There isn't an option for ordering the ciphers Your conf matches what I saw in the doc, and it makes sense (aside from DES-CBC3-SHA that's deprecated). But that's not what I see in my tests. The ELB negociate successfully the following ciphers: OK AES128-SHA OK AES256-SHA OK CAMELLIA128-SHA OK CAMELLIA256-SHA OK DES-CBC3-SHA OK DHE-RSA-AES128-SHA OK DHE-RSA-AES256-SHA OK DHE-RSA-CAMELLIA128-SHA OK DHE-RSA-CAMELLIA256-SHA OK ECDHE-RSA-AES128-SHA OK ECDHE-RSA-AES256-SHA OK ECDHE-RSA-DES-CBC3-SHA OK ECDHE-RSA-RC4-SHA OK EDH-RSA-DES-CBC3-SHA OK RC4-SHA You can test it yourself with 'openssl s_client -connect login.persona.org:443 -cipher ECDH' (with a recent openssl only). So, aside from the fact that the ELB doesn't let us order the ciphers, it seems that the configuration options aren't consistent with what runs in production. For discussion's sake: how are would it be to use the ELBs as L4 load balancer, and terminate the SSL on Nginx (or Apache, ...) ?
Assignee: nobody → gene
Blocks: tls-everything
Note : mmayo suggest this as a possible SSL terminator : https://github.com/bumptech/stud
Stud is a valid option. I'm happy with pretty much anything that has a good track record (apache, nginx, stud, stunnel, ...) and uses OpenSSL. Bonus points if we can use a recent version of OpenSSL and enable AES-GCM.
So I looked into Stud a little bit closer. It works as advertised, and I managed to build it with the latest openssl version rather easily. https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456#SSL%26TLSCiphersuites-Stud But it doesn't support OCSP stapling, and judging but the backlog of issues on the github repository, isn't actively developed. If OCSP stapling is wanted, then Nginx and Apache are the only SSL servers that support it.
I've found that this is a bug in AWS that we can workaround for the time being. I've copied you on the bug report to AWS (Case 108267411) > aside from DES-CBC3-SHA that's deprecated Would you like me to disable this cipher? > aside from the fact that the ELB doesn't let us order the ciphers How significant of a security impact is it if we are unable to order the ciphers (assuming we can constrian what ciphers are available for use in negotiation)? > For discussion's sake: how are would it be to use the ELBs as L4 load balancer, and terminate the SSL on Nginx (or Apache, ...) ? This would be non-trivial so let's explore your questions with the ELBs first and move onto a more drastic solution like this if we reach an impass.
I'm developing code now to create a policy that matches what you mention in the mana page linked in the description and to bind it to our listeners when they're provisioned.
(In reply to Gene Wood [:gene] from comment #8) > I've found that this is a bug in AWS that we can workaround for the time > being. I've copied you on the bug report to AWS (Case 108267411) > Sounds good. AWS default conf probably doesn't enforce any ciphersuite, and that's why ECDH shows up until you actually force one. Too bad that this cipher is gone now. > > aside from DES-CBC3-SHA that's deprecated > > Would you like me to disable this cipher? > Yes please, and RC4-MD5 as well. > > aside from the fact that the ELB doesn't let us order the ciphers > > How significant of a security impact is it if we are unable to order the > ciphers (assuming we can constrian what ciphers are available for use in > negotiation)? > At that point, ordering isn't even going to help us, because our choices are limited to AES in CBC mode (vulnerable to BEAST), or RC4 (recently weakened). The priority is to enable more, better, ciphers. But this isn't a "we're all gonna die" kind of issue. It's "opsec wishlist". Persona is a critical service that provide state of the art security, and it would be nice to serve it via state of the art TLS. > > For discussion's sake: how are would it be to use the ELBs as L4 load balancer, and terminate the SSL on Nginx (or Apache, ...) ? > > This would be non-trivial so let's explore your questions with the ELBs > first and move onto a more drastic solution like this if we reach an impass. Agreed. ELBs would be the preferred solution. Firefox will change it's TLS preferences soon, so it would be nice to provide the same level of security: https://groups.google.com/forum/#!topic/mozilla.dev.tech.crypto/gFfKw3EOffo
Julien, I'm having trouble translating the list of recommended cipher's here : https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456#SSL%26TLSCiphersuites-Generalpurposeciphersuite into the right list of checked ciphers in the ELB in Comment 3 Can you provide the subset of the list in Comment 3 that I should check and enable?
This should help: https://mana.mozilla.org/wiki/download/attachments/35069456/elbssl1.png?version=1&modificationDate=1374251764477&api=v2 SSL Protocols: SSLv3, TLSv1 AES128-SHA AES256-SHA CAMELLIA128-SHA CAMELLIA256-SHA DHE-DSS-AES128-SHA DHE-DSS-AES256-SHA DHE-DSS-CAMELLIA128-SHA DHE-DSS-CAMELLIA256-SHA RC4-SHA
Ahh, I saw those images in the doc but didn't realize they indicated the prefered config because the text above it says "For future reference, below is a list of ciphers supported by the ELBs." you may want to change that to something indicating that it's a visual representation of the prefered config, not just a screenshot of the "ciphers supported by the ELBs"
Good point. I'll update the doc. Thanks for the feedback!
Julien, I've created a tool to build a policy based of your doc and apply it to an ELB https://github.com/mozilla/identity-ops/blob/master/aws-tools/apply_security_assurance_elb_ciphersuite_policy.py You're welcome to refernce or steal it for your mana page. I'll add this into our provisioning logic so all future stacks have this applie to all ELBs. I'll need to work with QA to change these ciphers and confirm that they work on all browsers.
Love it! I added it to the doc.
Julien, The note on the mana page that says " As of July 2013, ELBs SSL support is considered UNSAFE and should not be used to serve production traffic. " do you mean that ELBs in their default configuaration are UNSAFE but if you change the Ciphersuite to what you've indicated they are SAFE? Or do you mean their still UNSAFE even with the Ciphersuite change?
They are still unsafe even with the ciphersuite change. ELBs provide the bare minimum as far as SSL goes. SSLLabs has an automated test that lists the issues rather well: https://www.ssllabs.com/ssltest/analyze.html?d=login.persona.org&s=54.244.1.203 ELBs are dragging being on the SSL front. To be truly current, AWS needs to: 1. add TLS1.1 and TLS1.2, including AES-GCM ciphersuites 2. add Ciphersuites ordering, to make sure the strongest ciphers are preferred when the client supports many of them 3. remove client-side renegociation 4. add Elliptic Curves (ECC, ECDHE) 5. add OCSP Stapling If AWS fixes 1/2/3, we could consider it safe.
OpSec will talk to AWS about ELB SSL. We'll try to obtain a resolution timeline for the issues listed above.
You'll now be able to see the new ciphersuite in production. Feel free to re-open this ticket if there's an issue.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Julien: any reason why opsec recommends keeping SSLv3 turned ON? (We only need to support IE >= 8.)
Flags: needinfo?(jvehent)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
This is true that Persona is different from the other services we expose, and we could potentially use a stricter ciphersuite. But the ELBs are not good enough at SSL to support that right now. We will want to revisit this when AWS updates their ELBs.
Flags: needinfo?(jvehent)
Ok, let's reopen the bug when AWS is updated (or if we decide to stop terminating TLS on ELB).
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Comments above say that DHE-RSA-AES128-SHA is available. Considering that BEAST has been mitigated on the client side, it would make more sense to prefer forward secrecy than ciphersuites that are immune to BEAST. Or does ELB not let you configure enough bits for the DH parameters for DHE? As for performance, surely for a service of Persona's nature, the perf hit of DHE should be accepted. Considering that Persona is a privacy-sensitive service, it's pretty bad to host it on infrastructure that limits the pro-privacy measures that can be taken. It seems to me that using ciphersuites with forward secrecy is even more important when the server private key is held by Amazon and, therefore, Mozilla alone can't protect it from disclosure.
This is interesting. When I looked at the ELB SSL capabilities [1], DHE-RSA ciphersuites weren't available. But :gene does mention it in comment 3, which I missed. :gene, how did you obtain the list in comment 3? Is this from an API call? As far as prioritizing DHE, this is unfortunately not possible. ELBs limit the configuration options that are exposed through their API, and prioritizing ciphers isn't possible. See comment 18 for a full wishlist. Before we move away from ELBs entirely, I want to work with AWS support and see where these issue stand on their roadmap. We already have evidence that ELBs support ECC ciphersuites, but are not exposed for configuration. So let's talk to them before rearchitecting everything. That being said, if DHE is available, then we should enable it. Now there is another debate on preferring AES-CBC over RC4. DHE-RSA-AES128-SHA uses AES in CBC mode and is therefore exposed to BEAST until TLS1.1, which applies here because ELBs talk TLS1. I don't know which of the two evil is preferable, and neither does the crypto community from what I've seen. But SSLLabs will cap the grade to B if the preferred ciphersuite is vulnerable to Beast, and that's one reason to put RC4 before it. (yes, people do look at SSLlabs to evaluate the quality of our SSL, for better or worse). I'll be happy to discuss it (in a separate bug) if you think the ciphersuite should be changed. [1] https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=35069456#SSL%26TLSCiphersuites-ElasticLoadBalancer
Flags: needinfo?(gene)
> Now there is another debate on preferring AES-CBC over RC4. > DHE-RSA-AES128-SHA uses AES in CBC mode and is therefore exposed to BEAST > until TLS1.1, which applies here because ELBs talk TLS1. I don't know which > of the two evil is preferable, and neither does the crypto community from > what I've seen. But SSLLabs will cap the grade to B if the preferred > ciphersuite is vulnerable to Beast, and that's one reason to put RC4 before > it. (yes, people do look at SSLlabs to evaluate the quality of our SSL, for > better or worse). > Well, looks like SSLLabs just changed its rating, so the above comment doesn't necessarily apply anymore. "Yesterday I changed the SSL Labs rating criteria to stop penalizing sites that do not implement server-side mitigations for the BEAST attack. That means that we now consider this attack sufficiently mitigated client-side, but, there are still some things you should now." http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html However, the issue at the core of Beast, predictable IVs, is inherent to TLS1 and cannot be fixed client side. It's a protocol issue. What changed since the first announcement of Beast is the way browsers deal with Same-Origin Policy. From a protocol standpoint, when TLSv1 is used, AES-CBC still uses weak IVs.
> :gene, how did you obtain the list in comment 3? Is this from an API call? That was a copy paste from the Web UI in the amazon console > That being said, if DHE is available, then we should enable it. Let me know specifically which ciphers you'd like enabled or disabled.
Flags: needinfo?(jvehent)
Case opened with AWS: https://aws.amazon.com/support/case?caseId=112269541
Flags: needinfo?(gene)
:gene, DHE-RSA should be enabled. I submitted a pull request on your script https://github.com/mozilla/identity-ops/pull/129 Is there a way to extract the list of supported ciphers programmatically? On the ELB API maybe?
Flags: needinfo?(jvehent)
> Is there a way to extract the list of supported ciphers programmatically? On the ELB API maybe? I've looked at results from the "DescribeLoadBalancerPolicies" API call http://docs.aws.amazon.com/ElasticLoadBalancing/latest/APIReference/API_DescribeLoadBalancerPolicies.html And the best I can do is create a test ELB and set it to use a default cipher policy, then query the policy to list the ciphers. Are you looking to establish a monitor that looks when the list of supported ciphers changes? > :gene, DHE-RSA should be enabled. I submitted a pull request on your script https://github.com/mozilla/identity-ops/pull/129 Looks good. I requested one change then I can merge it.
Julien, thanks for the PR. Is it ok to wait for the next stage train to go out to see these changes in the live ELBs?
Yes, it's fine. Since we can't sort the cipher, I'm not even sure they will be in use anyway. But I'm curious about the result!
Enable TLS 1.2 and related ciphersuites: https://github.com/mozilla/identity-ops/pull/132 For testing in staging, and eventually push to production.
Status: RESOLVED → REOPENED
Flags: needinfo?(gene)
Resolution: FIXED → ---
Thanks for merging the pull request. Let me know when you get a chance to deploy the new conf, so I can test it afterward.
Group: mozilla-corporation-confidential
(In reply to Julien Vehent [:ulfr] from comment #10) > (In reply to Gene Wood [:gene] from comment #8) > > How significant of a security impact is it if we are unable to order the > > ciphers (assuming we can constrian what ciphers are available for use in > > negotiation)? > > > > At that point, ordering isn't even going to help us, because our choices are > limited to AES in CBC mode (vulnerable to BEAST), or RC4 (recently > weakened). The priority is to enable more, better, ciphers. > But this isn't a "we're all gonna die" kind of issue. It's "opsec wishlist". > Persona is a critical service that provide state of the art security, and it > would be nice to serve it via state of the art TLS. This server (and any server that serves sensitive information) should always prefer the ECDHE cipher suites whenever they are offered and more generally it should prefer DHE cipher suites over RSA key exchange cipher suites since private key compromise is a serious concern and ECDHE and DHE miminize the damage of private key compromise when implemented correctly. Also, CBC-mode ciphers are only vulnerable to BEAST with TLS 1.0 or earlier versions. There are other CBC-mode attacks like Lucky 13 though. So, generally, I do think the ordering is important, though I don't think the ordering that servers pick needs to match exactly the ordering I suggested for Firefox and other browsers in my proposal. My proposal is for clients, not servers.
Brian: ordering is still not available in AWS ELBs, but the latest release of ELBs enables TLS1.2, GCM and DHE ciphers. https://github.com/mozilla/identity-ops/pull/132/files I'm waiting for :gene to push this live. As far as ordering, the opsec recommendation at https://wiki.mozilla.org/Security/Server_Side_TLS is partly based on your proposal. ECDHE and DHE ciphers are at the top. If you would like to review it, please post your comments on the tracker bug at https://bugzilla.mozilla.org/show_bug.cgi?id=901393
Any update on pushing the ciphersuite reordering live? The current setup only support TLSv1. $ ./cipherscan login.persona.org:443 ........ prio ciphersuite protocols pfs_keysize 1 DHE-RSA-AES256-SHA SSLv3,TLSv1 DH,1024bits 2 AES256-SHA SSLv3,TLSv1 3 CAMELLIA256-SHA SSLv3,TLSv1 4 DHE-RSA-AES128-SHA SSLv3,TLSv1 DH,1024bits 5 AES128-SHA SSLv3,TLSv1 6 CAMELLIA128-SHA SSLv3,TLSv1 7 RC4-SHA SSLv3,TLSv1
Ya, I continue to be blocked by an AWS bug (same one I've been dealing with for the last few months). Best I was able do was get the new cipher suites up in 1 region https://bugzilla.mozilla.org/show_bug.cgi?id=958719#c2 I'm going to try to find some time to put together code to reproduce the problem that I can submit to amazon in a support ticket. But at least later today half of our users will get the new cipher suite
Can we move these ELBs under the new SSL policy from AWS ? Their ciphersuite would be better than the current one. For the record: $ ./cipherscan login.persona.org:443 ...... prio ciphersuite protocols pfs_keysize 1 AES256-SHA SSLv3,TLSv1 2 CAMELLIA256-SHA SSLv3,TLSv1 3 AES128-SHA SSLv3,TLSv1 4 CAMELLIA128-SHA SSLv3,TLSv1 5 RC4-SHA SSLv3,TLSv1 Certificate: trusted, 2048 bit, sha256WithRSAEncryption signature
Now tracked in bug 1077566
Status: REOPENED → RESOLVED
Closed: 11 years ago → 10 years ago
Flags: needinfo?(gene)
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.