Closed Bug 1475432 Opened 6 years ago Closed 2 years ago

Unappropriate TLS config on accounts.firefox.com

Categories

(Cloud Services :: Server: Firefox Accounts, task)

Product:
Cloud Services
Component:
Server: Firefox Accounts
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: jan, Unassigned)

References

Details

(Keywords: nightly-community, Whiteboard: [fxa-waffle-ignore]) 

https://www.hardenize.com/report/accounts.firefox.com/1531444991#www_tls
Compare with: https://www.hardenize.com/report/accounts.google.com/1531446246#www_tls

Which was the last Firefox version that only supported DHE 1024 bit as maximum, and which commonly used browser still connects via TLSv1 with a non-PFS cipher? Should corpses really login?

Please
* use DHE 2048 bit, if you need to support it at all
* drop non-PFS ciphersuites
* prefer PFS AEAD over PFS AES CBC SHA2

Thanks ;)
I feel like we already have a bug about this config, but I can't seem to find it; :g-k could you please direct this to the right person on your team for further comment?
Flags: needinfo?(gguthe)
See Also: → 1219547
(In reply to Ryan Kelly [:rfkelly] from comment #1)
> I feel like we already have a bug about this config, but I can't seem to
> find it; :g-k could you please direct this to the right person on your team
> for further comment?

Normally I'd redirect to :ulfr, but he's on parental leave so I'll leave the NI and check on IRC when people are awake in the US.

Bug 1219547 says we're using the intermediate compatibility TLS config to support older android devices and browsers and that ELBv1 didn't let us set the DH params.

However, https://forums.aws.amazon.com/ann.jspa?annID=3061 suggests that we can switch from AWS-ELBSecurityPolicy-2015-03 to AWS-ELBSecurityPolicy-2015-05 to get 2048 bit DHE params. Switching to a custom security policy or ELBv2/ALBs might let us set the ciphersuites.

rfk, do you know if we still need to support the older devices? Bug 1219547 is three years old at this point. Ideally we could switch right to a modern TLS config.
>  do you know if we still need to support the older devices?

I recall someone did a breakdown to see what % of traffic they were - :jbuck was that you by any chance?  We should repeat that analysis and see what the current state of affairs is.
Flags: needinfo?(jbuckley)
(In reply to Ryan Kelly [:rfkelly] from comment #3)
> >  do you know if we still need to support the older devices?
> 
> I recall someone did a breakdown to see what % of traffic they were - :jbuck
> was that you by any chance?  We should repeat that analysis and see what the
> current state of affairs is.

Samsung S-Browser prevented us from moving forward with newer ciphers. We should
check with Legal, there may be some contractual obligations here.
(In reply to Greg Guthe [:g-k] from comment #2)
>
> However, https://forums.aws.amazon.com/ann.jspa?annID=3061 suggests that we
> can switch from AWS-ELBSecurityPolicy-2015-03 to
> AWS-ELBSecurityPolicy-2015-05 to get 2048 bit DHE params. Switching to a
> custom security policy or ELBv2/ALBs might let us set the ciphersuites.

I misread that. 2015-05 drops DHE support.

:jbuck can you check with our AWS TAM or try redeploying the 2015-03 policy in dev or stage to see if that bumps us to the larger DHE params?
:darkspirit's recommendations look good to me, but I'm not an expert.

:keeler would you recommend using DHE 2048 bit if we have to support DHE, dropping non-PFS ciphersuites, and preferring PFS AEAD over PFS AES CBC SHA2 if possible?
Flags: needinfo?(gguthe) → needinfo?(dkeeler)
(In reply to Greg Guthe [:g-k] from comment #5)
> :jbuck can you check with our AWS TAM or try redeploying the 2015-03 policy
> in dev or stage to see if that bumps us to the larger DHE params?

:jbuck you can ignore this. I filed a support request with Case ID 5218010961. Still interested in the traffic analysis for https://bugzilla.mozilla.org/show_bug.cgi?id=1475432#c3 though
https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html
Amazon's preferred ciphersuite order is absurd.

comment 0 basically asks for something nearer to Mozilla Intermediate without messing everything up with even more legacy ciphersuites. https://wiki.mozilla.org/Security/Server_Side_TLS#Intermediate_compatibility_.28default.29

To visualize the request:
https://www.hardenize.com/report/accounts.firefox.com/1531444991#www_tls
vs.
TLSv1.2, TLSv1.1, TLSv1
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-SHA256
ECDHE-RSA-AES256-SHA384
ECDHE-RSA-AES128-SHA
ECDHE-RSA-AES256-SHA
DHE-RSA-AES128-SHA (DHE 2048 bit)

Non-PFS shouldn't be needed: DES-CBC3-SHA is the best IE on WinXP supports.
Firefox has supported DHE-RSA-AES128-SHA since 2003.
Amazon doesn't support Chacha and I even haven't changed the AES128 over 256 preference to not change CPU usage, despite I'd clearly prefer AES256.^^
TLS cipher suite & protocol analysis for fxa auth servers is viewable at https://docs.google.com/spreadsheets/d/1iLn1mh4QcQyQm96U6UIXx3Ty8gVhIgf2sKZcnTw6khI/edit?usp=sharing for Mozilla employees

I also did the User Agent analysis for the DHE-RSA-AES128-SHA cipher suite which is the reason we haven't upgraded ELB policies yet https://docs.google.com/spreadsheets/d/1PTbDdc_rHMV6rv7JGcrdWrrZtOZuijdmWtp_BiNLkI4/edit#gid=0
Flags: needinfo?(jbuckley)
(In reply to Greg Guthe [:g-k] from comment #6)
> :darkspirit's recommendations look good to me, but I'm not an expert.
> 
> :keeler would you recommend using DHE 2048 bit if we have to support DHE,
> dropping non-PFS ciphersuites, and preferring PFS AEAD over PFS AES CBC SHA2
> if possible?

In general that seems reasonable. I would refer to https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility if we don't need to have compatibility with older clients (although that might be a bit out of date - we could presumably enable some TLS 1.3 ciphersuites if we have TLS 1.3 on that server...)
Flags: needinfo?(dkeeler)
> Samsung S-Browser prevented us from moving forward with newer ciphers. 

For those with access to :jbuck's doc shared above, the following user-agents (at least) are SBrowser:

 * Apache-HttpClient/UNAVAILABLE (java 1.5)
 * Firefox AndroidSync 1.@MOZ_APP_VERSION@.0 (Firefox)
([:keeler] (use needinfo) from comment #11)
> In general that seems reasonable. I would refer to https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility if we don't need to have compatibility with older clients

I strongly support this. With PCI's TLS 1.0 deprecation, Art. 32 EU GDPR and https://datatracker.ietf.org/doc/draft-moriarty-tls-oldversions-diediedie/ in mind:

What percentage of "legitimate" requests still use TLS 1.0 + 1.1 and could you deprecate them for FxA?

https://website-archive.mozilla.org/www.mozilla.org/firefox_releasenotes/en-US/firefox/27.0/releasenotes/
Firefox supports TLS 1.2 with ECDHE-RSA-AES128-GCM-SHA256 since 2014 (Firefox 27, bug 861266 + bug 937789).
Note that it's already impossible to use Firefox 27 for regular browsing as there are too many sec_error_unknown_issuer  errors. Warranty has expired, I would say. :)

(Ryan Kelly [:rfkelly] from comment #12)
> > Samsung S-Browser prevented us from moving forward with newer ciphers. 
> 
> For those with access to :jbuck's doc shared above, the following user-agents (at least) are SBrowser:
>  * Apache-HttpClient/UNAVAILABLE (java 1.5)
>  * Firefox AndroidSync 1.@MOZ_APP_VERSION@.0 (Firefox)

If they only support DHE-RSA-AES128-SHA with DHE 1024 bits (are we talking about Android 2.3.7?) and if you can't deprecate FxA for them right away (a creepy contract which has become illegal?), then please at least try to deprecate non-PFS ciphersuites and fix ciphersuite order until you have green light:

TLSv1.2, TLSv1.1, TLSv1
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-SHA256
ECDHE-RSA-AES256-SHA384
ECDHE-RSA-AES128-SHA
ECDHE-RSA-AES256-SHA
DHE-RSA-AES128-SHA (DHE 1024 bit 🤮)
(In reply to Greg Guthe [:g-k] from comment #7)
> (In reply to Greg Guthe [:g-k] from comment #5)
> > :jbuck can you check with our AWS TAM or try redeploying the 2015-03 policy
> > in dev or stage to see if that bumps us to the larger DHE params?
> 
> :jbuck you can ignore this. I filed a support request with Case ID
> 5218010961. Still interested in the traffic analysis for
> https://bugzilla.mozilla.org/show_bug.cgi?id=1475432#c3 though

Heard back from AWS, apparently they rolled it back since it broke Java 6 clients:

> On May 28th and May 29th, it came to our attention that clients using Java 6 were unable to negotiate an SSL connection with a load balancer when traditional Diffie-Hellman ciphers were enabled and the key length was set to 2048 bits. Once we understood the problem, we immediately rolled back the staging of the new key length and began the process of rollback for load balancers that has received the update. 
> ... 
> We incorrectly believed that older version of Java would not attempt to negotiate using traditional Diffie-Hellman and would use another cipher, with the update in key length having no impact on existing clients. This was true for other Java versions but not Java 6. 
> ... 
> No solution was ever found, so the parameters remained as is.

They said they might be able to try a workaround, but couldn't promise anything.

Another option is terminating TLS in nginx ourselves like we do for Sync.
Heard back from AWS:

> The internal team were actively working on this case and also received a final update from the team. 
>
> At this time we will not be able to increase the key-size from 1024 for 2048 for DH ciphers. We recommend that customers migrate to ECDH ciphers as these are significantly more secure and perform better. 

So yeah.


Looks like our options are:

A. update to a newer predefined AWS ELB policy; drop support for older clients

B. create a custom ELB policy; set attributes for specific ciphers to enabled and the Server-Defined-Cipher-Order attribute to true; drop support for DH ciphers and older clients using them (like the example in https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/ssl-config-update.html)

C. terminate TLS ourselves like we do for Sync; update DH params and cipher order as discussed above


:jbuck :jrgm do you have a preference or an idea of how much bandwidth you'll have to work on this? C. would be the most work for ops but provide the most backwards compatibility for older clients. We could also try B. and see if setting a better cipher order reduces the traffic from ciphers we want to drop.
Flags: needinfo?(jrgm)
Flags: needinfo?(jbuckley)
Our current plan is:

1) Switch to a new ELB cipher policy that moves DHE-RSA-AES128-SHA to the bottom of the priority list, meaning as few clients as possible will use it
2) Remove the use of the DHE-RSA-AES128-SHA cipher on November 1st 2018 by switching to the 2016-08 ELB security policy
Flags: needinfo?(jrgm)
Flags: needinfo?(jbuckley)
Depends on: 1508390
Whiteboard: [fxa-waffle-ignore]
See Also: → 1537701
  1. Remove the use of the DHE-RSA-AES128-SHA cipher on November 1st 2018 by switching to the 2016-08 ELB security policy

Did we do this and can now close out this bug?

Re-running the hardenize report from Comment 0 seems to show a lot of green and no orange or red.

Flags: needinfo?(jbuckley)

Yep, this was done on 2019-03-26.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(jbuckley)
Resolution: --- → FIXED

https://www.hardenize.com/report/accounts.firefox.com#www_tls
Confirmed this step, you've deployed predefined policy 2016-08 and safely dropped support for DHE. Thank you!

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/ssl-config-update.html
Now please configure a custom policy for accounts.firefox.com that removes support for non-PFS ciphersuites (plain RSA):

SSL Protocols [x] TLS 1.0
              [x] TLS 1.1
              [x] TLS 1.2
SSL Options   [ ] (No server order preference, as AWS enforces the following HTTP2-unfriendly ciphersuite order:)
SSL Ciphers   [x] ECDHE-RSA-AES128-GCM-SHA256 (supported with TLS 1.2 since Firefox 27)
              [x] ECDHE-RSA-AES128-SHA
              [x] ECDHE-RSA-AES256-GCM-SHA384
              [x] ECDHE-RSA-AES256-SHA (i.e. supported with TLS 1.0 by Firefox 4.0 build 2010-12-12 and by Android 4.0.4)

Risk: Small. This could break clients who use (in)security software to scan their connections, and only if that software is old, buggy and only supports plain RSA. That is a security problem of its own, affecting other services as well, noticing is desired and users should be advised to uninstall or update such software.

Background of the general deprecation process:
https://blog.mozilla.org/security/2018/10/15/removing-old-versions-of-tls/
TLS 1.0 and TLS 1.1 will be disabled with Nightly 72 or 73 in 5-7 months and ride the trains.

Status: RESOLVED → REOPENED
Type: defect → task
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 5 years ago → 2 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.