591523 - Disable TLS false start by default

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[Wan-Teh Chang]

In Chromium issue 50650 (http://crbug.com/50650) we discovered
a TLS false start incompatible server.

We should maintain a blacklist of TLS false start incompatible
sites and disable the SSL_ENABLE_FALSE_START option when
connecting to the sites on the blacklist.

Comment 1

[Robert Sayre]

I don't think we should maintain a blacklist unless we have evidence that such an approach will work with a large user base.

Comment 2

[Daniel C]

Is there a specific server setup (software/version/config) that causes it to fail?

Comment 3

[Kai Engert (:KaiE:)]

(In reply to comment #0)
> 
> We should maintain a blacklist of TLS false start incompatible
> sites

I disagree.

Blacklists embedded in software don't work.

New broken sites appear, problem sites get fixed.
One always play catch-up with blacklists.

In my opinion, if the protocol code is unable to detect incompatible server and work around it, then the false start code is not yet ready.

Comment 4

[Kai Engert (:KaiE:)]

I agree we need to find a solution for TLS false-start incompatible sites, therefore it makes sense to have this bug blocking2.0

There a multiple possible solutions:
- add a blacklist (and we two votes against the approach)
- automatic workaround / fallback
- disable the new false-start feature before FF 4 final release

I'm updating the summary, to invite us to broaden the horizon when searching for a solution.

I understand one reported problem site is
  http://www.picnik.com/app

Are we able to reproduce the failure with FF 4 beta on Windows?

Comment 5

[Robert Sayre]

(In reply to comment #3)
>
> I disagree.
> 
> Blacklists embedded in software don't work.

Ideally, we would measure the effectiveness of the blacklist. If something as popular as http://www.picnik.com is broken with TLS False Start, there are probably more sites, but perhaps not.

Comment 2 hints at the question we need to answer.

Comment 6

[Wan-Teh Chang]

Attachment: Sample ssl_false_start_blacklist_data.cc

On Monday we discovered the second SSL false start incompatible
site: https://moneycenter.yodlee.com (see http://crbug.com/53690).
The site is using the same SSL accelerator that www.picnik.com uses.

We decided to disable SSL false start in Google Chrome 6.0 (but
leave it enabled on the Chromium trunk), talk to the SSL
accelerator manufacturer, and start probing the top 1 million
sites to assess the extent of the problem.

The prober found several hundred false start incompatible sites.
So Adam Langley wrote a program to precompile the blacklist into
a hash table: http://codereview.chromium.org/3303008/show
I attached a sample precompiled ssl_false_start_blacklist_data.cc
for your reference.

In addition to the blacklist, we want to eliminate the dependency
of false start failures on TCP packet arrival times, so that failures
will be deterministic.  Adam Langley wrote a patch for Chromium:
http://codereview.chromium.org/3331005/show
I will look into a patch for NSS:
https://bugzilla.mozilla.org/show_bug.cgi?id=589047

So this is what we plan to do in Chromium.  Please let me know
if you want me to integrate the blacklist with Mozilla, or
disable SSL false start by default.

Note: It is hard for a browser to detect false start incompatible
server.  The most common failure mode is that the server drops the
application data sent by the client and waits for the client to send
the application data in vain, resulting in a hang.  If the browser
uses a timeout to detect an incompatible server, it runs the risk
of sending a duplicate request to a slow server.

Comment 7

[Kai Engert (:KaiE:)]

(In reply to comment #6)
> 
> ... We decided to disable SSL false start in Google Chrome 6.0 (but
> leave it enabled on the Chromium trunk) ...
> 
> ... Please let me know
> if you want me to integrate the blacklist with Mozilla, or
> disable SSL false start by default.

My preference is to disable false start for now, and find a better solution than the blacklist.

My tendency is to veto a blacklist of sites embedded in software.

A minor speedup on initial connection does not justify a step backwards in compatibility that would require a maintenance nightmare to keep it usable.

Comment 8

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

IMO, False Start without explicit server-side opt-in should be treated very similar to the TLS secure renegotiation compatibility mode--a temporary hack used while waiting for server implementations to catch up. The most reliable solution will be to work with the OpenSSL team and others (probably through the IETF TLS WG) to create a way for the server to explicitly opt-in to False Start (and Snap Start) in a secure way. I think it would be easy to get support for such an extension into OpenSSL and I imagine TLS Accelerator manufacturers will be forced to implement it as well.

Such an explicit server opt in would allow clients like Chrome and Firefox to respect servers' security policies, which is not possible with the current False Start and proposed Snap Start mechanisms, as I briefly touched on here in my message "Limited rollback attacks in False Start and Snap Start" on the tls@ietf mailing list: 
http://www.ietf.org/mail-archive/web/tls/current/msg06933.html

Comment 10

[Jason Duell]

So this is marked beta7--what's our plan for that?   It seems like the only realistic 'fix' in the new day or two is to disable false-start in the FF4 beta 7.  Correct?  If so, we need to land that ASAP.

If I read the comments correctly, a possibly viable strategy after that is to 1) use a blacklist like chrome's until 2) we get some sort of server opt-in working, per comment 8.  (Or possibly just #2 if blacklists are indeed the devil).

Comment 11

[Wan-Teh Chang]

A blacklist is also a realistic fix.  Do you want to use a
blacklist in FF4?  If not, I can change the default value of
the security.ssl.enable_false_start preference to 'false' in
mozilla/netwerk/base/public/security-prefs.js.

Comment 12

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

How often do you plan to update the chromium blacklist?  Would there be a feed we could grab from?

Comment 13

[Jason Duell]

WTC:

The decision isn't mine to make, other than that we presumably need to do something ASAP for beta 7 (I'm assuming the ostrich algorithm--just letting things fail for the 100 or so sites on the web--is not an option for the beta).

How quickly do you think you could get a blacklist working?  Do you already have a patch and a working list?  Barring that, I think given the time/risk tradeoff, we should probably just turn off false-start for beta 7, and we can see if we can get a blacklist (or other solution) working by FF 4's release.

Kai:  what do you think?

Comment 14

[Johnathan Nightingale [:johnath]]

Talked with Sayre about this today - his assertion was that we should just turn it off at this point for FF4 - that it's interesting, and was worth trying out, but without a good solution for compat, we should punt. I have a hard time disagreeing.

Comment 15

[Wan-Teh Chang]

Attachment: Disable SSL false start by default

I forgot that it is also important to fix NSS bug 589047 or
make an equivalent change to Mozilla so that false start
failures are deterministic.

Since I can't write that patch quickly, it'd be prudent to
disable false start.

Re: blacklist updates: we will update it whenever we receive
a bug report.

Comment 16

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I think it should be disabled until a safe solution can be found (if there is one).

I commented in bug 589047 but I will repeat the short version here: while implementing the suggestion in bug 589047 is helpful, it will not make of False Start failures deterministic, only much more likely.

Really, this problem is very similar to the problem with bad HTTP proxies causing that break pipelining, bad HTTP proxies are just more common than bad TLS implementations.

A blacklist would be a more attractive if we knew that all vendors of the incompatible hardware/software had already fixed the incompatibility on their end. Otherwise, they will be actively selling their software/devices to more sites indefinitely. But, right now it isn't even clear that they are aware of the issue or that they even want to support False Start.

I am also concerned about compatibility issues with things like TLS intercepting proxies that would cause similar incompatibility for all sites that the "protected" clients are attempting to access over TLS. I think it is too risky to enable False Start by default until we are confident that these types of products do not cause any problem.

Comment 17

[Adam Langley]

(In reply to comment #16)
> A blacklist would be a more attractive if we knew that all vendors of the
> incompatible hardware/software had already fixed the incompatibility on their
> end. Otherwise, they will be actively selling their software/devices to more
> sites indefinitely. But, right now it isn't even clear that they are aware of
> the issue or that they even want to support False Start.
> 
> I am also concerned about compatibility issues with things like TLS
> intercepting proxies that would cause similar incompatibility for all sites
> that the "protected" clients are attempting to access over TLS. I think it is
> too risky to enable False Start by default until we are confident that these
> types of products do not cause any problem.

We believe that we have found the sole vendor who's products are causing issues. They are working on a fix. While they do so, Chrome 7 will test the waters for other incompatible sites.

Afterwards, Firefox can follow in the path that we clear.

Comment 18

[Wan-Teh Chang]

Patch pushed to mozilla-central for FF 4.0b7:
http://hg.mozilla.org/mozilla-central/rev/8b252736a650

Comment 19

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

The bug to enable False Start (again) is bug 658222.