989062 - Regression: TLS 1.2 connection failure / Cisco WebEx

Status: RESOLVED INVALID

(Core :: Networking, defect)

Description

[will69]

A connection cannot be established with a site offering TLS 1.2, in particular current Cisco WebEx, when using Firefox 28 and above. The error is:

"Peer could not decode an SSL handshake message.
(Error code: ssl_error_decode_error_alert)"

You can try any current Cisco WebEx site to reproduce.
Example: https://www.wbxmeeting.com
(Disclaimer: We are not associated with this site. The error actually occurs on our extranet.)

Connection was successful with Firefox versions 24 to 27.

Here is a report for your convenience:
https://www.ssllabs.com/ssltest/analyze.html?d=www.wbxmeeting.com

Note that TLS 1.2 and TLS 1.0 are enabled, while TLS 1.1 is not.

Workaround (Caution: disables TLS 1.2 for all sites):
security.tls.version.max=2

Connection can be established successfully with Chrome 33, Opera 12, and IE.

Comment 1

[Sylvestre Ledru [:Sylvestre]]

Brian, does it ring a bell? (sorry if you are not the right person to contact)

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #1)
> Brian, does it ring a bell? (sorry if you are not the right person to
> contact)

I can reproduce it.

Chrome loads the page correctly. My guess was that Chrome *is* affected but that their looser TLS intolerance fallback logic caused them to fall back to an earlier version. However, Chrome's UI normally indicates when non-secure fallback was used, and the UI doesn't say anything about it, so my next guess would be that the order of the cipher suites in Firefox causes the server to choose a cipher suite that the server doesn't correctly implement. This is going probably to require some wiresharking and enabling/disabling of cipher suites and whatnot for the person who gets assigned the bug.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

David or Camilo, could you take care of that? It is pretty urgent if we want to see that bug fixed for 29. Thanks

Comment 4

[Camilo Viecco (:cviecco)]

I will start diagnosis

Comment 5

[Camilo Viecco (:cviecco)]

From wireshark:
An initial connection to www.wbxmeeting.com:443 is completed and data start to flow from server to client, but a few byes later the server sends an 'Encrypted alert message'.

On seing this:
chrome: falls back to start tls 1.0 connection
FF: Tries to fallback, since a handshake was completes using tls 1.2 it refuses to downgrade. 

I would also state that both FF client hello messages are different when doing the second connection. But those extra extensions are also pressent when chrome tries 1.0 and succeeds.

Comment 6

[Camilo Viecco (:cviecco)]

Attachment: ff_webexfail.pcap

Comment 7

[Patrick McManus [:mcmanus]]

camilo - you set needinfo without asking a question. what's up?

Comment 8

[Patrick McManus [:mcmanus]]

what Camilo described to me in irc was

1] a successful connection to webex on tls 1.2 with a full handshake
2] a little bit of HTTP
3] an encrypted Alert from webex (reason unknown)
4] firefox making a new connection to webex in order to keep going
5] webex replying to our second tls 1.2 client hello with a TLS error (this is different from the first time where we handshook ok)
6] we don't fallback to retrying with < 1.2 because of #1 - this is viewed as a downgrade attack. Chrome appears not to care.

I'm going to put out feelers for a webex contact.

Comment 9

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #2)
> (In reply to Sylvestre Ledru [:sylvestre] from comment #1)
> > Brian, does it ring a bell? (sorry if you are not the right person to
> > contact)
> 
> However, Chrome's UI normally indicates when non-secure
> fallback was used, and the UI doesn't say anything about it

Actually, I take it back. Chrome's UI does indicate that it did non-secure fallback.

This seems like the same type of problem as bug 944465. However, in that bug, Wan-Teh indicated that Chrome's behavior was indicative of a bug in Chrome that needed to be fixed. I also got the impression that it was fixed. Perhaps Chrome undid its fix for that?

It would be good to figure out which cipher suite is being negotiated and then see if disabling that cipher suite and similar (e.g. if it's an AES-CBC cipher suite, disable all AES-CBC cipher suites) fixes the problem.

Comment 10

[Wan-Teh Chang]

https://www.wbxmeeting.com cannot decode a TLS 1.2 ClientHello with
a padding extension produced by NSS. However, it can decode a TLS 1.1
ClientHello with a padding extension produced by NSS. I have found a
workaround in NSS, but we should report this bug to the vendor of the
SSL/TLS library used by https://www.wbxmeeting.com.

will69, can you report this bug?

Details:

https://www.wbxmeeting.com uses session tickets. So a session
resumption ClientHello would exceed 256 bytes and NSS would
pad it to 512 bytes.

Brian: I removed Chrome's insecure fallback on errors received
after the initial handshake is done. This error (decode_error
alert) occurs during the handshake, so it still triggers an
insecure fallback in Chrome. I can remove the insecure fallback
on decode_error alert as well.

Comment 11

[Sylvestre Ledru [:Sylvestre]]

If you have a contact at Cisco/webex, I contact them about this issue.

Comment 12

[Patrick McManus [:mcmanus]]

(In reply to Sylvestre Ledru [:sylvestre] from comment #11)
> If you have a contact at Cisco/webex, I contact them about this issue.

private followup made.

Comment 13

[will69]

I have forwarded all information to our accredited Cisco contact.

Comment 14

[Wan-Teh Chang]

Attachment: [Do not check this in yet] NSS workaround: reorder the TLS extensions in ClientHello

This seems to work in Firefox. We should not check
this in until we have heard from Cisco. I'd like to
know if the ClientHello generated by NSS is somehow
malformed. If the bug is in the server side, we
should ask the server vendor to recommend a workaround.

Comment 15

[will69]

The Cisco bug reference is "CSCuo03023". They are aware of this bug.

@Cisco: Please comment (see above).

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Camilo, what do you think about patch in comment #14? It would be nice to have this bug fixed for 29.

Comment 17

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Comment on attachment 8405176 [details] [diff] [review]
[Do not check this in yet] NSS workaround: reorder the TLS extensions in ClientHello

Review of attachment 8405176 [details] [diff] [review]:
-----------------------------------------------------------------

::: security/nss/lib/ssl/ssl3ext.c
@@ +270,5 @@
>   */
>  static const 
>  ssl3HelloExtensionSender clientHelloSendersTLS[SSL_MAX_EXTENSIONS] = {
> +    { ssl_signature_algorithms_xtn, &ssl3_ClientSendSigAlgsXtn },
> +    { ssl_cert_status_xtn,        &ssl3_ClientSendStatusRequestXtn },

Please add a comment explaining the ordering. Maybe you don't know the precise rule that we need to follow, but at least it would be good to have a comment referencing this bug and explaining that the order matters for compatibility.

Comment 18

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #16)
> Camilo, what do you think about patch in comment #14? It would be nice to
> have this bug fixed for 29.

This is not a super-low-risk change. We are supposed to be able to put these entries in the handshake in any order, but this implementation requires a specific order. However, if there are other implementations that require a *different* specific order, then we may be breaking those other websites.

It would be a good idea to look at what order MSIE11 uses. We should consider duplicating their ordering. However, MSIE11 is very liberal regarding non-secure TLS version intolerance fallback so that isn't a no-brainer either.

(In reply to Wan-Teh Chang from comment #10)
> I can remove the insecure fallback
> on decode_error alert as well.

I think that would be great.

Comment 19

[Camilo Viecco (:cviecco)]

Sylvestre, as Brian mentioned, this is not a low risk change. While the patch is OK, we dont know what other site with break with this order. Cisco is working on this, (notice that this is not a cisco URL even tough they are using a cisco 'orion' product). I would wait on more info from cisco (not via bugzilla).

Comment 20

[Camilo Viecco (:cviecco)]

Comment on attachment 8405176 [details] [diff] [review]
[Do not check this in yet] NSS workaround: reorder the TLS extensions in ClientHello

I am not an NSS peer, so I cannot give an official r+. For what is worth, this code is OK, However we dont knot what other TLS backends will break with this change.

Comment 21

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Camilo Viecco (:cviecco) from comment #20)
> I am not an NSS peer, so I cannot give an official r+. For what is worth,
> this code is OK, However we dont knot what other TLS backends will break
> with this change.

There is a simple way to estimate whether this increases or decreases version intolerance fallback:

1. Add a pref to Gecko that controls whether non-secure fallback due to suspected TLS version intolerance is enabled.
2. Run mwobensmith's compatibility tests with that pref set to "disable non-secure fallback".

Even easier than that, we can simply run mwobensmith's compatibility tests to determine whether we've broken any of the sites on his list even when non-secure version fallback is enabled. I think both are important to do, not for this change, but really any change.

And/or, perhaps Ivan (CC'd) has some information about how the ordering of extensions in the client hello affects compatibility. At least, I think Ivan may be interested in knowing that the ordering of extensions in the client hello is a factor in compatibility, because it SSL Labs and other sites may be over-estimating pure TLS version intolerance.

Comment 22

[Sylvestre Ledru [:Sylvestre]]

OK. It is now likely that 29 will not ship with this bug fix. Is that a big deal?

Comment 23

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #22)
> OK. It is now likely that 29 will not ship with this bug fix. Is that a big
> deal?

I think it is a big deal. I suggest that for Firefox 29 only, we do what Google Chrome is doing: add SEC_ERROR_DECODE_ERROR_ALERT to the whitelist of errors that trigger TLS version fallback. This will be an unfortunate but safe fix that we should undo as soon as we've verified that a better fix (Wan-Teh's patch, or another one) works well.

There is a thread on this issue on the ietf-tls mailing list. Here's my message:

    http://www.ietf.org/mail-archive/web/tls/current/msg12145.html

I filed a bug 998135 in NSS to fix this within NSS.

Comment 24

[Wan-Teh Chang]

Comment on attachment 8405176 [details] [diff] [review]
[Do not check this in yet] NSS workaround: reorder the TLS extensions in ClientHello

This patch is not intended for checkin. I don't even know why
it seems to solve the problem, which means when we implement
the next TLS extension, we won't know how to avoid breaking
the workaround.

We need to hear from the vendor of the server about this
problem. Until then, I suggest allowing the decode_error alert
to trigger a protocol version fallback.

Comment 25

[Sylvestre Ledru [:Sylvestre]]

(In reply to Brian Smith (:briansmith, was :bsmith; NEEDINFO? for response) from comment #23)
> (In reply to Sylvestre Ledru [:sylvestre] from comment #22)
> > OK. It is now likely that 29 will not ship with this bug fix. Is that a big
> > deal?
> 
> I think it is a big deal. 
OK. Thanks. If we do a second build for 29, we might take a potential fix. I will leave the "affected" for 29 in case Cisco fix the issue on their side and tracking it for 30.

Comment 26

[Lukas Blakk [:lsblakk] use ?needinfo]

It looks like bug 998135 has the NSS workaround and there's other work that needs to be done on Cisco's side so this doesn't seem to be a tracking-worthy bug as we've already shipped this since FF28

Comment 27

[Ben Watt]

Anything that can be done to resolve this, Cisco are not tracking this as something they need to follow up on anymore, and as Firefox is the only browser that won't work with their WebEx sites it's a big problem.

Comment 28

[Patrick McManus [:mcmanus]]

comment 27

Comment 29

[Eric Rescorla (:ekr)]

Cullen, can you help here?

Comment 30

[Cullen Jennings]

Note: AFAIC this is the same thing as https://bugzilla.mozilla.org/show_bug.cgi?id=998135 

Bit if background... Cisco makes a product internally called orion that a modification of webex to run on your own enterprise networks instead of having Cisco run it for you in the cloud. You install it as a server and on your network and it will run without any connection to the internet. It's typically run by enterprises or other organizations that want to make sure all their data is on their own servers. Orion uses the OpenSSL code for TLS. 

The https://www.wbxmeeting.com site does not actually belong to Cisco. It is a old version of orion and running by a small bank (you can look at the cert to get all the details of who runs it). Cisco would recommend upgrading to a newer version for many reasons including fixes to OpenSSL. It looks like www.wbxmeeting.com is version 1.5 of orion which used some version of OpenSSL 0.98. You can try connecting to a recent version of orion at https://orionmeet.cisco.com which is version 2.5 of orion and a much more recent version openssl. It seems to work fine with recent versions of FF (and thus why Cisco bug got closed)

We don't know what sort of "extra" TLS munging things might be in front of www.wbxmeeting.com but assuming there is none, the general feeling around Cisco is this is the bug where there was some versions of OpenSSL that had borked processing of options in certain orders and other browser either reordered options to work around the OpenSSL problem or did not complain about when this caused a downgrade. That seems to have been fixed in later versions of OpenSSl cisco is using in current products. (I want to add that I have not actually gone and verified that a given version of OpenSSL caused this problem with FF but that seems to what the folks on these products think - perhaps someone here knows exactly what is going on with that.)

My expectation is most orion customers have upgraded to a later version that would not have this problem. Lots of Cisco products use OpenSSL but similarly with then, I would expect them to be using a version that is not a problem. Because of this, I would not expect widespread problems but given the huge number of products based on openssl from many vendors, and the slowness to upgrade lots of servers on the internet, I imagine there will be many other sites that have this issue. Things like heartblead have probably caused widespread upgrade to of servers to versions of OpenSSL that don't have this problem. 

Hope that helps. Cullen

Comment 31

[Ben Watt]

Hi Cullen,

Thanks for the information. Orion I believe is the internal name for "Cisco WebEx Meetings Server".

It's good to know that this issue may be fixed in version 2.5, but this version hasn't been released publically yet. The company I work for is using the latest version of CWMS, which is 2.0 MR4.

This release suffers from the problems outlined in this bug report, so it's actually the case that there is no publically available of CWMS that works with Firefox without adjusting the config of Firefox.

The release notes of each version of CWMS also confirm that this bug is present.

If Cisco's own documentation shows that the bug is present, and customers have reported it as such, the Cisco bug should not be terminated. And if it is resolved in 2.5, it should be listed on the Cisco bug.

I look forward to trying the new release when it comes out from Cisco to see if this problem is resolved, it's certainly been causing us a lot of issues.

Can you arrange for the bug report to be updated?

Comment 32

[Cullen Jennings]

Hi Ben - what you are saying makes sense.

Comment 33

[Cullen Jennings]

So I have confirmed what Ben said. This bug still exists in latest Cisco released version (See http://www.cisco.com/c/en/us/td/docs/collaboration/CWMS/2_0/Release_Notes.html ). Technically speaking I believe this is a bug in the Cisco product not Firefox and specifically it is the openssl code we are using. The Cisco bug was closed but I expect that to change - I will keep Ben in the loop on what happens with the bug and perhaps we can update here as it makes sense.

All that said, given the root cause is a bug in openssl and that probably impacts many other things, it seems to me that it might be a good idea for Firefox to reorder the stuff such that it does not trigger this bug.

Comment 34

[Cullen Jennings]

The Cisco side is getting fixed up - I don't have an ETA but the new cisco bugs numbers are CSCuq98568 and CSCur02612. I hope to see this in CWMS version 2.0 MR5