945210 - libssl cannot negotiate a TLS 1.2 connection to billingcp.us whereas some other implementations can

Status: RESOLVED WORKSFORME

(Tech Evangelism Graveyard :: English US, defect, P3)

Description

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

See https://www.ssllabs.com/ssltest/analyze.html?d=billingcp.us.

The ssllabs test for TLS 1.2 support passes. The IE and Chrome handshake simulations fail. IE11, Chrome 31, the current Chrome Canary, and Firefox also fail to create a TLS 1.2 connection; the non-Firefox browsers end up doing insecure fallback to SSL 3.0, and Firefox fails due to bug 945195.

The Wireshark trace provided by a Mozillazine forum user indicates that the server is choking on our client hello. I have not investigated this in detail, but my suspicion is that this server either has an issue with the cipher suites offered (and/or the ordering of them), and/or it may have a problem with the TLS version numbers we use in the record layer:

15 3.988740000 192.168.0.102 198.23.156.12 TCP 66 ispipes > https [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=4 SACK_PERM=1
16 4.100948000 198.23.156.12 192.168.0.102 TCP 66 https > ispipes [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1452 SACK_PERM=1 WS=128
17 4.100982000 192.168.0.102 198.23.156.12 TCP 54 ispipes > https [ACK] Seq=1 Ack=1 Win=261360 Len=0
18 4.102699000 192.168.0.102 198.23.156.12 SSL 207 Client Hello
19 4.213028000 198.23.156.12 192.168.0.102 TCP 60 https > ispipes [ACK] Seq=1 Ack=154 Win=15744 Len=0
20 4.214430000 198.23.156.12 192.168.0.102 TCP 60 https > ispipes [FIN, ACK] Seq=1 Ack=154 Win=15744 Len=0
21 4.214557000 192.168.0.102 198.23.156.12 TCP 54 ispipes > https [ACK] Seq=154 Ack=2 Win=261360 Len=0
22 4.215926000 192.168.0.102 198.23.156.12 TCP 54 ispipes > https [FIN, ACK] Seq=154 Ack=2 Win=261360 Len=0

Comment 1

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

This server is intolerant to OCSP stapling. Its cert is missing an OCSP URI in the AIA extension; it is possible that this causes the server to fail, when trying to process the extension. Of all the extensions that Firefox sends, this is the only extension that it chokes on.

I checked the ssllabs.com report above, and I found that all of the clients that don't support OCSP stapling connected successfully with TLS 1.2, and all of the clients that do support OCSP stapling failed to connect. This indicates that this is unlikely to be a problem with only libssl's implementation of OCSP stapling.

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I used their "pre-sales contact us" form to send the following message:

Hi, my name is Brian Smith. I lead the work on TLS, certificates, and similar things for Firefox. Please see:

https://bugzilla.mozilla.org/show_bug.cgi?id=945210
https://www.ssllabs.com/ssltest/analyze.html?d=billingcp.us

It appears that your server's TLS implementation has a serious bug that causes it to close the connection when it receives a TLS ClientHello message from any client that supports OCSP stapling, which is almost every modern browser now. As a result, browsers downgrade the connection, insecurely, from TLS 1.2 to TLS 1.1 to TLS 1.0 to SSL 3.0. This causes a huge amount of latency. Also, it is likely that at least some browsers (including, hopefully, Firefox) will soon stop downgrading connections to SSL 3.0 like this, which will cause sites with this problem to break.

I noticed that your website's certificate does not have an OCSP URI in its AIA extension. This could be why your TLS implementation is working badly--it may be trying to get an OCSP response, failing to do so, and then closing the connection. I recommend that, in addition to reporting this bug to your TLS server vendor, that you also contact your the CA that issued your certificate and ask them for a new certificate that has the OCSP AIA URI.

Also, I would very much appreciate it if you could reply to this message (to [e-mail] or [e-mail]) with information about what server you are using. That way, we can also work with that vendor to help them resolve the problem with their software.

Thank you very much for your time.

Cheers,
Brian Smith
Mozilla Corporation

Comment 3

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Kathleen,

This certificate appears to be issued in 2013, so it should have an OCSP URI in its AIA extension unless the server successfully staples an OCSP response, but it doesn't. The CA is "AlphaSSL" that chains to "GlobalSign Root CA." Can one of you poke GlobalSign about the general badness of not including an OCSP URI in the certificate, and about this specific site in particular?

Comment 4

[Jean-Marc Desperrier]

They are using a SSL seal linked to Global Sign, through Alpha SSL :

The seal code loads data from : https://seal.globalsign.com

But the pop-up URL is : https://seal.alphassl.com/SiteSeal/siteSeal/profile/profile.do?p1=9d530320&p2=b8c0708f0aee50ce64a4b396dd34e4c4b2f78e93f8956fdf54b14ca865a2b1888576bcddb1a3bbccbac2759055a90c97&p3=f63aed55313cae0fca033af2cc4682ccd730677c

Love how the seal says to check the adress in the URL bar starts with https://seal.alphassl.com/ , but has such a long url that the start can be hidden depending on how you load it (initially it's visible, but if I copy/paste to load it, it's isn't).

Comment 5

[Kathleen Wilson]

(In reply to Brian Smith (:briansmith, was :bsmith; Please NEEDINFO? me if you want a response) from comment #3)
> Kathleen,
> 
> This certificate appears to be issued in 2013, so it should have an OCSP URI
> in its AIA extension unless the server successfully staples an OCSP
> response, but it doesn't. The CA is "AlphaSSL" that chains to "GlobalSign
> Root CA." Can one of you poke GlobalSign about the general badness of not
> including an OCSP URI in the certificate, and about this specific site in
> particular?


Which certificate or website is causing the problems?

Comment 6

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

The website that was having the trouble is https://billingcp.us.

However, I just tried it now and the problem is fixed. The CA worked with the website to get a replacements certificate that contains an OCSP AIA URI. See https://twitter.com/rmhrisk/status/408602088332349440 (Ryan Hurst ‏from GlobalSign):
   
   "@pzb @BRIAN_____ @AlphaSSL @globalsign this is an older pre BR cert,
    we will call the customer and offer them a replacement / extension"

Not only does the website correctly negotiate TLS 1.2, but it also staples an OCSP response!

This means that there is probably some server implementation that chokes whenever OCSP stapling is enabled and the end-entity certificate does not contain an OCSP AIA URI.

Ryan, were you able to find out any more information about this that you could share, publicly here in bugzilla or privately in email? In particular, do you know what server implementation the site is using and/or if my diagnosis of the problem's cause is correct?

http://toolbar.netcraft.com/site_report?url=https://billingcp.us says that the TLS implementation supports the following extensions:

   RFC4366 server name,
   RFC5746 renegotiation info,
   RFC5077 session ticket,
   RFC4366 status request,
   RFC6520 heartbeat

And it says the server is "Apache". The Server header in the HTTPS response is "Apache" too. The home page was generated with "PHP/5.4.21," according to X-Powered-By.

If this is a bug in Apache then we should make sure it has already been fixed and/or help Apache fix it.

I will put this on the list of things to talk about in the CABForum, especially in the SSL Performance Working Group.

Thanks Ryan, for your help with this!

Comment 7

[ryan_hurst]

My pleasure; I did not have support ask but I noticed the same thing (responding as Apache). If you like I can have our customer support staff call and verify.

Comment 8

[ryan_hurst]

I should add, AlphaSSL is our retail brand; its operated by us 100%; as I mention on Twitter this was issued prior to BR, we were relying on CRLs for AlphaSSL at the time we have made the switch to include AIA:OCSP for all SSL certificates.

Comment 9

[Kurt Roeckx]

It seems that they only have 1 suite enabled (TLS_RSA_WITH_RC4_128_SHA), so on that I also can't guess which ssl implementation they are using.  It would be good to know if it's apache which version and wether it's mod_ssl or mod_gnutls that's being used.

Comment 10

[ryan_hurst]

I have asked the vetting manager in the US who worked with them on the replacement to reach out and get this information, we will also offer to help them improve their suite configuration.

Comment 11

[Kaspar]

(In reply to ryan_hurst from comment #8)
> I should add, AlphaSSL is our retail brand; its operated by us 100%; as I
> mention on Twitter this was issued prior to BR, we were relying on CRLs for
> AlphaSSL at the time we have made the switch to include AIA:OCSP for all SSL
> certificates.

For the sake of transparency, "at the time" apparently means January 2013:

-----BEGIN CERTIFICATE-----
MIIEfDCCA2SgAwIBAgISESF2NXrGRDzvaCItMutrgQvUMA0GCSqGSIb3DQEBBQUA
MC4xETAPBgNVBAoTCEFscGhhU1NMMRkwFwYDVQQDExBBbHBoYVNTTCBDQSAtIEcy
MB4XDTEzMDExMDIwMTIzN1oXDTE0MDExMTIwMTIzN1owPjEhMB8GA1UECxMYRG9t
YWluIENvbnRyb2wgVmFsaWRhdGVkMRkwFwYDVQQDExB3d3cuYmlsbGluZ2NwLnVz
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwyC2jBhqPPMH/UFQWrR4
ktlJk96rmN816VmEmEeck+QppeZyRLV6CQx+KGyMVlfVYeB3b6bf8UYw5qpfdiRZ
aDpJqMJA03H1y+6QxyuUykXNPaL48YbwLNn59LLb1vU9zOtWjdQIwUvaHdO5O0J3
kYyKdsilaxWlByl8YuDIMIUMk525XT1ykRUtUNsq7bC0Xkz8wfNzQmpxaIS3TJXw
5ujMFcMTKinkjYVTg7GJlubER+aIDMq4iNe1VdsJyhbSE/B9RzzpxFGpNbcSmBc5
MmA9kGKwrqFhngmbD4NFjX25whMRllEaeT9HsrUmGACYsJRgQNh9BXyqa9zrPye7
AwIDAQABo4IBgjCCAX4wDgYDVR0PAQH/BAQDAgWgME0GA1UdIARGMEQwQgYKKwYB
BAGgMgEKCjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNv
bS9yZXBvc2l0b3J5LzApBgNVHREEIjAgghB3d3cuYmlsbGluZ2NwLnVzggxiaWxs
aW5nY3AudXMwCQYDVR0TBAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH
AwIwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDovL2NybDIuYWxwaGFzc2wuY29tL2dz
L2dzYWxwaGFnMi5jcmwwTAYIKwYBBQUHAQEEQDA+MDwGCCsGAQUFBzAChjBodHRw
Oi8vc2VjdXJlMi5hbHBoYXNzbC5jb20vY2FjZXJ0L2dzYWxwaGFnMi5jcnQwHQYD
VR0OBBYEFIFmC117IUJfsji01p5dFsbj0hQCMB8GA1UdIwQYMBaAFBTqGVXwDg0y
xh90M7eOZhpMEjEeMA0GCSqGSIb3DQEBBQUAA4IBAQB/BGHxBLK6pK3BKCLUjcHE
/5y7b3JWdNkCbKojzsqNK4s4kprNd9CSXq5vGJkjkCHZ6tMY1Iu00w09KirX/keU
JYDyh1KiC0z/RzmiQGq4VurhyoYWyUVAoXqWr08fHpOBaElhUN3cAELKKAXY0T68
BfXtxE7UgEeTB1/CJSDWTKPtnLGyemETtxJ5OSkGirIU3zKwkOcUL4d3aQx5rbeH
wuFKv6GtHXAoiyLN+7BfCKJTT191gHF1hzc29Qa9ojwsIYvOdvPG2zW2obuGnZq6
s3rXHc0YdCxUFge+ptRWpCUp8F+1RSNu3kGiPXrHP2y5ZtqS+zaPWBOAu48qcweR
-----END CERTIFICATE-----

It is then interesting to read what "Management" asserts on 19 June 2013 according to https://cert.webtrust.org/SealFile?seal=1515&file=pdf::

> [...] in GlobalSign management's opinion, in providing its SSL certificate services [...]
> for the Root CAs [...] 'AlphaSSL G2' [...] during the period from April 1, 2012 to
> March 31, 2013, GlobalSign NV/SA has:
> • Disclosed its Certificate Practice Statement and its commitment to provide SSL certificates
>   in conformity with CA/Browser Forum Baseline Requirements for the Issuance and Management
>   of Publicly-Trusted Certificates [...]"

where the "CA/Browser Forum..." text links to https://cabforum.org/Baseline_Requirements_V1_1_3.pdf. In that document, we find under 13.2.1:

> The CA SHALL make revocation information for Subordinate Certificates and Subscriber Certificates
> available in accordance with Appendix B.
> If the Subscriber Certificate is for a high-traffic FQDN, the CA MAY rely on stapling,
> in accordance with [RFC4366], to distribute its OCSP responses. In this case, the CA SHALL
> ensure that the Subscriber “staples” the OCSP response for the Certificate in its TLS handshake.
> The CA SHALL enforce this requirement on the Subscriber either contractually, through the
> Subscriber or Terms of Use Agreement, or by technical review measures implement by the CA.

It's probably doubtful whether billingcp.us is really "high-traffic", but in any case, I'm quite interested in hearing how this requirement was enforced "either contractually" or "by technical review measures" (the subscriber agreement under https://www.globalsign.com/repository/globalsign-subscriber-agreement-digital-certificates-and-services.pdf does not have any stipulations to this effect, JFTR).

Comment 12

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Kaspar Brand from comment #11)
> For the sake of transparency, "at the time" apparently means January 2013:

Kaspar, if you think it is worthwhile to debate/discuss whether there is a CA policy compliance issue regarding this website and/or CA, please start a thread on dev-security-policy. This bug is about how we cope with the compatibility/security issue with that site and/or more generally with some versions of the server side software being used by the website on which we discovered this issue. Part of the the solution for this bug may be to ask and/or require CAs to replace certificates that are missing the OCSP URI in the AIA extension, but if so, the details of that CA communication would be better discussed/debated on the dev-security-policy mailing list.

Anyway, I'm not tell you to not discuss the issue. I'm just asking that we please not get into the compliance issue in this bug report. Thanks.

Comment 13

[Yuhong Bao]

Any update on which server software it was?

Comment 14

[Yuhong Bao]

Still RC4 only.

Comment 15

[Masatoshi Kimura [:emk]]

Why don't you file a new bug and block bug 1138101 instead of adding an off-topic comment?

Comment 16

[Yuhong Bao]

Asking again with a contact CCed what server software were you using when you used the old certificate from 2013.