1306481 - womenonwaves.org and wonderpush.com reject TLS handshakes from Firefox 51

Status: RESOLVED WORKSFORME

(Web Compatibility :: Site Reports, defect)

Description

[Matt Wobensmith [:mwobensmith][:matt:]]

The following sites have regressed and are now broken on Nightly 52, compared to Nightly 51:

https://ksd.org	pr_connect_reset_error
https://womenonwaves.org	pr_end_of_file_error
https://wonderpush.com	pr_end_of_file_error	

It is unclear if these are related or not.

I don't know how to further diagnose these sites. If we find a common cause, we can update this bug's title and description. We can also break these out into discrete bugs if they turn out to be caused by different problems.

Happy to dig into these further if anyone has any advice.

Comment 1

[Martin Thomson [:mt:]]

Taking a look at our ClientHello, the only differences are the addition of curve25519 and the new PSS signature schemes.  It's not the absence of P-521, since I tested both with and without it and got the same result.  I'll test with some changes and see what happens.

(As an aside, we need to disable DSA signature schemes.  We don't have cipher suites that use them, so it's just wasted bytes.)

Comment 2

[Martin Thomson [:mt:]]

OK, I've worked out what is causing the two pr_end_of_file_error cases.  Both are objecting to the presence of new PSS codepoints in the signature_algorithms extension.  Sadly, it doesn't seem like a change in ordering is going to make these servers happy.  We can work around this by disabling PSS for the moment, but that's going to make TLS 1.3 deployment very difficult.

Sadly, this also happens in NSS 3.27, which landed in Firefox 51.

Comment 3

[Martin Thomson [:mt:]]

No joy on ksd.org.  The connection establishes fine with NSS on the command line.  And it appears to connect but receive an alert.

Comment 4

[Masatoshi Kimura [:emk]]

Chrome Canary could not connect these sites, either.

Comment 5

[Matt Wobensmith [:mwobensmith][:matt:]]

Sorry, changed the title without seeing the details in comment 2. Changed it back. 

FWIW the reason I didn't catch this in Nightly 51 is probably because I ran the pass before the change was introduced. I may need to do canary passes multiple times per cycle if this becomes a problem.

Comment 6

[Martin Thomson [:mt:]]

Nick, do you think that the error on ksd.org could be a problem with the change to the header table size?

I see the connection dropping immediately after getting established and NSS connects happily without the rest of our HTTP stack sitting on top.

Comment 7

[u408661]

Looks like it - if I change the HPACK receive buffer size back to 4k, it works. Unlike in bug 1305321 though, that's the *only* value that works - others also fail to connect. I would imagine this is an apache mod-h2 bug (that's the server ksd.org claims to be running), but I'll poke around a bit more to be sure before reaching out to Stefan.

Comment 8

[u408661]

So ksd.org is just receiving our opening preamble, settings, prio tree, and first request, and then sending us back a GOAWAY/PROTOCOL_ERROR with no debug info. Not much I can do with that info without knowing what they're complaining about.

I'll try setting up an apache + mod_h2 locally to see if I can reproduce with logging, but in the meantime (in case I can't repro locally), Matt do you know anyone at KSD that might be able to help debug from their end, or is it just a site you use? (I can always try reaching out if you don't know anyone).

Comment 9

[Matt Wobensmith [:mwobensmith][:matt:]]

(In reply to Nicholas Hurley [:nwgh][:hurley] from comment #8)
> Matt do
> you know anyone at KSD that might be able to help debug from their end, or
> is it just a site you use? (I can always try reaching out if you don't know
> anyone).

This site was surfaced with my regular testing via TLS canary: 
https://tlscanary.mozilla.org

So, I don't know anyone associated with this site. It's just been culled from Alexa's top sites list, so I use it and 300k others for regular compatibility testing.

Also, it appears that this is a different issue and could be split from this bug. Is it correct to say that it would be an Apache issue and not a Firefox one?

Comment 10

[u408661]

(In reply to Matt Wobensmith [:mwobensmith][:matt:] from comment #9) 
> Also, it appears that this is a different issue and could be split from this
> bug. Is it correct to say that it would be an Apache issue and not a Firefox
> one?

Yes, we should split the ksd.org bug out from this one. While I do believe this an apache bug, with the amount of information I have currently, I'm not comfortable ruling out a firefox bug just yet. I'll file a new bug for the h2 issue with apache.

Comment 11

[Martin Thomson [:mt:]]

General consensus here is to treat this as an evangelism problem.  Apparently the two affected sites are running an erlang TLS stack which is incapable of dealing with new things.  In this case, it's our new signature algorithms.

Changes have been made upstream:
https://github.com/erlang/otp/commit/ebfd862f47611fa17be72cad1afcd6a13f14bc4d
https://github.com/erlang/otp/commit/ae7347bfdcab2486bb55dfe54918a0c994d8b7c7

Comment 12

[Masatoshi Kimura [:emk]]

wonderpush.com has been fixed.

Comment 13

[Olivier Favre]

Hello.
Thank you for reporting, especially Martin Thomson for reaching us personnaly via email!
We have indeed upgraded our erlang webserver stack to fix the issue.
Again, many thanks!
Best

Comment 14

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 15

[sjw]

(In reply to Olivier Favre from comment #13)
> Hello.
> Thank you for reporting, especially Martin Thomson for reaching us
> personnaly via email!
> We have indeed upgraded our erlang webserver stack to fix the issue.
> Again, many thanks!
> Best

Thanks, I can confirm that the issue is no longer reproducible.