Closed Bug 1306481 Opened 3 years ago Closed 2 years ago

womenonwaves.org and wonderpush.com reject TLS handshakes from Firefox 51

Categories

(Web Compatibility :: Desktop, defect, major)

Firefox 51
defect
Not set
major

Tracking

(firefox49 unaffected, firefox-esr45 unaffected, firefox50 unaffected, firefox51 affected, firefox52 wontfix)

RESOLVED WORKSFORME
Tracking Status
firefox49 --- unaffected
firefox-esr45 --- unaffected
firefox50 --- unaffected
firefox51 --- affected
firefox52 --- wontfix

People

(Reporter: mwobensmith, Unassigned)

References

Details

(Keywords: compat)

The following sites have regressed and are now broken on Nightly 52, compared to Nightly 51:

https://ksd.org	pr_connect_reset_error
https://womenonwaves.org	pr_end_of_file_error
https://wonderpush.com	pr_end_of_file_error	

It is unclear if these are related or not.

I don't know how to further diagnose these sites. If we find a common cause, we can update this bug's title and description. We can also break these out into discrete bugs if they turn out to be caused by different problems.

Happy to dig into these further if anyone has any advice.
Taking a look at our ClientHello, the only differences are the addition of curve25519 and the new PSS signature schemes.  It's not the absence of P-521, since I tested both with and without it and got the same result.  I'll test with some changes and see what happens.

(As an aside, we need to disable DSA signature schemes.  We don't have cipher suites that use them, so it's just wasted bytes.)
OK, I've worked out what is causing the two pr_end_of_file_error cases.  Both are objecting to the presence of new PSS codepoints in the signature_algorithms extension.  Sadly, it doesn't seem like a change in ordering is going to make these servers happy.  We can work around this by disabling PSS for the moment, but that's going to make TLS 1.3 deployment very difficult.

Sadly, this also happens in NSS 3.27, which landed in Firefox 51.
Has Regression Range: --- → yes
Summary: Broken secure sites in Nightly 52.0a1 → Broken secure sites in Firefox 51
Version: 52 Branch → 51 Branch
No joy on ksd.org.  The connection establishes fine with NSS on the command line.  And it appears to connect but receive an alert.
Chrome Canary could not connect these sites, either.
Summary: Broken secure sites in Firefox 51 → Broken secure sites in Firefox 52
Sorry, changed the title without seeing the details in comment 2. Changed it back. 

FWIW the reason I didn't catch this in Nightly 51 is probably because I ran the pass before the change was introduced. I may need to do canary passes multiple times per cycle if this becomes a problem.
Summary: Broken secure sites in Firefox 52 → Broken secure sites in Firefox 51
Nick, do you think that the error on ksd.org could be a problem with the change to the header table size?

I see the connection dropping immediately after getting established and NSS connects happily without the rest of our HTTP stack sitting on top.
Flags: needinfo?(hurley)
Looks like it - if I change the HPACK receive buffer size back to 4k, it works. Unlike in bug 1305321 though, that's the *only* value that works - others also fail to connect. I would imagine this is an apache mod-h2 bug (that's the server ksd.org claims to be running), but I'll poke around a bit more to be sure before reaching out to Stefan.
Flags: needinfo?(hurley)
So ksd.org is just receiving our opening preamble, settings, prio tree, and first request, and then sending us back a GOAWAY/PROTOCOL_ERROR with no debug info. Not much I can do with that info without knowing what they're complaining about.

I'll try setting up an apache + mod_h2 locally to see if I can reproduce with logging, but in the meantime (in case I can't repro locally), Matt do you know anyone at KSD that might be able to help debug from their end, or is it just a site you use? (I can always try reaching out if you don't know anyone).
Flags: needinfo?(mwobensmith)
(In reply to Nicholas Hurley [:nwgh][:hurley] from comment #8)
> Matt do
> you know anyone at KSD that might be able to help debug from their end, or
> is it just a site you use? (I can always try reaching out if you don't know
> anyone).

This site was surfaced with my regular testing via TLS canary: 
https://tlscanary.mozilla.org

So, I don't know anyone associated with this site. It's just been culled from Alexa's top sites list, so I use it and 300k others for regular compatibility testing.

Also, it appears that this is a different issue and could be split from this bug. Is it correct to say that it would be an Apache issue and not a Firefox one?
Flags: needinfo?(mwobensmith)
(In reply to Matt Wobensmith [:mwobensmith][:matt:] from comment #9) 
> Also, it appears that this is a different issue and could be split from this
> bug. Is it correct to say that it would be an Apache issue and not a Firefox
> one?

Yes, we should split the ksd.org bug out from this one. While I do believe this an apache bug, with the amount of information I have currently, I'm not comfortable ruling out a firefox bug just yet. I'll file a new bug for the h2 issue with apache.
See Also: → 1307190
General consensus here is to treat this as an evangelism problem.  Apparently the two affected sites are running an erlang TLS stack which is incapable of dealing with new things.  In this case, it's our new signature algorithms.

Changes have been made upstream:
https://github.com/erlang/otp/commit/ebfd862f47611fa17be72cad1afcd6a13f14bc4d
https://github.com/erlang/otp/commit/ae7347bfdcab2486bb55dfe54918a0c994d8b7c7
Component: Security: PSM → Desktop
Keywords: compat
Product: Core → Tech Evangelism
Summary: Broken secure sites in Firefox 51 → womenonwaves.org and wonderpush.com reject TLS handshakes from Firefox 51
Version: 51 Branch → Firefox 51
wonderpush.com has been fixed.
Hello.
Thank you for reporting, especially Martin Thomson for reaching us personnaly via email!
We have indeed upgraded our erlang webserver stack to fix the issue.
Again, many thanks!
Best
Too late for firefox 52, mass-wontfix.
(In reply to Olivier Favre from comment #13)
> Hello.
> Thank you for reporting, especially Martin Thomson for reaching us
> personnaly via email!
> We have indeed upgraded our erlang webserver stack to fix the issue.
> Again, many thanks!
> Best

Thanks, I can confirm that the issue is no longer reproducible.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WORKSFORME
Product: Tech Evangelism → Web Compatibility
You need to log in before you can comment on or make changes to this bug.