1346392 - server that tries ntlm over h2 needs an h1 fallback

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Kerem Yuceturk]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393

Steps to reproduce:

Repro Steps:
1.	Install SharePoint Server 2016 on Windows Server 2016.
2.	Set up a SharePoint web application and site so that it's hosted over SSL/TLS and uses NTLM authentication.
3.	Install Firefox 52.0 on a Windows 10 v. 1607 client computer.
4.	Launch Firefox 52.0.
5.	Browse to the SharePoint site that's hosted over SSL/TLS.
6.	When prompted for your NTLM credentials, provide the credentials.



Actual results:

Result:
The SharePoint site doesn't load.  Instead, you get the following Firefox error message:

Secure Connection Failed
The connection to the server was reset while the page was loading.
•	The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
•	Please contact the website owners to inform them of this problem.



Expected results:

Expected Result:
Authentication should succeed and the site should load.

Notes:
1.	This does not repro with Firefox 51.0.
2.	If I set the "security.tls.version.max" preference in Firefox 52.0 to "2" (TLS 1.1), then Firefox is able to complete this scenario successfully.  But this scenario fails if the "security.tls.version.max" preference in Firefox 52.0 is set to its default value of "3" (TLS 1.2).
3.	Firefox 52.0. is able to successfully NTLM authenticate to SharePoint Server 2016 sites using HTTP instead of SSL/TLS.
4.	Firefox 52.0 is able to successfully browse to SharePoint Server 2016 content over SSL/TLS that doesn’t require NTLM authentication.

Comment 1

[Kerem Yuceturk]

Some additional comments from a SharePoint+FF customer: https://social.technet.microsoft.com/Forums/en-US/76d6fc5f-aa82-40fd-8e9d-fd9efc70314b/firefox-v52-breaks-sharepoint-2016-ssl-connection?forum=SP2016

Comment 2

[Loic]

Kerem, could you run the tool mozregression to narrow down a regression range in FF52. It would help to find the regressing bug and debug.
See http://mozilla.github.io/mozregression/ for details.

Run the command "mozregression --good=51", then copy here the final pushlog from the repository mozilla-inbound. For each build downloaded by the tool, make the test and enter if it's good or bad.

If you need to set up a testing profile with mozregression, it's possible (--profile). You can use the --help command.

Comment 3

[Troy Starr]

Hi Loic, Kerem may not be able to look at this soon so I've gone ahead and used the mozregression tool to repro this bug.  Here's the output:

pushlog_url: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=d26ac63f1b81c3fce35448a7c502e95e0b5c56c0&tochange=41a2caef2b66d3dca1debe1ac5b4fea58d0fe6f5

Let us know if you need any more information.  Thanks!

- Troy Starr [MSFT]

Comment 4

[Loic]

Thanks for the pushlog. I guess a good culprit is bug 486508.

Honza, could you check this issue, please.

Comment 5

[Troy Starr]

After further troubleshooting, I don't think TLS 1.2 is what directly triggers this regression.  Instead, I believe it's HTTP/2 support in Firefox.

If I leave the security.tls.version.max preference at its default value of "3" (TLS 1.2), but set the network.http.spdy.enabled.http2 preference to "false", then Firefox 52 is able to successfully authenticate and render the SharePoint Server 2016 site over TLS 1.2.  But if I leave network.http.spdy.enabled.http2 at its default value of "true" under the same conditions, then this bug repros in Firefox 52.

This makes sense with the original repro steps since HTTP/2 over TLS requires TLS 1.2 or higher.  And Windows Server 2016 is the only Windows Server release that currently supports HTTP/2 in IIS.

NTLM isn't compatible with HTTP/2 due to the connection-oriented nature of NTLM authentication and the potential ambiguity of the user's identity when multiplexing HTTP/2 streams over a single TCP connection.  If NTLM authentication will be performed, the connection needs to be downgraded from HTTP/2 to HTTP/1.x.

- Troy Starr [MSFT]

Comment 6

[Honza Bambas (:mayhemer)]

We had to uplift that bug...

Comment 7

[Honza Bambas (:mayhemer)]

Sylvester, why exactly have you marked bug 1315332 with status-firefox52: affected → wontfix on 2017-01-25?

Comment 8

[Kohei Yoshino [:kohei]]

Posted the site compatibility doc: https://www.fxsitecompat.com/en-CA/docs/2017/ntlm-authentication-fails-in-certain-cases/

Comment 9

[Sylvestre Ledru [:Sylvestre]]

Honza, Usually, I explain why I am doing such changes and I didn't here. I apologize for that. 

Here, I think it I did because no uplift request (or anything else) happened on the bug for 2 months and I wontfixed it to remove it from our radar.

Comment 10

[Julien Cristau [:jcristau] (back April 22)]

Loic, bug 1315332 is supposed to be fixed in 53, was the status here checked, or just based on the regression range?

Comment 11

[Loic]

The flags I set in my comment #4 were based on the regression range found by Troy in comment #3 (I didn't test myself as I don't have such a setup at home). 

If this bug is already fixed, when using mozregression, nightlies after 2016-11-30 should have been tested as good because they contain the bugfix from bug 1315332.

Indeed, I think the reporter should test Beta 53 to confirm if it's fixed or not.

Kerem or Troy, could you download Beta and confirm if it's fixed or not, please.
Link: https://www.mozilla.org/en-US/firefox/beta/all/

Comment 12

[Troy Starr]

Hi Loic, I've downloaded and installed Firefox 53.0b2 64-bit on Windows 10 v. 1607.  The bug still repros with 53.0b2.  So it may have been premature to resolve this bug as a duplicate of bug 1315332.

As with Firefox 52, if I set the network.http.spdy.enabled.http2 preference to "false", then Firefox 53.0b2 is able to successfully authenticate and render the SharePoint Server 2016 site over TLS 1.2.

Comment 13

[Loic]

Ok, let's reopen it.

Comment 14

[Loic]

Honza, it looks like this bug is not a duplicate of 1315332. Could it be an edge case?

Comment 15

[Honza Bambas (:mayhemer)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #9)
> Honza, Usually, I explain why I am doing such changes and I didn't here. I
> apologize for that. 
> 
> Here, I think it I did because no uplift request (or anything else) happened
> on the bug for 2 months and I wontfixed it to remove it from our radar.

Yep, I had to request an uplift :(  it escaped me that there were a merge in between...

Comment 16

[Honza Bambas (:mayhemer)]

(In reply to Troy Starr from comment #12)
> Hi Loic, I've downloaded and installed Firefox 53.0b2 64-bit on Windows 10
> v. 1607.  The bug still repros with 53.0b2.  So it may have been premature
> to resolve this bug as a duplicate of bug 1315332.
> 
> As with Firefox 52, if I set the network.http.spdy.enabled.http2 preference
> to "false", then Firefox 53.0b2 is able to successfully authenticate and
> render the SharePoint Server 2016 site over TLS 1.2.

Troy, thanks.  Just to make it clear, switching network.http.spdy.enabled.http2 to false fixes this issue in both 53 and 52, or only in 53?

Comment 17

[Troy Starr]

(In reply to Honza Bambas (:mayhemer) from comment #16)
> Troy, thanks.  Just to make it clear, switching
> network.http.spdy.enabled.http2 to false fixes this issue in both 53 and 52,
> or only in 53?

Hi Honza, that fixes this issue in both 53 and 52.

Comment 18

[Honza Bambas (:mayhemer)]

(In reply to Troy Starr from comment #17)
> (In reply to Honza Bambas (:mayhemer) from comment #16)
> > Troy, thanks.  Just to make it clear, switching
> > network.http.spdy.enabled.http2 to false fixes this issue in both 53 and 52,
> > or only in 53?
> 
> Hi Honza, that fixes this issue in both 53 and 52.

Thanks!  Then it seems unrelated to the changes we have landed in 53.

Patrick, Nick, Dragana, h2 is your area, can you look at this please?

Comment 19

[Patrick McManus [:mcmanus]]

tony lays it out correctly in comment 5

"NTLM isn't compatible with HTTP/2 due to the connection-oriented nature of NTLM authentication and the potential ambiguity of the user's identity when multiplexing HTTP/2 streams over a single TCP connection."

So I think this is primarily a sever bug - asking for NTLM over h2.

"If NTLM authentication will be performed, the connection needs to be downgraded from HTTP/2 to HTTP/1.x.

I'm not sure where that 'needs' language comes from - the server has asked for something rather impossible. I do think that the client could see it coming however, and I'd take a patch for this.

maybe nick would write something.

not a regression.

Comment 20

[Honza Bambas (:mayhemer)]

Thanks Patrick, I completely missed comment 5.

Do you think we can do something on our side?

Comment 21

[Patrick McManus [:mcmanus]]

Yeah.. I think we could mark the origin as non-h2 capable and retry the request (as is needed by the auth failure anyhow) on h1 as suggested. I don't really think we're in violation of anything by not doing it, but its a reasonable approach.

this would be a new feature - so we shouldn't track this as a bug or regression. Easier to fix it on the server imo - but maximum interop is always a goal.

Comment 22

[Troy Starr]

(In reply to Patrick McManus [:mcmanus] from comment #21)
> Yeah.. I think we could mark the origin as non-h2 capable and retry the
> request (as is needed by the auth failure anyhow) on h1 as suggested. I
> don't really think we're in violation of anything by not doing it, but its a
> reasonable approach.
> 
> this would be a new feature - so we shouldn't track this as a bug or
> regression. Easier to fix it on the server imo - but maximum interop is
> always a goal.

Patrick, I'm not sure I follow your line of reasoning.  This scenario was working fine with Firefox 51.0, including the HTTP/2 to HTTP/1.x downgrade.  The scenario only started failing with Firefox 52.0.  Is that not the definition of a regression?

Also note that I make no claim that either Firefox or IIS is at fault.  Since the requests and responses are being sent over an encrypted connection and I can't see the unencrypted application data being sent, it's unclear to me where the problem lies.  I'm only pointing out the behavior change when setting the http2 preference in Firefox 52+ and mentioning the known compatibility issue between NTLM and HTTP/2.

The Mozilla team is of course welcome to approach this bug report as they wish, but I think a more productive approach would be to identify the change that caused this regression and then try to understand why it broke the scenario.  Since we know that there was at least one NTLM-focused change in the regression range reported by the mozregression tool, that would seem like a prime candidate to start with.

Comment 23

[u408661]

We can probably detect if the server wants NTLM and we're using h2, then shutdown the connection (potentially with a nasty error code, since the server did something wrong) and dial back using h1, while marking the host as h1-only (at least for the browser session). We already have support for the re-dialing, so it shouldn't be too absurd to get this done. I'll put it in my queue.

Comment 24

[Troy Starr]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #23)
> We can probably detect if the server wants NTLM and we're using h2, then
> shutdown the connection (potentially with a nasty error code, since the
> server did something wrong) and dial back using h1, while marking the host
> as h1-only (at least for the browser session). We already have support for
> the re-dialing, so it shouldn't be too absurd to get this done. I'll put it
> in my queue.

OK, but I feel the need to stress one last time that I don't think we have evidence that Firefox had a long-term issue with downgrading HTTP/2 connections for NTLM.  Firefox was already downgrading HTTP/2 connections to HTTP/1.x for NTLM authentication pre-52, and that was working fine.  Something changed with Firefox 52.

Comment 25

[Honza Bambas (:mayhemer)]

Patrick, this is a regressions (comment 22).

Comment 26

[Patrick McManus [:mcmanus]]

if it ever worked, it was a race condition.. or perhaps the problem is misdiagnosed?

Comment 27

[Patrick McManus [:mcmanus]]

anyhow - this isn't a sensible thing for the server to be asking for, and nick's going to write some code to do the best we can - but that's not fixing a bug, so not much more to discuss..

Comment 28

[Patrick McManus [:mcmanus]]

(In reply to Troy Starr from comment #24)
>Firefox was already downgrading HTTP/2 connections to
> HTTP/1.x for NTLM authentication pre-52, and that was working fine. 
> Something changed with Firefox 52.

do you feel confident that it was downgrading them to h1 or is it possible the use case was somehow working in another way?

Comment 29

[Troy Starr]

Attachment: IIS 10 log - Firefox 51.0 repro steps

Comment 30

[Troy Starr]

Attachment: IIS 10 log - Firefox 52.0 repro steps

Comment 31

[Troy Starr]

(In reply to Patrick McManus [:mcmanus] from comment #28)
> do you feel confident that it was downgrading them to h1 or is it possible
> the use case was somehow working in another way?

I feel confident that the connections were getting downgraded from HTTP/2 to HTTP/1.1 in Firefox 51 based on the IIS logs from the repro environment (see attachments).  The first anonymous request/response is via HTTP/2, but the next two request/response pairs for NTLM authentication are via HTTP/1.1.  It's not clear to me whether the switch to HTTP/1.1 was initiated by the client or the server from these logs.

In the Firefox 52 case, the IIS logs just show the first anonymous request/response via HTTP/2.  It's not showing the subsequent expected request/response pairs for NTLM authentication.  I don't know if that means the request never got sent in the first place or if the server never responded.

Comment 32

[Dragana Damjanovic [:dragana]]

Can you make a http logging for both Firefox versions 51 and 52.

You can fine details at:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging


thanks

Comment 33

[Honza Bambas (:mayhemer)]

(In reply to Dragana Damjanovic [:dragana] from comment #32)
> Can you make a http logging for both Firefox versions 51 and 52.
> 
> You can fine details at:
> https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging
> 
> 
> thanks

And please add negotiateauth:5 to the list of modules (MOZ_LOG).  Thanks.

Comment 34

[Honza Bambas (:mayhemer)]

And BTW, I still believe this is cased by the changes from bug 486508, which changes how we manage connections bound to NTLM.  I believe that the bug actually allowed the downgrade - it was not a feature, we have had a code to explicitly downgrade from h2 to h1.  But now the connection is bound tight to the first 401/NTLM response, so that downgrade is not possible.

Comment 35

[Troy Starr]

Attachment: Firefox direct NTLM network logs

Hi folks, Firefox network logs attached.

Since my repro environment is a production SharePoint server on our corporate network, I've decided to set up a separate test environment so I won't have to redact data in these logs.  I've set up a minimal repro web site on IIS 10 that requires NTLM authentication.  SharePoint is not installed on this test environment because its presence is not required to repro the bug.

Looking at the logs, it appears that both Firefox and IIS are behaving correctly in terms of what's getting sent over the wire.  This includes both Firefox 51 and 52.  In both cases, Firefox starts by sending an anonymous GET request over HTTP/2:

GET /
upgrade-insecure-requests: 1
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.5
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0
authority: ntlm.troystarr.net

IIS correctly returns an HTTP 401 status over HTTP/2 indicating that this resource requires authentication, and letting the client know that it supports NTLM authentication.

HTTP/2.0 401
content-type: text/html
server: Microsoft-IIS/10.0
www-authenticate: NTLM
date: Fri, 17 Mar 2017 09:21:18 GMT
content-length: 1293
X-Firefox-Spdy: h2

Both versions of Firefox then begin the NTLM handshake over HTTP/2 by resending the request with an Authorization header:

GET /
Host: ntlm.troystarr.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Authorization: NTLM ********************************************************

Here's where we diverge.  In Firefox 52, the network log doesn't contain a response from IIS.  The connection seems to terminate at that point.

But in Firefox 51, the network log shows that IIS responds correctly with a downgrade to HTTP/1.1 and the NTLM handshake completes as expected:

HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/10.0
WWW-Authenticate: NTLM TlRMTVNTUAACAAAADgAOADgAAAAFgomiCYU/hlZoe6wAAAAAAAAAAJgAmABGAAAACgA5OAAAAA9GAE8AVQBOAEQAUgBZAAIADgBGAE8AVQBOAEQAUgBZAAEADgBXAEkATgAyADAAMQA2AAQAGABmAG8AdQBuAGQAcgB5AC4AaABvAG0AZQADACgAVwBpAG4AMgAwADEANgAuAGYAbwB1AG4AZAByAHkALgBoAG8AbQBlAAUAGABmAG8AdQBuAGQAcgB5AC4AaABvAG0AZQAHAAgA9KzV2/+e0gEAAAAA
Date: Fri, 17 Mar 2017 09:21:28 GMT
Content-Length: 1293

GET / HTTP/1.1
Host: ntlm.troystarr.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Authorization: NTLM ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Fri, 17 Mar 2017 08:59:51 GMT
Accept-Ranges: bytes
Etag: "1e32b2d6fc9ed21:0"
Server: Microsoft-IIS/10.0
Persistent-Auth: true
Date: Fri, 17 Mar 2017 09:21:28 GMT
Content-Length: 80

Comment 36

[Troy Starr]

(In reply to Honza Bambas (:mayhemer) from comment #34)
> And BTW, I still believe this is cased by the changes from bug 486508, which
> changes how we manage connections bound to NTLM.  I believe that the bug
> actually allowed the downgrade - it was not a feature, we have had a code to
> explicitly downgrade from h2 to h1.  But now the connection is bound tight
> to the first 401/NTLM response, so that downgrade is not possible.

That seems likely to me as well.

It's worth mentioning that it's correct for the very first 401 response to the first anonymous request to remain HTTP/2.  Even though the server adds the "WWW-Authenticate: NTLM" header to the response, you're not actually performing NTLM authentication yet.  The server could have advertised support for other forms of authentication like Basic or Digest alongside NTLM in that response, and it would premature to downgrade the connection at that point since the server doesn't know what the client is going to do next.

The NTLM auth handshake only begins when the client sends a request with the Authorization: NTLM header.  At that point, the server knows you're performing NTLM authentication and can downgrade the connection to HTTP/1.1.

Comment 37

[Dragana Damjanovic [:dragana]]

(In reply to Troy Starr from comment #35)
> Created attachment 8848468 [details]
> Firefox network logs
> 
> Hi folks, Firefox network logs attached.
> 
> Since my repro environment is a production SharePoint server on our
> corporate network, I've decided to set up a separate test environment so I
> won't have to redact data in these logs.  I've set up a minimal repro web
> site on IIS 10 that requires NTLM authentication.  SharePoint is not
> installed on this test environment because its presence is not required to
> repro the bug.
> 
> Looking at the logs, it appears that both Firefox and IIS are behaving
> correctly in terms of what's getting sent over the wire.  This includes both
> Firefox 51 and 52.  In both cases, Firefox starts by sending an anonymous
> GET request over HTTP/2:
> 
> GET /
> upgrade-insecure-requests: 1
> accept-encoding: gzip, deflate, br
> accept-language: en-US,en;q=0.5
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0)
> Gecko/20100101 Firefox/51.0
> authority: ntlm.troystarr.net
> 
> IIS correctly returns an HTTP 401 status over HTTP/2 indicating that this
> resource requires authentication, and letting the client know that it
> supports NTLM authentication.
> 
> HTTP/2.0 401
> content-type: text/html
> server: Microsoft-IIS/10.0
> www-authenticate: NTLM
> date: Fri, 17 Mar 2017 09:21:18 GMT
> content-length: 1293
> X-Firefox-Spdy: h2
> 
> Both versions of Firefox then begin the NTLM handshake over HTTP/2 by
> resending the request with an Authorization header:
> 
> GET /
> Host: ntlm.troystarr.net
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0)
> Gecko/20100101 Firefox/51.0
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate, br
> Connection: keep-alive
> Upgrade-Insecure-Requests: 1
> Authorization: NTLM ********************************************************
> 
> Here's where we diverge.  In Firefox 52, the network log doesn't contain a
> response from IIS.  The connection seems to terminate at that point.
> 

This request is sent in 52 and in 51 over H2, but in both cases the request stream is reset and the transactions close with 0x804b0014 which is NS_ERROR_NET_RESET. In 51 the transaction is restarted:

017-03-17 09:21:28.912000 UTC - [Socket Thread]: D/nsHttp nsHttpTransaction::Close [this=16bd19dac00 reason=804b0014]
2017-03-17 09:21:28.912000 UTC - [Socket Thread]: D/nsHttp restarting transaction @16bd19dac00
2017-03-17 09:21:28.912000 UTC - [Socket Thread]: V/nsHttp nsHttpConnectionMgr::AddTransaction [trans=16bd19dac00 -10]

and in 52 it is just closed:

2017-03-17 09:45:07.774000 UTC - [Socket Thread]: D/nsHttp nsHttpTransaction::Close [this=23880f9c400 reason=804b0014]
2017-03-17 09:45:07.774000 UTC - [Socket Thread]: D/nsHttp nsHttpTransaction 23880f9c400 request context set to null in ReleaseBlockingTransaction() - was 238958fb660

....

2017-03-17 09:45:07.791000 UTC - [Main Thread]: D/nsHttp nsHttpTransaction::DeleteSelfOnConsumerThread [this=23880f9c400]
2017-03-17 09:45:07.791000 UTC - [Main Thread]: D/nsHttp Destroying nsHttpTransaction @23880f9c400


I will look into why it is not restarted.

Comment 38

[Dragana Damjanovic [:dragana]]

I think something with NS_HTTP_STICKY_CONNECTION has changed therefore bug 1337826 started to appear more often in 52 (1337826 is about Pipelining therefore I have not dig into thinks deeper there :) )

Comment 39

[Dragana Damjanovic [:dragana]]

Bug 486508 landed in 52 and it is the reason for this :)

Comment 40

[Dragana Damjanovic [:dragana]]

We should not dispatch a NTLM negotiation transaction over h2. That would fix this.

Comment 41

[Dragana Damjanovic [:dragana]]

(In reply to Dragana Damjanovic [:dragana] from comment #38)
> I think something with NS_HTTP_STICKY_CONNECTION has changed therefore bug
> 1337826 started to appear more often in 52 (1337826 is about Pipelining
> therefore I have not dig into thinks deeper there :) )

Just for completeness: but 486508 doe not influence 1337826.

Comment 42

[Mikhaylo]

Hello.
I have the same issue with SharePoint2016
all works on 51.* and early
Doesn't work on 52.*

All works fine after I set the network.http.spdy.enabled.http2 preference to "false".
But we have a lot people in our network, so this solving is not comfortable for us.
Waiting for fix.

Comment 43

[Liz Henry (:lizzard) (relman/hg->git project)]

If we can fix this I'm happy to take the patch in 53 (and maybe for esr52.1)
53 release date is April 18th. 

If we need to push out a fix for 52 before then, we might be able to do it through a system add-on. 
Please let Julien know about 52 & esr.

Comment 44

[Kohei Yoshino [:kohei]]

Found another report on SUMO: https://support.mozilla.org/t5/Firefox/As-of-today-Firefox-won-t-let-me-access-SharePoint-2016-Get-401/m-p/1380115

Comment 45

[Dragana Damjanovic [:dragana]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #43)
> If we can fix this I'm happy to take the patch in 53 (and maybe for esr52.1)
> 53 release date is April 18th. 
> 
> If we need to push out a fix for 52 before then, we might be able to do it
> through a system add-on. 
> Please let Julien know about 52 & esr.

Nick, can you make the patch? You only need to mark such transaction as disable-spdy.

Comment 46

[Troy Starr]

Attachment: Firefox NTLM via Negotiate network logs

Comment 47

[Troy Starr]

Attachment: Firefox Kerberos via Negotiate network logs

Comment 48

[Troy Starr]

I've tested some additional, related authentication scenarios:

1. NTLM authentication via Negotiate (i.e. no Service Principal Name exists, so the client should perform NTLM authentication via Negotiate)
2. Kerberos authentication via Negotiate (i.e. a Service Principal Name exists, so the client should perform Kerberos authentication via Negotiate)

These additional authentication scenarios also suffer from the same bug as direct NTLM authentication.  They all work fine with HTTP/2 over TLS 1.2 in Firefox 51.0, but they all fail with HTTP/2 over TLS 1.2 in Firefox 52.0.  If I set the network.http.spdy.enabled.http2 preference to "false" in Firefox 52.0, then they all work again.

I've uploaded new attachments if you'd like to see the networking logs for these scenarios.

Hopefully this helps ensure any patch created to fix this bug considers all of these authentication scenarios.

Comment 49

[u408661]

Attachment: Bug 1346392 - force non-spdy on sticky auth connections.

Review commit: https://reviewboard.mozilla.org/r/124490/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/124490/

Comment 50

[Dragana Damjanovic [:dragana]]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

https://reviewboard.mozilla.org/r/124490/#review127466

::: netwerk/protocol/http/nsHttpChannel.cpp:5622
(Diff revision 1)
>      }
>  
> +    // This is going to be connection-based auth, and we can't do that over spdy
> +    // connections, so just go ahead and disable spdy on our transaction.
> +    mTransaction->DisableSpdy();
> +

This is wrong place to disableSpdy. CloseStickyConnection is called on auth errors not in the normal behavior. Also there is no CloseStickyConnection in the log that is attached by the reporter. Have you tested this?

Comment 51

[u408661]

(In reply to Dragana Damjanovic [:dragana] from comment #50)
> Have you tested this?

No, and this is our problem with NTLM - we have no support for testing it, at all. So we keep having these bugs that totally break NTLM which we could've caught, but never do.

I'll go back through the auth code again and try to find the correct place.

Comment 52

[Dragana Damjanovic [:dragana]]

search for authRetry in nsHttpChannel.

also maybe you need to unset NS_HTTP_STICKY_CONNECTION.

Comment 53

[Troy Starr]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #51)
> (In reply to Dragana Damjanovic [:dragana] from comment #50)
> > Have you tested this?
> 
> No, and this is our problem with NTLM - we have no support for testing it,
> at all. So we keep having these bugs that totally break NTLM which we
> could've caught, but never do.
> 
> I'll go back through the auth code again and try to find the correct place.

Let me know if you need help testing a fix.  I can test a private build in my repro environment if you'd like.

Comment 54

[Patrick McManus [:mcmanus]]

(In reply to Troy Starr from comment #53)
> (In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail])
> from comment #51)
> > (In reply to Dragana Damjanovic [:dragana] from comment #50)
> > > Have you tested this?
> > 
> > No, and this is our problem with NTLM - we have no support for testing it,
> > at all. So we keep having these bugs that totally break NTLM which we
> > could've caught, but never do.
> > 
> > I'll go back through the auth code again and try to find the correct place.
> 
> Let me know if you need help testing a fix.  I can test a private build in
> my repro environment if you'd like.

thank you!

Comment 56

[Kohei Yoshino [:kohei]]

What's the status here? Beta->Release merge is next Monday, so the patch has to be baked and landed ASAP.

Comment 57

[u408661]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/124490/diff/1-2/

Comment 58

[Honza Bambas (:mayhemer)]

Nit: can you add a comment why we disallow h2 and why here exactly?  Also would be good to note in that comment that when the connection is already h2-disabled the call is a no-op.  thanks.

Comment 59

[Dragana Damjanovic [:dragana]]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

https://reviewboard.mozilla.org/r/124490/#review129382

::: netwerk/protocol/http/nsHttpChannel.cpp:5641
(Diff revision 2)
> +    mCaps |= NS_HTTP_DISALLOW_SPDY;
> +
> +    if (!(mTransaction->Caps() & NS_HTTP_DISALLOW_SPDY)) {
> +        mTransaction->DisableSpdy();
> +        // Force the transaction to restart
> +        mTransaction->Close(NS_ERROR_NET_RESET);

I think you should not restart transaction here. It will be restarted later be the nsHttpConnection.

I think the code path where ForceNoSpdy is called is also the success path and we do not want to restart transaction for that. Although this will be called only if spdy is not disallowed and it will be by the first 401 response.

Can you please make a build and ask Troy Starr to try it?

Comment 60

[Dragana Damjanovic [:dragana]]

You should also remove NS_HTTP_STICKY_CONNECTION. This is set later, after GetCredentialForChallenge, and it is going to be set (I think even with your change) and the transaction is going to keep the same connection. I think we will never check if spdy is disabled and that connection is h2, it is just going to be dispatch.

NS_HTTP_STICKY_CONNECTION is set in nsHttpChannel::DoAuthRetry

Comment 61

[Liz Henry (:lizzard) (relman/hg->git project)]

This missed the 53 RC build today. We could still potentially take a patch for 53 if we (maybe) do an RC2 build later in the week. But the timing is really tight. 
 
I'd like to see this some get testing before we land on release. Hopefully Nick can work with Troy or one of the other people who reported this problem  (in comment 42 and comment 44).

Comment 62

[u408661]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/124490/diff/2-3/

Comment 63

[u408661]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

https://reviewboard.mozilla.org/r/124490/#review129382

> I think you should not restart transaction here. It will be restarted later be the nsHttpConnection.
> 
> I think the code path where ForceNoSpdy is called is also the success path and we do not want to restart transaction for that. Although this will be called only if spdy is not disallowed and it will be by the first 401 response.
> 
> Can you please make a build and ask Troy Starr to try it?

So I kept the close here - we know we're going to need to at this point, so why drag things out? Plus, we'll only close if we had enabled spdy to begin with. I did, however, put a check in to ensure we don't set STICKY_CONNECTION.

Comment 64

[u408661]

Troy, once the builds with the latest patch are done in a few hours (or whenever you're next available), could you try one of them from https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-4389f6c913d5ee922e8115767b6d4ab0dc182c65/ to ensure the issue is fixed? Thanks.

Comment 65

[Troy Starr]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #64)
> Troy, once the builds with the latest patch are done in a few hours (or
> whenever you're next available), could you try one of them from
> https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-
> 4389f6c913d5ee922e8115767b6d4ab0dc182c65/ to ensure the issue is fixed?
> Thanks.

You bet - I'll check back in a couple hours to see if the test build is ready.  If so, I'll test the fix with my repro environment.

Comment 66

[Troy Starr]

I used my repro environment to test Nick's fix identified as:

20170410171034
https://hg.mozilla.org/try/rev/4389f6c913d5ee922e8115767b6d4ab0dc182c65

SUMMARY:

The testing was successful - the fix addressed the bug and no other regressions were found.  I was able to successfully browse to content using NTLM and Kerberos authentication, even if both the client and server were configured to support HTTP/2 over TLS 1.2.  Firefox successfully downgraded the connection to HTTP/1.1 and was able to retrieve the resources.

DETAILS:

+----------------------+------------------------+--------------------+------------------+
| AuthN / Connection   | HTTP/1.1 (unencrypted) | HTTP/1.1 (TLS 1.2) | HTTP/2 (TLS 1.2) |
+----------------------+------------------------+--------------------+------------------+
| Anonymous            | PASS (1)               | PASS (1)           | PASS (2)         |
+----------------------+------------------------+--------------------+------------------+
| Basic                | PASS (1)               | PASS (1)           | PASS (2)         |
+----------------------+------------------------+--------------------+------------------+
| Digest (MD5-sess)    | FAIL (3)               | FAIL (3)           | FAIL (4)         |
+----------------------+------------------------+--------------------+------------------+
| Kerberos (Negotiate) | PASS (5)               | PASS (5)           | PASS (6)         |
+----------------------+------------------------+--------------------+------------------+
| NTLM (Direct)        | PASS (5)               | PASS (5)           | PASS (7)         |
+----------------------+------------------------+--------------------+------------------+
| NTLM (Negotiate)     | PASS (5)               | PASS (5)           | PASS (7)         |
+----------------------+------------------------+--------------------+------------------+

1: The requests were initiated as HTTP/1.1, and the responses were successfully received as HTTP/1.1 as expected.

2: The requests were initiated as HTTP/2, and the responses were successfully received as HTTP/2 as expected.

3: The initial request was initiated as HTTP/1.1, and the response was successfully received as HTTP/1.1 as expected.  However, when accessing additional resources defined in the HTML page (images), the client included an Authorization header as HTTP/1.1 that was rejected by the server with an HTTP 401 response.  This resulted in a new authentication prompt to the end user.  When the end user provided the same credentials again, the request was retried and was successful.  I believe this is a known issue, tracked by Bugzilla bug 270447.  This failure is unrelated to this change.

4: The initial request was initiated as HTTP/2, and the response was successfully received as HTTP/2 as expected.  However, when accessing additional resources defined in the HTML page (images), the client included an Authorization header as HTTP/2 that was rejected by the server with an HTTP 401 response.  This resulted in a new authentication prompt to the end user.  When the end user provided the same credentials again, the request was retried and was successful.  I believe this is a known issue, tracked by Bugzilla bug 270447.  This failure is unrelated to this change.

5: The requests were initiated as HTTP/1.1, and the responses were successfully received as HTTP/1.1 as expected.  In addition, the existing connection was reused when requesting additional content, so no further re-authentication attempts were necessary.

6: The initial request was initiated as HTTP/2, and the initial HTTP 401 response was successfully received as HTTP/2.0 as expected.  The client then resent the request with an Authorization header over HTTP/1.1 as expected.  The content was received over HTTP/1.1 as expected.  However, when requesting additional resources defined in the HTML page (images), the client initiated a new HTTP/2 connection rather than reusing the existing HTTP/1.1 connection.  All resources were successfully retrieved, but at a cost of repeating the anonymous -> authenticated handshake for each additional resource.

7: The requests were initiated as HTTP/2, and the initial HTTP 401 response was successfully received as HTTP/2 as expected.  The client then resent the request with an Authorization header over HTTP/1.1 as expected.  The remaining steps of the NTLM handshake were completed over HTTP/1.1 as expected and the content was received over HTTP/1.1 as expected.  However, when requesting additional resources defined in the HTML page (images), the client initiated a new HTTP/2 connection rather than reusing the existing HTTP/1.1 connection.  All resources were successfully retrieved, but at a cost of repeating the NTLM handshake for each additional resource.

Comment 67

[Dragana Damjanovic [:dragana]]

Troy, thank you very very much for testing!!!!!
Very appreciated.

Comment 68

[Dragana Damjanovic [:dragana]]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

https://reviewboard.mozilla.org/r/124490/#review131366

Comment 69

[Dragana Damjanovic [:dragana]]

Nick, can you make a patch for case 6 and 7? We need to make connection entry not use h2.
We could open another bug for this, since we should act fast on this regression so it can make it in 53. But if you can extend this patch, I am fine with that too.

Comment 70

[u408661]

Troy, thanks for all the testing!

Dragana - I'll open a bug for that followup. I'm fine with sub-optimal behaviour, at least we have the regression fixed, and I want to get it landed ASAP to increase our chances of making it into 53.

Comment 71

[u408661]

Ugh, try failures (assertions) in test_authentication.js, definitely related to this patch. Back to ye olde drawing board.

Comment 72

[u408661]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/124490/diff/3-4/

Comment 73

[u408661]

OK, just pushed another version. I should've listened to Dragana and not force-closed the transaction in ForceNoSpdy - that happens on the main thread, but nsHttpTransaction::Close needs to be called on the socket thread.

Troy, sorry to keep going around on this, but would it be possible for you to test new builds to ensure the removal of the force-close keeps things working? Once again, they'll be available in a few hours, this time at https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-d04b75958c69ffe88e1d2427b6c8916d58ac0842/

Comment 74

[Troy Starr]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #73)
> OK, just pushed another version. I should've listened to Dragana and not
> force-closed the transaction in ForceNoSpdy - that happens on the main
> thread, but nsHttpTransaction::Close needs to be called on the socket thread.
> 
> Troy, sorry to keep going around on this, but would it be possible for you
> to test new builds to ensure the removal of the force-close keeps things
> working? Once again, they'll be available in a few hours, this time at
> https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-
> d04b75958c69ffe88e1d2427b6c8916d58ac0842/

No problem, I'll look for the new build to test in a couple hours.

Comment 75

[Troy Starr]

Attachment: Firefox Nightly bug fix test network logs

I used my repro environment to test Nick's fix identified as:

20170411094441
https://hg.mozilla.org/try/rev/d04b75958c69ffe88e1d2427b6c8916d58ac0842

SUMMARY:

Like last time, the testing was successful - the fix addressed the bug and no other regressions were found.  I was able to successfully browse to content using NTLM and Kerberos authentication, even if both the client and server were configured to support HTTP/2 over TLS 1.2.  Firefox successfully downgraded the connection to HTTP/1.1 and was able to retrieve the resources.

I've uploaded Firefox Nightly network logs for the NTLM and Kerberos tests in case anyone wants to review them.

DETAILS:

+----------------------+------------------------+--------------------+------------------+
| AuthN / Connection   | HTTP/1.1 (unencrypted) | HTTP/1.1 (TLS 1.2) | HTTP/2 (TLS 1.2) |
+----------------------+------------------------+--------------------+------------------+
| Anonymous            | Pass [1]               | Pass [1]           | Pass [2]         |
+----------------------+------------------------+--------------------+------------------+
| Basic                | Pass [1]               | Pass [1]           | Pass [2]         |
+----------------------+------------------------+--------------------+------------------+
| Digest (MD5-sess)    | Fail [3]               | Fail [3]           | Fail [4]         |
+----------------------+------------------------+--------------------+------------------+
| Kerberos (Negotiate) | Pass [5]               | Pass [5]           | Pass [6]         |
+----------------------+------------------------+--------------------+------------------+
| NTLM (Direct)        | Pass [5]               | Pass [5]           | Pass [7]         |
+----------------------+------------------------+--------------------+------------------+
| NTLM (Negotiate)     | Pass [5]               | Pass [5]           | Pass [7]         |
+----------------------+------------------------+--------------------+------------------+

1: The requests were initiated as HTTP/1.1, and the responses were successfully received as HTTP/1.1 as expected.

2: The requests were initiated as HTTP/2, and the responses were successfully received as HTTP/2 as expected.

3: The initial request was initiated as HTTP/1.1, and the response was successfully received as HTTP/1.1 as expected.  However, when accessing additional resources defined in the HTML page (images), the client included an Authorization header as HTTP/1.1 that was rejected by the server with an HTTP 401 response.  This resulted in a new authentication prompt to the end user.  When the end user provided the same credentials again, the request was retried and was successful.  I believe this is a known issue, tracked by Bugzilla bug 270447.  This failure is unrelated to this change.

4: The initial request was initiated as HTTP/2, and the response was successfully received as HTTP/2 as expected.  However, when accessing additional resources defined in the HTML page (images), the client included an Authorization header as HTTP/2 that was rejected by the server with an HTTP 401 response.  This resulted in a new authentication prompt to the end user.  When the end user provided the same credentials again, the request was retried and was successful.  I believe this is a known issue, tracked by Bugzilla bug 270447.  This failure is unrelated to this change.

5: The requests were initiated as HTTP/1.1, and the responses were successfully received as HTTP/1.1 as expected.  In addition, the existing connection was reused when requesting additional content, so no further re-authentication attempts were necessary.

6: The initial request was initiated as HTTP/2, and the initial HTTP 401 response was successfully received as HTTP/2 as expected.  The client then resent the request with an Authorization header over HTTP/1.1 as expected.  The content was received over HTTP/1.1 as expected.  However, when requesting additional resources defined in the HTML page (images), the client initiated a new HTTP/2 connection rather than reusing the existing HTTP/1.1 connection.  All resources were successfully retrieved, but at a cost of repeating the anonymous -> authenticated handshake for each additional resource.

7: The requests were initiated as HTTP/2, and the initial HTTP 401 response was successfully received as HTTP/2 as expected.  The client then resent the request with an Authorization header over HTTP/1.1 as expected.  The remaining steps of the NTLM handshake were completed over HTTP/1.1 as expected and the content was received over HTTP/1.1 as expected.  However, when requesting additional resources defined in the HTML page (images), the client initiated a new HTTP/2 connection rather than reusing the existing HTTP/1.1 connection.  All resources were successfully retrieved, but at a cost of repeating the NTLM handshake for each additional resource.

Comment 76

[u408661]

Thanks so much, Troy. Looks like this is good to land. I've filed a followup (bug 1355858) to fix the reauths you noted in 6 and 7 above. We'll have to ask :lizzard about making this into 53, but based on my discussions with her previously, it doesn't look good at this point.

Comment 77

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88f793b6a03c
force non-spdy on sticky auth connections. r=dragana

Comment 78

[Troy Starr]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #76)
> Thanks so much, Troy. Looks like this is good to land. I've filed a followup
> (bug 1355858) to fix the reauths you noted in 6 and 7 above. We'll have to
> ask :lizzard about making this into 53, but based on my discussions with her
> previously, it doesn't look good at this point.

Thanks Nick, I appreciate everyone's engagement to fix this issue.  It would be good to pursue porting this fix to Firefox 52 ESR as well.

Comment 79

[u408661]

(In reply to Troy Starr from comment #78)
> Thanks Nick, I appreciate everyone's engagement to fix this issue.  It would
> be good to pursue porting this fix to Firefox 52 ESR as well.

Definitely. Once the patch hits nightly, I'll be requesting uplifts to all appropriate branches (including esr52).

Comment 80

[Honza Bambas (:mayhemer)]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #79)
> appropriate branches (including esr52).

You may have a hard time to persuade the esr drivers to include this.  not a sec/crash fix.  but still worth a try, this bug is mainly affecting users of esr and release branches.

Comment 81

[u408661]

(In reply to Honza Bambas (:mayhemer) from comment #80)
> (In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail])
> from comment #79)
> > appropriate branches (including esr52).
> 
> You may have a hard time to persuade the esr drivers to include this.  not a
> sec/crash fix.  but still worth a try, this bug is mainly affecting users of
> esr and release branches.

True. That's why I said I'd request. Can't promise any approvals :)

Comment 82

[Patrick McManus [:mcmanus]]

ntlm is largely an enterprise feature, so its a significant esr52 regression fix.. my .02 says that's worth the backport.

Comment 83

[u408661]

Comment on attachment 8852629 [details]
Bug 1346392 - force non-spdy on sticky auth connections.

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
User impact if declined: Fixes some NTLM breakage, and NTLM is exceedingly common in enterprises.
Fix Landed on Version: nightly 55
Risk to taking this patch (and alternatives if risky): low
String or UUID changes made by this patch: none

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.


Approval Request Comment
[Feature/Bug causing the regression]: 486508
[User impact if declined]: inability to use NTLM with servers that speak HTTP/2
[Is this code covered by automated tests?]: no (none of our ntlm stuff is)
[Has the fix been verified in Nightly?]: no (but has been verified from try builds)
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: not particularly
[Why is the change risky/not risky?]: small fix targeting ntlm auth connections only
[String changes made/needed]: none

Comment 84

[Liz Henry (:lizzard) (relman/hg->git project)]

OK, I'd like to get this into a 53 RC2 build. It sounds like it affects many enterprise users (ESR and release) and our fallback behavior is incorrect. We can follow up on bug 1355858 later for the issues still outstanding (since the followup bug doesn't sound worse than what we're already shipping).
I agree this is also worth bringing to ESR52 especially since we are already planning an esr52 build 2.

Comment 85

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/a4b7a2a843e5

Comment 86

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/110f5d85f2ac
https://hg.mozilla.org/releases/mozilla-release/rev/110f5d85f2ac

Comment 87

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/955ee0340ab7

Comment 88

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/88f793b6a03c

Comment 89

[Andrei Vaida [:avaida]]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #83)
> [Needs manual test from QE? If yes, steps to reproduce]: no

Setting qe-verify- based on Nicholas' assessment on manual testing needs.

Comment 90

[Liz Henry (:lizzard) (relman/hg->git project)]

Since folks are asking, I want to make it clear this fix did land (on 4/12) for 52.1.0esr, which we built on 4/13.

Comment 91

[Krzysztof]

Since 4/13 on nightly I got specific issue. I am behind a proxy and I use to have manual configuration of it. Since 4/13 almost all requests are answered 407 (needs authentication) and barely none 200 nor 304 (not modified). But turning on automatic proxy configuration makes all requests answered 200 and 304. Could this patch be responsible for it? Thanks.

Comment 92

[Honza Bambas (:mayhemer)]

(In reply to Krzysztof from comment #91)
> Since 4/13 on nightly I got specific issue. I am behind a proxy and I use to
> have manual configuration of it. Since 4/13 almost all requests are answered
> 407 (needs authentication) and barely none 200 nor 304 (not modified). But
> turning on automatic proxy configuration makes all requests answered 200 and
> 304. Could this patch be responsible for it? Thanks.

Please file a new bug with all necessary details and make it blocking this one.  That is the best way for us to take a look at that issue.  Thanks.

Comment 93

[Troy Starr]

Although this bug is already fixed, I'm adding this comment in the interest of knowledge sharing about the HTTP/2 protocol.

The HTTP/2 protocol provides an official mechanism for the server to signal to the client that it should downgrade to HTTP/1.1.  Per https://tools.ietf.org/html/rfc7540#section-7, it can send a RST_STREAM frame in response to a request and set the error code within the frame to:

0xD HTTP_1_1_REQUIRED The endpoint requires that HTTP/1.1 be used instead of HTTP/2.

This is what IIS 10 on Windows Server 2016 does in response to an NTLM authentication attempt over HTTP/2.  Firefox 51.0 was handling that downgrade request from the server while Firefox 52.0 was aborting due to the NTLM-related code changes that were added.

Comment 94

[lgbmontenegroii]

Issue still exists on Firefox 54.

Summary:

We are using SQL Server Reporting Services 2016 in Native Mode, upon accessing the Web Portal, user will be presented
with the authentication dialog. After providing the credentials, Firefox will respond with a 'Secure Connection Failed' message. By setting 'network.http.spdy.enabled.http2' to false, the authentication will completed successfully.

Other browsers (Chrome, IE) are working properly.

Comment 95

[Honza Bambas (:mayhemer)]

Nick, comment 94?

Comment 96

[David]

I too am seeing the issue re-appearing in v53.0.3 (64-bit). I don't have a version before this so can't confirm when it re-appeared.

Comment 97

[u408661]

Yes, I'm aware that this continues to be an issue (for some people, but apparently not everyone) on release versions. However, nightly behaves just fine for me (where release doesn't - Dragana has confirmed this, as well, so it's not just me). There is the problem that I don't know *what* changed to make this work (which bears investigating), but given the (obvious) fragility of this code, I would be loath to backport the change, even if I knew what it was.

Comment 98

[Honza Bambas (:mayhemer)]

Thanks for the explanation Nick.  I agree.  We've just released 54, not sure what the situation is there.

Comment 99

[David]

For what its worth I tested other releases for a successful connection for Windows NTLM authentication to IIS 10/Win2016,

52.1.0 ESR Success
51.1.1 ESR Success
52.1.2 ESR Fails
52.2.0 ESR Fails

53.0.0 Success
53.0.2 Success
53.0.3 Fails
54.0b9 Fails
54.0 Fails

55.0b1 - Success
56.0a1 - Success

Comment 100

[Honza Bambas (:mayhemer)]

So 54 is broken as well.. hm...  there is not much to do with it now and frankly I would not uplift anything here.  This is sensitive.  

The 'success' results seem to be erratic, I think this issue is intermittent.

We've moved 55 from mozilla-central (Nightly) to mozilla-beta (Beta) few days ago.  According Nick's claim this was fixed on Nightly (assuming 55), I presume beta is in a good shape now, which is good enough for me.  Will move to Release in some 8 weeks.

Comment 101

[Troy Starr]

FYI, this issue no longer repro's for me in Firefox 55.0.1.  When connecting to a web server that supports HTTP/2 over TLS 1.2, but requires NTLM or Kerberos authentication, Firefox correctly falls back to HTTP/1.1 and completes the NTLM/Kerberos authentication successfully.

Comment 102

[Frank Taffelt]

FYI, this seems not really solved in some cases. We can reproduce the issue with current Firefox 55.0.3 (64bit).

If I set the "security.tls.version.max" preference in Firefox 55.0.3 to "2" (TLS 1.1) the authentication works correctly. will provide network logs as attachment.

Comment 103

[Frank Taffelt]

Attachment: Networking logs of failing NTLM requests in Firefox 55.0.3

Comment 104

[Frank Taffelt]

In my view this problem hasn't been solved yet. It would be great if you could bring this on the agenda. It's important for our product and i'm sure its relevant for other firefox users! in case this is the wrong ticket, please give me a pointer or suggest other steps.

Comment 105

[Honza Bambas (:mayhemer)]

(In reply to Frank Taffelt from comment #104)
> In my view this problem hasn't been solved yet. It would be great if you
> could bring this on the agenda. It's important for our product and i'm sure
> its relevant for other firefox users! in case this is the wrong ticket,
> please give me a pointer or suggest other steps.

Since this bug has been patched and marked fixed, can you rather open a new bug, provide all the necessary information like symptoms etc and retest with current nightly to eliminate that the bug has already been fixed in the meantime?  Logs from a more recent version would be good too.

Thanks.