1366824 - NTLM auth fails over HTTP/2

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Gary Pendlebury]

Attachment: Firefox log.txt-main.12784

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063

Steps to reproduce:

Since the update to Firefox 53.0.3, I've seen two NTLM-HTTP/2-related problems. These did not occur with 53.0.2, so I assume these problems are related to <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1360574">bug 1360574</a>.

Both issues only appear to occur when the pages are issued over HTTP/2.

1. Attempt to load a page that requires NTLM authentication, and a valid entry for the site exists in network.automatic-ntlm-auth.trusted-uris, but the Windows credentials aren't available to the browser (e.g. the PC is off-domain or accessing a page on another domain).

2. Attempt to load a page that requires NTLM authentication, without a valid entry for the site in network.automatic-ntlm-auth.trusted-uris, and the Windows credentials aren't available to the browser.


Actual results:

1. A "Secure Connection Failed" page is displayed, with no further information returned. It isn't possible to load the page.

2. A "Secure Connection Failed" page is displayed, with no further information returned. However, in this case, clicking on the "Report errors like this to help Mozilla identify and block malicious sites" tick-box and selecting "Try Again" causes the page to load. It doesn't matter whether the tick-box is on or off, or even whether the value changes: as long as it's clicked-on, the page will subsequently load.


Expected results:

1. The browser should have prompted for credentials and loaded the page. This is the behaviour that occurred with Firefox 53.0.2 and earlier.

2. The browser should have prompted for credentials and loaded the page. This is the behaviour that occurred with Firefox 53.0.2 and earlier. The browser also should not change its authentication behaviour based on the selection of the tick-box.

Comment 1

[Julien Cristau [:jcristau] (back April 22)]

ni? Nicholas who has been dealing with the recent NTLM/H2 bugs (first 1346392, then 1360574).

Comment 2

[Gary Pendlebury]

I can confirm that the problem doesn't occur when "network.http.spdy.enabled.http2" is set to false.

This appears to be related to bug 1346392. Was there a regression of this fix in 53.0.3?

Comment 3

[u408661]

So I was able to reproduce this yesterday using an old nightly build (possibly from last week, I don't know for sure). However, attempting to dig in further today with most recent nightly (as well as a local build from m-c tip), I have been unable to reproduce this. Honza, I know you've had your fingers in some NTLM-related stuff recently, too, have you landed anything in the last week or so NTLM-related? There could very likely be some strange interplay going on between our work.

Comment 4

[Honza Bambas (:mayhemer)]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #3)
> So I was able to reproduce this yesterday using an old nightly build
> (possibly from last week, I don't know for sure). 

Do you still have it?  If you have the cset I could say for sure.  Otherwise I need to dig in hg logs.

Comment 5

[Honza Bambas (:mayhemer)]

Only bug I can quickly find is bug 1345910, fixed 4-27.

Comment 6

[u408661]

Hrm... I don't still have the cset (I would've bisected easily, otherwise), and I don't *think* my old build was pre-april 27. I'll download some other old nightlies and see if I can repro/bisect.

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Too late to fix for 53, we can keep the bug open for 54/55 to see if anyone reproduces the issue.

Comment 8

[Troy Starr]

FWIW, I'm still able to reproduce this issue in the RTM release of Firefox 54.0.

Comment 9

[u408661]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=1346392#c97 - this appears to be fixed on nightly, but we are still unclear as to what changed that would make this work on nightly but not release. As such, I'm leaving this open to investigate/bisect what's going on, but there does not appear to be any action needed to fix the problem in the longer term.

Comment 11

[Gary Pendlebury]

From my testing with 54.0:

When the credentials are passed automatically from Windows, this problem no-longer exists. In this case, it appears that HTTP/1.1 is used. Is this correct though? Shouldn't HTTP/2 be used where it's available?

When credentials are manually entered in the browser, the problem still exists.

Comment 12

[Gary Pendlebury]

I've also tested with Nightly (56.0a1), and have found that when credentials are both passed through from Windows or manually entered in the browser, the problem no-longer exists. However, all connections use HTTP/1.1. It seems that HTTP/2 is not used, even when it's available.

Comment 13

[Honza Bambas (:mayhemer)]

(In reply to Gary Pendlebury from comment #12)
> I've also tested with Nightly (56.0a1), and have found that when credentials
> are both passed through from Windows or manually entered in the browser, the
> problem no-longer exists. However, all connections use HTTP/1.1. It seems
> that HTTP/2 is not used, even when it's available.

That is expected and correct.  Thanks for confirming this is fixed on Nightly.

Comment 14

[Gary Pendlebury]

In my testing of FF 55.0, this problem no-longer exists.

Comment 15

[Troy Starr]

(In reply to Gary Pendlebury from comment #14)
> In my testing of FF 55.0, this problem no-longer exists.

Same here - this issue no longer repro's for me in Firefox 55.0.1.  When connecting to a web server that supports HTTP/2 over TLS 1.2, but requires NTLM or Kerberos authentication, Firefox correctly falls back to HTTP/1.1 and completes the NTLM/Kerberos authentication successfully.

Comment 16

[Julien Cristau [:jcristau] (back April 22)]

Great news, thanks for testing.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Does something need to be fixed on ESR52 yet? Not really sure what the status is given the different bugs involved.

Comment 18

[u408661]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #17)
> Does something need to be fixed on ESR52 yet? Not really sure what the
> status is given the different bugs involved.

I don't believe so, from everything I can tell, this is covered by the fix to bug 1346392 (which is in esr52)

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Thanks for clarifying.

Comment 20

[Troy Starr]

(In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail]) from comment #18)
> (In reply to Ryan VanderMeulen [:RyanVM] from comment #17)
> > Does something need to be fixed on ESR52 yet? Not really sure what the
> > status is given the different bugs involved.
> 
> I don't believe so, from everything I can tell, this is covered by the fix
> to bug 1346392 (which is in esr52)

This might not be the case... I believe that fix was checked in for ESR 52.1, but according to bug 1346392 comment 99, this scenario broke again in later ESR 52.x releases:

> 52.1.0 ESR Success
> 51.1.1 ESR Success
> 52.1.2 ESR Fails
> 52.2.0 ESR Fails

Comment 21

[Chris Mills [:cmills]]

This is a regression, so I don't see why it needs documentation. Removing ddn

Comment 22

[Julien Cristau [:jcristau] (back April 22)]

(In reply to Troy Starr from comment #20)
> (In reply to Nicholas Hurley [:nwgh][:hurley] (also [e-mail])
> from comment #18)
> > (In reply to Ryan VanderMeulen [:RyanVM] from comment #17)
> > > Does something need to be fixed on ESR52 yet? Not really sure what the
> > > status is given the different bugs involved.
> > 
> > I don't believe so, from everything I can tell, this is covered by the fix
> > to bug 1346392 (which is in esr52)
> 
> This might not be the case... I believe that fix was checked in for ESR
> 52.1, but according to bug 1346392 comment 99, this scenario broke again in
> later ESR 52.x releases:
> 
> > 52.1.0 ESR Success
> > 51.1.1 ESR Success
> > 52.1.2 ESR Fails
> > 52.2.0 ESR Fails

[Assuming 51.1.1 above is 52.1.1, as there was no 51.1.1]
The one relevant change in 52.1.2 is bug 1360574 which backed out the initial change from bug 1346392 (which itself was in 52.1.0).

Comment 23

[u408661]

Ugh... so we have conflicting reports. Because over in bug 1360574, the patch which landed to replace the one that was backed out has verification from a few people (including me) that everything works fine in that state. But now we're hearing reports that things *don't* work. That suggests some sort of different configuration, either client or server side (or perhaps something else landed somewhere that made this work/not work appropriately). I'll try looking at this, but Jason, any word on getting someone more knowledgeable about NTLM? This doesn't look likely to go away entirely without more ntlm expertise...

Comment 24

[Troy Starr]

I just tried testing this scenario with Firefox 55.0.3 and Firefox 52.3.0 ESR.  Firefox 55.0.3 works fine.

Firefox 52.3.0 ESR still repros the bug when you don't have the DNS domain name in your network.automatic-ntlm-auth.trusted-uris or network.negotiate-auth.trusted-uris preferences.  In that situation you're prompted for credentials, and once you provide them you get the "Secure Connection Failed" page.  If you have the appropriate DNS domain name in those preferences, then you're silently authenticated and the page renders successfully.