Since latest release NTLM/SPNEGO no longer works

RESOLVED FIXED in Firefox 49

Status

()

Core
Networking: HTTP
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: Rob Kleijwegt, Assigned: mayhemer)

Tracking

({regression})

48 Branch
mozilla51
regression
Points:
---

Firefox Tracking Flags

(firefox49 fixed, firefox50 fixed, firefox51 fixed)

Details

(Whiteboard: [necko-active][ntlm])

Attachments

(4 attachments)

(Reporter)

Description

2 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160726073904

Steps to reproduce:

Access some company websites.


Actual results:

I had configured Firefox for SSO to a couple of company websites. The configuration as I had it is still there, but since release 48 was installed I'm no longer able to watch the company websites.


Expected results:

The company websites (as configured for NTLM/SPNEGO) should have remained available.

Updated

2 years ago
Component: Untriaged → Networking: HTTP
Product: Firefox → Core
can you please create an http log 
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

and attach it to this bug?
(Assignee)

Comment 2

2 years ago
Jan, could this be bug 1248564?  Or anything you may know we have changed around NTLM in 48?
Flags: needinfo?(jhorak)

Comment 3

2 years ago
(In reply to Honza Bambas (:mayhemer) from comment #2)
> Jan, could this be bug 1248564?  Or anything you may know we have changed
> around NTLM in 48?
Hm, no idea. I didn't push any other code into 48. Is it possible to get some logs in Windows like when setting environment variable 'NSPR_LOG_MODULES=negotiateauth:5' in unix?

I'll try to get some ntlm/spnego server, but I'm unsure if I will be successful.
Flags: needinfo?(jhorak)
(Assignee)

Comment 4

2 years ago
(In reply to Jan Horak from comment #3)
> (In reply to Honza Bambas (:mayhemer) from comment #2)
> > Jan, could this be bug 1248564?  Or anything you may know we have changed
> > around NTLM in 48?
> Hm, no idea. I didn't push any other code into 48. Is it possible to get
> some logs in Windows like when setting environment variable
> 'NSPR_LOG_MODULES=negotiateauth:5' in unix?

Sure, it's multiplatform ;)  And now it's better to use MOZ_LOG env var to setup your modules.

We can also ask Rob (the reporter) to add this module to his log.

Rob, see also comment 1.

> 
> I'll try to get some ntlm/spnego server, but I'm unsure if I will be
> successful.
Flags: needinfo?(robkleijwegt)
(Reporter)

Comment 5

2 years ago
Created attachment 8777665 [details]
firefox-v48-log.zip

This is the log file I was asked for in the first comment. I tried to access our Topdesk system, which I was Always able to access without a problem, but doing that now brings me to a login screen where I can get no further. Of course I have restarted Firefox and even the computer, alas to no avail.
Attachment #8777665 - Flags: feedback+
(Reporter)

Comment 6

2 years ago
The entries to allow my access in the configuration are:
* network.automatic-ntlm-auth.trusted-uris: farmfrites.local,farmnet.farmfrites.com
* network.negotiate-auth.trusted-uris: servicedesk.farmfrites.com
This last entry refers to Topdesk. The first entries refer to intranet sites. All those sites are unavailable now.

Comment 7

2 years ago
Maybe David could help us with reproducer/server settings?

Comment 8

2 years ago
This is a Windows client, isn't it? So it'll be using SSPI for both NTLM and Negotiate(SPNEGO) authentication. That's mostly outside my knowledge (although I did 'accidentally' write that support for OpenConnect VPN once, and even fixed the interoperability between Wine's SSPI and Samba/winbind's ntlm_auth helper.)

Looking at the logs, it looks like the failing request was a Negotiate/SPNEGO authentication, at around 05:12:42.648000 ? The server sends a 401 with 'WWW-Authenticate: Negotiate', and we never attempt to do so. I note we *did* successfully use SPNEGO to authenticate to the proxy.

Can you obtain a ticket manually for this host? Is 'kvno' available for the Windows command line?
 kvno -S HTTP servicedesk.farmfrites.com

I know 'klist' is available for Windows. If you find and run it, does it show that you have a ticket for that host?

2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetAuthenticator [this=17375820 channel=21b21b9c]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquiring lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquired lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetCredentialsForChallenge [this=17375820 channel=21b21b9c proxyAuth=0 challenges=Negotiate]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Released lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetIdentityFromURI [this=17375820 channel=21b21b9c]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquiring lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquired lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpAuthCache::GetAuthEntryForDomain [key=http://servicedesk.farmfrites.com:-1 realm=]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Released lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp unable to authenticate
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: V/nsHttp nsHttpConnectionMgr::OnMsgReclaimConnection [conn=149047bc]
2016-08-04 05:12:42.648000 UTC - [Main Thread]: V/nsHttp ProcessAuthentication failed [rv=80004004]
(Reporter)

Comment 9

2 years ago
Apart from the new version of Firefox nothing has changed. Since I have to use Topdesk (servicedesk.farmfrites.com) extensively, I'm obliged to use another browser (either IE or Chrome) to do my work. There's no login problem there.
'knvo' is not available. When running 'klist' I do not see a ticket specifically for this host. However, since other browsers work correctly, it seems that it is not necessary. Furthermore 'servicedesk.farmfrites.com' (the only one with a '.com' extension) is not the only host I cannot access: our project portal, 'farmnet.farmfrites.com', is inaccessible from Firefox also.
(Reporter)

Comment 10

2 years ago
When I go to the companu project portal I'm confronted with a login popup. There I can at least login. I shall add the http log for this request also.
(Reporter)

Comment 11

2 years ago
Created attachment 8777763 [details]
firefox-v48-farmnet.zip

Another log, this time from trying to access the company project portal.
Duplicate of this bug: 1291789

Comment 13

2 years ago
OK, if you have no ticket even after successfully authenticating with another browser, that implies that you're using NTLMSSP within SPNEGO (not Kerberos in SPNEGO) to authenticate to that machine.

So maybe something changed in Firefox such that it's not falling back to NTLM correctly when it's doing SPNEGO auth? Perhaps it's *not* doing SPNEGO, and asking for only Kerberos? I can't see from the (successful) proxy authentication, because the ticket there is "helpfully" turned into all asterisks.

Out of interest, if you download/build the OpenConnect VPN client on Windows, does *that* work? Obviously it won't make a VPN connection but at its heart it starts as an HTTPS client so it would be interesting to see if it can authenticate using SSPI. It would need the a test page it can fetch over HTTPS though; it doesn't do bare HTTP. We have nice simple SSPI code there, so we can experiment.

The other thing I'd be doing at this point is just 'git bisect' between the latest working, and first non-working version. It takes wall-clock and CPU time, but saves thinking until you are staring at the offending commit and *then* you can start properly paying attention again.
Keywords: regression
Whiteboard: [necko-active]

Comment 14

2 years ago
We're having the same issue in Firefox 48.0 x64;
the setting user_pref("network.automatic-ntlm-auth.trusted-uris", "<several url's>" does not work anymore.

It worked fine in the previous version we used, 46.0.1 x64.

I found a similar bugthread where it was fixed in an update: 1121843
Is it possible to fix this issue in 48.0 x64 in an update as well?

Comment 15

2 years ago
(In reply to David Woodhouse from comment #8)
> ...
> 2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp unable to
> authenticate
> 2016-08-04 05:12:42.648000 UTC - [Socket Thread]: V/nsHttp
> nsHttpConnectionMgr::OnMsgReclaimConnection [conn=149047bc]
> 2016-08-04 05:12:42.648000 UTC - [Main Thread]: V/nsHttp
> ProcessAuthentication failed [rv=80004004]
On the other hand, according to the log the ProcessAuthentication fails with NS_ERROR_ABORT and changes in https://hg.mozilla.org/mozilla-central/rev/0793f99fca8f#l1.61 returns also NS_ERROR_ABORT when 'allowed == false'. I have no idea why TestNotInPBMode would return false for non-PB window. However this still can be a false lead.

Comment 16

2 years ago
Yeah. I didn't see that "ChallengeReceived URI blocked" message in the log, but perhaps there are other reasons for that.
(Reporter)

Comment 17

2 years ago
Created attachment 8777788 [details]
openconnectvpn-log.txt

This is the log when using the openconnect vpn client to access the same page as in the first attachment. The authentication seems to work OK.

I'm not sure how to use 'git disect' here.
Flags: needinfo?(robkleijwegt)

Comment 18

2 years ago
That successfully authenticated to the proxy, but didn't manage to connect to the HTTPS port on the real server at all; the TLS negotiation fails. I note you were accessing it over HTTP before, not HTTPS. Does HTTPS normally work? That might not turn out to be so helpful a test after all.

'git bisect' on a checked-out copy of the Firefox source would keep offering you intermediate versions (between v47 and v48) to build and test, and would relatively quickly lead you to the commit which actually broke it. But we have a suspect, even if we don't quite understand how it's broken. Can you check out v48 and *just* revert the one commit mentioned in comment 15, and see if that fixes the problem for you?
(Reporter)

Comment 19

2 years ago
I'm afraid we have no internal HTTPS sites. External HTTPS sites work as expected.

Before trying to work on 'git bisect' I have started a VirtualBox with Firefox being still version 47. There everything works fine. This is a machine with the same Windows version (7), part of the same domain, using the same login and essentially a mirror of my physical PC.

I'm not at all familiar with 'git bisect', so I shall have to work my way through getting this done.

Comment 20

2 years ago
Don't bother with 'git bisect' for now. That helps you find the offending commit if you have no idea which it was, and you are just doing a binary chop search between the last-known-good and first-known-bad versions.

In this case we have a suspect already. Just take the v48 source and revert that one patch, and see if that 'fixes' the issue. (And test that your build *without* that patch reverted hasn't magically avoided the problem for some other reason, of course).
(Reporter)

Comment 21

2 years ago
That's something I not familiar with either. Other new territories to explore.

Comment 22

2 years ago
I'm having the same issue starting with version 48. But I've found that the SPNEGO auth doesn't work only if using private browsing mode or "Never remember history" settings (the same issue also occurs with NTLM auth).
(Reporter)

Comment 23

2 years ago
That's something I tried immediately before diving into the source and patch history and I have to second that, Alex. I always use "Never remember history" and hadn't even thought of that being related to this problem.

Comment 24

2 years ago
Yeah, me too. And AFAIK it shouldn't.

Comment 25

2 years ago
I see, so in this case the culprit of this bug is really bug 1248564 because it is doing exactly this. It disables to send credentials in private browsing mode. I didn't know that "Never remember history" is technically the same as private browsing mode. So in this case is up to someone to decide if this issue is a bug or not and how can we fix this because in bug 1248564 we agreed that leaking credentials in private browser mode is undesirable.
(Assignee)

Comment 26

2 years ago
(In reply to Rob Kleijwegt from comment #23)
> That's something I tried immediately before diving into the source and patch
> history and I have to second that, Alex. I always use "Never remember
> history" and hadn't even thought of that being related to this problem.

Aha!  "Never remember history" internally actually means "Run in Private Browsing more all the time".  OK, this is a conceptual problem.  I may check if we can recognize this somehow (and bypass) or suggest change to how we treat the "Never remember history" option altogether (harder problem).
Assignee: nobody → honzab.moz
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Updated

2 years ago
Attachment #8777665 - Flags: feedback+

Comment 27

2 years ago
The URL's in network.automatic-ntlm-auth.trusted-uris still don't work with Never Remember History off (in 48.0).

BTW: in version 46.0.1 and other versions the URL's in network.automatic-ntlm-auth.trusted-uris DID work with Never Remember History ON.

Comment 28

2 years ago
(In reply to Jan Horak from comment #25)
> I see, so in this case the culprit of this bug is really bug 1248564 because
> it is doing exactly this. It disables to send credentials in private
> browsing mode. I didn't know that "Never remember history" is technically
> the same as private browsing mode. So in this case is up to someone to
> decide if this issue is a bug or not and how can we fix this because in bug
> 1248564 we agreed that leaking credentials in private browser mode is
> undesirable.

IMHO the mix of these two subjects, private browsing and authentication, is a little bit odd. 
After all, private browsing mode (or "Never remember history") is not the same as anonymous browsing.
Private browsing is about not saving temporary navigation files, history, passwords, form data, autocomplete entries, cookies, etc. This is only for local privacy not anonymity. If we were talking about anonymous browsing...

Comment 29

2 years ago
(In reply to s.jousma from comment #27)
> The URL's in network.automatic-ntlm-auth.trusted-uris still don't work with
> Never Remember History off (in 48.0).
> 
> BTW: in version 46.0.1 and other versions the URL's in
> network.automatic-ntlm-auth.trusted-uris DID work with Never Remember
> History ON.

"Never remember history" is equivalent to Private Browsing mode but you have these two settings available individually. In version 48 if you don't use neither of these you should be able to use both SPNEGO auth (network.negotiate-auth.trusted-uris) and NTLM auth (network.automatic-ntlm-auth.trusted-uris). Or at least it works to me...
(Reporter)

Comment 30

2 years ago
For me "remembering history" also allows me to access all company sites.

Indeed I would expect private browsing to be something different than remembering history. Not only would it be somewhat superfluous to have two settings to do the same thing, but I also agree with Alex that I would have expected not remembering history to be something for local privacy and not for anonymity.

Comment 31

2 years ago
I can confirm that this is the case on FF 48.

Rolling back to v47 results in everything working correctly, even with private mode/don't remember history.

Comment 32

2 years ago
I can confirm that using regular browsing mode in Firefox 48 (64-bit) works with negotiate authentication for me.  But if I use Private Browsing, it no longer works.  This is a recent change, because until very recently, Private Browsing worked find with negotiate auth.

Comment 33

2 years ago
Experiencing the same issue. Have FF config parameter network.automatic-ntlm-auth.trusted-uris set for intranet site, and FF History set to Never Remember History. Worked fine until v48.
(Assignee)

Updated

2 years ago
Blocks: 1248564
(Assignee)

Updated

2 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 34

2 years ago
Created attachment 8781548 [details] [diff] [review]
v1

when a channel is found to be in the PB mode, we also check the never remember history pref.  if found true, we treat the channel as not in PB mode
Attachment #8781548 - Flags: review?(jduell.mcbugs)
Comment on attachment 8781548 [details] [diff] [review]
v1

Review of attachment 8781548 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks Honza!
Attachment #8781548 - Flags: review?(jduell.mcbugs) → review+
(Assignee)

Updated

2 years ago
Keywords: checkin-needed

Comment 36

2 years ago
Thank you for the quick response to this v48 bug. I am not able to understand the logic in Honza's patch, but as long as it allows automatic NTLM Authentication to work with History set to Never Remember History, like it did prior to v48, I am a happy camper. Thanks again!

Comment 37

2 years ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1c68bc94ea52
Allow negotiate/ntml to work when in the 'Never remember history' mode. r=jduell
Keywords: checkin-needed

Comment 38

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/1c68bc94ea52
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox51: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
(Assignee)

Comment 39

2 years ago
Comment on attachment 8781548 [details] [diff] [review]
v1

Approval Request Comment
[Feature/regressing bug #]: bug 1248564
[User impact if declined]: inability to use NTLM negotiation when the browser is set to "Never remember history" mode, which seems to be _not_ a rare case
[Describe test coverage new/current, TreeHerder]: just landed, none (no NTLM tests)
[Risks and why]: zero, this just allows certain NTLM code to go forward when in the above mentioned code, no other impact at all expected
[String/UUID change made/needed]: none
Attachment #8781548 - Flags: approval-mozilla-beta?
Attachment #8781548 - Flags: approval-mozilla-aurora?

Comment 40

2 years ago
I received an update to FF this morning 48.0.1. SSO NTLM Authentication is still broken with History set to Never Remember History.  Can someone here please confirm for me that this patch wasn't in the FF update that I received today?  Thanks!
(Reporter)

Comment 41

2 years ago
I'm afraid I didn't receive a patch yet, so I don't know.
(Assignee)

Comment 42

2 years ago
It's only in Firefox 51 (=Nightly channel).  Will be uplifted to 50 and 49 if approval is granted.  Not on Release channel sooner than on 09-14.

Comment 43

2 years ago
Thank you for the update Honza, and the quick turnaround on the patch you delivered to address this bug. I hope the rest of the community shares your brilliance and moves it quickly through the vetting process.
Comment on attachment 8781548 [details] [diff] [review]
v1

Fix for private browsing regression from 48. Please uplift - this should make it into the beta 6 build.
Attachment #8781548 - Flags: approval-mozilla-beta?
Attachment #8781548 - Flags: approval-mozilla-beta+
Attachment #8781548 - Flags: approval-mozilla-aurora?
Attachment #8781548 - Flags: approval-mozilla-aurora+

Comment 45

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/28353c3a97ed
status-firefox50: --- → fixed

Comment 46

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/cdf6a5d69566
status-firefox49: --- → fixed

Comment 47

2 years ago
SSO NTLM Authentication is not fixed in the latest available/downloadable x64 version 48.0.2.

In this thread in TrackingFlags it says: "status-firefox49: fixed";
Does that mean that SSO NTLM Authentication will not be fixed until 49.0?
And if so: when will 49.0 be available?
(Assignee)

Comment 48

2 years ago
(In reply to s.jousma from comment #47)
> SSO NTLM Authentication is not fixed in the latest available/downloadable
> x64 version 48.0.2.
> 
> In this thread in TrackingFlags it says: "status-firefox49: fixed";
> Does that mean that SSO NTLM Authentication will not be fixed until 49.0?
> And if so: when will 49.0 be available?

49 will be released tomorrow.  The fix for this bug will be there.
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1298336
(Reporter)

Comment 50

2 years ago
As I have enjoyed a holiday it took some time to react, but now I'm back I'm glad everything works again as expected. Thanks for the effort.
(Assignee)

Updated

2 years ago
See Also: → bug 1315680
(Assignee)

Updated

a year ago
Whiteboard: [necko-active] → [necko-active][ntlm]
You need to log in before you can comment on or make changes to this bug.