Closed Bug 1291700 Opened 7 years ago Closed 7 years ago

Since latest release NTLM/SPNEGO no longer works

Categories

(Core :: Networking: HTTP, defect)

Product:
Core
Component:
Networking: HTTP
Version:
48 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla51
Tracking Flags:
Tracking Status
firefox49 --- fixed
firefox50 --- fixed
firefox51 --- fixed

People

(Reporter: robkleijwegt, Assigned: mayhemer)

References

Details

(Keywords: regression, Whiteboard: [necko-active][ntlm])

Attachments

(4 files)

firefox-v48-log.zip
1.07 MB, application/x-zip-compressed
Details
firefox-v48-farmnet.zip
1.59 MB, application/x-zip-compressed
Details
openconnectvpn-log.txt
2.04 KB, text/plain
Details
v1
3.25 KB, patch
jduell.mcbugs
: review+
lizzard
: approval-mozilla-aurora+
lizzard
: approval-mozilla-beta+
Details | Diff | Splinter Review 
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160726073904

Steps to reproduce:

Access some company websites.


Actual results:

I had configured Firefox for SSO to a couple of company websites. The configuration as I had it is still there, but since release 48 was installed I'm no longer able to watch the company websites.


Expected results:

The company websites (as configured for NTLM/SPNEGO) should have remained available.
Component: Untriaged → Networking: HTTP
Product: Firefox → Core
can you please create an http log 
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

and attach it to this bug?
Jan, could this be bug 1248564?  Or anything you may know we have changed around NTLM in 48?
Flags: needinfo?(jhorak)
(In reply to Honza Bambas (:mayhemer) from comment #2)
> Jan, could this be bug 1248564?  Or anything you may know we have changed
> around NTLM in 48?
Hm, no idea. I didn't push any other code into 48. Is it possible to get some logs in Windows like when setting environment variable 'NSPR_LOG_MODULES=negotiateauth:5' in unix?

I'll try to get some ntlm/spnego server, but I'm unsure if I will be successful.
Flags: needinfo?(jhorak)
(In reply to Jan Horak from comment #3)
> (In reply to Honza Bambas (:mayhemer) from comment #2)
> > Jan, could this be bug 1248564?  Or anything you may know we have changed
> > around NTLM in 48?
> Hm, no idea. I didn't push any other code into 48. Is it possible to get
> some logs in Windows like when setting environment variable
> 'NSPR_LOG_MODULES=negotiateauth:5' in unix?

Sure, it's multiplatform ;)  And now it's better to use MOZ_LOG env var to setup your modules.

We can also ask Rob (the reporter) to add this module to his log.

Rob, see also comment 1.

> 
> I'll try to get some ntlm/spnego server, but I'm unsure if I will be
> successful.
Flags: needinfo?(robkleijwegt)
Attached file firefox-v48-log.zip — 
This is the log file I was asked for in the first comment. I tried to access our Topdesk system, which I was Always able to access without a problem, but doing that now brings me to a login screen where I can get no further. Of course I have restarted Firefox and even the computer, alas to no avail.
Attachment #8777665 - Flags: feedback+
The entries to allow my access in the configuration are:
* network.automatic-ntlm-auth.trusted-uris: farmfrites.local,farmnet.farmfrites.com
* network.negotiate-auth.trusted-uris: servicedesk.farmfrites.com
This last entry refers to Topdesk. The first entries refer to intranet sites. All those sites are unavailable now.
Maybe David could help us with reproducer/server settings?
This is a Windows client, isn't it? So it'll be using SSPI for both NTLM and Negotiate(SPNEGO) authentication. That's mostly outside my knowledge (although I did 'accidentally' write that support for OpenConnect VPN once, and even fixed the interoperability between Wine's SSPI and Samba/winbind's ntlm_auth helper.)

Looking at the logs, it looks like the failing request was a Negotiate/SPNEGO authentication, at around 05:12:42.648000 ? The server sends a 401 with 'WWW-Authenticate: Negotiate', and we never attempt to do so. I note we *did* successfully use SPNEGO to authenticate to the proxy.

Can you obtain a ticket manually for this host? Is 'kvno' available for the Windows command line?
 kvno -S HTTP servicedesk.farmfrites.com

I know 'klist' is available for Windows. If you find and run it, does it show that you have a ticket for that host?

2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetAuthenticator [this=17375820 channel=21b21b9c]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquiring lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquired lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetCredentialsForChallenge [this=17375820 channel=21b21b9c proxyAuth=0 challenges=Negotiate]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Released lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpChannelAuthProvider::GetIdentityFromURI [this=17375820 channel=21b21b9c]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquiring lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Acquired lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp nsHttpAuthCache::GetAuthEntryForDomain [key=http://servicedesk.farmfrites.com:-1 realm=]
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: D/nsSocketTransport Released lock on thread 618680
2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp unable to authenticate
2016-08-04 05:12:42.648000 UTC - [Socket Thread]: V/nsHttp nsHttpConnectionMgr::OnMsgReclaimConnection [conn=149047bc]
2016-08-04 05:12:42.648000 UTC - [Main Thread]: V/nsHttp ProcessAuthentication failed [rv=80004004]
Apart from the new version of Firefox nothing has changed. Since I have to use Topdesk (servicedesk.farmfrites.com) extensively, I'm obliged to use another browser (either IE or Chrome) to do my work. There's no login problem there.
'knvo' is not available. When running 'klist' I do not see a ticket specifically for this host. However, since other browsers work correctly, it seems that it is not necessary. Furthermore 'servicedesk.farmfrites.com' (the only one with a '.com' extension) is not the only host I cannot access: our project portal, 'farmnet.farmfrites.com', is inaccessible from Firefox also.
When I go to the companu project portal I'm confronted with a login popup. There I can at least login. I shall add the http log for this request also.
Another log, this time from trying to access the company project portal.
OK, if you have no ticket even after successfully authenticating with another browser, that implies that you're using NTLMSSP within SPNEGO (not Kerberos in SPNEGO) to authenticate to that machine.

So maybe something changed in Firefox such that it's not falling back to NTLM correctly when it's doing SPNEGO auth? Perhaps it's *not* doing SPNEGO, and asking for only Kerberos? I can't see from the (successful) proxy authentication, because the ticket there is "helpfully" turned into all asterisks.

Out of interest, if you download/build the OpenConnect VPN client on Windows, does *that* work? Obviously it won't make a VPN connection but at its heart it starts as an HTTPS client so it would be interesting to see if it can authenticate using SSPI. It would need the a test page it can fetch over HTTPS though; it doesn't do bare HTTP. We have nice simple SSPI code there, so we can experiment.

The other thing I'd be doing at this point is just 'git bisect' between the latest working, and first non-working version. It takes wall-clock and CPU time, but saves thinking until you are staring at the offending commit and *then* you can start properly paying attention again.
Keywords: regression
Whiteboard: [necko-active]
We're having the same issue in Firefox 48.0 x64;
the setting user_pref("network.automatic-ntlm-auth.trusted-uris", "<several url's>" does not work anymore.

It worked fine in the previous version we used, 46.0.1 x64.

I found a similar bugthread where it was fixed in an update: 1121843
Is it possible to fix this issue in 48.0 x64 in an update as well?
(In reply to David Woodhouse from comment #8)
> ...
> 2016-08-04 05:12:42.648000 UTC - [Main Thread]: D/nsHttp unable to
> authenticate
> 2016-08-04 05:12:42.648000 UTC - [Socket Thread]: V/nsHttp
> nsHttpConnectionMgr::OnMsgReclaimConnection [conn=149047bc]
> 2016-08-04 05:12:42.648000 UTC - [Main Thread]: V/nsHttp
> ProcessAuthentication failed [rv=80004004]
On the other hand, according to the log the ProcessAuthentication fails with NS_ERROR_ABORT and changes in https://hg.mozilla.org/mozilla-central/rev/0793f99fca8f#l1.61 returns also NS_ERROR_ABORT when 'allowed == false'. I have no idea why TestNotInPBMode would return false for non-PB window. However this still can be a false lead.
Yeah. I didn't see that "ChallengeReceived URI blocked" message in the log, but perhaps there are other reasons for that.
Attached file openconnectvpn-log.txt — 
This is the log when using the openconnect vpn client to access the same page as in the first attachment. The authentication seems to work OK.

I'm not sure how to use 'git disect' here.
Flags: needinfo?(robkleijwegt)
That successfully authenticated to the proxy, but didn't manage to connect to the HTTPS port on the real server at all; the TLS negotiation fails. I note you were accessing it over HTTP before, not HTTPS. Does HTTPS normally work? That might not turn out to be so helpful a test after all.

'git bisect' on a checked-out copy of the Firefox source would keep offering you intermediate versions (between v47 and v48) to build and test, and would relatively quickly lead you to the commit which actually broke it. But we have a suspect, even if we don't quite understand how it's broken. Can you check out v48 and *just* revert the one commit mentioned in comment 15, and see if that fixes the problem for you?
I'm afraid we have no internal HTTPS sites. External HTTPS sites work as expected.

Before trying to work on 'git bisect' I have started a VirtualBox with Firefox being still version 47. There everything works fine. This is a machine with the same Windows version (7), part of the same domain, using the same login and essentially a mirror of my physical PC.

I'm not at all familiar with 'git bisect', so I shall have to work my way through getting this done.
Don't bother with 'git bisect' for now. That helps you find the offending commit if you have no idea which it was, and you are just doing a binary chop search between the last-known-good and first-known-bad versions.

In this case we have a suspect already. Just take the v48 source and revert that one patch, and see if that 'fixes' the issue. (And test that your build *without* that patch reverted hasn't magically avoided the problem for some other reason, of course).
That's something I not familiar with either. Other new territories to explore.
I'm having the same issue starting with version 48. But I've found that the SPNEGO auth doesn't work only if using private browsing mode or "Never remember history" settings (the same issue also occurs with NTLM auth).
That's something I tried immediately before diving into the source and patch history and I have to second that, Alex. I always use "Never remember history" and hadn't even thought of that being related to this problem.
Yeah, me too. And AFAIK it shouldn't.
I see, so in this case the culprit of this bug is really bug 1248564 because it is doing exactly this. It disables to send credentials in private browsing mode. I didn't know that "Never remember history" is technically the same as private browsing mode. So in this case is up to someone to decide if this issue is a bug or not and how can we fix this because in bug 1248564 we agreed that leaking credentials in private browser mode is undesirable.
(In reply to Rob Kleijwegt from comment #23)
> That's something I tried immediately before diving into the source and patch
> history and I have to second that, Alex. I always use "Never remember
> history" and hadn't even thought of that being related to this problem.

Aha!  "Never remember history" internally actually means "Run in Private Browsing more all the time".  OK, this is a conceptual problem.  I may check if we can recognize this somehow (and bypass) or suggest change to how we treat the "Never remember history" option altogether (harder problem).
Assignee: nobody → honzab.moz
Status: UNCONFIRMED → NEW
Ever confirmed: true
Attachment #8777665 - Flags: feedback+
The URL's in network.automatic-ntlm-auth.trusted-uris still don't work with Never Remember History off (in 48.0).

BTW: in version 46.0.1 and other versions the URL's in network.automatic-ntlm-auth.trusted-uris DID work with Never Remember History ON.
(In reply to Jan Horak from comment #25)
> I see, so in this case the culprit of this bug is really bug 1248564 because
> it is doing exactly this. It disables to send credentials in private
> browsing mode. I didn't know that "Never remember history" is technically
> the same as private browsing mode. So in this case is up to someone to
> decide if this issue is a bug or not and how can we fix this because in bug
> 1248564 we agreed that leaking credentials in private browser mode is
> undesirable.

IMHO the mix of these two subjects, private browsing and authentication, is a little bit odd. 
After all, private browsing mode (or "Never remember history") is not the same as anonymous browsing.
Private browsing is about not saving temporary navigation files, history, passwords, form data, autocomplete entries, cookies, etc. This is only for local privacy not anonymity. If we were talking about anonymous browsing...
(In reply to s.jousma from comment #27)
> The URL's in network.automatic-ntlm-auth.trusted-uris still don't work with
> Never Remember History off (in 48.0).
> 
> BTW: in version 46.0.1 and other versions the URL's in
> network.automatic-ntlm-auth.trusted-uris DID work with Never Remember
> History ON.

"Never remember history" is equivalent to Private Browsing mode but you have these two settings available individually. In version 48 if you don't use neither of these you should be able to use both SPNEGO auth (network.negotiate-auth.trusted-uris) and NTLM auth (network.automatic-ntlm-auth.trusted-uris). Or at least it works to me...
For me "remembering history" also allows me to access all company sites.

Indeed I would expect private browsing to be something different than remembering history. Not only would it be somewhat superfluous to have two settings to do the same thing, but I also agree with Alex that I would have expected not remembering history to be something for local privacy and not for anonymity.
I can confirm that this is the case on FF 48.

Rolling back to v47 results in everything working correctly, even with private mode/don't remember history.
I can confirm that using regular browsing mode in Firefox 48 (64-bit) works with negotiate authentication for me.  But if I use Private Browsing, it no longer works.  This is a recent change, because until very recently, Private Browsing worked find with negotiate auth.
Experiencing the same issue. Have FF config parameter network.automatic-ntlm-auth.trusted-uris set for intranet site, and FF History set to Never Remember History. Worked fine until v48.
Blocks: 1248564
Status: NEW → ASSIGNED
Attached patch v1 — — Splinter Review 
when a channel is found to be in the PB mode, we also check the never remember history pref.  if found true, we treat the channel as not in PB mode
Attachment #8781548 - Flags: review?(jduell.mcbugs)
Comment on attachment 8781548 [details] [diff] [review]
v1

Review of attachment 8781548 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks Honza!
Attachment #8781548 - Flags: review?(jduell.mcbugs) → review+
Keywords: checkin-needed
Thank you for the quick response to this v48 bug. I am not able to understand the logic in Honza's patch, but as long as it allows automatic NTLM Authentication to work with History set to Never Remember History, like it did prior to v48, I am a happy camper. Thanks again!
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1c68bc94ea52
Allow negotiate/ntml to work when in the 'Never remember history' mode. r=jduell
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/1c68bc94ea52
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
status-firefox51: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
Comment on attachment 8781548 [details] [diff] [review]
v1

Approval Request Comment
[Feature/regressing bug #]: bug 1248564
[User impact if declined]: inability to use NTLM negotiation when the browser is set to "Never remember history" mode, which seems to be _not_ a rare case
[Describe test coverage new/current, TreeHerder]: just landed, none (no NTLM tests)
[Risks and why]: zero, this just allows certain NTLM code to go forward when in the above mentioned code, no other impact at all expected
[String/UUID change made/needed]: none
Attachment #8781548 - Flags: approval-mozilla-beta?
Attachment #8781548 - Flags: approval-mozilla-aurora?
I received an update to FF this morning 48.0.1. SSO NTLM Authentication is still broken with History set to Never Remember History.  Can someone here please confirm for me that this patch wasn't in the FF update that I received today?  Thanks!
I'm afraid I didn't receive a patch yet, so I don't know.
It's only in Firefox 51 (=Nightly channel).  Will be uplifted to 50 and 49 if approval is granted.  Not on Release channel sooner than on 09-14.
Thank you for the update Honza, and the quick turnaround on the patch you delivered to address this bug. I hope the rest of the community shares your brilliance and moves it quickly through the vetting process.
Comment on attachment 8781548 [details] [diff] [review]
v1

Fix for private browsing regression from 48. Please uplift - this should make it into the beta 6 build.
Attachment #8781548 - Flags: approval-mozilla-beta?
Attachment #8781548 - Flags: approval-mozilla-beta+
Attachment #8781548 - Flags: approval-mozilla-aurora?
Attachment #8781548 - Flags: approval-mozilla-aurora+
SSO NTLM Authentication is not fixed in the latest available/downloadable x64 version 48.0.2.

In this thread in TrackingFlags it says: "status-firefox49: fixed";
Does that mean that SSO NTLM Authentication will not be fixed until 49.0?
And if so: when will 49.0 be available?
(In reply to s.jousma from comment #47)
> SSO NTLM Authentication is not fixed in the latest available/downloadable
> x64 version 48.0.2.
> 
> In this thread in TrackingFlags it says: "status-firefox49: fixed";
> Does that mean that SSO NTLM Authentication will not be fixed until 49.0?
> And if so: when will 49.0 be available?

49 will be released tomorrow.  The fix for this bug will be there.
As I have enjoyed a holiday it took some time to react, but now I'm back I'm glad everything works again as expected. Thanks for the effort.
See Also: → 1315680
Whiteboard: [necko-active] → [necko-active][ntlm]
You need to log in before you can comment on or make changes to this bug.