Closed Bug 804605 Opened 12 years ago Closed 11 years ago

Kerberos authentication does not work with CNAME

Categories

(Core :: Networking: HTTP, defect)

Product:
Core
Component:
Networking: HTTP
Version:
20 Branch
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla21
Tracking Flags:
Tracking Status
firefox17 + fixed
firefox18 + fixed
firefox19 + verified

People

(Reporter: andriys, Assigned: mcmanus)

References

Details

Attachments

(2 files)

backout for beta
28.64 KB, patch
lsblakk
: approval-mozilla-beta+
Details | Diff | Splinter Review
backout for beta FF 18
30.89 KB, patch
lsblakk
: approval-mozilla-beta+
Details | Diff | Splinter Review 
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/17.0 Firefox/17.0
Build ID: 20121017073013

Steps to reproduce:

My setup:
* Windows 7 workstation joined to Active Directory.
* A couple of Apache 2.2 web servers (running on different OSs) configured to accept Kerberos authentication (mod_auth_kerb 5.4) against the same AD domain.
* Each web server is configured to serve several virtual hosts. Some of the names of the virtual hosts are registered in the DNS as CNAMEs.

I'm using Firefox 17 beta on my Windows 7 PC:
* In about:config the following parameters were tweaked:
   - network.negotiate-auth.allow-non-fqdn = true
   - network.negotiate-auth.trusted-uris = our.ad.realm.name

Steps to trigger the problem:
* Try to access any page on a site that has host name being a CNAME and uses Kerberos to authenticate users.



Actual results:

Depending on which web server I access and whether I use FQDN or just a host name one of the following two things may happen:

a) Kerberos authentication fails and Firefox asks for username and password.
b) Web server returns error 500 "Internal Server Error".

At the same time one of the following errors appear in the error log file on the server:

* gss_display_name() failed:  An invalid name was supplied (, unknown mech-code 0 for mech unknown)
* gss_accept_sec_context() failed:  An unsupported mechanism was requested (, unknown mech-code 0 for mech unknown)
* gss_accept_sec_context() failed: No credentials were supplied, or the credentials were unavailable or inaccessible (, Unknown error)

The problem manifests itself only when using CNAMEs. Kerberos authentication works as expected when using A-records.

The problem first appeared right after the first update from FF16 to FF17 a couple of weeks ago.



Expected results:

FF automatically authenticates the user.
Component: Untriaged → Networking: HTTP
Product: Firefox → Core
Assignee: nobody → mcmanus
Thanks for the report andriy - I can confirm from the logs that something is amiss with the negotiate canonicalization. (it is happening on the dns level, but it doesn't seem to be reflected in the protocol tokens).

Interestingly IIS does not care.. I have a cname set up and auth proceeds normally with it - so my testing say the DNS happen, and it saw the test pass.

767158 is on both beta-17 and aurora-18. Tomorrow I'll make a patch to back it out of beta; this patch was the leading edge of a bunch of jank removing patches that mostly landed on 18 so I don't mind waiting for 18 to hit the field to see the benefit of 767158. 

I'll also work on a real fix, of course :) I will probably need you to validate it.

The basic problem is that nsHttpNegotiateAuth calls nsIAuthModule->init out of ChallengeReceived, while nsHttpNTLMAuth calls it later out of GenerateCredentials. The canonicalized host is not available until GenerateCredentials(). Hopefully there is no reason that nsHttpNegotiateAuth cannot do the same.
Blocks: 767158
Attached patch backout for beta — — Splinter Review 
[Approval Request Comment]
this problem is caused by 767158. This patch backs that patch (and a correct spot fix 785050) out of FF17-beta.

The backout is not purely mechanical due to the treewide nullptr/NULL and PRUint/uint_t changes causing some conflicts. but there was nothing very tricky involved.

The patches removed chromehang issues. I hope they can be easily spot fixed in aurora - but we can decide that when I've got a patch.
Attachment #675238 - Flags: approval-mozilla-beta?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Andriy,can you test the build at the link below and confirm it fixes your issue?

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.com-628a4205b2de/try-win32/
Flags: needinfo?(andriys)
(In reply to Patrick McManus [:mcmanus] from comment #3)
> Andriy,can you test the build at the link below and confirm it fixes your
> issue?
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.
> com-628a4205b2de/try-win32/

Following the link you've provided I've picked this file: firefox-19.0a1.en-US.win32.zip. In short, it does NOT work as expected.

Well, the Kerberos auth does seem to work, actually. Everything is fine when accessing just a single simple file (for example an image, or an HTML file that does not reference any other files). When accessing a more complex pages, however, Firefox unexpectedly starts requesting authentication info (login/password) somewhere in the middle of the process. If I enter the valid login/password pair, FF uses this pair to load everything that's remaining. If I click "Cancel" button, FF skips one file and continues, and while continuing it sometimes asks for a login/password again.

Interestingly enough, this new problem is not CNAME-related any more.

Web-server logs does not show any Kerberos-related errors. Error log is actually empty, and the access log shows something similar to the following:

192.168.22.100 - - [26/Oct/2012:16:49:36 +0300] "GET /vote/cinema/ HTTP/1.1" 401 401 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - andriys [26/Oct/2012:16:49:36 +0300] "GET /vote/cinema/ HTTP/1.1" 304 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/vote.css HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/jquery.cluetip.css HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/js/jquery.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/js/jquery-ui.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/js/jquery.cluetip.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/js/vote.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - andriys [26/Oct/2012:16:49:37 +0300] "GET /vote/cinema/vote.css HTTP/1.1" 304 - "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"

Here FF asks for login/password. And when I click "Cancel" button the following appears in the access log:

192.168.22.100 - - [26/Oct/2012:16:49:43 +0300] "GET /vote/cinema/js/jquery-ui.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - andriys [26/Oct/2012:16:49:43 +0300] "GET /vote/cinema/js/jquery-ui.js HTTP/1.1" 304 - "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:43 +0300] "GET /vote/cinema/js/jquery.cluetip.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - andriys [26/Oct/2012:16:49:43 +0300] "GET /vote/cinema/js/jquery.cluetip.js HTTP/1.1" 304 - "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - - [26/Oct/2012:16:49:44 +0300] "GET /vote/cinema/js/vote.js HTTP/1.1" 401 401 "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
192.168.22.100 - andriys [26/Oct/2012:16:49:44 +0300] "GET /vote/cinema/js/vote.js HTTP/1.1" 304 - "http://www01.xxx.intra/vote/cinema/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/19.0 Firefox/19.0"
Flags: needinfo?(andriys)
Comment on attachment 675238 [details] [diff] [review]
backout for beta

looks like this backout doesn't fix the issue so minusing for branch uplift.
Attachment #675238 - Flags: approval-mozilla-beta? → approval-mozilla-beta-
Comment on attachment 675238 [details] [diff] [review]
backout for beta

renominating the backout because rejection misunderstood the comment trail.

comments 3 and 4 are about a possible (but appranently incomplete) fix for the trunk without backing anything out. They have nothing to do with the backout.

I'm confindent the backout will fix the problem for beta.
Attachment #675238 - Flags: approval-mozilla-beta- → approval-mozilla-beta?
Andriy, thanks for your help with this. This is still works for me, so I will have to lean on you for testing.

Can you try running a build from here http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.com-c1314acc511a/try-win32/

That is a nightly build patched for the CNAME issue as well as a race condition in between populating the auth cache and starting other sessions that is due to the asynchrnonous changes introduced by the original patch. I'm just guessing that's your issue based on the information you have provided and reading the code again. Your continued help is appreciated. (this is the patch https://tbpl.mozilla.org/?tree=Try&rev=c1314acc511a)

If that doesn't work can you make an HTTP log and attach it to this bug

It should be similar to https://developer.mozilla.org/en-US/docs/HTTP_Logging except the NSPR_LOG_MODULES like should contain negotiateauth:5

i.e.

set NSPR_LOG_MODULES=timestamp,nsHttp:5,nsSocketTransport:5,nsHostResolver:5,negotiateauth:5

and set NSPR_LOG_FILE to something which you will upload.


thanks!
Status: NEW → UNCONFIRMED
Ever confirmed: false
Flags: needinfo?(andriys)
Comment on attachment 675238 [details] [diff] [review]
backout for beta

Thanks for the clarification, in that case approving the backout and will set tracking flags on this so we keep it on our radar.
Attachment #675238 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
(In reply to Patrick McManus [:mcmanus] from comment #7)
> Andriy, thanks for your help with this. This is still works for me, so I
> will have to lean on you for testing.
> 
> Can you try running a build from here
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.
> com-c1314acc511a/try-win32/

I've tried that. Unfortunately, the problem still persists in this version.


> If that doesn't work can you make an HTTP log and attach it to this bug

Following your instructions, I've prepared two log files (for the case with CNAME as well as A record). But since they contain some internal host and domain names of our company, I'd like to submit them in-private, if possible. Is it ok if I e-mail them to you directly?
Flags: needinfo?(andriys)
> But since they contain some internal host and
> domain names of our company, I'd like to submit them in-private

Or should I possibly obfuscate the logs in some way and attache to this bug report anyway? Please advise.
Andriy, let's start with just emailing me pmcmanus@mozilla.com
Patrick, should we be doing the same for FF18?
Just auto-updated to 18 beta, and the problem returned.
Patrick, have you received the logs I E-mailed to you on Oct. 30?
Version: 17 Branch → 18 Branch
Andriy, yes I got the mail. Thanks. Its just queued behind some other things right now.

We won't let this go out on the release channel like this - definitely on my radar to (at least) back out of 18 this week.
[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: 
Testing completed (on m-c, etc.): 
Risk to taking this patch (and alternatives if risky): 
String or UUID changes made by this patch: 

I haven't had time to work out the right fix here.. I have one that is close, but not quite right.

So we'll back this out of 18 (like we did for 17) to buy a little more breathing room. Try confirmation of backout https://tbpl.mozilla.org/?tree=Try&rev=d535f49c7fed

Removing jank is important, but this is a pretty lightly used code path so deferring the improvement isn't the end of the world.
Attachment #687435 - Flags: approval-mozilla-beta?
Comment on attachment 687435 [details] [diff] [review]
backout for beta FF 18

Glad you're getting closer to the proper fix, but yes - let's get this backed out of 18 now well ahead of release.
Attachment #687435 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Just auto-updated to 19 beta, the problem is till here.
I'm updating the version field to be '19 Branch'. Should the tracking flags be updated as well, or has this problem been fixed already in v20 and v21?
Version: 18 Branch → 19 Branch
(In reply to Patrick McManus [:mcmanus] from comment #18)
> backout of beta 18
> https://hg.mozilla.org/releases/mozilla-beta/rev/94efb16f90df

Patrick, please just perform the backout on all branches. We want to get this off of our tracking list once and for all.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
https://hg.mozilla.org/mozilla-central/rev/4a1188e7f538
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla21
This seems to be fixed in the update I've auto-received today morning (FF 19.0, Build ID 20130123083802).

Thanks.
Marking this verified on FF 19 based on comment 24.
status-firefox19: fixed → verified
Just auto-updated to 20 beta (build ID 20130220104816), and discovered that it's still affected by the problem. The backout must have been incomplete or incorrect (at least on the 20 branch).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Version: 19 Branch → 20 Branch
Andriy, I'm going to ask you to retest with 20 beta because I've just confirmed that the backout on this branch is identical to the one on 19 that you confirmed was fine.

Please test with a fresh profile to be sure.

If the problem persists please provide the "built from" information from about:buildconfig and double check that your test case still works with firefox 19 (to rule out the possibility of a breaking change elsewhere).

We don't have a test that covers this so chasing it down beyond confirming the backout will be very hard - so I want to double check the report first. Thanks for your help.
(In reply to Patrick McManus [:mcmanus] from comment #27)
> Andriy, I'm going to ask you to retest with 20 beta because I've just
> confirmed that the backout on this branch is identical to the one on 19 that
> you confirmed was fine.
> 
> Please test with a fresh profile to be sure.

I've just done what you've asked me to do. Downloaded fresh copies of FF19 and FF20b1, and tested them with fresh profiles (I was creating a fresh profile each time I started FF of another version).

I confirm that my previous report was correct. FF19 works as expected, but FF20b1 does not. The symptoms of the problem are exactly the same as were originally reported.

> If the problem persists please provide the "built from" information from
> about:buildconfig

Built from http://hg.mozilla.org/releases/mozilla-beta/rev/f9348d70933c
status-firefox20: fixed → ---
Still a problem with FF 20.0 - downgrading to 19.02 helps - please fix 20.0 (release 20.01)
Depends on: 857483
Chiming in to confirm I have this bug on 20.0 release channel.
(In reply to Andriy Syrovenko from comment #28)
> Built from http://hg.mozilla.org/releases/mozilla-beta/rev/f9348d70933c

The backout has landed later then the changeset you were testing with. 

The backout cs: http://hg.mozilla.org/releases/mozilla-beta/rev/4a1188e7f538

Please check again (with fresh profile) with a build you find at [1] that by examining the sources doesn't contain the fix (so, the patches for 766973 767158 785050 are backed out).

And, I unfortunately don't have an environment to test with and don't have resources to build one.


[1] http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/20.0-candidates/build1/
(In reply to Honza Bambas (:mayhemer) from comment #31)
> Please check again (with fresh profile) with a build you find at [1]
> [1]
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/20.0-candidates/
> build1/

I am on the beta update channel, and the problem still persist in the latest betas I received automatically.

Still, I will test the build you are referring to soon.
(In reply to Honza Bambas (:mayhemer) from comment #31)
> Please check again (with fresh profile) with a build you find at [1]

I've just checked with this build:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/20.0-candidates/build1/win32/en-US/firefox-20.0.zip
buildID: 20130326150557
Built from http://hg.mozilla.org/releases/mozilla-release/rev/c90d44bfa96c

The problem is there, easily reproducible, symptoms are the same, as originally reported.
(In reply to Andriy Syrovenko from comment #33)
> (In reply to Honza Bambas (:mayhemer) from comment #31)
> The problem is there, easily reproducible, symptoms are the same, as
> originally reported.

Thanks.

I confirmed the patches we have suspected are correctly backed out from 20.

This must be something else.
No longer blocks: 767158
Andriy, thank you very much for the time you are willing to spend on this issue!

I will ask you to do some more tests since none of us in mozilla have currently a functional test environment to test NTLM with Kerberos.

So, first test, please check the following two builds:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/12/2012-12-25-03-07-57-mozilla-central/firefox-20.0a1.en-US.win32.installer.exe - I assume a failure

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/12/2012-12-24-03-09-02-mozilla-central/firefox-20.0a1.en-US.win32.installer.exe - I assume a success

Thanks.
Blocks: 807678
(In reply to Honza Bambas (:mayhemer) from comment #35)
> So, first test, please check the following two builds:

Unfortunately, both build failed. The problem is the same- Kerb auth fails for hosts that are referred to by CNAMEs in DNS, but work fine for A-records.

Just in case you need it, the first build I tested is this:
--
Build ID: 20121225030757
Built from http://hg.mozilla.org/mozilla-central/rev/dc2abccc2adb
Test failed.

The second build:
--
Build ID: 20121224030902
Built from http://hg.mozilla.org/mozilla-central/rev/d348dbf1dab4
Test failed.
Andriy, I've created an experimental build that backs out bug 807678.  According bug 857291 comment 10 it seems to fix bug 857291, now we need to confirm whether it fixes this bug too.

Please try this build and let me know:
https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/honzab.moz@firemni.cz-24c4514bc842/try-win32/firefox-20.0.en-US.win32.installer.exe

Thanks for help!
Flags: needinfo?(andriys)
(In reply to Honza Bambas (:mayhemer) from comment #37)
> Please try this build and let me know:
> https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/honzab.
> moz@firemni.cz-24c4514bc842/try-win32/firefox-20.0.en-US.win32.installer.exe

This build works fine for me with both CNAMEs and As. Thanks.
Flags: needinfo?(andriys)
(In reply to Andriy Syrovenko from comment #38)
> (In reply to Honza Bambas (:mayhemer) from comment #37)
> > Please try this build and let me know:
> > https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/honzab.
> > moz@firemni.cz-24c4514bc842/try-win32/firefox-20.0.en-US.win32.installer.exe
> 
> This build works fine for me with both CNAMEs and As. Thanks.

Thanks for testing it.  I've submitted the patch to bug 857291 that seems to be more proper for it.
I've just auto-updated to FF21 (through beta update channel). Posting this comment to let you know that FF21 beta is affected by this bug as well.
status-firefox21: fixed → ---
I'm going to declare that this bug is fixed with checkin for comment 22.

The new issue, which has the same symptoms for Andriy but is caused by a different issue, is tracked in bug 857291.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Hi!

Isn't this bug a duplicate of bug #537903? If this is the case, this bug is still present in Firefox 23.0.1 on Linux.

Cheers
Hi!

I'm sorry for my last comment, I think I did a mistake, I can't reproduce the bug and everything seems to work just fine with CNAMEs.

All my apologies
Cheers
If you are like me and found this issue using Google: It's probably back in the 35.0 beta build. I'll see if I can create a new bug report.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: