535193 - DNS resolution in MakeSN of nsAuthSSPI causing issues for proxy servers that support NTLM auth

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Jim Mathies [:jimm]]

Attachment: http log

This is bug is a spin off of a mozillazine post. Attached are the logs.

MakeSN uses dns resolution on the host of the new service name we pass in. It seems in some cases the local dns resolver won't be able to resolve these host names, causing a failure in authSSPI. 

http://forums.mozillazine.org/viewtopic.php?f=23&t=1648755

Comment 1

[Jim Mathies [:jimm]]

Attachment: http + dns log

Comment 2

[Jim Mathies [:jimm]]

Seems to me we don't need the canonical lookup since service names identify services, not hosts. Multiple services can run on a single canonical host.

wtc, any comments?

Comment 3

[Jim Mathies [:jimm]]

Sent some email to the original poster. There seems to be some confusion as it was reported that the generic fall back pref fixed the problem, but then it was also reported to be a problem in 3.5.6, which didn't get that patch. Will post back once I figure out what's going on.

Comment 4

[Wan-Teh Chang]

Jim: Takehiro Takahashi of IBM told me that NTLM does not need
the canonical name.  Recall that in your NTLM fix we decided
to call MakeSN to share code with the Negotiate auth scheme,
not because NTLM needs it.

Comment 5

[Jim Mathies [:jimm]]

(In reply to comment #4)
> Jim: Takehiro Takahashi of IBM told me that NTLM does not need
> the canonical name.  Recall that in your NTLM fix we decided
> to call MakeSN to share code with the Negotiate auth scheme,
> not because NTLM needs it.

Ok, that's what I was hoping. I pushed an update that removes the dns lookups to try, and I'll see if I can get the reporter to test that. If it fixes it I'll post the patch.

Comment 6

[Wan-Teh Chang]

I remember NTLM just needs to pass a string in the
right format as the 'sn' argument (which used to be
NULL for NTLM).  Takehiro Takahashi believed that
the actual host name in the 'sn' argument doesn't
matter, as he observed in experiments that IE seems
to just use the host name in the URL as the host
name component of the 'sn' argument, without
converting it to a FQDN or the canonical name.

Comment 7

[Daniel Veditz [:dveditz]]

This presumably affects Firefox 3.5.6 and 3.0.16 as well.

Comment 10

[Jim Mathies [:jimm]]

Note to drivers, this should block the final release of 1.9.2. I should have a patch ready later today.

Comment 11

[Mike Beltzner [:beltzner, not reading bugmail]]

Jim, got someone on deck to review that patch? Also, please post patches for trunk, 1.9.2, 1.9.1 and 1.9.0.

Comment 12

[Jim Mathies [:jimm]]

Wan-Teh hopefully with have the time.  (In reply to comment #6)
> I remember NTLM just needs to pass a string in the
> right format as the 'sn' argument (which used to be
> NULL for NTLM).  Takehiro Takahashi believed that
> the actual host name in the 'sn' argument doesn't
> matter, as he observed in experiments that IE seems
> to just use the host name in the URL as the host
> name component of the 'sn' argument, without
> converting it to a FQDN or the canonical name.

wtc, we also pass kerberos through nsAuthSSPI. The docs on MSDN point to the same SPN docs for both, so I'm assuming at this point that kerberos won't have issues without the canonical host lookups. Curious what your thoughts are on that. I'm going to try and get kerberos working on my local IIS server to test as well.

Comment 13

[Jim Mathies [:jimm]]

Attachment: skip canonical lookups patch.v1

Patch that removes the MakeSN functionality.

Comment 15

[kitchin]

In Attachment 418191 [details] [diff]
+    // this to the service prinicpal name SSPI expects: '<service class>/<host>'

Typo: prinicpal for principal

Comment 16

[Wan-Teh Chang]

Jim: I am not that familiar with Kerberos, but I heard that
it is Kerberos that requires the canonical name.

Comment 17

[Wan-Teh Chang]

Comment on attachment 418191 [details] [diff] [review]
skip canonical lookups patch.v1

Jim, the canonical name requirement comes from Kerberos,
so we should not remove the MakeSN function.  We should
continue to use MakeSN for the Negotiate scheme, and
use your new code in this patch for the NTLM scheme.
Can you do that?  Thanks.

Comment 18

[Jim Mathies [:jimm]]

Attachment: skip canonical lookups patch.v2

Comment 19

[Jim Mathies [:jimm]]

Comment on attachment 418271 [details] [diff] [review]
skip canonical lookups patch.v2

Ok, here's a new rev that keeps MakeSN for kerberos.

Comment 20

[Wan-Teh Chang]

Comment on attachment 418271 [details] [diff] [review]
skip canonical lookups patch.v2

r=wtc.

>+    if (mPackage == PACKAGE_TYPE_NTLM) {
>+        // For NTLM, just use the uri host, do not do canonical host lookups. (bug 535193)
>+        mServiceName.Assign(serviceName);
>+        PRInt32 index = mServiceName.FindChar('@');
>+        if (index == kNotFound)
>+            return NS_ERROR_INVALID_ARG;
>+        mServiceName.Replace(index, 1, '/');
>+    }

Portions of the following comments from MakeSN should be duplicated
here so that the reader know why you're replacing @ by /.

    // The service name looks like "protocol@hostname", we need to map
    // this to a value that SSPI expects.  To be consistent with IE, we
    // need to map '@' to '/' and canonicalize the hostname.

You should ask someone familiar with the Mozilla string classes
to check your use of Assign, FindChar, and Replace.  It looks
correct to me.

Comment 21

[Wan-Teh Chang]

Comment on attachment 418271 [details] [diff] [review]
skip canonical lookups patch.v2

>+        // For NTLM, just use the uri host, do not do canonical host lookups. (bug 535193)
>+        mServiceName.Assign(serviceName);
>+        PRInt32 index = mServiceName.FindChar('@');
>+        if (index == kNotFound)
>+            return NS_ERROR_INVALID_ARG;

I wonder if we should return NS_ERROR_UNEXPECTED here to
match MakeSN.

Comment 22

[Jim Mathies [:jimm]]

Attachment: skip canonical lookups patch.v3

bz, can you give us a final sr on this?

Comment 23

[Jim Mathies [:jimm]]

http://hg.mozilla.org/mozilla-central/rev/ebaf721a5015
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/68f268906598

Comment 24

[Al Billings [:abillings - ex-MoCo]]

What is the scenario/setup necessary to reproduce this bug to check the fix? We don't have a proxy environment set up anywhere that I know of within MoCo.

Comment 25

[Jim Mathies [:jimm]]

Attachment: 1.9.1 patch

Comment 26

[Jim Mathies [:jimm]]

(In reply to comment #24)
> What is the scenario/setup necessary to reproduce this bug to check the fix? We
> don't have a proxy environment set up anywhere that I know of within MoCo.

Some of the dupes, especially bug 535506 list environments that use open source proxies that experienced trouble. We'd need to setup a test nt domain with a set of proxies similar to what's used in corp environs to test. Really, we need that generally as well not just for this bug, NTLM is sorely lacking in moz testing, we've mostly relied on feedback from users to do it.

Comment 27

[Jim Mathies [:jimm]]

Attachment: 1.9.0 patch

Comment 29

[Daniel Veditz [:dveditz]]

Comment on attachment 418309 [details] [diff] [review]
1.9.1 patch

Approved for 1.9.1.7 and 1.9.1.8, a=dveditz for release-drivers

1.9.1.8 is the normal 1.9.1 branch, 1.9.1.7 requires checking into the FF3.5.6 relbranch GECKO1916_20091130_RELBRANCH

Comment 30

[Daniel Veditz [:dveditz]]

Comment on attachment 418312 [details] [diff] [review]
1.9.0 patch

please use "cvs diff -pu8" for review in the future (amount of context as appropriate).

Approved for 1.9.0.17 and 1.9.0.18, a=dveditz for release-drivers

1.9.0.18 is the normal cvs HEAD, 1.9.0.17 is now on the FF3.0.6 relbranch GECKO190_20091130_RELBRANCH

Comment 31

[Daniel Veditz [:dveditz]]

Please set status1.9.1 to .7-fixed when it's landed on BOTH the relbranch and the main 1.9.1 branch. QA will verify on both before adding the verified keyword.

For the 1.9.0 tree please add the fixed1.9.0.17 keyword when you land on the relbranch, and the fixed1.9.0.18 keyword when you land on cvs HEAD.

Comment 32

[Mike Beltzner [:beltzner, not reading bugmail]]

Jim - please check these fixes in to the release branches indicated in comment 29 and comment 30; as soon as that's done we can start spinning builds.

Comment 33

[Jim Mathies [:jimm]]

(In reply to comment #29)
> (From update of attachment 418309 [details] [diff] [review])
> Approved for 1.9.1.7 and 1.9.1.8, a=dveditz for release-drivers
> 
> 1.9.1.8 is the normal 1.9.1 branch, 1.9.1.7 requires checking into the FF3.5.6
> relbranch GECKO1916_20091130_RELBRANCH

http://hg.mozilla.org/releases/mozilla-1.9.1/rev/bcd93c25ee72
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/9a04b31d5b46

Comment 34

[Jim Mathies [:jimm]]

Attachment: cvd diff -pu8

(In reply to comment #30)
> (From update of attachment 418312 [details] [diff] [review])
> please use "cvs diff -pu8" for review in the future (amount of context as
> appropriate).
> 
> Approved for 1.9.0.17 and 1.9.0.18, a=dveditz for release-drivers
> 
> 1.9.0.18 is the normal cvs HEAD, 1.9.0.17 is now on the FF3.0.6 relbranch
> GECKO190_20091130_RELBRANCH

Actually I don't have cvs setup right to checkin, I didn't receive checkin access until we switched to mercurial. Can someone with cvs checkin rights land this for me?

Comment 36

[:Gavin Sharp [email: [e-mail]]]

Checked in on CVS HEAD and GECKO190_20091130_RELBRANCH:

mozilla/extensions/auth/nsAuthSSPI.cpp 	1.15 
mozilla/extensions/auth/nsAuthSSPI.cpp 	1.14.2.1

Comment 37

[mxh]

I am a Firefox user.  I see the status is "RESOLVED FIXED".  Can someone please explain to me in laymans terms ow to apply this fix?  As it stands I have just two options- use an older insecure firefox or another browser like chrome..

Comment 38

[chris hofmann]

fixes are applied to the latest nightly builds.

I'd recommend grabbing one of the install packages from
ftp://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-1.9.2/

then update to the next Firefox 3.6 beta or release candidate when it comes out.

If you can please confirm that the new build works for you to fix this problem.

thanks for your participation.

Comment 39

[Al Billings [:abillings - ex-MoCo]]

Actually, Chris, it would be more helpful to have the Firefox 3.5 or 3.0 version of this fix verified than to get people on the 3.6 beta.

For Firefox 3.5.6 users, they can go to http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.5.7-candidates/build1/ and install the Firefox 3.5.7 candidate build for their locale (probably en-US). If people do so, it would be nice if they reported back here if it fixed the issue for them.

For those seeing the problem in Firefox 3.0.16, they can try the 3.0.17 candidate build at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0.17-candidates/build1/.

Comment 40

[T.R. Finneran]

Fix seems to work without problems in corporate environment using Firefox 3.5.7 candidate for en-US on Windows XP. Problem occurs in 3.5.6. Problem does not occur in 3.5.7 or 3.5.5.

Comment 41

[Al Billings [:abillings - ex-MoCo]]

Thank you, T.R. It is good to hear a verification of the fix. Is there any way we could convince you to try with the Firefox 3.0.17 candidate in your local environment as well?

Comment 42

[T.R. Finneran]

Firefox 3.0.17 candidate also seems to work without problems in corporate environment. en-US version.

BTW - Thanks everyone for the quick fix. I look forward to seeing this new version released. The Mozilla team rocks!

Comment 43

[Al Billings [:abillings - ex-MoCo]]

That's a big help. Thank you for checking with both versions.

Comment 44

[mxh]

I can confirm that the hanging problem seems to be fixed in  Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2b6pre) Gecko/20091221 Namoroka/3.6b6pre (.NET CLR 3.5.30729) ID:20091221162325 too.  Thanks.

Comment 45

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #44)
> I can confirm that the hanging problem seems to be fixed in  Mozilla/5.0
> (Windows; U; Windows NT 5.1; en-US; rv:1.9.2b6pre) Gecko/20091221
> Namoroka/3.6b6pre (.NET CLR 3.5.30729) ID:20091221162325 too.  Thanks.

Thank you for testing the 3.6 build and for the verification. It's really appreciated. Marking as verified fixed on 1.9.2.

Comment 46

[T.R. Finneran]

An occasional Proxy error occured when using Firefox 3.5.7 candidate for en-US on Windows XP when accessing a new https site. Proxy error occured three times until I loaded same URL in IE 8. After doing so, FF 3.5.7 no longer had any issues with that site or any other https URL. The problem may or may not be with our company's proxy server itself. Sorry, but this problem is very hard to reproduce and I do not remember the exact error message. Is there anywhere in the code where the protocol https vs http would make a difference? I would guess that the problem is not Mozilla browser related, but is just a flakey proxy server issue which would not be unusual for our company's network.

Comment 47

[beLIEve]

I have an NTLM authentication problem reported since Firefox 3.5.5. Despite installing Firefox 3.5.7, I still notice a difference between Internet Explorer 8 and Firefox 3.5.7.

Internet Explorer 8 sends the IP address of the Proxy in its NTLMv2 Response, while Firefox 3.5.7 sends the hostname of the URL being accessed. Can anyone comment if this will introduce problem with authentication?


Internet Explorer 8 :
            NTLM Message Type: NTLMSSP_AUTH (0x00000003)

                NTLMv2 Response:
...
                    Name: Unknown, HTTP/10.10.1.131
                        Name type: Unknown (9)
                        Name len: 32
                        Name: HTTP/10.10.1.131				<----- IP address of the HTTP Proxy
                    Name: End of list
                        Name type: End of list (0)
                        Name len: 0

Firefox 3.5.7 :

            NTLM Message Type: NTLMSSP_AUTH (0x00000003)

                NTLMv2 Response:
...
                    Name: Unknown, HTTP/en-us.start3.mozilla.com	<----- hostname of the URL being accessed
                        Name type: Unknown (9)
                        Name len: 58
                        Name: HTTP/en-us.start3.mozilla.com

Comment 48

[Wan-Teh Chang]

beLIEve: thanks a lot for the info.  Do you specify your
proxy server using its IP address or host name?

Jim: I think we're sending the wrong host name when the
target of the NTLM authentication is a proxy server.
In that case we should use the host name or IP address
of the proxy server rather than the destination server.

Comment 49

[Jim Mathies [:jimm]]

(In reply to comment #48)
> beLIEve: thanks a lot for the info.  Do you specify your
> proxy server using its IP address or host name?
> 
> Jim: I think we're sending the wrong host name when the
> target of the NTLM authentication is a proxy server.
> In that case we should use the host name or IP address
> of the proxy server rather than the destination server.

Hmm, what's your basis for that, observing IE? (I don't have an ntlm proxy to test with unfortunately.) We have a number of outstanding bugs related to ntlm and proxy authentication, so this might not be related the sn.

Comment 50

[Wan-Teh Chang]

Jim: the basis is who we're doing the NTLM challenge response with.
When a proxy server requests NTLM authentication using a
Proxy-Authenticate header, we send our NTLM response in a
Proxy-Authorization header, which is intended for the proxy,
not the server we want to visit ultimately.  The server we
want to visit ultimately is not part of the NTLM authentication.

Comment 51

[Jim Mathies [:jimm]]

(In reply to comment #50)
> Jim: the basis is who we're doing the NTLM challenge response with.
> When a proxy server requests NTLM authentication using a
> Proxy-Authenticate header, we send our NTLM response in a
> Proxy-Authorization header, which is intended for the proxy,
> not the server we want to visit ultimately.  The server we
> want to visit ultimately is not part of the NTLM authentication.

Isn't the sn supposed to be a unique identifier of the service we are attempting to connect to? In which case it should be the host running the service.

Comment 52

[beLIEve]

Thanks a lot for your response Wan-Teh and Jim.

Wan-Teh : The proxy is specified using its IP address. And btw, this is an explicit HTTP Proxy.

Jim : I compared against Internet Explorer because NTLM Authentication worked with Internet Explorer but failed with Firefox on Vista and Windows 7. This problem is not easily reproducible - we relied on data from a customer.

Firefox 3.5.5 did not send any of such hostname/IP address in its NTLMv2 Response while Firefox 3.5.7 sends the hostname of the server being accessed. As this is a recent fix for NTLM, I posted here to make an inquiry instead of filing a new bug.

Comment 53

[Wan-Teh Chang]

Jim: a proxy server is also a service.  Another way to
look at this issue is which computer will consume the
NTLM blob we generate.  The NTLM blob in a
Proxy-Authorization header will be consumed by the
proxy server, so the sn should identify the proxy server.

Comment 54

[Jim Mathies [:jimm]]

(In reply to comment #53)
> Jim: a proxy server is also a service.  Another way to
> look at this issue is which computer will consume the
> NTLM blob we generate.  The NTLM blob in a
> Proxy-Authorization header will be consumed by the
> proxy server, so the sn should identify the proxy server.

Ok, so I will try to put together a patch for this and maybe we can get some folks to experiment with a try build. My only concern is that we might be mixing past issues into this, we have existing bugs related to repeated proxy auth prompting that are quite old - specifically bug 318253, and bug 439463.

Comment 55

[beLIEve]

Thanks Jim. I can get the people affected to test it out.

The problem I reported happens only with Firefox on Vista and Windows 7. It works fine on XP. I compared the Wireshark capture between Firefox and Internet Explorer 8 and that was the difference I found. And of course, it happened on a GET and not CONNECT. Hence, I'd rule out the 2 bugs you listed.

Comment 56

[David :Bienvenu]

fixed for 2.0.24

Comment 57

[Rationale]

I have a proxy authentication with NTLM configured and impacted with this where the username and password prompt seen even though the Firefox is sending NTLM 1 and 3 automatically. I confirm that the behavior seen on PCAP level is consistent with this bug. Could anyone point me to a download link for 2.0.24?

Comment 58

[Jim Mathies [:jimm]]

(In reply to comment #57)
> I have a proxy authentication with NTLM configured and impacted with this where
> the username and password prompt seen even though the Firefox is sending NTLM 1
> and 3 automatically. I confirm that the behavior seen on PCAP level is
> consistent with this bug. Could anyone point me to a download link for 2.0.24?

Possibly bug 542318?

Comment 59

[mxh]

Ok I'm back, I wanted to add that there's definitely something different about Firefox's handling over IE or even Chrome.  We have some intranet sites that now simply refuse to open on FF since 3.5.7.  That bug is now carried over to 3.5.6 as well.  These sites include a new web app we are building as well as things like SSRS.  Let me know if you need me to test anything on my setup, and if so, how.

Comment 60

[mxh]

The above should read "That bug is now carried over to 3.6 as well."  fwiw even Opera can open these intranet sites.
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 ID:20100115144158
Adblock Plus 1.1.3
Add to Search Bar 2.0
AutoPager 0.6.0.12
Context Search 0.4.5
Default Full Zoom Level 4.3
FEBE 6.3.2
Flashblock 1.5.11.2
Handytag 2.1
Image Zoom 0.4.2
LastPass 1.64.4
Linkification 1.3.7
Nightly Tester Tools 2.0.3
Read It Later 2.0.3
Session Manager 0.6.7.4
SkipScreen 0.3.20091214_AMO
Zoom toolbar 0.9

Comment 61

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Can you please stop messing up with flags? Thanks!

If you see the issue again with Firefox 3.6 while 3.6b6 was WFM, then run Firefox in Safe Mode (http://support.mozilla.com/kb/Safe+Mode) for testing purposes.

Comment 62

[mxh]

So how do we get this bug reopened?

Comment 63

[Al Billings [:abillings - ex-MoCo]]

Open a new bug.

Comment 64

[mxh]

(In reply to comment #63)
> Open a new bug.

According to https://bugzilla.mozilla.org/page.cgi?id=fields.html#status that's not the way to do it.  We need to Reopen, not create a new bug.

Comment 65

[Al Billings [:abillings - ex-MoCo]]

Only if it is *exactly* the same issue as the original issue. In other words, if the exact original issue has not actually been fixed. If this is a variant issue, which is what it sounded like to me, you'll get more traction by opening a new bug given the number of comments on the existing bug.