Closed Bug 1057266 Opened 10 years ago Closed 9 years ago

Firefox sends NTLM phase 3 message after a successful NTLM authentication

Categories

(Core :: Networking, defect)

31 Branch
x86_64
Windows 7
defect
Not set
normal

Tracking

()

RESOLVED DUPLICATE of bug 486508

People

(Reporter: gerrit.hohl, Unassigned)

References

()

Details

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0 (Beta/Release)
Build ID: 20140716183446

Steps to reproduce:

I'm using Firefox v31.0 (Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0) on Windows 7 (64 Bit, All patches installed). I request the following public PDF document from our Typo3 website using a proxy which we programmed at our own. The proxy requires authentication via NTLM.
http://www.aurenz.de/fileadmin/user_upload/Downloads/Infopool/WebFox/Aurenz_WebFox_Proxy_Filtering.pdf


Actual results:

1.) Firefox opens a connection to the proxy and sends the request. The proxy replies with an authentication request. The connection is closed by the proxy afterwards (Reply is HTTP/1.1, "Connection: close" and "Proxy-Connection: close" are set).

2.) Then Firefox opens a connection again and performs a NTLM authentication (Phase 1 message from browser, phase 2 message from proxy, phase 3 message from browser) successfully. As response to the phase 3 request the requested resource is returned (which is the PDF document). Afterwards the connection is closed by the browser.

3.) Firefox opens another connection which is denied again with a authentication request. The connection is closed by the proxy.

4.) Then Firefox opens another connection and performs again a NTLM authentication which is also successfully. Also in this case it gets the resource (which is the favicon) as reply to the request containing the phase 3 NTLM message.

5.) But then the proxy gets a request for another resource from Firefox containing a phase 3 NTLM message. The proxy denies the request with an authentication request as the proxy didn't expected any authentication and also isn't able to start a new authentication handshake (maybe it wants to use a different user, but the same connection) with a phase 3 message. The proxy closes the connection afterward sending the reply.

6.) Now the steps 4.) and 5.) are repeated for every resource needed. The requests are for partial content of the PDF document. Why Firefox requests these resources (PDF document, favicon, partial content of the PDF document) and why in this order has to be discussed with the team of the embedded PDF viewer of Firefox (which I used in this example).

The NTLM messages from step 4.) and 5.) look pretty similar and also have a similar content (same target name, same workstation name, same flags, etc.). So I guess (but only guess) that the difference are some timestamp related informations.

The responses of step 2.) and step 4.) Firefox gets from the proxy (not from the target webserver) look pretty similar:

HTTP/1.1 200 OK
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 899715
Date: Fri, 22 Aug 2014 07:18:57 GMT
Content-Type: application/pdf
ETag: "2b40d9e-dba83-4f6c4e0992b40"
Server: Apache
Last-Modified: Fri, 11 Apr 2014 14:09:09 GMT
Accept-Ranges: bytes
Keep-Alive: timeout=3, max=25
Via: 1.1 SRVISA1, 1.1 webfoxcore (webfoxcore/4.5 (Build 54))

HTTP/1.1 200 OK
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 1150
Date: Fri, 22 Aug 2014 07:18:59 GMT
Content-Type: image/x-icon
ETag: "27cd12a-47e-47ce3272502c0"
Server: Apache
Last-Modified: Mon, 11 Jan 2010 13:02:43 GMT
Accept-Ranges: bytes
Keep-Alive: timeout=3, max=25
Via: 1.1 SRVISA1, 1.1 webfoxcore (webfoxcore/4.5 (Build 54))

I don't recognize anything which could trigger that behaviour.


Expected results:

I would have expected that Firefox would behave exact the same way it does with any other website using our proxy:

1.) Open a connection and request a resource. Getting an authentication request as response. Connection is closed.

2.) Open a new connection, perform the NTLM handshake, getting the resource and sending more request without having an authentication data in it using that connection (keyword: persistent connections) - or simply closing it, if it is not needed anymore.

3.) Performing the steps 1.) and 2.) for any needed resource / connection.

An alternative would be that Firefox would perform an authentication not only for every connection, but for every requested resources (which would be an unnecessary overhead for all involved components - Firefox and the proxy and/or web server). And then it should start with NTLM phase 1 each time.
Can you reproduce this with something simpler than a PDF, like a single HTML document (perhaps requesting a single image as well, to trigger a third load) ? As it is, the setup is quite complicated and hard to replicate (in fact, I guess the proxy software is not publicly available in the first place?), which makes it more difficult to figure out where any buggy behaviour comes from and/or how to reproduce it.
Flags: needinfo?(gerrit.hohl)
Hello Gijs,

thanks for the very fast reply. :)

Okay, I did another few tests. It seems that Firefox v31.0 as well as Internet Explorer v11.0 are doing re-authentications of already authenticated connections every now and then, also Firefox does it a lot more often, while IE stops doing it after some time (maybe some kind of warm-up phase?). I tested it with different web site (HTML containing links to pictures, Flash and so on). I don't know the reason why they do it. Maybe the NTLM messages from the Windows Server contain some information when the NTLM client should authenticate again. But it's only a guessing. I know NTLM only as far as described here (without the decryption and cipher things): http://davenport.sourceforge.net/ntlm.html

I tried to request different PDF files (everytime I cleared the cache, closed the browser, open it [default site: about:config] and opened directly the link to the PDF file):
- http://www.w3.org/Protocols/HTTP/1.1/rfc2616.pdf
- http://www.zueblin.de/databases/internet/_public/files.nsf/SearchView/00357272630EE648C1257CDF003710A3/$File/Screen_ES%2014962_Geschaeftsbericht_2013.pdf?OpenElement
There I don't have the same problems. But I also have to admit that the requests patterns are not exactly the same as when I try to open the PDF file I mentioned first.

I also tried to access the problem candidate using the Adobe Acrobat Adobe 11.0.8.4 plug-in. In this case there weren't any problems.

No, unfortunately it's not publicly available. It would be possible to give you a 30-days-trial version. But the problem seems very specific as it only appears with that single PDF file deliviered by Typo3 from that Apache web server.
I will perform a Wireshark session on a Squid proxy using NTLM. Unfortunately it is an older one (2.7.STABLE6). But it should be okay just for comparison.

I'll write a comment here again after I've done that.
Flags: needinfo?(gerrit.hohl)
Okay, I set up a Squid 2.7.STABLE6 with NTLM. And I get exactly the same behaviour of Firefox v31.0 if I request the named PDF document: It also sends a NTLM type 3 message after requesting the favicon on the same connection. There is only one difference between our proxy and Squid (at least in that version):

Our proxy starts an authentication every time it gets a credentials (means "Proxy-Authorization") from the client. And if that authentication fails it closes the connection after the response that an authentication is required (means "Proxy-Authenticate").

Squid gets an error message from the external NTLM authenticator in this case. But it only writes it to its cache.log file and processes the request if nothing happened.

2014/08/22 11:52:06| authenticateNTLMHandleReply: Error validating user via NTLM. Error returned 'BH 2:-2 80090308'

This 'BH 2:-2 80090308' error is from the NTLM authenticator I used. That error code 0x80090308 can be found in the description of the "AcceptSecurityContext (NTLM) function":
http://msdn.microsoft.com/en-us/library/windows/desktop/aa374707%28v=vs.85%29.aspx
SEC_E_INVALID_TOKEN - 0x80090308L - The function failed. The token passed to the function is not valid.
Which - I guess - means that the type of the NTLM message is wrong (because it's a type 3 without receiving a corresponding type 1 before it).



Somehow there seems to be a constellation in which Firefox (or the NTLM library it uses or the NTLM API of Windows which is used by the NTLM library) performs a wrong NTLM authentication in this single case. And the proxy doesn't seem to be the problem.

If you want to try it: As far as I know e.g. Ubuntu 9.10 Karmic contains Squid 2.7.STABLE6.
http://old-releases.ubuntu.com/releases/9.10/
Or maybe you can use the Windows port of that Squid version. I guess it would be easier than to set up the NTLM authentication. Otherwise you would need also to install Samba and get it working with Squid and your Windows Active Directory.
I have a few more URLs, all of them are PDF files. But it doesn't happen only with our Typo3 site, it also happens with other sites. And all of these were open with Firefox v31.0 using it's built-in internal PDF viewer.

http://www.aurenz.de/fileadmin/user_upload/Downloads/Infopool/WebFox/Aurenz_WebFox_Proxy.pdf
http://www.aurenz.de/fileadmin/user_upload/Downloads/WebFox_Summary.pdf
http://www.aurenz.de/fileadmin/user_upload/Downloads/ReleaseNotes/Aurenz_WebFox_Deltaliste.pdf
http://www.fortinet.com/sites/default/files/productdatasheets/FortiGate-60D.pdf
One hint for folks looking to reproduce issues like this with samba and ntlm_auth.  The --password option to ntlm_auth allows you to set a forced password, so there is no need to hook up winbindd and AD, it can just tests against that fixed password. 

For example:

auth_param ntlm program /data/samba/samba4/prefix/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp --password=password
Depends on: 1160000
I'm really sorry for letting this bug get lost. Maybe Dragana can help move this forward.
Flags: needinfo?(dd.mozilla)
Product: Firefox → Core
Component: Untriaged → Networking
Sorry for a delay, i will take a look next week.
Can you still reproduce this?

I have tried with Windows XP(I only have a old windows machine) and with squid3 and I cannot reproduce it. It uses the same connection for the next request.
Flags: needinfo?(dd.mozilla) → needinfo?(gerrit.hohl)
I have tried it with Nightly.
Hello Dragana,

I just happened yesterday - at least my log says that. We have Windows 7 and the browser was Mozilla Firefox v38.0. I haven't tested it yet with Squid, but with our own written proxy. But in this aspect the behavior is the same like using Squid.
Flags: needinfo?(gerrit.hohl)
I have tried with Firefox 38 a well, but no luck reproducing it.

may I ask you to send me a Http log (made with your test Squid proxy)
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

and squid config file
Flags: needinfo?(gerrit.hohl)
Hello Dragana,

I'm sorry for my late reply. I saw your comment, but I didn't have the time to reply. Unfortunately I working on a different project at the moment. But I will post the data you requested (log and config file) as soon as possible.
I will mark this as dup of bug 486508. Probably fix for that bug should fix this too.
Status: UNCONFIRMED → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Flags: needinfo?(gerrit.hohl)
You need to log in before you can comment on or make changes to this bug.