Closed Bug 486508 Opened 14 years ago Closed 6 years ago
An error during SSL transfer can cause Firefox to send an invalid Proxy NTLM Sequence on auto retry
8.43 KB, application/octet-stream
158 bytes, text/plain
34.92 KB, application/octet-stream
2.66 KB, patch
|Details | Diff | Splinter Review|
104.96 KB, text/plain
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:184.108.40.206) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1 Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:220.127.116.11) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1 I am developing a proxy and found an issue with firefox's retry/NTLM code. Browsers doing NTLM authentication on a new connection must first send a Proxy-Authorization header with a NTLM type 1 packet, then get a challenge from the proxy, and finally send a NTLM type 3 packet to complete the handshake. Here's what can happen with firefox, and is in the attached packet trace. Connection A: Does 3 way NTLM handshake to proxy for a "CONNECT" request to get a ssl-encrypted file. Proxies final header reply is 200 Connected. Then, the ssl sequence is negotiated successfully with the web server. Finally, the file from the server doesn't transfer successfully and instead the connection is closed and the proxy closes the connection to the browser with a socket close() TCP FIN (That's all it can really do at this point in a CONNECT tunnel). Then firefox does this: Opens a new connection B as a retry of connection A, and immediately sets the Proxy-Authorization header to the last NTLM type 3 packet from Connection A, which is invalid. NTLM authentication must always start with type 1. Thus, the samba helper I am using gives this error: libsmb/ntlmssp.c:ntlmssp_update(333) got NTLMSSP command 3, expected 1 By the way, this may be what is causing bug 366562. I have a packet trace and a 100% reproducible case. This bug is on linux and windows clients: ua=Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:18.104.22.168) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 3.5.30729) Reproducible: Always Steps to Reproduce: 0. Configure to use a NTLM authenticated proxy 1. On an apache web server, make a test-file.cgi script that when called, kills httpd to simulate a network/server error during ssl transfer 2. in firefox, browse to https://apache_server/test-file.cgi Actual Results: 3. you will probably get a popup for auth, or on my proxy will receive a 400 bad request error. 4. also, you can confirm via packet trace that firefox attempted a retry connection and started it with the NTLM type 3 packet of the first connection. Expected Results: Firefox should either give up after the first download attempt failed, or retry properly with the correct NTLM type 1 handshake sequence.
This is the test case script, it runs under mod_python on apache. #!/usr/bin/python import os def index(req): # this won't kill the parent, which has root uid, # but it will kill us os.system("killall -TERM httpd") #req.write("Hello\n")
Firefox 3.1 b3 still has this problem, on windows anyway. O On Linux however, using the fedora 11 distro build, it doesn't appear to retry.
Oops, I forgot I turned off NTLM authentication temporarily the other day for my Linux box. Turning it back on, 3.1b3 still has the problem on Linux too. Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3) Gecko/20090327 Fedora/3.1-0.11.beta3.fc11 Firefox/3.1b3
I can confirm Firefoxes behaviour here. We have the same issue here and I assume it's quite sure that Bug 366562 is about the same thing. This topic definately needs attention. If you're in a company doing let's say IWA or NTLM authentication and you're receiving a RST as a client for whatever reasons, there shouldn't be a user unfriendly authentication popup, which -on top- won't accept your (correct) credentials anyway. I can upload a pcap as well.
This topic definately needs some attention payed by someone from the Mozilla engineering team! https://bugzilla.mozilla.org/show_bug.cgi?id=318253, created 30.11.2005, Status: New https://bugzilla.mozilla.org/show_bug.cgi?id=366562, created 10.01.2007, Status: Unconfirmed https://bugzilla.mozilla.org/show_bug.cgi?id=501361, created 30.06.2009, Status: Unconfirmed https://bugzilla.mozilla.org/show_bug.cgi?id=486508, created 02.04.2009, Status: Unconfirmed In my opinion, the Mozilla team did a really good job in all the way, Firefox developed – but here is a glitch and that thing is really obvious – if Firefox is used in a business environment together with NTLM authentication.
Was extended protection installed on the system? (http://www.microsoft.com/technet/security/advisory/973811.mspx) We are noticing problems with Firefox IWA authn when that is installed.
Reporter, please retest with Firefox 3.6.12 or later in a fresh profile (http://support.mozilla.com/kb/Managing+profiles). Also update your plugins (flash, adobe reader, java, quicktime, silverlight, etc.) Go to the developer's website and download the latest version from there. If you no longer see this issue, please close this bug as RESOLVED, WORKSFORME. If you do see the bug, please post a comment.
Whiteboard: [CLOSEME 2010-12-01]
No reply, INCOMPLETE. Please retest with Firefox 3.6.12 or later and a new profile (http://support.mozilla.com/kb/Managing+profiles). If you continue to see this issue with the newest firefox and a new profile, then please comment on this bug.
Status: UNCONFIRMED → RESOLVED
Closed: 12 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: INCOMPLETE → ---
Using the attached script, I can reproduce the issue 100%. I used Squid as an NTLM auth proxy, configured with: auth_param ntlm program /data/samba/samba4/prefix/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp --password=password auth_param ntlm children 20 startup=0 idle=1 auth_param ntlm keep_alive on I reproduced this with current nightly, with essentially the same result.
Updated capture to show the current failure. The behaviour looks much like what is expected after fix for bug 734229.
This is I think part of the correct fix for this issue. It is never correct to send an NTLM or Negotiate header more than once, so remove it from the HTTP request once we write it out in the proxy CONNECT. A similar patch will be needed in the bare HTTP case, once per TCP socket, but I can't figure out where to do that.
Attachment #8628170 - Flags: feedback?(honzab.moz)
(In reply to Andrew Bartlett from comment #13) > Created attachment 8628170 [details] [diff] [review] > A proof of concept for clearing the NTLM header once sent > > This is I think part of the correct fix for this issue. It is never correct > to send an NTLM or Negotiate header more than once, so remove it from the > HTTP request once we write it out in the proxy CONNECT. > > A similar patch will be needed in the bare HTTP case, once per TCP socket, > but I can't figure out where to do that. I do not see bug 734229, so I do not know what it has changed. I don't think you need to change the function that you have change in the patch. Can you explain what you want to do? Is it the cause of the problem the fact that after a ssl error the first packet already contains a ntlm header and it should be just plain request, no ntlm?
Attachment #8628170 - Flags: feedback?(dd.mozilla) → feedback-
(In reply to Dragana Damjanovic [:dragana] from comment #14) > (In reply to Andrew Bartlett from comment #13) > > Created attachment 8628170 [details] [diff] [review] > > A proof of concept for clearing the NTLM header once sent > > > > This is I think part of the correct fix for this issue. It is never correct > > to send an NTLM or Negotiate header more than once, so remove it from the > > HTTP request once we write it out in the proxy CONNECT. > > > > A similar patch will be needed in the bare HTTP case, once per TCP socket, > > but I can't figure out where to do that. > > I do not see bug 734229, so I do not know what it has changed. bug 734229 was partially fixed with https://hg.mozilla.org/integration/mozilla-inbound/rev/4ea6bc9e2fd5 > I don't think you need to change the function that you have change in the > patch. Can you explain what you want to do? I want to remove the NTLM header from Mozilla's knowledge, once used it can never be re-used (even on the same TCP socket). Additionally, to make a new one, we need to start with an un-authenticated request. > Is it the cause of the problem the fact that after a ssl error the first > packet already contains a ntlm header and it should be just plain request, > no ntlm? Yes. After any NTLM header is sent, we must never send it again, even if retrying the connection. Andrew Bartlett
(In reply to Andrew Bartlett from comment #15) > > I want to remove the NTLM header from Mozilla's knowledge, once used it can > never be re-used (even on the same TCP socket). Additionally, to make a new > one, we need to start with an un-authenticated request. > > > Is it the cause of the problem the fact that after a ssl error the first > > packet already contains a ntlm header and it should be just plain request, > > no ntlm? > > Yes. After any NTLM header is sent, we must never send it again, even if > retrying the connection. > > Andrew Bartlett My suggestion is to reset the ntlm state (headers and everything else) once 200 is received on an error after the last NTLM message - so at the point of receiving a response. It can happen that for some reason connection breaks and in that case we can still resend the last ntlm header (receiver had not received that header at all) You should delete the header in function nsHttpChannel::ProcessResponse(): http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpChannel.cpp#1339
(In reply to Dragana Damjanovic [:dragana] from comment #17) > > My suggestion is to reset the ntlm state (headers and everything else) once > 200 is received on an error after the last NTLM message - so at the point of > receiving a response. > It can happen that for some reason connection breaks and in that case we can > still resend the last ntlm header (receiver had not received that header at > all) No, this won't work. NTLM is tied to the TCP socket, it is *never* valid to resend an NTLM authenticate header on a different socket. Indeed, it is in particularly the case that we must remove the header if the connection breaks for any reason. We could get into games around the first 'negotiate' header vs the third 'authenticate' header, but even then we get into nasty failure modes (potential loops), so we should on failure go back to pretending not to know any authentication and let the whole process retry. > You should delete the header in function nsHttpChannel::ProcessResponse(): > http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/ > nsHttpChannel.cpp#1339 I think this would be too late, we need to remove it as soon as it is written to the socket, because it is tied to the TCP socket. Can you indicate where that is handled? Thanks, Andrew Bartlett
The headers are write to a socket at: http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpConnection.cpp#1542 and they are prepared in: http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpConnection.cpp#1792 or maybe the best place to do it is in: http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpTransaction.cpp#1161 this is case when a connection is restarted. This scenario I have seen in two logs (bug 318253). The socket is reset and at that point we try to reconnect, to open a new socket. The headers will not be written twice except on reconnect (previous case) or when it try to resolve an error: http://mxr.mozilla.org/mozilla-central/source/netwerk/base/nsSocketTransport2.cpp#1471
There are 3x2x2 cases: - plain HTTP, plain PROXY, HTTPS+plain proxy - a new connection, reused connection - no cached challenge+credentials, cached challenge+credentials In all cases we must send Proxy-Authorization and WWW-Authorization only for the first use time (per connection lifetime) and always start with NTLM1. Seems like two things are broken: 1. we are sending NTLM3 instead of NTLM1 *in case of cached credentials*, looks like a bad reuse of continuationState (the auth module), we must drop mAuthContinuationState at a certain point 2. we are adding the Authorization headers more than needed (needs to be bypassed for an already authenticated connection) ; this may effectively make problem 1 moot I think we need to expose flags on nsHttpConnection like NTLMProxyAuthenticated (!mCompletedProxyConnect will do) and NTLMWWWAuthenticated. These two will drive adding the headers, best somewhere in nsHttpNegotiate/NTLMAuth services. Andrew, do you mind if I steal this bug from you?
Assignee: abartlet → honzab.moz
Also, this could be as easy as drop of the mProxyAuthContinuationState... Will check.
So, we are definitely not re-sending WWW-Authorization header on an NTLM connection because the nsHttpHTLMAuth authenticator doesn't set the REUSABLE_CREDENTIALS flag. However the proxy auth NTLM3 message on CONNECT is a bit mystery for me. Andrew, I'm trying to config squid according your comment 10 on my linux box. The proxy asks for a password but I'm not sure what to enter. I added a test user but the proxy doesn't accept that credentials. I don't run AD on that linux box. Can you please give me an advice how to setup a test account that ntlm_auth sees? Thanks.
Comment on attachment 8628170 [details] [diff] [review] A proof of concept for clearing the NTLM header once sent Review of attachment 8628170 [details] [diff] [review]: ----------------------------------------------------------------- Definitely not the proper way, but I believe it fixed the problem. It's however somewhere else. When I have a way to reproduce I'll find out quickly.
Attachment #8628170 - Flags: feedback?(honzab.moz) → feedback-
(In reply to Honza Bambas (:mayhemer) from comment #22) > So, we are definitely not re-sending WWW-Authorization header on an NTLM > connection because the nsHttpHTLMAuth authenticator doesn't set the > REUSABLE_CREDENTIALS flag. > > However the proxy auth NTLM3 message on CONNECT is a bit mystery for me. As it it an an identical string, I think the transaction is being re-played as-is. > Andrew, I'm trying to config squid according your comment 10 on my linux > box. The proxy asks for a password but I'm not sure what to enter. I added > a test user but the proxy doesn't accept that credentials. I don't run AD > on that linux box. The password is password :-) Back in 2003 (samba git adc07646a3057e5c572edb3ebdac2367206aaeef), I thought it might be handy to just have a fixed password able to be configured directly in ntlm_auth, so I re-used the --password option :-) That is what the --password=password part does :-) > Can you please give me an advice how to setup a test account that ntlm_auth > sees? Thanks. Any username will work :-) Andrew Bartlett
I won't have much time to deal with this right now.
Assignee: honzab.moz → nobody
Component: General → Networking
Product: Firefox → Core
Whiteboard: [http-conn] → [http-conn][necko-backlog]
Firefox 48 is generating unexpected prompts for credentials when using an NTLM authenticated proxy server.
Do you have "Never remember history" option in Options/Advanced/History set? There is a known regression, already fixed in Firefox 49 (currently in Beta, bug 1291700).
Presuming that you are referring to Options>Privacy>History, I have the following settings: *Tracking protection enabled *Use custom settings for history **Remember my browsing and download history **Accept cookies from sites ***Accept third-party cookies: Always ***Keep until: They expire I tried setting the update channel to "beta" in about:config and restarted Firefox multiple times, but it wouldn't grab the 49 beta version of Firefox. I ended up installing Firefox 49.0b6 manually. After doing so, I couldn't immediately reproduce the problem, so I'll have other affected users try this beta version.
(In reply to raynebc from comment #29) > Presuming that you are referring to Options>Privacy>History, I have the > following settings: > *Tracking protection enabled > *Use custom settings for history > **Remember my browsing and download history > **Accept cookies from sites > ***Accept third-party cookies: Always > ***Keep until: They expire > > I tried setting the update channel to "beta" in about:config and restarted > Firefox multiple times, but it wouldn't grab the 49 beta version of Firefox. > I ended up installing Firefox 49.0b6 manually. After doing so, I couldn't > immediately reproduce the problem, so I'll have other affected users try > this beta version. Thanks for the update.
So far I haven't seen that this bug occurs in the latest beta versions.
The user indicates he's still getting unexpected prompts to authenticate with the proxy server, despite frequently getting new beta builds.
For those this might help: I to got the problem with NTLM authentication on Windows 7.0 / FireFox connecting to a Linux server in an Windows - based network after updating to version 48.0.2. There was an reset program on the Mozila website, I can not find it anymore, perhaps it's only shown when it detects the problem. After clicking on that it did do a number of resets within our FireFox installations on the problem PC's. Afterwards I only needed to add the apropriate Linux machine in de network.automatic-ntlm-auth.trusted-uris and it all works fine again.
To clarify, this isn't just happening with websites that use integrated authentication (ie. ones that can automatically use the logged on user's credentials). It's happening with any website because the proxy server itself requires the credentials.
User indicates this still occurs in the latest beta version (50.0b4). He works around it by cancelling the prompt and usually it resumes working normally for a while. Occasionally he instead receives a "connection refused" error and has to reload the page.
I've just set up a server that consistently replicates the issue. Applying Andrews patch fixes the issue. From the comments, it would appear that this is not the right place for the fix, and bug 1309438 which the patch does not fix. I've got time to take a look at this if someone wants give me some direction. Alternatively I can open up the server, currently only for nominated source IP's.
I've done a bit more testing and Andrews patch also mostly fixes bug 1309438. It's not perfect in that there is one extra prompt for the proxy auth details.
It looks like I spoke to soon, muddled the http and https test cases. Will investigate more and update when I've got more definitive data.
I applied Andrew's patch incorrectly, have reapplied correctly the patch and can confirm that it resolves the issue.
(In reply to Gary Lockyer from comment #39) > I applied Andrew's patch incorrectly, have reapplied correctly the patch and > can confirm that it resolves the issue. See comment 14 and 23. We can't take the patch in the form as it is. The fix needs to be done in more proper way. Although, it seems like you are able to reproduce, right? If you are willing to help, I may ask you for logs (if so, I will provide details later - have to think first). Or at least give you patches for testing, but I won't have time for working on this very soon :( The best would be to have a test case in hands, so I could reproduce. That would very much speed up the fix here.
I can reproduce consistently, be more than happy to provide logs. Alternatively I can open up the test server, only to known source IP addresses though.
Have done some further investigation. It appears to be in the nsHttpTransaction restart logic, the Connection request is sent immediately on socket connection, which in this case contains the NTLM type 3 header. So it looks like there needs to be some way to either trigger an NTLM type 1 header, or clear the existing type 3 header. As a quick check I modified nsHttpTransaction::Restart to return NS_ERROR_NET_RESET immediately, this stops the password re-prompting loop.
Patch to fix the issue, essentially it's just Andrews patch moved closer to the source of the issue. This is not intended as a real solution based on the comments for the original patch, but it does serve to document where the issue is.
The sequence of events that I can see is as follows. 1) NTLM proxy auth succeeds, ProxyAuthorization header remains set in the httpTransaction and contains the NTLM type 3 message 2) https connection is reset 3) httpTransaction::Restart gets called 4) The connection is reestablished, and the proxy connection string is rebuilt containing the NTLM type 3 message from step 1, which is then sent. Based on what I can see in the logs the original connection is released in nsHttpTransaction::Close and a new one created.
Gary, thanks for all the effort and narrowing down the cause! I intend to take a look at this bug today or tomorrow.
Assignee: nobody → honzab.moz
Whiteboard: [http-conn][necko-backlog] → [http-conn][necko-active]
OK, Gary, I will ask you to provide logs, just make sure I follow the right paths. I assume you are on the current Nightly. You can just go to about:networking, then the Logging section of it and then just press the [Start Logging] button. Then reproduce the problem. Best would be to have two logs, one without the patch (vanilla nightly) and one with your patch. Finally, send me the log files (there will be a path to it shown under Current Log File on the about:networking page when logging is started). Just send them to my bugzilla email, since it may contain some sensitive information. To make things a bit easier, please refer URLs you navigate to, so I can find them more easily in the log. Thanks!
Submitted wrong version of the patch initially, this is the correct version
Attachment #8802365 - Attachment is obsolete: true
Thanks Gary. I was testing locally, debug build, local IIS 10 server, a page referring 60 images, all of them behind NTLM (same credentials). I can see our parallelism code simply mixes connections, channels that drive the authentication loops among each other. In other words, connections are not sticky to the channel anymore from some reason (a different bug I will file shortly).
Anyway, one solution to the problem you identified in this bug is to never reset the transaction (it means to deploy it on a new connection). That doesn't make any sense when doing the NTLM auth loop. The state of the auth process is held in nsHttpChannel (it's mAuthProvider member). That (AFAIK) cannot be restarted from scratch (an implementation limitation) other way than to redirect the http channel artificially to the same URL and start over with the authentication. To fix this bug depends on fixing the sticky connection problem.
Patch to prevent the reprompt on a restart of an NTLM transaction. Note this patch is not a intended as fix it just serves to document my observations. And it just fixes the symptoms not the root cause On receipt of NTLM 1 challenge for a connection that has an identity cached in the AuthCache. The code assumes that the cached identity must be incorrect, and reprompts. This is not the case in a transaction restart. The patch attempts to detect a restart, and prevent the password reprompt. Note that while this does stop the password reprompt on a connection restart. It does not correctly handle invalid proxy passwords. On an invalid proxy password it just prompts once more, and then cancels the attempt. On the current release and nightly it continually prompts for the password, even if the correct password is entered. And needs to be manually cancelled.
This really depends on bug 1311720 and precise presence of the sticky flag. I realized that we simply must not allow restart of the transaction. It's perfectly OK to just report the error to the user, the server is resetting the connection and the transaction must not ever use a different connection - it's bound to the one it already have via the NS_HTTP_STICKY_CONNECTION flag.
Gary, thanks for the last patch, but I found a simpler solution lower. Can you please try to apply the patch from bug 1311720 (v2) and the patch I submitted here (v1) and see if that fixes the problem? Thanks.
Status: NEW → ASSIGNED
Attachment #8803374 - Flags: review?(mcmanus) → review+
Gary, just a note the patch in bug 1311720 updated to v2.1, which should be the final version. You may test with it. Thanks.
Applied the patches and tested. On connection reset the browser displays the connection failed and does not re-prompt for password. So the patches appear to have fixed that. However you now need to supply the proxy password twice, i.e. on startup you get prompted for it twice
Appears to be performing two authentication attempts in parallel.
Attached packet trace of the duplicate password prompt. Have log files available if required
(In reply to Gary Lockyer from comment #57) > Have log files available if required Yes, please! that will probably be more useful than packet logs. thanks.
logs emailed separately.
(In reply to Gary Lockyer from comment #55) > However you now need to supply the proxy password twice, i.e. on startup you > get prompted for it twice Thanks for the logs. It's a different bug. In the logs I can see the first request firefox makes is http://detectportal.firefox.com/success.txt (note an insecure request) and the other one is made for https://search.services.mozilla.com/ (note a SECURE request this time). Normally, we coalesce authentication prompts , . The only thing that differs between the two requests is "level", which is also part of the coalescing hash key. We need to fix this at . Instead of checking mUsingSSL (reflecting the origin server) we need to check if the proxy is ssl.  https://dxr.mozilla.org/mozilla-central/rev/c845bfd0accb7e0c29b41713255963b08006e701/toolkit/components/passwordmgr/nsLoginManagerPrompter.js#656  https://dxr.mozilla.org/mozilla-central/rev/c845bfd0accb7e0c29b41713255963b08006e701/toolkit/components/passwordmgr/nsLoginManagerPrompter.js#1536  https://dxr.mozilla.org/mozilla-central/rev/c845bfd0accb7e0c29b41713255963b08006e701/netwerk/protocol/http/nsHttpChannelAuthProvider.cpp#818
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/mozilla-inbound/rev/f576244a2513 Prevent restart of connection-sticky transactions to not break NTLM negotiation. r=mcmanus
Yes, it work and I'm very happy for that. Nobody imagine the happiness that I feel. Hip Hip Hurrah Hip Hip Hurrah Thank you so much to all people involved on it.
I have to say big thanks to Gary! Without his help, insight, testing and logs fixing this would not be possible that easily. Thanks!
Whiteboard: [http-conn][necko-active] → [http-conn][necko-active][ntlm]
This is marked as resolved, but the user still reports getting this behavior.
(In reply to raynebc from comment #68) > This is marked as resolved, but the user still reports getting this behavior. Please file a new bug with all detailed data you can provide, thanks.
raynebc, did you file a follow up bug?
I added bug #1351462 reporting the problem as described by the user.
Depends on: 1351462
Whiteboard: [http-conn][necko-active][ntlm] → [necko-active][ntlm]
You need to log in before you can comment on or make changes to this bug.