366562 - NTLM authentication to proxy may fail while connected to secure (https, SSL) site

Status: RESOLVED DUPLICATE of bug 486508

(Core :: Networking, defect)

Description

[shattered]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9

Firefox may fail to authenticate to a proxy server (in my case, both NTLM and Basic methods are offered by Squid proxy) in the "middle" of a session with secure site.  That is, initial connection succeeds, and I may browse any number of pages after that; but after some time proxy refuses next request and FF displays "enter password" dialog box (actually, two in my case, one for NTLM method and another for Basic).

Reproducible: Sometimes




I used Wireshark to sniff the interaction between FF and proxy server.  Normally, there's a sequence of HTTP requests that looks like this:

No.     Time        Source         Destination           Protocol Info
      4 0.000305    client:3978        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1
      6 0.052950    proxy             client:3978        HTTP     HTTP/1.0 407 Proxy Authentication Required (text/html)
     14 0.008956    client:3979        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1, NTLMSSP_NEGOTIATE
     16 0.002776    proxy             client:3979        HTTP     HTTP/1.0 407 Proxy Authentication Required, NTLMSSP_CHALLENGE (text/html)
     17 0.000017    proxy             client:3979        HTTP     Continuation or non-HTTP traffic
     19 0.012160    client:3979        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1, NTLMSSP_AUTH, User: FOO\bar
     21 0.505952    proxy             client:3979        HTTP     HTTP/1.0 200 Connection established

But the failing sequence is:

   1093 8.835104    client:4040        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1
   1095 0.002670    proxy             client:4040        HTTP     HTTP/1.0 407 Proxy Authentication Required (text/html)
   1103 0.015343    client:4041        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1, NTLMSSP_NEGOTIATE
   1105 0.007575    proxy             client:4041        HTTP     HTTP/1.0 407 Proxy Authentication Required, NTLMSSP_CHALLENGE (text/html)
   1106 0.000027    proxy             client:4041        HTTP     Continuation or non-HTTP traffic
   1111 0.019539    client:4042        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1
   1113 0.002653    proxy             client:4042        HTTP     HTTP/1.0 407 Proxy Authentication Required (text/html)
   1118 0.015873    client:4041        proxy             HTTP     CONNECT metalink.oracle.com:443 HTTP/1.1, NTLMSSP_NEGOTIATE
   1119 0.003330    proxy             client:4041        HTTP     HTTP/1.0 407 Proxy Authentication Required (text/html)

Note that third request was sent on a new connection.

Comment 1

[shattered]

Still happens in 3.0.

Comment 2

[Joe Tiedeman]

Just to confirm that you're not the only one experiencing this.

We're running a Squid proxy server (2.6 Stable 20) running on 2003 Server SP1 using ntlm auth for clients and frequently when accessing metalink (and other ssl enabled sites, gmail.google.com for example) we get the authentication prompts.

We've confirmed this in FF3.0.3 & 3.0.4.

If I can provide any diagnostic information, please let me know. I'll be happy to help

Comment 3

[Joe Tiedeman]

irony of ironies, my boss just got the same issue when browsing this exact page!

Comment 4

[Yann]

Same Here. On some computers (very same Firefox 3.06 version) the proxy auth pops up. 
I did compare pnetwork capture from two host using the same login, as far as I can tell the http/NTML request looks the same (except for the actual encrypted part of course). 
For some reason, and for the non-working hosts only,  wiresharks paquet dissector truncates the domain/login/host NTLMSSP fields idisplayed to the first character. The actual paquet content is not truncated, and it does not do that for the computers that have no issue. I have no idea of what this means.

Comment 5

[Rapone]

Attachment: firefox ntlm negotiation error with proxy 407 request

when firefox processing ntlm authentication in IWA real with bluecoat proxy having a pop-up storm..

Comment 6

[Rapone]

I've solved problem.
is not firefox problem but is a plug-in js bug.
when i try to go on web site where are the js script that try to connect to anhoter site for send counter data for web navigation, proxy send 407 request, and ff pass ntlm negotiation, but jc cannot use it, then ff pass basic but js cannot use it.
the page that js script try to connect is unaccessible (if try with ie you have access deny from ocs) with ntlm/basic authentication.

this script it's mapped from varius AV like a spyware, but is false positive, it's only a counter page that some web site habe in their http code.
if you set on proxy, for irmworldwide.com, a rule where is access deny, you solve pop-up problem.

i belive there are some site where are js script that try to connect to irmworldwide.com (a server spyrewere) and if you have a firefox with a js plug-in you have pop-up storm with proxy can occurr session authentication.

bye :-)

Comment 7

[Alex]

First of all we also do experience this problem and it's bad we haven't found a solution yet.
We're having BlueCoat as a proxy and FF 3.x on our clients. The problem is like Yann described that FF is sending a WRONG NTLM authentication to the proxy - which is then prompting for valid credentials (a fall-back from NTLM to Basic).
We weren't able to identify the source or the exact circumstances when the issue occurs. It seems as if FF is trying to reach the destination but it's temporarily unreachable. Then FF tries to access the destination once more - but hands out only the first character of a valid NTLM user name (example, real user is 'amuelle', then FF will only send 'a')
I think this problem is quite a urgent one. If you were able to find out SOMETHING about this, please let everyone know!
Thanks.

Comment 8

[Rapone]

add possible solution for Alex.
I finally resolved the pop-up FF set to "proxy ip" authentication for firefox agent.
But I've understood what was the real problem (in my case).

In my case FF (js plug-in) try to connect to a web site (robot info site) and Bluecoat send the normally 407, but for any reasons, FW and IPS after BC, reset tcp connection from OCS (legacy spyreware!!). so when bluecoat pass the reset to FF, plug-in re-try to connect to OCS, so BC send 407 for session auth (proxy mode) and we have a tcp reset again... during this situation have many pop-up (failed obviusly)
When I try with IE i don't have pop-up because the tcp reset "was passed" to a js plug-in... like a close session!!

in my case I used "proxy ip" auth for firefox because is only way to don't have pop-up.

the problem, i belive, is from ff e js plug-in when it recive a tcp reset (http code 500) from proxy.

Comment 9

[Alex]

I understand what you mean choosing Proxy-IP as authentication mode but this is no real solution but a workaround. There must be a reason for the resets in the connection and also a reason why FF only sends the first letter of the real username to the Bluecoat Proxy. The proxy itself is free of guilt as it prohibits internet access for non authenticated users and prompts the user for correct credentials as the provided ones (handed over by the browser) can't be validdted against the Auth server.
So what's the real cause of this issue? It can't be JS only. You can surf several sites containing JS without getting a single prompt so I assume it must be somethign else. 
The issue is definately reproducible accessing Google Mail but there might be more sites. Please test Google Mail and post more URLs where the issue is definately reproduceable.
Regards Alex

Comment 10

[shattered]

Username is not actually truncated, it's a problem in wireshark's dissector.

Comment 11

[Alex]

are you sure? usually you can see the username the way it should be, then, you only see one letter and the proxy refuses the connection. makes sense to me.
What else do you think causes the problem or what could be the solution?

Comment 12

[Yann]

As I said in my first comment, yes, there is an issue with wireshark's dissector. The actual paquet bytes are the same (at least for the username/domain part). However, the dissector has no problem when transparent authentication is successfull : it always display the full credentials correctly. When authentification fails, it displays only the first characters, every time.
I suppose this correlation is no coincidence and that I am missing something here.

Comment 13

[Rapone]

confirm what yann said, and add that wireshark trunk also the ntlm key...

for alex.
yes in my case It's a workaround, but the real issues is this:
in my case
if I have for any reason(java,javascript,other) a tcp reset from ocs, when Bluecoat pass this info the ntlm mechanism fail..(no with ie)

In your case I belive you must set "always verify" for the OCS target (after you create the rule flush the cache) because if proxy cache the http object cokies for that connection, of course (we have been this problem in varius customer) try to use for another and when OCS send the "expired object" BC and also squid re-connect to ocs and you have auth-407, but at this point can't pass session-cookie (pass not valid session-coockie to proxy) and request you with pop-up.

set a rule with destination is ocs and action is always-verify+allow and you shouldn't be show pop-up auth... :-)

Comment 14

[Alex]

Hi Rapone, 
this really is not workaround. The only possible workaround for me is to do an authentication mode like Proxy-IP. This authenticates the user as usual but adds like a timer, a surrogate which actually is the Client's IP. In a definated timeframe, the proxy will use the Client's IP as an authentication surrogate and treat all of the Client's requests as valid, as authenticated. 

But anyway, how come this Bug is still unconfirmed?

Comment 15

[Rapone]

Hi Alex,
using proxy-ip authentication mode you solve problem because only first 407 request is authenticate with ntlm ssp method(whit o without pop-up), so after any other auth request are authenticate verifying if there is an ip(client ip source) with an user authenticated (is indifferent who user is). that means if you have a Nat   proxy-ip is not a good authentication mode...(first user authenticated is good for all behind ip natted!) 

in session mode every new session must be authenticate, and each session require 407 and ntlm handshake.. 
if your problem can be solved whit proxy-ip auth mode ok, but remember how to work it.

You told about client-ip surrogate, beware, because for session authentication that right is cookie session surrogate.. 

About my problem whit FF pop-up storm using session-authenticate method I really found a probable problem of BC to manage http tcp-reset o silent-drop http packet from an IPS, so this generate a re-request from js into FF for the same session..and i show pop-up storm..
But i cannot consedering a real bug of FF(only 1/2 bug!), because there are other tecnology that modify OCS http response.. :-) but IE cannot have the same problem. :-(. i.e.

bye

Comment 16

[Alex]

Just a quick note (I'll leave for weekend)
Proxy-IP is a solution for us as we don't use NAT here. I was aware of the big hole I'd have created this way, but thanks anyway :-)
And using cookies as a surrogate could have worked, true.. I'll see if I use that. Currently we only use Proxy-IP for all Firefox / Mozilla User Agents. Ok, User Agent can be faked, but nevertheless the Client has to authenticate at least one single time. 
I don't quite understand what you say about BC and whether it might be a failure on their side, but I'm sure this is because I'm in a hurry right now. Write me if you want, I'll reply. Promise.

Comment 17

[Alex]

This topic definately needs some attention payed by someone from the Mozilla engineering team!

https://bugzilla.mozilla.org/show_bug.cgi?id=318253, created 30.11.2005, Status: New
https://bugzilla.mozilla.org/show_bug.cgi?id=366562, created 10.01.2007, Status: Unconfirmed
https://bugzilla.mozilla.org/show_bug.cgi?id=501361, created 30.06.2009, Status: Unconfirmed
https://bugzilla.mozilla.org/show_bug.cgi?id=486508, created 02.04.2009, Status: Unconfirmed

In my opinion, the Mozilla team did a really good job in all the way, Firefox developed – but here is a glitch and that thing is really obvious – if Firefox is used in a business environment together with NTLM authentication.

Comment 18

[Andrew Bartlett]

I can reproduce this at-will with the instructions in bug 486508 so marking as a duplicate.