828236 - issue with NTLM and some SSL Proxies

Status: VERIFIED FIXED

(Core :: Networking, defect)

Description

[Michael Howard]

User Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1250.0 Iron/22.0.2150.0 Safari/537.4

Steps to reproduce:

Updated to Firefox 18.0


Actual results:

unable to access the internet through proxy. (keeps asking for login) mainly on HTTPS sites but also effects normal HTTP sites.  


Expected results:

should not ask for authentication.

Comment 1

[Michael Howard]

This has been confirmed by multiple uses within the company I work at.

This issue is not apparent in Firefox 17.0.*

Comment 2

[peter.gransdorfer]

Same here:
Unable to access HTTPS sites through our company proxy. After the update to Firefox 18 the browser asking for login.
 
We use squid 2.7 with Basic and NTLM authentification.

Comment 3

[Matthias Versen [:Matti]]

Do you use a pac file, let firefox determine the system proxy configuration or have you configured a direct http(s) proxy in Firefox?

It asks for the authentication but doesn't accept them or it keep asking for multiple resources on the page but the authentication itself is accepted ? 

Could you please attach a http log where this issue happens ?
https://developer.mozilla.org/en-US/docs/HTTP_Logging

Comment 4

[peter.gransdorfer]

I have tried the pac file and the direct configuration of the proxy (proxy:3128)

I tried the login several times, but it looks like that the authentication not works. The squid log says also TCP_DENIED on HTTPS request.

Attached you find the http log from my firefox.

Comment 5

[peter.gransdorfer]

Attachment: http log proxy auth

direct proxy configuration (proxy:3128), request webpage "google.at"

Comment 6

[peter.gransdorfer]

Comment on attachment 699926 [details]
http log proxy auth

direct proxy configuration (proxy:3128), request webpage "google.at"

Comment 7

[Matthias Versen [:Matti]]

marking new based on 2 independent reports

Comment 8

[Michael Howard]

Attachment: http log proxy auth ([e-mail])

attempted to access https://google.co.uk

Comment 9

[Michael Howard]

Have been informed that this issues may be due to 'network.http.proxy.keep-alive' no longer being supported?

Comment 10

[Patrick McManus [:mcmanus]]

(In reply to howardm from comment #9)
> Have been informed that this issues may be due to
> 'network.http.proxy.keep-alive' no longer being supported?

we still do keep-alives.. the pref that lets you stop that has been removed.

Comment 12

[Chris Dewar]

As per https://bugzilla.mozilla.org/show_bug.cgi?id=570283 

From what I can see for the CONNECT method no keep-alive header is sent after the proxy NTLM challenge: 

CONNECT google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0
Host: google.com
Proxy-Authorization: NTLM <SNIPPED HASH>

In previous versions FF would send a Proxy-Connection header: 

CONNECT www.google.co.uk:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0
Proxy-Connection: keep-alive
Host: www.google.co.uk
Proxy-Authorization: NTLM <SNIPPED HASH>

Comment 13

[Patrick McManus [:mcmanus]]

(In reply to Chris Dewar from comment #12)
> As per https://bugzilla.mozilla.org/show_bug.cgi?id=570283 
> 
> From what I can see for the CONNECT method no keep-alive header is sent
> after the proxy NTLM challenge: 
> 
> CONNECT google.com:443 HTTP/1.1

Keep-Alive is implicit in a HTTP/1.1 request.. but maybe that's the issue here. I'd actually expect we'd see the normal connection header.

I'd like to see that squid is closing the connection after sending the 407.. do you have a packet trace for this (you can make one with wireshark)

Comment 14

[Thomas]

I have the same issue for my company.

I don't know if this information will help you but I send :
#Here is the log on my squid proxy server for HTTPS :
1357832158.979      0 10.0.0.54 TCP_DENIED/407 1737 CONNECT www.google.fr:443 - NONE/- text/html
1357832159.012      0 10.0.0.54 TCP_DENIED/407 1711 CONNECT www.google.fr:443 - NONE/- text/html

#Here is the log on my squid proxy server for HTTP :
1357832185.361    117 10.0.0.54 TCP_MISS/200 21160 GET http://www.google.fr/ MY_DOMAIN\my_username DIRECT/173.194.67.94 text/html
1357832185.624      0 10.0.0.54 TCP_DENIED/407 1800 GET http://clients1.google.fr/generate_204 - NONE/- text/html
1357832185.662      0 10.0.0.54 TCP_DENIED/407 1922 GET http://clients1.google.fr/generate_204 - NONE/- text/html
1357832185.707      5 10.0.0.54 TCP_NEGATIVE_HIT/204 328 GET http://clients1.google.fr/generate_204 MY_DOMAIN\my_username NONE/- text/html
1357832185.812     53 10.0.0.54 TCP_MISS/204 395 GET http://www.google.fr/csi? MY_DOMAIN\my_username DIRECT/173.194.67.94 image/gif

(10.0.0.54 is client IP)

So Firefox doesn't send authentication to proxy for HTTPS seb site.

Comment 15

[Chris Dewar]

(In reply to Patrick McManus [:mcmanus] from comment #13)
> (In reply to Chris Dewar from comment #12)
> > As per https://bugzilla.mozilla.org/show_bug.cgi?id=570283 
> > 
> > From what I can see for the CONNECT method no keep-alive header is sent
> > after the proxy NTLM challenge: 
> > 
> > CONNECT google.com:443 HTTP/1.1
> 
> Keep-Alive is implicit in a HTTP/1.1 request.. but maybe that's the issue
> here. I'd actually expect we'd see the normal connection header.
> 
> I'd like to see that squid is closing the connection after sending the 407..
> do you have a packet trace for this (you can make one with wireshark)

Yes version of squid is HTTP/1.0 so would epxlain it - and yes squid closes the connection after the request: 

HTTP/1.0 407 Proxy Authentication Required
Server: squid/2.6.STABLE9
Date: Thu, 10 Jan 2013 13:28:06 GMT
Content-Type: text/html
Content-Length: 1265
Expires: Thu, 10 Jan 2013 13:28:06 GMT
X-Squid-Error: ERR_ACCESS_DENIED 0
Proxy-Authenticate: NTLM
Proxy-Authenticate: Basic realm="Internet via 100W"
X-Cache: MISS from apollo
Via: 1.0 apollo:8881 (squid/2.6.STABLE9)
Proxy-Connection: close

I can attach the entire trace in pcap format if it helps?

Comment 16

[Patrick McManus [:mcmanus]]

(In reply to Chris Dewar from comment #15)

> Proxy-Connection: close
> 
> I can attach the entire trace in pcap format if it helps?


please do attach. I'll try and cobble something together for you to test.

Comment 18

[Chris Dewar]

Attachment: Shows NTLM auth failing due to non peristant connections

Firefox/18.0

Comment 19

[Thomas]

For information, her we are using squid/2.7.STABLE7

Comment 20

[Michael Howard]

Here us what Bloxx (Our Proxy System Provider) has said the cause of the issue may be:

"It looks like Firefox latest update to 18.0 breaks NTLM authentication.

As they no longer send the ”Proxy-Connection: Keep-Alive” when negotiating NTLM auth squid (the core proxy service we use) responds with: “Proxy-Connection: close” at all stages of the NTLM challenge, breaking the negotiate, challenge, authenticate sequence. 

The challenge and authenticate message should always be done in the same connection but because the connection is closed firefox will now use a new connection for each stage which it should not."

Comment 21

[Patrick McManus [:mcmanus]]

so 570283 can't be easily reverted - too much bitrot.

but the only piece that's really a problem is on the CONNNECT tunnel setup. Elsewhere it sends the normal "Connection: keep-alive" which is fine in practice - its just with the CONNECT it doesn't currently send either variant and HTTP/1.0 proxies will do the wrong thing.

So we can do that with a very targeted micro-revert of 570283 (adding in both variants as P-C is not standards defined).

I have a copy of squid 2.6 available and this seems to work, but I don't have NTLM configured with it so I'll need reporters on this bug to verify.

I'll have the CI make some builds. stay tuned.

diff --git a/netwerk/protocol/http/nsHttpConnection.cpp b/netwerk/protocol/http/nsHttpConnection.cpp
--- a/netwerk/protocol/http/nsHttpConnection.cpp
+++ b/netwerk/protocol/http/nsHttpConnection.cpp
@@ -1442,16 +1442,20 @@ nsHttpConnection::SetupProxyConnect()
 
     // CONNECT host:port HTTP/1.1
     nsHttpRequestHead request;
     request.SetMethod(nsHttp::Connect);
     request.SetVersion(gHttpHandler->HttpVersion());
     request.SetRequestURI(buf);
     request.SetHeader(nsHttp::User_Agent, gHttpHandler->UserAgent());
 
+    // a CONNECT is always persistent
+    request.SetHeader(nsHttp::Proxy_Connection, NS_LITERAL_CSTRING("keep-alive"));
+    request.SetHeader(nsHttp::Connection, NS_LITERAL_CSTRING("keep-alive"));
+
     val = mTransaction->RequestHead()->PeekHeader(nsHttp::Host);
     if (val) {
         // all HTTP/1.1 requests must include a Host header (even though it
         // may seem redundant in this case; see bug 82388).
         request.SetHeader(nsHttp::Host, nsDependentCString(val));
     }
 
     val = mTransaction->RequestHead()->PeekHeader(nsHttp::Proxy_Authorization);

Comment 22

[Patrick McManus [:mcmanus]]

Attachment: patch 0

570283 removed proxy-connection and that was the only connection oriented header we were sending on a CONNECT request. HTTP/1.0 squid didn't realize we supported K-A and therefore fell down on doing NTLM auth (which requires multiple transactions on the same connection).

This patch restores proxy-connection in that case and also adds in the standards compliant connection.

Comment 23

[Patrick McManus [:mcmanus]]

(In reply to Patrick McManus [:mcmanus] from comment #21)

> 
> I'll have the CI make some builds. stay tuned.
> 

Builds will appear in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-95a9f44c52fd

I just started them moments ago, so it could take anywhere from 30 minutes to several hours for them to appear (depending on load and platform being built).

Chris, Peter, Michael, Thomas, (anyone else with the problem), 

can you try a build from there and report on if it works for you?

Comment 24

[Chris Dewar]

(In reply to Patrick McManus [:mcmanus] from comment #23)
> (In reply to Patrick McManus [:mcmanus] from comment #21)
> 
> > 
> > I'll have the CI make some builds. stay tuned.
> > 
> 
> Builds will appear in
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.
> com-95a9f44c52fd
> 
> I just started them moments ago, so it could take anywhere from 30 minutes
> to several hours for them to appear (depending on load and platform being
> built).
> 
> Chris, Peter, Michael, Thomas, (anyone else with the problem), 
> 
> can you try a build from there and report on if it works for you?

Good Job! Works for me: 

After the first 407 the FF test build sends this when i request https://www.google.co.uk: 

CONNECT www.google.co.uk:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20130110 Firefox/17.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: www.google.co.uk
Proxy-Authorization: NTLM <NTLM AUTH TOKEN>

To which squid keeps the connection alive until the client fully authenticates. 

Side note (just incase it helps with the reasoning behind https://bugzilla.mozilla.org/show_bug.cgi?id=570283): I'm not 100% sure you need both 
request.SetHeader(nsHttp::Proxy_Connection, NS_LITERAL_CSTRING("keep-alive"));	
AND
request.SetHeader(nsHttp::Connection, NS_LITERAL_CSTRING("keep-alive"));

Squid(squid/2.6.STABLE9) at least, will keep the connection alive even with the Connection: keep-alive header alone (it does with a standard http GET anyway)

Comment 25

[mramon79]

For me also works!!!
thanks

Comment 26

[Thomas]

(In reply to Patrick McManus [:mcmanus] from comment #23)
> (In reply to Patrick McManus [:mcmanus] from comment #21)
> 
> > 
> > I'll have the CI make some builds. stay tuned.
> > 
> 
> Builds will appear in
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/mcmanus@ducksong.
> com-95a9f44c52fd
> 
> I just started them moments ago, so it could take anywhere from 30 minutes
> to several hours for them to appear (depending on load and platform being
> built).
> 
> Chris, Peter, Michael, Thomas, (anyone else with the problem), 
> 
> can you try a build from there and report on if it works for you?

Hi,

I try

seems to be working now :-)

here is my proxy log :
1357890740.485      0 10.10.10.101 TCP_DENIED/407 1743 CONNECT mail.google.com:443 - NONE/- text/html
1357890740.515      0 10.10.10.101 TCP_DENIED/407 1865 CONNECT mail.google.com:443 - NONE/- text/html
1357890740.638      0 10.10.10.101 TCP_DENIED/407 1755 POST http://ocsp.thawte.com/ - NONE/- text/html
1357890740.678      0 10.10.10.101 TCP_DENIED/407 1877 POST http://ocsp.thawte.com/ - NONE/- text/html
1357890740.808     91 10.10.10.101 TCP_MISS/200 1593 POST http://ocsp.thawte.com/ MY_DOMAIN\my_username DIRECT/199.7.55.72 application/ocsp-response
1357890740.902      0 10.10.10.101 TCP_DENIED/407 1755 CONNECT accounts.google.com:443 - NONE/- text/html
1357890740.933      0 10.10.10.101 TCP_DENIED/407 1877 CONNECT accounts.google.com:443 - NONE/- text/html
1357890741.156    113 10.10.10.101 TCP_MISS/200 1593 POST http://ocsp.thawte.com/ MY_DOMAIN\my_username DIRECT/199.7.55.72 application/ocsp-response


there is "TCP_DENIED/407" twice, then firefox send authentication "MY_DOMAIN\my_username" .

You will add this fix on the standard firefox setup ? (I am newbie here)

Kind regards

Comment 27

[peter.gransdorfer]

I have tried the nightly build and it works!

Thanks

Comment 28

[Michael Howard]

I can confirm that this has been resolved in the nightly build and no longer get this issue.

Comment 29

[Michael Howard]

One user has reported it promptly crashed their graphics driver.

No other issues have been found and I personally have had no issues and everything seems to be working.

Comment 30

[Patrick McManus [:mcmanus]]

(In reply to Michael Howard from comment #29)
> One user has reported it promptly crashed their graphics driver.

definitely unrelated to the change in here.

> 
> No other issues have been found and I personally have had no issues and
> everything seems to be working.

thanks!

Comment 31

[Patrick McManus [:mcmanus]]

  https://hg.mozilla.org/integration/mozilla-inbound/rev/fbc20626801f

Comment 32

[Patrick McManus [:mcmanus]]

Comment on attachment 700613 [details] [diff] [review]
patch 0

[Approval Request Comment]
Regression caused by (bug #): 570283
User impact if declined: Users behind some proxies that are configured to do windows integrated authentication (NTLM) won't be able to use the proxy; the proxy is often a requirement for network access.
Testing completed (on m-c, etc.): on m-i now. Passes CI tests of try. Independent reporter confirmations of a test build based on FF18 in Comments 24, 25, 26, 27, and 28.
Risk to taking this patch (and alternatives if risky): very low - it reverts a portion of 570283 and simply adds 2 headers to a CONNECT tunnel transaction (one of which we've had in <= FF17 and the other which is the open standards equivalent)

String or UUID changes made by this patch: none

Comment 33

[Patrick McManus [:mcmanus]]

 https://hg.mozilla.org/releases/mozilla-aurora/rev/9c175e30dd94
 https://hg.mozilla.org/releases/mozilla-beta/rev/0a35a304b737

Comment 34

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/fbc20626801f

Comment 36

[Patrick McManus [:mcmanus]]

  https://hg.mozilla.org/releases/mozilla-release/rev/2e7cc15ea6d1

Comment 37

[juan becerra [:juanb]]

Michael, could you help us verify this on the latest beta and release candidates?

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/19.0b2-candidates/build1/ (beta)

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/18.0.1-candidates/build1/ (release)

We were able to verify a couple of proxy related issues, but I don't know how to go about this one. Per comment #28 I'll mark this as verified in fx21 (nightly).

Comment 38

[Michael Howard]

I can confirm that this issue has been resolved in the following versons.

18.0.1
19.0b2
21.0a1

this issue is still showing in 20.0a1

Comment 39

[Patrick McManus [:mcmanus]]

(In reply to Michael Howard from comment #38)

> this issue is still showing in 20.0a1

hmm, this doesn't make any sense to me unless maybe you tested an old build? Can you retest and supply the about:buildconfig revision?

The same code is on all those branches (I just confirmed - indeed 21 and 20 actually still have byte for byte equivalency between nsHttpConnection) and I also confirmed that it does the expected thing on aurora-20 (i..e sends the connection header when making a CONNECT).. I did not confirm that NTLM actually works because I don't have that setup in my proxy, but I did confirm that the connect header is sent which is what was diagnosed as the root of this bug.

If it persists for you can we see a wireshark packet capture (in pcap format)?

Comment 40

[Michael Howard]

Attachment: About:BuildConfign from Firefix 20.0a1

Comment 41

[Patrick McManus [:mcmanus]]

(In reply to Michael Howard from comment #40)
> Created attachment 703307 [details]
> About:BuildConfign from Firefix 20.0a1

That's not the current firefox 20 code, which is found on the aurora channel and declares itself as 20.0a2

I believe you have code from when 20 was on nightly because your about:buildconfig shows a mozilla-central branch with the last changeset dated 1/7.. the fix for this bug was not made until 1/9 so I would not expect it to be there.

Can you try out aurora?

Comment 42

[Michael Howard]

Ok i can confirm this works with 20.0a2

Comment 43

[Patrick McManus [:mcmanus]]

(In reply to Michael Howard from comment #42)
> Ok i can confirm this works with 20.0a2

thanks!