766817 - websockets causing repeated auth prompts?

Status: RESOLVED FIXED

(Core :: Networking: WebSockets, defect)

Description

[Jason Duell]

We've got some reports here, specifically about sockets.ny.stackexchange.com:

  http://meta.stackoverflow.com/questions/130053/my-corporate-proxy-doesnt-support-web-sockets-which-makes-se-a-pain-to-use
  
  http://meta.stackoverflow.com/questions/124132/getting-authentication-prompt-for-sockets-ny-stackexchange-com-on-every-page-loa

This might be related too.

  http://meta.stackoverflow.com/questions/123648/console-error-when-visiting-meta

Comment 1

[Jason Duell]

Kim:

none of the stackoverflow reports give me an exact URL to try so I can see if I can reproduce this.  Can you give me a URL that's showing the problem?

Thanks

Comment 2

[Kim Sullivan]

Hi, actually, all of the stack overflow URLs (even the title page) do this, its not one specific page. Fortunately, I've been able to reproduce the exact same behaviour on the websockets site, as suggested on one of the the stackoverflow reports:
http://www.websocket.org/echo.html

The site says "websockets supported", but as soon as i click the "connect" button, firefox shows me a proxy dialog with the following message (the URL of our proxy is redacted), with the login information already filled in:

The proxy moz-proxy://proxy.example.com:8080 is requesting a username and password. The site says: "moz-proxy://proxy.example.com:8080"

After I enter the correct login information and hit OK, i get the dialog again (tried it for about 6 times, i just repeatedly get the dialog). When I enter the wrong login information a few times, my account gets locked and I have to call our IT support. When I hit cancel, a different proxy authentication dialog appears (again, the proxy and company specific string have been redacted). This one doesn't have the login information filled in (both fields are blank):

The proxy moz-proxy://proxy.example.com.com:8080 is requesting a username and password. The site says: "company specific string"

When I hit cancel, the proxy dialog doesn't appear again (from websocket.org), but it does appear again when on stack overflow (maybe for each connection attempt?). When I hit OK (with or without filling in the correct information), it loops to the first dialog.

When I switch to manual configuration and input the same proxy that is configured automatically (and have "use proxy for all" checked), I don't get the proxy dialog, but I don't get the echo either. After a short while, the site shows me the following log:
ERROR: undefined
DISCONNECTED

When I uncheck "use this for all protocols" and delete all other protocol entries except HTTP, the page doesnt finish loading and doesn't show the message that websockets are supported.

To sum up
* automatic proxy script - websockets are supported, don't work + authentication dialog
* manual proxy for all protocols - websockets are supported, don't work but no auth dialog (just error message on the site)
* manual proxy just for http - site doesn't finish loading

Comment 3

[Jason Duell]

Kim,

Thanks for the great bug report.  I'd like to get to the bottom of this.

> When I uncheck "use this for all protocols" and delete all other 
> protocol entries except HTTP, the page doesnt finish loading and 
> doesn't show the message that websockets are supported.

This is the weirdest case to me, and makes me suspect that you're somehow not getting the proxy setting right somehow.  Are you setting both HTTP/SSL, and leaving FTP/SOCKS blank? (I'd recommend doing that, in case the page has any HTTPS elements).  Are you able to browse other, non-websockets pages OK?  

Let's try to

1) make sure you can set just HTTP/HTTPS proxy settings, manually, and get regular HTTP/SSL pages OK (yahoo, google, etc should all work).

2) Once that works, see if you can fully load 
  
  http://www.websocket.org/echo.html

The "This browser supports websockets" message should happen so far as JavaScript can see the WebSocket class available, so that should really work unless something is really messed up (like page not loading fully).

3) Note that unless you're using a nightly build of Firefox, bug 713026 will prevent the echo test from working over any HTTP proxy unless you use TLS.  So make sure you test only with the "use Secure Websockets (TLS)" button checked for now.   (Or upgrade to nightly).

If we can boil down the test case to this, that'll make life simpler.  If we can't get your manually-set HTTP/HTTPS settings to work, I'll think some more about how to approach this.

Again, thanks--I can't seem to reproduce this problem with squid, at least, so having you available to help with this is really helpful. (Do you know what proxy server your company is using?)

Comment 4

[Kim Sullivan]

Ok, I tried it again as you suggested - setting a manual proxy for http and https, clearing FTP and SOCKS.

1. Normal websockets - I get the proxy authentication dialog 

So i switched to secure sockets and tried it. I didn't get the auth dialog anymore:

2. Manual proxy, FTP and SOCKS cleared - works fine
3. Manual proxy, FTP and SOCKS filled in - error after timeout
4. Manual proxy, "use for all protocols" - error after timeout 
5. Automatic proxy configuration - echo works

So, the auth dialog appears only when using non-secure websockets, I'll have to try the nightly build to see how bug 713026 changes things.

Comment 5

[Kim Sullivan]

The error persists in Nightly 16.0a1 (2012-06-21). Let me know if i can do something more to help diagnose it, would a packet trace using Wireshark help? Or is there something I can use directly in firefox?

I tried finding out something about our corporate proxy, but I'm not sure if I have access to that information.

Comment 6

[Kim Sullivan]

Sorry for spamming with comments, one more finding: the nightly Firefox displays only the first type of proxy authentication dialog (The site says: "moz-proxy://proxy.example.com:8080"). After I hit cancel, the second dialog (The site says: "company specific string) doesn't appear anymore.

Comment 7

[Jason Duell]

Thanks for the info, Kim.  Here's where we're at:

2. Manual proxy, FTP and SOCKS cleared - works fine
5. Automatic proxy configuration - echo works

Good!  :)

3. Manual proxy, FTP and SOCKS filled in - error after timeout
4. Manual proxy, "use for all protocols" - error after timeout 

This is bug 713023, which should be fixed in the nightly build by tomorrow or the day after.

1. Normal websockets - I get the proxy authentication dialog 

So this is still the weird case.  It's not weird that it fails--most proxies are currently configured to fail CONNECT requests to any port but 443.  But when I do this with squid, I simply get a failure, not an auth dialog.  Presumably either we've omitted some auth info from non-SSL request, or the proxy is erroneously asking for auth when it shouldn't.

A wireshark trace of your connection to the proxy for case #1 would be great: it's probably the best way to proceed from here.  If it contains non-encrypted password info, you can probably just scramble the hash content in the header (I doubt the issue is our hashing/encryption), or contact me at jduell at mozilla dotcom to figure out a way to transfer the file securely.

Thanks!

Comment 8

[Jason Duell]

P.S. Make sure to use nightly for the trace--we do CONNECTs all wrong for non-SSL websockets in earlier versions. (this is probably why you see different behavior for them)

Comment 9

[Kim Sullivan]

I got information about the proxy we're using - BlueCoat SG8100 using SGOS 5.4.8.1, probably not reproducible unless you have the actual hardware.

Since 16.0a1 (2012-06-21) I get the proxy auth dialog even for secure websockets, I tried it again in nightly 16.0a1 (2012-06-24), both secure and non secure websockets fail.

So I ran wireshark and tried to find out what changed between the two versions for secure sockets. Apparently, the current nightly sends the authentication information using a different connection than the NTLMSSP_NEGOTIATE message. It goes like this:

Working in beta:
1. TCP connect port from 3050 to HTTP proxy port 8080
2. HTTP CONNECT echo.websocket.org:443 HTTP/1.1
3. HTTP/1.1 407 Proxy Authentication Required
4. TCP close connection
5. TCP connect port from 3051 to HTTP proxy port 8080
6. CONNECT echo.websocket.org:443 HTTP/1.1 , NTLMSSP_NEGOTIATE
7. HTTP/1.1 407 Proxy Authentication Required
8. CONNECT echo.websocket.org:443 HTTP/1.1 , NTLMSSP_AUTH, User: DOMAIN\sullivan
9. HTTP/1.1 200 Connection established
10. ...

Not working at least since nightly 16.0a1 (2012-06-21):
1. TCP connect port from 3094 to HTTP proxy port 8080
2. HTTP CONNECT echo.websocket.org:443 HTTP/1.1
3. HTTP/1.1 407 Proxy Authentication Required
4. TCP close connection
5. TCP connect port from 3097 to HTTP proxy port 8080
6. CONNECT echo.websocket.org:443 HTTP/1.1 , NTLMSSP_NEGOTIATE
7. HTTP/1.1 407 Proxy Authentication Required
8. TCP close connection
9. TCP connect port 3098 to HTTP proxy port 8080
8. CONNECT echo.websocket.org:443 HTTP/1.1 , NTLMSSP_AUTH, User: DOMAIN\sullivan
9. HTTP/1.1 407 Proxy Authentication Required
10. ...

I don't know much about proxy specs, but it seems like the proxy doesn't like to negotiate access with one connection and then being provided the actual login information in a different connection, so it asks again.

Comment 10

[Jason Duell]

Patrick:  I assume the "extra" connection(s) here (at least the first one in beta) is one of our speculative preconnnects (do they do a CONNECT if configured to?).   I'm puzzled as to why we're dropping the connection twice in the nightly build, though.

Honza: So this is NTLM auth for the proxy.  Anything you can guess that might be going on?

Generally, I'd expect that a proxy would be fine accepting the above behavior (client gets a 407 response, shuts down connection, reopens a new connection with auth headers already present): it's not efficient, and wastes a TCP connection latency, but it would generally be OK.  But 1) the auth info might be wrong; or 2) the proxy might expect a challenge/response specially for CONNECT (but I'd still have guessed a client could show up with the credentials already: seems odd); or 3) maybe there's something NTML-specific going on here.

Kim:  Thanks again for the great info.  There are a few more ways you can help if you're up for it:

1) Was the result above for a ws:// or wss:// (i.e SSL or not) web socket?  I recently changed some of the logic that handles non-secure connecting, so I'm wondering if that's the cause of the change in nightly.

2) Could you run your example above again on nightly, this time with both a wireshark capture for your proxy port (8080), and an NSPR log?  For NSPR instructions, see

   https://developer.mozilla.org/en/HTTP_Logging

except for NSPR_LOG_MODULES use

  NSPR_LOG_MODULES=timestamp,nsHttp:5,nsWebSocket:5,proxy:5

3) If you've still got time and energy after that (totally understand if not), you could get a more precise regression range (i.e figure out which is the first nightly that the behavior here changed), either by using

   http://harthur.github.com/mozregression/

or by just manually grabbing nightlies until you find which one broke things.

Thanks!

Comment 11

[Patrick McManus [:mcmanus]]

(In reply to Jason Duell (:jduell) from comment #10)
> Patrick:  I assume the "extra" connection(s) here (at least the first one in
> beta) is one of our speculative preconnnects 

beta/14 does not have any of the speculative preconnect code.

Comment 12

[Kim Sullivan]

Attachment: nspr log of working websockets from beta

Comment 13

[Kim Sullivan]

Attachment: broken websockets nspr log nightly

Comment 14

[Kim Sullivan]

> 1) Was the result above for a ws:// or wss:// (i.e SSL or not) web socket? 
> I recently changed some of the logic that handles non-secure connecting, so
> I'm wondering if that's the cause of the change in nightly.

Only secure, since I could get it to work in beta14, so I could compare the two.

> 
> 2) Could you run your example above again on nightly, this time with both a
> wireshark capture for your proxy port (8080), and an NSPR log?  For NSPR
> instructions, see

Whireshark is the same as last time, the nightly terminates the connection before sending auth information which prompts the proxy to disregard the previous negotiation. NSPR logs are attached, I tried to compare them side by side, the only thing that seems different is that the broken version doesn't seem to log anything about nsHttpPipeline and instead has a lot more nsConnectionManager, like this, directly before the second CONNECT with:
2012-06-26 06:55:09.690000 UTC - 3624[210f530]:   adding transaction to pending queue [trans=7bbdf80 pending-count=1]
2012-06-26 06:55:09.690000 UTC - 3624[210f530]: nsHttpConnectionMgr::OnMsgReclaimConnection [conn=669f060]
2012-06-26 06:55:09.690000 UTC - 3624[210f530]: nsHttpConnectionMgr::ConditionallyStopReadTimeoutTick armed=1 active=1
2012-06-26 06:55:09.690000 UTC - 3624[210f530]:   connection cannot be reused; closing connection
2012-06-26 06:55:09.690000 UTC - 3624[210f530]: nsHttpConnection::Close [this=669f060 reason=80004004]

> 3) If you've still got time and energy after that (totally understand if
> not), you could get a more precise regression range (i.e figure out which is
> the first nightly that the behavior here changed), either by using

mozregression seems kinda broken, i was actually able to do a manual bisection quicker than it installed itself (and I started halfway through, not counting the time to install mozillabuild...). Anyway: the first nightly that is broken is 2012-06-07-02-57-55-mozilla-central, or more precisely:
this still works http://hg.mozilla.org/mozilla-central/rev/6338a8988917
this doesnt work http://hg.mozilla.org/mozilla-central/rev/7e4c2abb9fc9

I tried to look at the list of changes between those two revisions but it is a bit daunting. Maybe bug 760608?

Comment 15

[Josh Matthews [:jdm]]

http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=6338a8988917&tochange=7e4c2abb9fc9

Comment 16

[Josh Matthews [:jdm]]

Yes, bug 760608 and bug 761775 appear to be the only changes related to networking.

Comment 17

[Jason Duell]

Patrick:  see comment 16.   I'll try to look more into what's going on with those changesets if you don't get to it first, but may not be for a few days.

Kim--thanks again! (jdm too!)

Comment 18

[Honza Bambas (:mayhemer)]

Jason, are you working on this?

Comment 19

[Jason Duell]

Not at the moment--feel free to grab

Comment 20

[Jason Duell]

fingers crossed that perhaps bug 318253 will fix this...

Comment 21

[michaeldunn]

Please note that some(?) WebSocket functionality requires proxy support.
http://en.wikipedia.org/wiki/WebSocket#Proxy_traversal

Kim, you are using BlueCoat proxy.  Does not officially support websockets.  Can you retry your test on direct connection?

https://kb.bluecoat.com/index?page=content&id=FAQ1731&cat=PROXYSG&actp=LIST

IMHO, bug 318253 is not related to WebSockets but is related to TCP closing the connection and opening without re-authenticating from the beginning.

Comment 22

[andreas.hoelzlwimmer]

I have been redirected here from bug 318253, as I initially thought it would be that one. Is there a way to fix this problem without Proxy traversal, as I doubt I can convince the people responsible here to add such a firewall rule.

Comment 23

[Jason Duell]

I don't know of a workaround right now, but I haven't experienced it personally.  Has anyone else found a workaround?

Comment 24

[Kim Sullivan]

Michael, 

> Kim, you are using BlueCoat proxy.  Does not officially support websockets. 
> Can you retry your test on direct connection?

Never had problems with direct connections, only at work where (for obvious reasons) testing without the proxy wont work. Also, the problems start happening much sooner than actual websocktet specific stuff happens, it looks like the initial CONNECT is broken. Are there any other parts of Firefox where I could test HTTP CONNECT? The auth dialog also pops up when loading java applets (but it doesnt loop like websockets).

> IMHO, bug 318253 is not related to WebSockets but is related to TCP closing
> the connection and opening without re-authenticating from the beginning.

I think the root cause is the same - see mostly comment 9, afterwards I was just trying to narrow it down to a specific nightly. Basically, I came to the same conclusion as you did in bug 318253, comment 96 -  closing the TCP connection in midst of authentication seems to break things (on some proxies and authentication schemes, might not be specific to BlueCoat after all).

I'll re-test as soon as 318253 gets fixed.

Comment 26

[Nicolas Michel]

I also have the problem at work. We have a Bluecoat webproxy. The network team contacted the Blucoat helpdesk and they redirected us to their knowledge database saying that the best workaround is to disable websocket in firefox :)

https://kb.bluecoat.com/index?page=content&id=KB5274&actp=RSS

Comment 27

[Nicolas Michel]

Here is a tshark sample just after clicking on the OK button (153.89.254.3 is the bluecoat and 192.168.253.128 is my laptop). We see at the end that the bluecoat is saying WebSocket 1478 WebSocket Unknown Opcode (but as said in my previous message, Bluecoat support tells us the problem is not their side since it works with Google Chrome and Internet Explorer). Here are the TCP traces:

35.185353 192.168.253.128 -> 153.89.254.3 TCP 54 60821 > 8080 [FIN, ACK] Seq=874 Ack=1426 Win=17088 Len=0
 35.185760 153.89.254.3 -> 192.168.253.128 TCP 60 8080 > 60821 [ACK] Seq=1426 Ack=875 Win=64239 Len=0
 35.186635 192.168.253.128 -> 153.89.254.3 TCP 54 60822 > 8080 [FIN, ACK] Seq=845 Ack=1426 Win=17088 Len=0
 35.187007 153.89.254.3 -> 192.168.253.128 TCP 60 8080 > 60822 [ACK] Seq=1426 Ack=846 Win=64239 Len=0
 35.189603 192.168.253.128 -> 153.89.254.3 TCP 74 60823 > 8080 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1492674 TSecr=0 WS=128
 35.189825 192.168.253.128 -> 153.89.254.3 TCP 74 60824 > 8080 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1492674 TSecr=0 WS=128
 35.190918 153.89.254.3 -> 192.168.253.128 TCP 60 8080 > 60823 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460
 35.190944 192.168.253.128 -> 153.89.254.3 TCP 54 60823 > 8080 [ACK] Seq=1 Ack=1 Win=14600 Len=0
 35.190984 153.89.254.3 -> 192.168.253.128 TCP 60 8080 > 60824 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460
 35.190991 192.168.253.128 -> 153.89.254.3 TCP 54 60824 > 8080 [ACK] Seq=1 Ack=1 Win=14600 Len=0
 35.191122 192.168.253.128 -> 153.89.254.3 HTTP 956 GET / HTTP/1.1 
 35.191305 153.89.254.3 -> 192.168.253.128 TCP 60 8080 > 60823 [ACK] Seq=1 Ack=903 Win=64240 Len=0
 35.191367 192.168.253.128 -> 153.89.254.3 HTTP 927 GET / HTTP/1.1 
 35.191477 153.89.254.3 -> 192.168.253.128 TCP 60 8080 > 60824 [ACK] Seq=1 Ack=874 Win=64240 Len=0
 35.195210 153.89.254.3 -> 192.168.253.128 WebSocket 1478 WebSocket Unknown Opcode  
 35.195238 192.168.253.128 -> 153.89.254.3 TCP 54 60823 > 8080 [ACK] Seq=903 Ack=1426 Win=17088 Len=0
 35.196462 153.89.254.3 -> 192.168.253.128 WebSocket 1478 WebSocket Unknown Opcode  
 35.196526 192.168.253.128 -> 153.89.254.3 TCP 54 60824 > 8080 [ACK] Seq=874 Ack=1426 Win=17088 Len=0

Comment 28

[Kim Sullivan]

(In reply to Nicolas Michel from comment #26)
> I also have the problem at work. We have a Bluecoat webproxy. The network
> team contacted the Blucoat helpdesk and they redirected us to their
> knowledge database saying that the best workaround is to disable websocket
> in firefox :)
> 
> https://kb.bluecoat.com/index?page=content&id=KB5274&actp=RSS

Thanks, I can confirm that setting network.websocket.enabled to false in about:config resolves the issue on Stack Overflow (no idea what it breaks, but it makes the site usable again).

Comment 29

[Nicolas Michel]

You're right saying that BlueCoat doesn't support websocket. The behavior of Firefox is still not normal since asking in loop for a user/pass when the connection just cannot be done is not a good behavior. I suppose that Google Chrome/IE do the difference between a wrong authentication, only triggered when the Proxy is asking for authentication (so the user/pass box is showed again) and an impossible TCP connection.

Current behavior : when receiving the following message from the proxy (as in comment #27): "WebSocket 1478 WebSocket Unknown Opcode" firefox interprets it like it is a proxy authentication problem and so shows the user/pass input box.

Expected behavior : when receiving the said message: "WebSocket 1478 WebSocket Unknown Opcode", firefox should understand that it will not get that connection opened successfully and won't try to open it again (and it will never ask for user/pass since it is unrelated).

(In reply to michaeldunn from comment #21)
> Please note that some(?) WebSocket functionality requires proxy support.
> http://en.wikipedia.org/wiki/WebSocket#Proxy_traversal
> 
> Kim, you are using BlueCoat proxy.  Does not officially support websockets. 
> Can you retry your test on direct connection?
> 
> https://kb.bluecoat.com/index?page=content&id=FAQ1731&cat=PROXYSG&actp=LIST
> 
> IMHO, bug 318253 is not related to WebSockets but is related to TCP closing
> the connection and opening without re-authenticating from the beginning.

Comment 30

[Patrick McManus [:mcmanus]]

I suspect this is a regression from 766312 - which is a specific case of the connection being closed after a kerberos challenge because of the presence of an upgrade header.

I'll have something to test soon.

Kim, are you in a position to test if I provide a build?

Comment 31

[Robert Longson [:longsonr]]

I see this when inside my work authenticating proxy network. If you have a patch I can test it.

Comment 32

[Patrick McManus [:mcmanus]]

(In reply to Robert Longson from comment #31)
> I see this when inside my work authenticating proxy network. If you have a
> patch I can test it.

Thanks Robert.. do you by any chance know if you're using a bluecoat proxy at work?

Comment 33

[Patrick McManus [:mcmanus]]

Kim, Robert, there are builds with a work-in-progress patch here:

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-8d940fe83d31/

can you tell me if this helps with your problem. (It won't make your proxy understand websockets, of course). Robert - if you have a way to grab headers (see http log at https://developer.mozilla.org/en-US/docs/HTTP_Logging) that could be useful if it doesn't clear things up for you. The logs Kim provided above were key to understand the difference between Kim's it-works and it-doesnt-work scenario. (well, at least I think I understand it.)

Comment 34

[Nicolas Michel]

I just made some tests with that nightly build, going to http://dba.stackexchange.com/questions/17262/phpmyadmin-incredibly-slow.
I confirm I have the authentication box loop with the version 15. I don't have the problem anymore with the nightly build.

But I also have to say that trying with the version 16, I don't either have the problem anymore!

Comment 35

[Jarrod Lee Petz]

Hi All,

I have encountered this issue at my work place. Loading our new website which is in development causes multiple authentication popups. This seems to only occur when doing a force refresh of the page though with key combo CTRL+F5. Other browsers don't seem to have this problem.

We are running a Bluecoat proxy. So since I know this proxy does not support websocket I thought to try find another proxy that would. I have now setup an Apache 2.2 forward proxy on our internal network. Using these test websites below and my test Apache 2.2 proxy(which does not have authentication at ALL). I can confirm that Chrome works fine but Firefox 15.0.1 does not.

TEST SITE 1 :  http://www.websocket.org/echo.html
TEST SITE 2 :  http://websocketstest.com/

RESULTS SITE 1:
----------------------------------------------
Firefox - No authentication popup. But I do get this in the log when I click the connect button
"
ERROR: undefined
DISCONNECTED
"

Chrome - Works as expected


RESULTS SITE 2:
----------------------------------------------
Firefox - Websockets do not work at all(comet does). Authentication popup happens.

Chrome - Websockets work on all ports except 8080. This makes sense as I havn't allowed it in the Apache 2.2 config.

Comment 36

[Jarrod Lee Petz]

Attachment: Apache 2.2 Forward Proxy config

Comment 37

[Jarrod Lee Petz]

Attachment: Chrome test 1 result

Comment 38

[Jarrod Lee Petz]

Attachment: Chrome test 2 result

Comment 39

[Jarrod Lee Petz]

Attachment: Firefox test 1 result

Comment 40

[Jarrod Lee Petz]

Attachment: Firefox test 2 result

Comment 41

[Jarrod Lee Petz]

Attachment: Firefox auth pop up

Comment 42

[Jarrod Lee Petz]

Attachment: Firefox proxy settings

Comment 43

[Jarrod Lee Petz]

How funny.

This seems to be fixed in Firefox 16.0.1 which just got released and my browser just auto updated.

Comment 44

[Jarrod Lee Petz]

Another interesting thing which could be part of the issue for people who switch proxies.

If I start FF 16.0.1 with my BlueCoat proxy set the TEST SITE 2 stuff fails as expected. However if I then change the proxy to Apache and retest it fails also. It is not until the browser is restarted with the new Apache Proxy config that it seems to take effect.

Comment 45

[Jarrod Lee Petz]

The revese of my previous comment is also true. If the browser is started with the Apache config then the 'TEST SITE 2' is successful. If I then change the proxy to my Blue Coat proxy the test is still successful until the browser is restarted.

Comment 46

[Patrick McManus [:mcmanus]]

kim, comment 33?

Comment 47

[Patrick McManus [:mcmanus]]

(In reply to Jarrod Lee Petz from comment #44)
> Another interesting thing which could be part of the issue for people who
> switch proxies.
> 
> If I start FF 16.0.1 with my BlueCoat proxy set the TEST SITE 2 stuff fails
> as expected. However if I then change the proxy to Apache and retest it
> fails also. It is not until the browser is restarted with the new Apache
> Proxy config that it seems to take effect.

that is a dup of 701562, which is fixed in FF 18

Comment 48

[Patrick McManus [:mcmanus]]

my hypothesis here is that the bluecoat is sensitive about the various challenge and responses of windows auth all being on the same connection. bug 766312 made a change that (by way of a few odd turns) said any transaction that is trying to upgrade to websockets should not allow connection reuse - the fear is that intermediaries might get confused as to the protocol state so we finished this connection even if a success upgrade to websockets did not happen even though strictly speaking only 101 should be required - it was the conservative thing to do.

however I think an exception should be made for 401/407 because of the stateful session management that is done with those response codes.

Comment 49

[Patrick McManus [:mcmanus]]

Attachment: patch 0

Comment 50

[Jason Duell]

Comment on attachment 671834 [details] [diff] [review]
patch 0

Review of attachment 671834 [details] [diff] [review]:
-----------------------------------------------------------------

Good idea.  Hope it fixes this.

::: netwerk/protocol/http/nsHttpConnection.cpp
@@ +867,5 @@
>      }
>      
>      const char *upgradeReq = requestHead->PeekHeader(nsHttp::Upgrade);
> +    // When trying to upgrade to another protocol don't allow http persistent
> +    // connections unless there is an auth failure

// Don't use persistent connection for Upgrade unless there's an auth failure:
   // some proxies expect to see auth response on persistent connection.

Feel free to modify/ignore my comment polishing :)

Comment 51

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b8b86cd19f38

Comment 52

[Kim Sullivan]

(In reply to Patrick McManus [:mcmanus] from comment #33)

> can you tell me if this helps with your problem. (It won't make your proxy
> understand websockets, of course)

Works like a charm. BTW, I'm not sure exactly why Bluecoat says it doesn't support websockets, the websocket echo demo works (I tried both SSL and non-SSL to be sure).

Also I tried both FF 16 and 16.0.1 which didn't resolve the problem, only the builds from the tryserver. Do you want me to do any network traces with the build using our bluecoat proxy?

Anyway, thanks a lot!

Comment 53

[Robert Longson [:longsonr]]

Fortunately for me my issue with StackOverflow seems to have disappeared with Firefox 16. I'm afraid I can't therefore do any testing here.

Comment 54

[(no longer active)]

https://hg.mozilla.org/mozilla-central/rev/b8b86cd19f38

Comment 55

[Patrick McManus [:mcmanus]]

(In reply to Kim Sullivan from comment #52)
> (In reply to Patrick McManus [:mcmanus] from comment #33)
> 
> > can you tell me if this helps with your problem. (It won't make your proxy
> > understand websockets, of course)
> 
> Works like a charm. BTW, I'm not sure exactly why Bluecoat says it doesn't
> support websockets, the websocket echo demo works (I tried both SSL and
> non-SSL to be sure).

It probably says that because it doesn't support them transparently, but when configured to allow an explicit tunnel through the proxy it (apparently) goes ok.

> 
> Also I tried both FF 16 and 16.0.1 which didn't resolve the problem, only
> the builds from the tryserver. 

Thanks Kim!

> Do you want me to do any network traces with
> the build using our bluecoat proxy?
> 

not needed at this point. Thanks.

Comment 56

[Patrick McManus [:mcmanus]]

Comment on attachment 671834 [details] [diff] [review]
patch 0

I'll nom this for backport based on comment 52 verifying the fix.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 766312
User impact if declined: Users behind bluecoat proxies that are configured to do integrated windows authorization will receive auth popups when visiting sites that use non ssl websockets. StackExchange.com is the high profile site.
Testing completed (on m-c, etc.): on m-c, confirmed by field with a nightly.
Risk to taking this patch (and alternatives if risky):  low
String or UUID changes made by this patch: none

Comment 57

[Ryan VanderMeulen [:RyanVM]]

Should this have a test?

Comment 58

[Alex Keybl [:akeybl]]

(In reply to Patrick McManus [:mcmanus] from comment #56)
> Comment on attachment 671834 [details] [diff] [review]
> patch 0
> 
> I'll nom this for backport based on comment 52 verifying the fix.
> 
> [Approval Request Comment]
> Bug caused by (feature/regressing bug #): 766312
> User impact if declined: Users behind bluecoat proxies that are configured
> to do integrated windows authorization will receive auth popups when
> visiting sites that use non ssl websockets. StackExchange.com is the high
> profile site.
> Testing completed (on m-c, etc.): on m-c, confirmed by field with a nightly.
> Risk to taking this patch (and alternatives if risky):  low
> String or UUID changes made by this patch: none

Can't we just back out bug 766312 from beta? network.http.pipelining is disabled by default, so it's not clear how that patch specifically benefits our beta/release users.

Comment 59

[Patrick McManus [:mcmanus]]

(In reply to Alex Keybl [:akeybl] from comment #58)

> 
> Can't we just back out bug 766312 from beta? network.http.pipelining is
> disabled by default, so it's not clear how that patch specifically benefits
> our beta/release users.

I did think about this when deciding to nom the patch, I should have addressed it.

I thinking backing out 766312 is actually riskier than promoting 766817 - though neither path is very risky.. one is probably "med-low" and the other "low".

I say this because 766312 addressed a low level latent bug in the "DontReuse()" logic. Pipelines happened to be what tripped over it but its possible that other new code in 16 or 17 also depends on that working correctly too. I can't point to anything, but its harder to say with confidence because its a pretty low level mechanism used in a bunch of different parts of http.

There is no known bug in the implementation of 766312, it just exposed the problem addressed in this bug. The scenario this patch applies to is much smaller - websockets services that are configured with http auth. Realistically that means via proxy with windows integrated auth because we don't have a mechanism for non-integrated authorization of subresources anyhow (if any servers are doing it they would be extremely small in number and intranet based) and we've tested the prominent ones here (bluecoat, TMG, ISA, wingate). So the surface is much smaller.

so that's my opinion, but there are arguments either way and I'm happy to have you make the decision.

Comment 60

[Patrick McManus [:mcmanus]]

 https://hg.mozilla.org/releases/mozilla-aurora/rev/6bfc36b5a53e

Comment 61

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 671834 [details] [diff] [review]
patch 0

Your explanation helps a lot and it makes sense to take this on Beta instead of looking to backout, please land this week.

Comment 62

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/releases/mozilla-beta/rev/06e3c0f40f48

Comment 63

[Daniel]

I´m still getting auth prompts in firefox 17 and 18. It´s not fixed yet.

Comment 64

[Kim Sullivan]

(In reply to Daniel from comment #63)
> I´m still getting auth prompts in firefox 17 and 18. It´s not fixed yet.

Please try it again with the latest 17 beta (build id 20121031065642 - released a few days ago), it seems it can take a while until a new beta with the patch included is released. I can confirm that with the latest beta, this specific problem (repeated prompts with websockets) is fixed for me.

Comment 65

[Daniel]

Confirmed solved in firefox 17 beta build 4. No more auth prompts. Good job!!!

Comment 66

[Daniel]

Sorry, but the auth prompts are here again. Tested with the same website www.peixeurbano.com.br. Few hours ago was ok, but now they still appearing.

Comment 67

[Patrick McManus [:mcmanus]]

daniel, there is a separate ntlm bug related to clicking  reload (separate than websockets). Can you confirm you are reporting something that does not involve reload?

Comment 68

[Daniel]

I can confirm that the prompt shows up without reloading and reloading too.

Comment 69

[Patrick McManus [:mcmanus]]

thanks

1 - what proxy do you use?

2 - pls file a separate bug and cc me and jduell .. as we probly want to keep the work already done here.

Comment 70

[Ioana (away)]

I tried to reproduce this issue on Firefox 16 (20120911232325), but it doesn't seem to reproduce unless you have a BlueCoat proxy.

If anyone does have access to a BlueCoat proxy, please try to verify this bug fix.

Comment 71

[Kim Sullivan]

(In reply to Ioana Budnar [QA] from comment #70)
> I tried to reproduce this issue on Firefox 16 (20120911232325), but it
> doesn't seem to reproduce unless you have a BlueCoat proxy.
> 
> If anyone does have access to a BlueCoat proxy, please try to verify this
> bug fix.

Please see comment 52, if anything more is needed, I'll be glad to try to help.

Comment 72

[Daniel]

I have BlueCoat proxy and still getting auth prompts with version 17.

Comment 73

[Christian]

Hi,

I have also a Bluecoat Proxy and can reproduce the issue with FF Version 17.0.1 on any Website when pressing ctrl+F5.

Any Ideas?
Tried everything from Bleucoat Support Website.

Thanks
Christian

Comment 74

[Daniel]

I have Bluecoat explicit and authenticated Proxy and can reproduce the issue just accessing www.peixeurbano.com.br for example. Don´t need to press ctrl+F5

Comment 75

[rcs]

This issue has recurred with Firefox ESR 24.0 against URL https://open.sap.com/ (even with network.websocket.enabled set to false).  Has the previous remediation been broken?

Thanks,
rcs

Comment 76

[rcs]

This persists with Firefox ESR 24.1.0 against URL https://open.sap.com/ even with the following settings:

network.auth.use-sspi false
network.automatic-ntlm-auth.allow-proxies false
network.automatic-ntlm-auth.trusted-uris false
network.websocket.enabled false

This issue does not seem to be RESOLVED FIXED in this release.

Thanks,
rcs

Comment 77

[Mike Kaply [:mkaply]]

Can you open a new bug with specific details on how to recreate the problem?

Comment 78

[stan]

This bug is still present in Firefox 38 ESR. 

Barracuda Webfilter appliance forward deployment HTTP PROXY with NTLM authentication.
WebSockets to port 80 (non secure) causes 1 prompt for Proxy username & password in Firefox.
WebSockets to poort 443 (secure) work, probably because the PROXY cannot inspect and passes.
In Chrome or IE there is no password prompt and WebSockets to port 80 fail silently.

Since the websocket.enabled false config pref is gone there is no workaround. dom.workers.websocket.enabled false does not seem to do anything.