Closed Bug 513659 Opened 16 years ago Closed 15 years ago

SSL over IPv6 does not work under Windows XP 32-bit when receive buffer size is 64kb+

Categories

(NSPR :: NSPR, defect, P2)

Product:
NSPR
Component:
NSPR
Platform:
x86
Windows XP
Type:
defect

Tracking

(firefox5-, blocking2.0 ?)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox5 - ---
blocking2.0 --- ?

People

(Reporter: ramon, Assigned: bugzilla.mozilla.simon)

References

(
URL
)

Details

(Whiteboard: [needed for IPv6 testday][tbwants])

Attachments

(5 files, 3 obsolete files)

firefox-ipv6-ssl-fail.pcap
572 bytes, application/octet-stream
Details
chrome-ipv6-ssl.pcap
48.41 KB, application/octet-stream
Details
PCAP Trace showing Firefox FINished the connection after TCP handshake
572 bytes, application/pcap
Details
NSPR log from Firefox showing IPv6 SSL connection failure
26.47 KB, text/plain
Details
Patch to automatically lower the SO_RCVBUF on AF_INET6 sockets to 65535 on Windows XP 32-bit so that getpeername() doesn't fail, v2, by Simon Arlott
2.71 KB, patch
Details | Diff | Splinter Review
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 If Firefox contacts an https URI over IPv6, the socket is immediatelly closed after TCP handshake: Firefox Server --------SYN-------------> <-------SYN, ACK -------- -------------ACK--------> --------FIN, ACK--------> 3.0.13 under Linux (Ubuntu) works fine. 3.5.2 under Linux (Ubuntu) works fine. The bug was reproduced on another Windows PC too. Reproducible: Always Steps to Reproduce: 1. get IPv6 2. start Wireshark to get a trace 3. enable ipv6 in about:config 4. surf to an IPv6 enabled https site, e.g. https://www.makhutov.org/ Actual Results: Firefox does not generate an error report but still shows the previous site. Expected Results: Initiate TLS handshake and load the site. IE8 works fine. Opera 9.22 works fine.
Possibly Bug 388117? At the least, this probably belongs in NSS.
Reproduced & Confirmed under Firefox 3.5.5, with a different website as well (http://timatio.com)
Both those sites are behind a tunnel with an MTU of 1480. Most likely this is an MTU issue. I can state that Firefox + Windows + IPv6 + optionally SSL has been working for at *minimum* for the last 8 years, that is as long as https://www.sixxs.net has been existing ;) Why are these reports against 3.5.2 anyway, 3.5.5 is out for quite some time already.
I can confirm this with: Firefox 3.6 under Windows XP SP3 running a sixxs aiccu tunnel on a linux 2.6 server with a tunnel MTU 1280 (as auto-configured by aiccu). Looking at the packet trace shows that Firefox is closing the connection to the server immediately after the connection is successfully established. No. Time Source Destination Protocol Info 1 0.000000 2001:770:1b7:0:c51b:9ae4:adc6:c031 2001:7b8:3:4f:202:b3ff:fe46:bec TCP applus > https [SYN] Seq=0 Win=16384 Len=0 MSS=1440 2 0.062151 2001:7b8:3:4f:202:b3ff:fe46:bec 2001:770:1b7:0:c51b:9ae4:adc6:c031 TCP https > applus [SYN, ACK] Seq=0 Ack=1 Win=5760 Len=0 MSS=1440 3 0.062178 2001:770:1b7:0:c51b:9ae4:adc6:c031 2001:7b8:3:4f:202:b3ff:fe46:bec TCP applus > https [ACK] Seq=1 Ack=1 Win=17280 Len=0 4 0.063129 2001:770:1b7:0:c51b:9ae4:adc6:c031 2001:7b8:3:4f:202:b3ff:fe46:bec TCP applus > https [FIN, ACK] Seq=1 Ack=1 Win=17280 Len=0 5 0.125968 2001:7b8:3:4f:202:b3ff:fe46:bec 2001:770:1b7:0:c51b:9ae4:adc6:c031 TCP https > applus [FIN, ACK] Seq=1 Ack=2 Win=5760 Len=0 6 0.126013 2001:770:1b7:0:c51b:9ae4:adc6:c031 2001:7b8:3:4f:202:b3ff:fe46:bec TCP applus > https [ACK] Seq=2 Ack=2 Win=17280 Len=0
As I mentioned before: works fine for me, and has never been broken as long as FF has supported IPv6 on Windows. Just upgraded to 3.6 and still works: Just check https://www.sixxs.net/tools/ipv6calc/ which nicely shows "User agent identification Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)" This ticket truly can be closed as invalid. (Btw: providing a TCPdump without the packet contents is far from useful as it effectively just shows that you did some TCP but that is it, it doesn't show what the content of the session was)
I understand that it works for you (and it has worked for me before on my old PC), so please let's track down what's different between these two copies of windows - What service pack level at you running at? My User agent shows as: "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 GTB6 (.NET CLR 3.5.30729)" I assume en-GB vs en-US makes no difference. I assume GTB6 means Google Toolbar. I have tried in safemode, and it makes no difference. I posted a *full* TCPdump, I did not strip any packets out. Maybe the newlines make things hard to see so I'm posting the same data but aliasing the address to make it easier to read: No. Time Source > Destination Protocol Info 1 0.000000 fx:port > server:https [SYN] Seq=0 Win=16384 Len=0 MSS=1440 2 0.062151 server:https > fx:port [SYN, ACK] Seq=0 Ack=1 Win=5760 Len=0 MSS=1440 3 0.062178 fx:port > server:https [ACK] Seq=1 Ack=1 Win=17280 Len=0 4 0.063129 fx:port > server:https [FIN, ACK] Seq=1 Ack=1 Win=17280 Len=0 5 0.125968 server:https > fx:port [FIN, ACK] Seq=1 Ack=2 Win=5760 Len=0 6 0.126013 fx:port > server:https [ACK] Seq=2 Ack=2 Win=17280 Len=0 As you can see from the timestamps, the connection was closed less than 1ms after the server confirmed it's opening. (RTT to the server was 60ms, and there were no retransmissions) Are there extra network debugging flags I can run with to get extra network debugging? I just noticed that www.sixxs.net have 4 IPv6 address, is it possible that we're testing against different addresses?
> What service pack level at you running at? It should be more than obvious that when one reports a problem with something that you should at least have applied all service packs etc. > I posted a *full* TCPdump, I did not strip any packets out. You only posted the summary of the tcpdump, not an actual pcap file which contains the packets. One can attach the file to the bug report. > As you can see from the timestamps, the connection was closed less than 1ms after the server confirmed it's opening. And? HTTP connections are very quick in most cases. As you didn't even bother of mentioning what exact URL you are connecting to, there is no way for anybody who would read this bug report for even remotely figuring out what you are trying to fetch (the full PCAP again would partially resolve that as then it might be included, then again in SSL style, it would be crypted ;) > Are there extra network debugging flags I can run with to get extra > network debugging? Maybe you want to at least look at Wireshark instead of tcpdump? > I just noticed that www.sixxs.net have 4 IPv6 address, is it possible > that we're testing against different addresses? Of course, and actually that DNS label will have addresses that change over time depending on availability and everything. The full pcap would at least show on the TCP level what the error would be though. A better question would be though: a) are you having a problem with firefox b) are you having a problem with firefox and a specific website?
I should have noticed this directly, but for some mysterious reason, your client is sending [FIN, ACK] after it actually got an ACK back from the server. Your client is thus closing the connection. So some other questions: - does a non-SSL connection work? - does a IPv4 SSL connection work? - did you try HTTPS + IPv6 against other websites?
Attached file chrome-ipv6-ssl.pcap —
> > What service pack level at you running at? > It should be more than obvious that when one reports a problem with something > that you should at least have applied all service packs etc. This is a work laptop, so it's patched with the latest updates. > > I posted a *full* TCPdump, I did not strip any packets out. > You only posted the summary of the tcpdump, not an actual pcap file which > contains the packets. One can attach the file to the bug report. Done. I have also posted a pcap from chrome on the same machine, connecting to the same website (www.sixxs.net) > > As you can see from the timestamps, the connection was closed less than 1ms > after the server confirmed it's opening. > And? HTTP connections are very quick in most cases. As you didn't even bother > of mentioning what exact URL you are connecting to, there is no way for anybody > who would read this bug report for even remotely figuring out what you are > trying to fetch (the full PCAP again would partially resolve that as then it > might be included, then again in SSL style, it would be crypted ;) As you noticed my RTT was 60ms and the local client was disconnecting. > > Are there extra network debugging flags I can run with to get extra > > network debugging? > Maybe you want to at least look at Wireshark instead of tcpdump? All caps have been made with Wireshark so far. > > I just noticed that www.sixxs.net have 4 IPv6 address, is it possible > > that we're testing against different addresses? > Of course, and actually that DNS label will have addresses that change over > time depending on availability and everything. > > The full pcap would at least show on the TCP level what the error would be > though. > > A better question would be though: > a) are you having a problem with firefox > b) are you having a problem with firefox and a specific website? I have only tested Firefox, IE8 and Chrome. Only Firefox shows this behavior. > I should have noticed this directly, but for some mysterious reason, your > client is sending [FIN, ACK] after it actually got an ACK back from the server. > Your client is thus closing the connection. Right, so I'm trying to track down what the mysterious reason is. > So some other questions: > - does a non-SSL connection work? Yes, both IPv4 and IPv6 with non-SSL work perfectly. > - does a IPv4 SSL connection work? Yes, also if I set network.dns.disableIPv6 then ssl works (because it's IPv4 only) > - did you try HTTPS + IPv6 against other websites? Yes, I'm using the sixxs dns servers for address resolution so I get IPv6 addresses for Google Mail, etc. They show this behavior too. This behavior also shows when I access https://www.darkskies.za.net/ which is a server on my local network. (i.e. it should have an MTU of 1500) BTW: The 'user visible' part of this experience is that after typing the address and hitting enter/go, the throbber spins for a fraction of a second and then stops. The previous web page continues to be displayed in the browser window. Additionally the darkskies server has a self-signed cert, which only shows a warning on IPv4 (which is 'obvious' from the packet traces, because SSL negotiation doesn't start on IPv6)
> This is a work laptop, so it's patched with the latest updates. In a lot of companies that statement would actually mean "the latest patches that the IS department approved", which tends to then be running several months behind... Nevertheless it seems to me it is a problem on the local host. Do you have any kind of 'firewalling' or 'anti-virus' software running or installed on your host, as that might also partially explain the behavior. As you state to have a 'work laptop', do they excert any kind of control over what sites you can/cannot connect to or is it in an Active Directory Domain? As this problem is also happening on a local server, the question would be if you have access to that and if you can capture the session packets there. Do also watch for ICMP flying around, they might indicate some issues that you don't see now.
(In reply to comment #12) > > This is a work laptop, so it's patched with the latest updates. > In a lot of companies that statement would actually mean "the latest patches > that the IS department approved", which tends to then be running several months > behind... With the recent global security concerns, they've had a recent blitz on new updates. > Nevertheless it seems to me it is a problem on the local host. > > Do you have any kind of 'firewalling' or 'anti-virus' software running or > installed on your host, as that might also partially explain the behavior. There's only Sophos. I wonder why it would affect only Firefox and not the other two browsers. > As you state to have a 'work laptop', do they excert any kind of control over > what sites you can/cannot connect to or is it in an Active Directory Domain? I don't think there's any white/black listing going on, otherwise it would apply to the other browsers too. Certainly it should affect IPv4 connections with the same hostname... > As this problem is also happening on a local server, the question would be if > you have access to that and if you can capture the session packets there. > Do also watch for ICMP flying around, they might indicate some issues that you > don't see now. I did some quick tests on my server, and didn't see any differences (besides the 1ms rrt difference in timestamps). I should also try this on the work network (which should also be IPv6 enabled if I remember correctly) Is there a command line tool that's built on the same backend that Firefox uses that I could use to try and debug this, alternatively is there an nightly/debug version of Firefox (Minefield?) that I could use to debug this further... (My guess is that there's some weird ipv6/nss/driver/??? mismatch causing this)
Maybe the problem is related to source IP addresses? I will upload a pcap trace which shows that I have the same problems as Norman. My environment is: Windows XP german, SP3, all patches installed. Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729) Windows firewall is deactivated. AntiVir virus scanner is deactivated. Problem happens only in Firefox, Opera and Internet Explorer are doing fine. Firefox IPv6 to http works fine. Firefox and IPv4+SSL also works fine. Only Firefox with IPv6+SSL shows the problem. The problem occurs also with other sites, e.g.: https://ipv6.google.com/, https://www.makhutov.org/, https://www.sixxs.net/.
I made a iframe matrix to assist with testing: http://darkskies.za.net/ip/ I can't get the bottom-right entry to display anything (not even the security warning)
Norman, thanks for the matrix. I retested with another PC with Windows 7 using also Firefox 3.5.7. This time everything worked fine. MAybe it is related to the Windows version. Is there somebody having IPv6+SSL working on Windows XPsp3?
See comment #5 "Windows NT 5.1" aka Windows XP SP3, working fine. The matrix for that just warns about the SSL certs being self-signed, but the rest works. (and I would really complain and try to figure out what was broken if it would not as I kind of use IPv6 a lot ;)
Jeroen, do you know a way to track down the problem. Is there a possibility in Firefox to activate SSL debugging?
No surprises to the last few comments. I seem to remember something about XP having two possible IPv6 stacks - either from pre-SP1 days (the research stack), or SP1 and onwards (the advanced networking stack). Does minefield have extra debugging that could help figure out the differences? Is there a command line tool that fetches a url with debugging information (like curl, but using the firefox network api's)
Good point, some related dll's, check if you have the same ones: tcpip6.sys = 5.1.2600.5625 6to4svc.dll = 5.1.2600.5512 Wship6.dll = 5.1.2600.5512 Ws2_32.dll = 5.1.2600.5512 Tunmp.sys = 5.1.2600.5512 Additionally there is another problem that might be the cause: there are two ways to install IPv6 on XP: "netsh int ipv6 install" which is the correct way, and "ipv6.exe install" which is the old way and apparently does not do everything that the former does. If one has installed it using the latter, try removing it (netsh int ipv6 uninstall) and then installing it again with the netsh command OR the third way, but apparently that is fine is to use is to use "Start" -> "Control Panel" -> "network connections" and remove/add it from the interfaces (right click, properties).
I have the same dll versions as above, I'm not sure how IPv6 was enabled because it was pre-installed. So I uninstall and reinstalled using netsh. No change. Network interface shows "Microsoft TCP/IP version 6" as a protocol.
I also have the same DLL versions. Is the problem really on IP stack level? Firefox+IPv6 works fine, but SSL+IPv6 does not work. I would assume that after TCP handshake the connection is handled over to Firefox. To me it looks like Firefox opens a TCP connection, and then something happens inside which causes him to the close the connections instead initiating the TLS handshake.
We've seen this problem - please try this: Instead of going to the IPv6 site by name, get it's IPv6 address from DNS, then try going to the https:// site using that IP address *but using the expanded IPv6 notation* - i.e., instead of https://[3ffe:1234::42], use https://[3ffe:1234:0:0:0:0:0:42] And see if Firefox connects.....
> https://[3ffe:1234:0:0:0:0:0:42] Using that kinda voids the whole idea of SSL, except for the crypto part, also note that 6bone is dead, if you want to provide an example address use an address/prefix from 2001:db8::/32 If you would actually look at the above tcpdump outputs and the provided pcaps you will notices that the people having these issues are connecting already over IPv6. The problem seems to be that the moment that the connection is established their side of the connection directly FIN's it and thus the connection closes. The problem that you might have seen is when a NAT box's DNS resolver is bogus or when Windows decides that it doesn't have proper enough IPv6 connectivity and thus does not even remotely try doing AAAA records let alone IPv6 connects. As we are talking about Windows XP SP3 though that little problem does not have effect. Note: I don't have the problem btw, FF just works like it always did. Only thing I can think of that might be causing issues is some kind of weird firewall/AV software.
No. No firewall, no AV. We've seen this on a LAN, browser to server. The point isn't to drag V6 back to the stone ages, but to show that we think Firefox has issues with V6 addresses and certs. Under the exact same operating scenario, Opera, Chrome and IE all connect fine.
> The point isn't to drag V6 back to the stone ages, but to show that we think > Firefox has issues with V6 addresses and certs. It's unlikely be certs because the connection is closed even before the ssl negotion starts. I will update my matrix to include IP addresses too and test tomorrow.
I just retested FF3.5.7 on another XPsp3 PC -> no SSL over IPv6. Then I removed Firefox completely and made a fresh installation of FF3.6. I also removed the Antivir Virus Scanner, but SSL over IPv6 still does not work on this PC :-(
ramon: I updated my matrix, do any of the v6/ssl blocks show anything?
no: it doesn't matter if the URL contains an IPv6 address or a domain. I both cases IPv6+SSL does not work on my XPsp3 PCs.
Dave: Testing with a raw ip address shows the same problem. The TCP connection is closed before SSL negotiation is started.
I just installed VirtualBox on my XPsp3 PC and started a clean XPsp2 Image without any applications except Firefox 3.6 and FF+IPv6+SSL works fine in the virtual machine. So I guess there must be somehting on my main Windows system which confuses Firefox. Unfortunately I can not uninstall software until it work on this PC too.
Can you confirm if a fully patched SP3 image works too? If possible track it down to a single update?
Hi, All! I would like to note that I have the same problem too. On HP 510 Notebook with Windows XP Home Edition preinstalled and updated to SP3 with all available patches. IE8 works. On another two PCs with Win XP Home and SP3 and all patches there is no problems with IPv6+SSL. Firefox in question: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729). Problem appeared with some 3.x version after activating ipv6 in September, 2009. When I try to connect to my home server, Apache writes to ssl_engine_log: [31/Jan/2010 10:58:11 07590] [info] Connection to child 0 established (server www.aws-net.org.ua:443, client 2001:15c0:6605:1:216:6fff:fec8:7253) [31/Jan/2010 10:58:11 07590] [info] Seeding PRNG with 1160 bytes of entropy [31/Jan/2010 10:58:11 07590] [trace] OpenSSL: Handshake: start [31/Jan/2010 10:58:11 07590] [trace] OpenSSL: Loop: before/accept initialization [31/Jan/2010 10:58:11 07590] [debug] OpenSSL: read 0/11 bytes from BIO#08320100 [mem: 083AC000] (BIO dump follows) +-------------------------------------------------------------------------+ +-------------------------------------------------------------------------+ [31/Jan/2010 10:58:11 07590] [info] Spurious SSL handshake interrupt[Hint: Usually just one of those OpenSSL confusions!?] Using ipv6 address instead of hostname in URL makes no difference. Pointing Firefox to any other ipv6 ssl-enabled site also reveal this problem. IMHO, it's really some problem with Windows... but which and how to solve it? Reinstalling Windows is a last resort but not a good one.
FYI: I tried enabling connections via a proxy server, and unsurprisingly it makes DNSv6+ssl work (Hostname with AAAA record). It fails to connect using a literal because it asks the squid to CONNECT to 2001:770::etc:etc:443. Squid tries to connect to 0.0.7.209:443 (2001=7*256+209). If I hand connect to squid and ask for CONNECT [2001:770::etc:etc]:443 then it works correctly.
ref to thread discussing this problem: http://www.gossamer-threads.com/lists/nsp/ipv6/20412
(In reply to comment #35) > It fails to connect using a literal because it asks the squid to CONNECT to > 2001:770::etc:etc:443. That seems bad. Can you file that as a separate bug? I think I have a simple fix.
(In reply to comment #37) > (In reply to comment #35) > > It fails to connect using a literal because it asks the squid to CONNECT to > > 2001:770::etc:etc:443. > > That seems bad. Can you file that as a separate bug? I think I have a simple > fix. see: bug 445168
Reporter, are you still seeing this issue with Firefox 3.6.13 or later in safe mode or a fresh profile? If not, please close. These links can help you in your testing. http://support.mozilla.com/kb/Safe+Mode http://support.mozilla.com/kb/Managing+profiles Also, ensure you update all your plugins (Flash, Reader, java, quicktime, etc) to the latest version available on the vendor's website.
Whiteboard: [CLOSEME 2011-2-25]
The problem persists in Firefox 3.6.13 (Windows XP). I assume it's a problem with the certificates. 1. Opening a site with https and v6 works, if the certificate is official (i.e. from Thawte). 2. Opening a site with https and v6 doesn't work, if the certificate is self-signed or missing. When using Safari 5.0.3 instead of Firefox 3.6.13 on the same machine the problem doesn't exist. Also when disabling IPv6 in the network or in Firefox on the same machine everything works as expected by using IPv4.
@Tyler: Yes, I tested with FF3.6.13 in safe mode and the problem is still there (at least on this XPsp3 machine). Yes, my plugins are up2date. I made a testsite: http://myip.pernau.at/ I consists of 4 frames and loads the same site using IPv4/v6 with and without SSL. You can see the result here: http://myip.pernau.at/chrome.9.0.597.84.gif http://myip.pernau.at/internetexplorer.8.0.gif http://myip.pernau.at/firefox.3.6.13-safemode.gif @Fredy: The problem I experience here and I have reported here is unrelated to certificate problems. Firefox closes the TCP connection before the SSL handshake (see the pcap trace above). If you have certificate problems you should open another bug report.
@Tyler: I also tested FF4ß10 and using a new profile. Same result, IPv6+HTTPS does not work. So it must be caused by something which is special on my PC.
Whiteboard: [CLOSEME 2011-2-25]
Version: unspecified → Trunk
This is occurring with Windows XP SP2 and Firefox 3.6.13/Thunderbird 3.1.7 too. Both clients connect and then immediately close the connection if IPv6 is used and the connection is to use SSL, regardless of destination port. Connections using Firefox over IPv6 using HTTP work which proves that it's not an OS problem. HTTPS connections using non-SSL ports (e.g. 80) exhibit the same behaviour, but HTTP works. This is going to cause problems if not fixed by June 8th (http://isoc.org/wp/worldipv6day/).
It's hard to tell if I have the right socket or not from the NSPR log, but this looks wrong: 3600[1927280]: recv -> 1, error = -5998, os error = 0 3600[1927280]: PR_Read returned [n=-1] 3600[1927280]: ErrorAccordingToNSPR [in=-5978 out=80004005] 3600[1927280]: nsSocketTransport::OnMsgInputClosed [this=41b0580 reason=80004005] 3600[1927280]: nsSocketInputStream::CloseWithStatus [this=41b0618 reason=0] 3600[1927280]: nsSocketInputStream::CloseWithStatus [this=41b0618 reason=80470002] -5998 is PR_WOULD_BLOCK_ERROR It should try again instead of closing the socket.
I ran "firefox -safe-mode" with the following set: NSPR_LOG_FILE=c:\nspr.txt NSPR_LOG_MODULES=all:5,timestamp SSLDEBUGFILE=c:\ssldebug.txt SSLDEBUG=99 SSLTRACE=99 I've attached part of the nspr.txt file but Firefox did not create an ssldebug.txt file. 2011-02-20 16:20:10.906000 UTC - 4064[830640]: nsSocketOutputStream::Write [this=2f4ee18 count=362] 2011-02-20 16:20:10.906000 UTC - 4064[830640]: calling PR_Write [count=362] netwerk/base/src/nsSocketTransport2.cpp nsSocketInputStream::Write Calls PR_Write() <NSS code calls SocketRecv> 2011-02-20 16:20:10.906000 UTC - 4064[830640]: recv: fd=37a5040 osfd=1192 buf=19ff8a4 amount=1024 flags=0 nsprpub/pr/src/io/prsocket.c SocketRecv Calls _PR_MD_RECV(), which returns 1 2011-02-20 16:20:10.906000 UTC - 4064[830640]: recv -> 1, error = -5998, os error = 0 Calls PR_GetError(), which returns -5998 (PR_WOULD_BLOCK_ERROR) <NSS code returns after calling SocketRecv> 2011-02-20 16:20:10.906000 UTC - 4064[830640]: PR_Write returned [n=-1] PR_Write() returns -1 2011-02-20 16:20:10.906000 UTC - 4064[830640]: ErrorAccordingToNSPR [in=-5978 out=80004005] nsSocketInputStream::Write Calls PR_GetError(), which returns -5978 (PR_NOT_CONNECTED_ERROR) There is special handling for PR_WOULD_BLOCK_ERROR, but this isn't used because the error code has changed after SocketRecv() returned. Without NSS debug logging I can't tell where it happens, but PR_SetError() must be getting called from somewhere changing the error from -5998 to -5978. The connection is still being established according to the OS so perhaps there is some operation that Win32 doesn't allow on in-progress IPv6 connections that it does allow for IPv4, and the SSL code is storing an error as a result of this.
I'm unable to run the 3.6 or 4.0 tinderbox builds. Even with the VC2005 debug DLLs I get "The application has failed to start because the application configuration is incorrect", so I can't provide an SSL debug/trace log.
When ssl2_BeginClientHandshake() calls ssl_GetPeerInfo(), Windows returns WSAENOTCONN. There's a comment that this can occur in HP-UX but that workaround (PR_Write() 0 bytes) doesn't work on WIN32: SSL: tracing set to 99 SSL: debugging set to 99 4028: SSL: grow buffer from 0 to 18432 4028: SSL: grow buffer from 0 to 18432 4028: SSL[61363104]: connect... 4028: SSL[61363104]: connect failed, errno=-5934 4028: SSL[61363104]: secure connect completed, rv == -1 4028: SSL[61363104]: SecureSend: sending 362 bytes 4028: SSL[61363104]: handshake loop 0... 4028: SSL[61363104]: ssl2 begin client handshake getting peer name 4028: SSL[61363104]: ssl2 begin client handshake getting peer name rv=-1 err=-5978 4028: SSL[61363104]: ssl2 begin client handshake getting peer name retry_rv=-1 err=-5978 4028: SSL[61363104]: handshake rv=-1 4028: SSL[61363104]: SecureSend: returning -1 count, error -5978 4028: SSL[61363360]: closing, rv=0 errno=-5978 The "connect failed" is normal as it just indicates a non-blocking WSAConnect(): WSAEALREADY A nonblocking connect or WSAConnect call is in progress on the specified socket.
http://www-01.ibm.com/support/docview.wss?uid=swg21317096 appears to refer to this issue too, but Firefox isn't setting the socket buffer size. It turns out that the bug in Windows doesn't require setting the socket size... I have two test apps that reproduce the getpeername() issue on Windows XP SP2: non-blocking code: http://s85.org/Qdez91lC non-blocking exe: http://s85.org/gduZVlkK blocking code: http://s85.org/8HMADvP3 blocking exe: http://s85.org/GzUyM4u7 Under Wine: --------------------------- test-ipv6-nonblock-getpeername --------------------------- IPv4 socket=76/0 (0ms) IPv4 connect=-1/10035 (0ms) IPv4 select=1/0 [r=0,w=1,e=0] (1ms) IPv4 getpeername=0/0 (0ms) IPv6 socket=84/0 (0ms) IPv6 connect=-1/10035 (0ms) IPv6 select=1/0 [r=0,w=1,e=0] (0ms) IPv6 getpeername=0/0 (0ms) --------------------------- test-ipv6-block-getpeername --------------------------- IPv4 socket=76/0 (0ms) IPv4 connect=0/0 (1ms) IPv4 getpeername=0/0 (0ms) IPv6 socket=84/0 (0ms) IPv6 connect=0/0 (0ms) IPv6 getpeername=0/0 (0ms) -> getpeername() works for both IPv4 and IPv6 on blocking and non-blocking sockets. Under Windows XP SP2: --------------------------- test-ipv6-nonblock-getpeername --------------------------- IPv4 socket=100/0 (93ms) IPv4 connect=-1/10035 (0ms) IPv4 select=1/0 [r=0,w=1,e=0] (63ms) IPv4 getpeername=0/0 (0ms) IPv6 socket=128/0 (31ms) IPv6 connect=-1/10035 (0ms) IPv6 select=1/0 [r=0,w=1,e=0] (47ms) IPv6 getpeername=-1/10057 (16ms) --------------------------- test-ipv6-block-getpeername --------------------------- IPv4 socket=100/0 (31ms) IPv4 connect=0/0 (63ms) IPv4 getpeername=0/0 (0ms) IPv6 socket=112/0 (15ms) IPv6 connect=0/0 (47ms) IPv6 getpeername=-1/10057 (0ms) -> getpeername() does not work for IPv6 for either blocking or non-blocking sockets (even after the socket is connected).
> Under Windows XP SP2: Please first try and upgrade to something current. XP SP3 was released on 21st of April 21 2008, which is nearly three years ago. You are missing those patches and most very likely every single one that have been added since that. A *LOT* of IPv6 related things where resolved in those versions. Off topic for Firefox: Your test programs is faulty, you really need to learn the use of getaddrinfo() which actually can fill in fields properly and at least use memset() to clear out everything else. Google(eva ipv6) for a proper tutorial. Also, do not compare Wine to Windows, it indeed might have similar API but the responses will not be the same. Windows is Windows, Wine is Wine (and I would not be surprised if the Wine folks did not spend too much time on IPv6 either). Nevertheless, welcome to 2011, the year that SP3 exists for 3 years already and upgrades have been flowing in for longer already. SP2 is 4 years old already, upgrade it already, most very likely your problems will be gone then. Also, check if you have non-microsoft firewalls/anti-virus tools installed on your machine, they tend to do weird things to IPv6 connections as they tend to not understand them properly.
> Please first try and upgrade to something current. I want to fix this for XP SP2 too otherwise Firefox and IPv6 will look broken to end users. Additional investigation reveals that the IBM issue is relevant. If the SO_RCVBUF of an AF_INET6 socket is greater than 65535 before the connection is initiated, then getpeername() will always fail. I haven't been able to cause Windows 2000 or Windows XP 64-bit SP2 to exhibit the same problem by increasing the SO_RCVBUF (both default to a size below 65536), but Windows XP 32-bit SP3 can be made to break getpeername() by setting SO_RCVBUF to 128000 before connecting. The Windows XP 32-bit SP2 system I have this problem on defaults the SO_RCVBUF to 128000 bytes. This works ok for AF_INET but not AF_INET6. Lowering the SO_RCVBUF to 65535 before calling connect() fixes the issue and getpeername() then works successfully. Adjusting the SO_RCVBUF after the connection has been established does not fix the problem. Updated report-only test app code: http://s85.org/89NElbX3 Updated report-only test app exe: http://s85.org/eJtKIzbo Raise-rcvbuf test app code: http://s85.org/kDz8P9fr (to 128000) Raise-rcvbuf test app exe: http://s85.org/KxZqzvA9 Lower-rcvbuf test app code: http://s85.org/8ZNGmez3 (to 65535) Lower-rcvbuf test app exe: http://s85.org/cbrUGkcn On XP 32-bit SP3: --------------------------- test-ipv6-block-getpeername --------------------------- IPv4 socket=96/0 (0ms) IPv4 connect=0/0 (0ms) IPv4 rbuf=8192 sbuf=8192 IPv4 getpeername=0/0 (0ms) IPv6 socket=108/0 (0ms) IPv6 connect=0/0 (90ms) IPv6 rbuf=8192 sbuf=8192 IPv6 getpeername=0/0 (0ms) --------------------------- test-ipv6-block-getpeername-raise-rcvbuf --------------------------- IPv4 socket=96/0 (0ms) IPv4 connect=0/0 (0ms) IPv4 rbuf=8192 sbuf=8192 IPv4 getpeername=0/0 (0ms) IPv6 socket=108/0 (0ms) IPv6 connect=0/0 (91ms) IPv6 rbuf=128000 sbuf=8192 <-- increased receive buf before connect() IPv6 getpeername=-1/10057 (0ms) <-- getpeername() fails after connect() Windows 7 32-bit is OK (default buffer size is 8192) Windows 7 64-bit is OK (default buffer size is 8192) Jeroen, could you try the raise-rcvbuf code on XP 32-bit SP3 to test if it breaks IPv6 getpeername()?
> I want to fix this for XP SP2 too otherwise Firefox and IPv6 will look broken > to end users. Users who are enabling IPv6 on XP should be smart enough to upgrade to a current version of XP, instead of a 3+ year old version. > Jeroen, could you try the raise-rcvbuf code on XP 32-bit SP3 to test if it > breaks IPv6 getpeername()? Works like a charm. Please note that the XP SP3 stack that is currently available to people who actually upgrade is quite different from the stack that was in RTM, SP1 and SP2. The SP2 was indeed deemed as 'production' but they ironed out more kinks in SP3 and the newer updates. Instead of trying to fix this problem in every single application, just upgrade. And actually, upgrade in that case should mean that one should upgrade to Vista or even better Seven which is thus two extra iterations of the Windows IPv6 stack and actually a completely new rewite of it all. But, for working IPv6 one does not need to go that far, I haven't bothered with that either. It all works on XP SP3 though. For the ticket folks: Windows issue, let people upgrade that instead of putting nonsense into Firefox and every single other application that tries to do IPv6.
>> Jeroen, could you try the raise-rcvbuf code on XP 32-bit SP3 to test if it >> breaks IPv6 getpeername()? > > Works like a charm. The great thing about web server logs is that I know that no one has downloaded any of the new code files or executables, so I know you're lying. If you read my previous comment you would see that the test on 32-bit XP SP3 reproduces the bug. > For the ticket folks: Windows issue The Mozilla codebase has workarounds for Windows NT bugs going back to 3.51, so don't assume that fixed Windows bugs should never be worked around where possible. However, this bug doesn't appear to be fixed yet in XP because I can't find a KB article for it.
> If you read my previous comment you would see that the test on 32-bit XP > SP3 reproduces the bug. Confirmed. Using Windows XP Professional here, Version 5.1 Build 2600.xpsp_sp3_gdr.101209-1647 : Service Pack 3 (if anyone cares), the bug persists. On the same machine Chrome, IE, Safari work fine.
> Confirmed. Using Windows XP Professional here, Version 5.1 Build > 2600.xpsp_sp3_gdr.101209-1647 : Service Pack 3 (if anyone cares), the bug > persists. On the same machine Chrome, IE, Safari work fine. Could you run test-ipv6-block-getpeername and test-ipv6-block-getpeername-lower-rcvbuf from http://proxima.lp0.eu/~simon/tmp/test-ipv6/ and copy+paste the output? (It can be compiled using mingw32 on cygwin if you don't want to run my binaries) It'll check that the getpeername() issue is occurring and also that it can be fixed by lowering the SO_RCVBUF.
Altough Mozilla doesn't change the SO_RCVBUF value, it changes SO_SNDBUF. http://mxr.mozilla.org/mozilla-central/source/netwerk/base/src/nsSocketTransport2.cpp#1134 Simon Arlott, what happens if you change the value of "network.tcp.sendbuffer" to 65535 (or lower) via about:config?
@Jeroen Massar: I have several PCs with XPsp3 and some have this bug, some not. So there is defintiely a usecase in fixing this also on SP3. @Masatoshi Kimura: I reduced the sendbuffer but it still does not work. @Simon: WinXPde with SP3 --------------------------- test-ipv6-block-getpeername --------------------------- Windows 5.1.2600 (3.0) IPv4 socket=1956/0 (0ms) IPv4 connect=0/0 (62ms) IPv4 rbuf=128000 sbuf=49152 IPv4 getpeername=0/0 (0ms) IPv6 socket=1944/0 (0ms) IPv6 connect=0/0 (63ms) IPv6 rbuf=128000 sbuf=49152 IPv6 getpeername=-1/10057 (0ms) --------------------------- OK --------------------------- --------------------------- test-ipv6-block-getpeername-lower-rcvbuf --------------------------- Windows 5.1.2600 (3.0) IPv4 socket=1956/0 (0ms) IPv4 connect=0/0 (63ms) IPv4 rbuf=65535 sbuf=49152 IPv4 getpeername=0/0 (0ms) IPv6 socket=1944/0 (0ms) IPv6 connect=0/0 (62ms) IPv6 rbuf=65535 sbuf=49152 IPv6 getpeername=0/0 (0ms) --------------------------- OK ---------------------------
> What happens if you change the value of "network.tcp.sendbuffer" to 65535 (or lower) via about:config? It doesn't fix the problem when the default receive buffer is >65535. Otherwise it has no effect. I can't cause the issue by raising the send buffer if the receive buffer is low enough. I've installed IPv6 on a 32-bit SP3 host that is updated from Windows Update and test it there: --------------------------- test-ipv6-block-getpeername --------------------------- Windows 5.1.2600 (3.0) IPv4 socket=96/0 (16ms) IPv4 connect=0/0 (0ms) IPv4 rbuf=8192 sbuf=8192 IPv4 getpeername=0/0 (0ms) IPv6 socket=108/0 (15ms) IPv6 connect=0/0 (0ms) IPv6 rbuf=8192 sbuf=8192 IPv6 getpeername=0/0 (0ms) <-- normal buffer sizes = ok --------------------------- test-ipv6-block-getpeername-raise-rcvbuf --------------------------- Windows 5.1.2600 (3.0) IPv4 socket=96/0 (0ms) IPv4 connect=0/0 (0ms) IPv4 rbuf=128000 sbuf=8192 IPv4 getpeername=0/0 (0ms) IPv6 socket=108/0 (16ms) IPv6 connect=0/0 (0ms) IPv6 rbuf=128000 sbuf=8192 IPv6 getpeername=-1/10057 (0ms) <-- high receive buffer = broken --------------------------- test-ipv6-block-getpeername-raise-sndbuf --------------------------- Windows 5.1.2600 (3.0) IPv4 socket=96/0 (0ms) IPv4 connect=0/0 (0ms) IPv4 rbuf=8192 sbuf=128000 IPv4 getpeername=0/0 (0ms) IPv6 socket=108/0 (15ms) IPv6 connect=0/0 (0ms) IPv6 rbuf=8192 sbuf=128000 IPv6 getpeername=0/0 (0ms) <-- high send buffer = ok I've come up with a patch but I'm unsure that I'm modifying the right socket() call as it doesn't appear to have any effect on the recompiled .dll files.
Attached patch Additional patch to w95sock.c (obsolete) — — Splinter Review
Compiles but not yet tested.
I've now tested the patch to w95sock.c and it allows HTTPS over IPv6 to work.
The following non-default registry settings exist, possibly created by the third party firewall or other unknown software. The receive buffer is set to 128000 bytes: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters] "SmallBufferSize"=dword:00000500 "InitialLargeBufferCount"=dword:00000064 "IgnorePushBitOnReceives"=dword:00000000 "DefaultSendWindow"=dword:00007fff "FastSendDatagramThreshold"=dword:00001000 "MaxFastTransmit"=dword:0000fac0 "TransmitWorker"=dword:00000020 "InitialMediumBufferCount"=dword:000000f0 "PriorityBoost"=dword:00000000 "MediumBufferSize"=dword:00003ac0 "DisableAddressSharing"=dword:00000001 "LimitSmallBufferCount"=dword:00000140 "DefaultReceiveWindow"=dword:0001f400 "LargeBufferSize"=dword:00014000 "InitialLargBufferCount"=dword:00000064
Attachment #516868 - Flags: review?(wtc)
Assignee: nobody → bugzilla.mozilla.simon
Confirming this bug given the number of people who claim to see this and the fact that a patch has been written by someone who confirmed. This seems like one of the most serious IPv6-related bugs we have on file. Would be great to see it resolved.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Since the proposed fix is XP-only, we need to verify that the problem doesn't occur on Vista and Windows 7 in similar circumstances (receive buffer larger than 64K).
Target Milestone: --- → Firefox 6
I've already checked Windows 2000, Windows XP (64-bit), and Windows 7 (32-bit and 64-bit) - none of them are affected. Windows Vista and Windows Server 2003 are the same OS version as Windows XP (64-bit). Windows Server 2003 has been observed to be unaffected too. Only Windows XP (32-bit) appears to be affected (including fully updated SP3).
Firefox 6 will be too late for http://isoc.org/wp/worldipv6day/ on the 8th of June. There's no way to know how many 32-bit XP users have default receive buffers that are too large...
It would be a shame if this bug would not be fixed before the ipv6day. The bug report is already 19 months old and luckily Simon spotted the cause.
Regarding the cause for the big default receive window: I know some UMTS/3G modems change AFD parameters to achieve better performance. I guess that's the cause for problems on my PC.
blocking2.0: --- → ?
tracking-firefox5: --- → ?
Summary: SSL over IPv6 does not work under Windows → SSL over IPv6 does not work under Windows XP 32-bit when receive buffer size is 64kb+
Whiteboard: [needed for IPv6 testday]
Target Milestone: Firefox 6 → Firefox 5
Comment on attachment 516868 [details] [diff] [review] Patch to automatically lower the SO_RCVBUF on AF_INET6 sockets to 65535 on Windows XP 32-bit so that getpeername() doesn't fail. Thank you for the patch. Does this bug affect Windows XP SP2 only? Or does it also affect Windows XP SP3?
According to comment 66 Windows XP SP3 is also affected.
Yes, I confirm that Windows XP SP3 is affected too. btw: you can test a browser easily by surfing to http://myip.pernau.at/ This website will have 4 frames: IPv4 | IPv6 ------------------- IPv4+SSL | IPv6+SSL If the bug occours, the right lower frame stays empty.
I don't think tracking this against FF5 specifically makes sense, since FF5 will be out after ipv6 day. Would be nice to get it on central soon though (and could even consider a nomination for prior releases once there's a reviewed and landed patch). WTC - any thoughts on a review ETA?
tracking-firefox5: + → -
Component: General → NSPR
Product: Firefox → NSPR
QA Contact: general → nspr
Target Milestone: Firefox 5 → ---
Version: Trunk → other
turns out wtc is not cc'd, so he didn't see the last comments :) fixing.
Simon: thanks a lot for the patch. I omitted the ntio.c changes (since I can't test them), edited the w95sock.c changes slightly for style, and checked it in on the NSPR trunk. Checking in w95sock.c; /cvsroot/mozilla/nsprpub/pr/src/md/windows/w95sock.c,v <-- w95sock.c new revision: 3.18; previous revision: 3.17 done
Attachment #516868 - Attachment is obsolete: true
Attachment #516868 - Flags: review?(wtc)
Priority: -- → P2
Target Milestone: --- → 4.8.8
Does a window size of 64k-1 lead to silly window syndrome? http://en.wikipedia.org/wiki/Silly_window_syndrome Would a window size of 64k-MAXMTU yield better results?
Nelson: I was also worried about a buffer size that's not a power of 2. I considered setting it to 32k, but my coworker Mike Belshe found that 64k gets the best performance on Windows XP: http://codereview.chromium.org/40168
Status: NEW → ASSIGNED
Target Milestone: 4.8.8 → 4.8.9
Whiteboard: [needed for IPv6 testday] → [needed for IPv6 testday][tbwants]
People who have ran into this problem, please try this patched build of Firefox and report back your results: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bsmith@mozilla.com-ab2a6b1548d7/try-win32/ (FWIW, I tried to reproduce the problem today, but I've been so far been unsuccessful.)
Brian, which build exactly? There are lots of them. If you want to reproduce the bug use the website http://myip.pernau.at/ for testing. I can easily turn the bug on/off by changing the DefaultReceiveWindow value in [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters]. e.g. "DefaultReceiveWindow"=dword:0001f400 will trigger the bug. Note: you have to reboot the PC to activate the new setting
I tried it and SSL-over-IPv6 works :-)
Thanks Ramon. I was also able to verify the fix using the instructions you gave. Wan-Teh, what do we need to do to get the patch r+d and checked in?
FWIW, the patch is r+d and checked in to NSPR. It just needs to get into mozilla-central. (But I don't know how that works these days)
We cut an NSPR tag and import that into m-c.
Comment on attachment 530218 [details] [diff] [review] Patch to automatically lower the SO_RCVBUF on AF_INET6 sockets to 65535 on Windows XP 32-bit so that getpeername() doesn't fail, v2, by Simon Arlott I pushed this patch to mozilla-central: http://hg.mozilla.org/mozilla-central/rev/6e1f4652252d
Great. Which will be the first offical release containing this bugfix?
Firefox 6 Aurora would be the most stable release with the fix on IPv6 test day.
tracking-firefox5: - → ---
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
@Brian: So there is no fix for the World IPv6 Day?
We will have a fix in "Aurora" builds (public betas) by that date, available from http://www.mozilla.com/firefox/channel/
Minusing for Firefox 5, but willing to talk more if I'm missing something.
tracking-firefox5: ? → -
It is just that it happens more and more that I have to switch to another browser as I experience problems with Firefox and IPv6+SSL. Of course I could also upgrade to Windows 7 but changing the browser is less work ;-) Bascically I do not understand why you do not fix it for Firefox 4+5.
So I agree that this would be nice to fix for Firefox 5. But Firefox 4 is not getting any updates anymore, the upgrade path is to Firefox 5.
Just as a reference for all the people who find this bugreport and want to fix the problem immediately without waiting for a new Firefox version: Note: This fix is only needed on Windows XP 1. Open Windows Registry and lower to default receive window below 65536 bytes, e.g: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AFD\Parameters]. "DefaultReceiveWindow"=dword:0000ffff 2. Reboot your PC
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: