787757 - Multiple startup hangs of >10 minutes

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Jared Wein [:jaws] (please needinfo? me)]

I brought my Win7 laptop home and connected it to the wifi network and have been having nonstop hangs, sometimes reaching >10 minutes, since connecting it to the network.

mozregression shows this pushlog:
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3f408698a03f&tochange=da8c6039c25e

My guess is that this regression is from bug 563169.

I will do a further bisect to confirm.

Comment 1

[Jared Wein [:jaws] (please needinfo? me)]

Bug 774242 seems similar to this bug, but I'm seeing this issue on Nightly 18 so I don't think it is the same.

Comment 2

[Jared Wein [:jaws] (please needinfo? me)]

Confirmed that this bug exists in Firefox Nightly 9/1/2012.

I used Firefox 15 Release to load the profile and switched the proxy setting to "No proxy" from "Use system proxy" and the hangs went away.

In Internet Explorer's Internet Options, the LAN Settings is set to "Automatically detect settings" and the proxy server checkbox is unchecked.

Comment 3

[Jared Wein [:jaws] (please needinfo? me)]

Switching back to "Use system proxy" reproduces the issue.

I did use Fiddler two days ago and I see that https://bugzilla.mozilla.org/show_bug.cgi?id=563169#c3 mentions Fiddler's use of the system proxy. Fiddler isn't currently running.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Hoping Jim can take a look and get this to the right person, we'll track this for Beta and branches.

Comment 5

[Jim Mathies [:jimm]]

Any chance you could get a stack of the hang?

Comment 6

[Jared Wein [:jaws] (please needinfo? me)]

Here's a stack of the main thread, http://pastebin.mozilla.org/1805836

Comment 7

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

The default is now to use the system proxy, so this should probably be pretty high priority -- I've already had one family member support request about the 16 beta being unusable due to this.

Comment 8

[Jim Mathies [:jimm]]

Thanks Jared, can you post your build id info?

Masatoshi, this appears to be additional fallout related to bug 563169 which might already be fixed, not sure.

Comment 9

[Jared Wein [:jaws] (please needinfo? me)]

This is from about:buildconfig, not sure if you were looking for something different.

http://pastebin.mozilla.org/1806037

Comment 10

[Masatoshi Kimura [:emk]]

Using INTERNET_PER_CONN_FLAGS_UI may solve the issue.
http://stackoverflow.com/questions/2871510/unable-to-query-proxy-automatically-detect-settings-on-windows-7
If it doesn't help, I propose reverting bug 563169 and bug 771115.

Comment 11

[(dormant account)]

I believe this is a duplicate of bug 769764

Comment 12

[Patrick McManus [:mcmanus]]

(In reply to Taras Glek (:taras) from comment #11)
> I believe this is a duplicate of bug 769764

not at 10 minutes its not :( ... moving that to another thread isn't going to help.

Comment 13

[Patrick McManus [:mcmanus]]

but to the extent that some delay is inherent, yes 769764 is the answer.

Comment 14

[Alex Keybl [:akeybl]]

(In reply to Masatoshi Kimura [:emk] from comment #10)
> Using INTERNET_PER_CONN_FLAGS_UI may solve the issue.
> http://stackoverflow.com/questions/2871510/unable-to-query-proxy-
> automatically-detect-settings-on-windows-7
> If it doesn't help, I propose reverting bug 563169 and bug 771115.

Can we spin up a try build for jaws to try out?

Comment 15

[Masatoshi Kimura [:emk]]

https://tbpl.mozilla.org/?tree=Try&rev=10c9d086524a
The build will come in <https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-10c9d086524a/>.

Comment 16

[Jared Wein [:jaws] (please needinfo? me)]

I am no longer at my parent's home and can't reproduce this on Nightly 9-06 with Use System Settings.

I think it is a combination of the Use System Settings and the network that is at my parent's home. Unfortunately I won't be back at their home again within the next couple months.

Comment 17

[Masatoshi Kimura [:emk]]

Attachment: Backout patch

Due to inability of testing, I propose backout.

Comment 18

[Masatoshi Kimura [:emk]]

Attachment: Backout patch (for aurora)

Comment 19

[Masatoshi Kimura [:emk]]

Attachment: Backout patch (for beta)

Comment 20

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

I thought I commented in the other bug, or maybe via email -- my dad tested the builds, and they fixed the problem for him.

Comment 21

[Masatoshi Kimura [:emk]]

Attachment: Use INTERNET_PER_CONN_FLAGS_UI if available

Hm, then let's try this.

Comment 22

[Jim Mathies [:jimm]]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #20)
> I thought I commented in the other bug, or maybe via email -- my dad tested
> the builds, and they fixed the problem for him.

I'm confused, which build did your dad test Vlad? I don't see try builds posted here with the INTERNET_PER_CONN_FLAGS_UI option, so I'm not sure why we expect that to fix the problem. Plus Jared can't test per comment 16.

Maybe a backout, at least on beta, is the right next step until we can figure out how to reproduce the hanging problem and use that to test potential fixes.

Comment 23

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

He tested the builds in https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-10c9d086524a/try-win32/ referenced in comment #15.

Comment 24

[Jim Mathies [:jimm]]

Comment on attachment 660104 [details] [diff] [review]
Use INTERNET_PER_CONN_FLAGS_UI if available

Ok!

Comment 25

[Patrick McManus [:mcmanus]]

it sounds like (from http://msdn.microsoft.com/en-us/library/windows/desktop/aa385145%28v=vs.85%29.aspx) that INTERNET_PER_CONN_FLAGS_UI is only avail on >= windows 7.

Shouldn't we fallback to the fast (but somewhat incomplete) old registry code if *_UI isn't available rather than the version that is causing this problem?

I'm all for trying to make this work when UI is available.

Comment 26

[Masatoshi Kimura [:emk]]

If the system doesn't support SmartWPAD, the hung should not occur. Are there any reports from <=Vista users?
@vlad: What version of Windows your father is using?

Comment 27

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Windows 7

Comment 28

[Patrick McManus [:mcmanus]]

josh and taras had thousands of chrome hangs.. I don't know if they have a OS breakdown

Comment 29

[Vladan Djeric (:vladan)]

(In reply to Patrick McManus [:mcmanus] from comment #28)
> josh and taras had thousands of chrome hangs.. I don't know if they have a
> OS breakdown

98% of hang reports involving ReadInternetOption or IsWpadEnabledForConnectedNetworks are from Windows 7, but we do have 472 reports of hangs with the following stacks from Windows Vista machines:

IsWpadEnabledForConnectedNetworks(int *) (in WININET.dll)
 -> NETWORK_MANAGER::GetWpadDecisionForRegKeyId(HKEY__ *,NETWORK_MANAGER::WPAD_NETWORK_INFO *) (in WININET.dll)
 -> ICSTRING::Strnicmp(char const *,unsigned long) (in WININET.dll)
 -> QueryPerConnOptions(void *,INTERNET_PER_CONN_OPTION_LISTA *) (in WININET.dll)
 -> ReadInternetOption (in xul.dll)
 -> nsWindowsSystemProxySettings::GetProxyForURI(nsIURI *,nsACString_internal &) (in xul.dll)
 -> nsProtocolProxyService::Resolve_Internal(nsIURI *,nsProtocolInfo const &,unsigned int,bool *,nsIProxyInfo * *) (in xul.dll)
 -> nsProtocolProxyService::Resolve(nsIURI *,unsigned int,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::LookupProxyInfo(nsIURI *,nsIURI *,unsigned int,nsCString *,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::NewChannelFromURIWithProxyFlags(nsIURI *,nsIURI *,unsigned int,nsIChannel * *) (in xul.dll)

and

ReadInternetOption (in xul.dll)
 -> nsWindowsSystemProxySettings::GetPACURI(nsACString_internal &) (in xul.dll)
 -> nsProtocolProxyService::Resolve_Internal(nsIURI *,nsProtocolInfo const &,unsigned int,bool *,nsIProxyInfo * *) (in xul.dll)
 -> nsProtocolProxyService::Resolve(nsIURI *,unsigned int,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::LookupProxyInfo(nsIURI *,nsIURI *,unsigned int,nsCString *,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::NewChannelFromURIWithProxyFlags(nsIURI *,nsIURI *,unsigned int,nsIChannel * *) (in xul.dll)
 -> nsIOService::NewChannelFromURI(nsIURI *,nsIChannel * *) (in xul.dll)
 -> NS_NewChannel(nsIChannel * *,nsIURI *,nsIIOService *,nsILoadGroup *,nsIInterfaceRequestor *,unsigned int,nsIChannelPolicy *) (in xul.dll)
 -> nsXMLHttpRequest::Open(nsACString_internal const &,nsACString_internal const &,bool,mozilla::dom::Optional<nsAString_internal> const &,mozilla::dom::Optional<nsAString_internal> const &) (in xul.dll)
 -> nsXMLHttpRequest::Open(nsAString_internal const &,nsAString_internal const &,bool,mozilla::dom::Optional<nsAString_internal> const &,mozilla::dom::Optional<nsAString_internal> const &,mozilla::ErrorResult &) (in xul.dll)


There are also ~200 hangs on Windows XP containing frames like these:

InternetGetConnectedStateExW (in WININET.dll)
 -> ReadInternetOption (in xul.dll)
 -> nsWindowsSystemProxySettings::GetPACURI(nsACString_internal &) (in xul.dll)
 -> nsProtocolProxyService::Resolve_Internal(nsIURI *,nsProtocolInfo const &,unsigned int,bool *,nsIProxyInfo * *) (in xul.dll)
 -> nsProtocolProxyService::Resolve(nsIURI *,unsigned int,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::LookupProxyInfo(nsIURI *,nsIURI *,unsigned int,nsCString *,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::NewChannelFromURIWithProxyFlags(nsIURI *,nsIURI *,unsigned int,nsIChannel * *) (in xul.dll)
 -> nsIOService::NewChannelFromURI(nsIURI *,nsIChannel * *) (in xul.dll)
 -> NS_NewChannel(nsIChannel * *,nsIURI *,nsIIOService *,nsILoadGroup *,nsIInterfaceRequestor *,unsigned int,nsIChannelPolicy *) (in xul.dll)
 -> nsXMLHttpRequest::Open(nsACString_internal const &,nsACString_internal const &,bool,mozilla::dom::Optional<nsAString_internal> const &,mozilla::dom::Optional<nsAString_internal> const &) (in xul.dll)

Comment 30

[Masatoshi Kimura [:emk]]

I'll attach a followup patch replacing fallback back to registry reading.

Comment 31

[Masatoshi Kimura [:emk]]

Comment on attachment 660101 [details] [diff] [review]
Backout patch (for beta)

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 563169
User impact if declined: Unusable (>10 minutes) hang
Testing completed (on m-c, etc.): backout
Risk to taking this patch (and alternatives if risky): backout
String or UUID changes made by this patch: no

Simple backout would be safer for branches.

Comment 32

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 660104 [details] [diff] [review]
Use INTERNET_PER_CONN_FLAGS_UI if available

Green on Try (I ran tests too just to be safe).
https://tbpl.mozilla.org/?tree=Try&rev=03f048437c71

https://hg.mozilla.org/integration/mozilla-inbound/rev/a09d350021f0

Comment 33

[Masatoshi Kimura [:emk]]

Branch patches also need to land.

Comment 34

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/a09d350021f0

Comment 35

[Jim Mathies [:jimm]]

http://hg.mozilla.org/releases/mozilla-aurora/rev/9e0406f7a307

Comment 36

[Jim Mathies [:jimm]]

http://hg.mozilla.org/releases/mozilla-beta/rev/6f6d971cd873

Comment 37

[u279076]

Jared, are you able to reproduce these hangs anymore on your laptop? This should be fixed with today's Aurora and Nightly builds. I'm not sure if it made the uplift to 16b4 in time.

Comment 38

[Alex Keybl [:akeybl]]

This is in b4.

Comment 39

[u279076]

Flagging [qa-] since we don't have a reproducible testcase. Jared, can you please test this with Firefox 16 and 17 on your laptop to see if has been resolved?

Comment 40

[Jared Wein [:jaws] (please needinfo? me)]

I think Vlad is in a better position than myself to confirm this.

Comment 41

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

I haven't heard about any further issues in 16, though I'll double check.