Last Comment Bug 787757 - Multiple startup hangs of >10 minutes
: Multiple startup hangs of >10 minutes
Status: RESOLVED FIXED
[qa-]
: hang, regression
Product: Core
Classification: Components
Component: Networking (show other bugs)
: 16 Branch
: x86_64 Windows 7
: -- critical (vote)
: mozilla18
Assigned To: Masatoshi Kimura [:emk]
:
Mentors:
: 792024 (view as bug list)
Depends on:
Blocks: 563169 781732
  Show dependency treegraph
 
Reported: 2012-09-01 21:19 PDT by Jared Wein [:jaws] (please needinfo? me)
Modified: 2016-02-18 11:11 PST (History)
16 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed
+
fixed
+
fixed


Attachments
Backout patch (7.27 KB, patch)
2012-09-11 08:50 PDT, Masatoshi Kimura [:emk]
no flags Details | Diff | Splinter Review
Backout patch (for aurora) (7.17 KB, patch)
2012-09-11 08:51 PDT, Masatoshi Kimura [:emk]
akeybl: approval‑mozilla‑aurora+
Details | Diff | Splinter Review
Backout patch (for beta) (7.17 KB, patch)
2012-09-11 08:51 PDT, Masatoshi Kimura [:emk]
akeybl: approval‑mozilla‑beta+
Details | Diff | Splinter Review
Use INTERNET_PER_CONN_FLAGS_UI if available (2.02 KB, patch)
2012-09-11 09:00 PDT, Masatoshi Kimura [:emk]
jmathies: review+
ryanvm: checkin+
Details | Diff | Splinter Review

Description Jared Wein [:jaws] (please needinfo? me) 2012-09-01 21:19:30 PDT
I brought my Win7 laptop home and connected it to the wifi network and have been having nonstop hangs, sometimes reaching >10 minutes, since connecting it to the network.

mozregression shows this pushlog:
> http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3f408698a03f&tochange=da8c6039c25e

My guess is that this regression is from bug 563169.

I will do a further bisect to confirm.
Comment 1 Jared Wein [:jaws] (please needinfo? me) 2012-09-01 21:41:43 PDT
Bug 774242 seems similar to this bug, but I'm seeing this issue on Nightly 18 so I don't think it is the same.
Comment 2 Jared Wein [:jaws] (please needinfo? me) 2012-09-01 22:38:22 PDT
Confirmed that this bug exists in Firefox Nightly 9/1/2012.

I used Firefox 15 Release to load the profile and switched the proxy setting to "No proxy" from "Use system proxy" and the hangs went away.

In Internet Explorer's Internet Options, the LAN Settings is set to "Automatically detect settings" and the proxy server checkbox is unchecked.
Comment 3 Jared Wein [:jaws] (please needinfo? me) 2012-09-01 22:51:16 PDT
Switching back to "Use system proxy" reproduces the issue.

I did use Fiddler two days ago and I see that https://bugzilla.mozilla.org/show_bug.cgi?id=563169#c3 mentions Fiddler's use of the system proxy. Fiddler isn't currently running.
Comment 4 Lukas Blakk [:lsblakk] use ?needinfo 2012-09-04 16:09:43 PDT
Hoping Jim can take a look and get this to the right person, we'll track this for Beta and branches.
Comment 5 Jim Mathies [:jimm] 2012-09-04 16:30:12 PDT
Any chance you could get a stack of the hang?
Comment 6 Jared Wein [:jaws] (please needinfo? me) 2012-09-05 10:59:49 PDT
Here's a stack of the main thread, http://pastebin.mozilla.org/1805836
Comment 7 Vladimir Vukicevic [:vlad] [:vladv] 2012-09-05 12:42:09 PDT
The default is now to use the system proxy, so this should probably be pretty high priority -- I've already had one family member support request about the 16 beta being unusable due to this.
Comment 8 Jim Mathies [:jimm] 2012-09-05 12:42:52 PDT
Thanks Jared, can you post your build id info?

Masatoshi, this appears to be additional fallout related to bug 563169 which might already be fixed, not sure.
Comment 9 Jared Wein [:jaws] (please needinfo? me) 2012-09-05 12:52:33 PDT
This is from about:buildconfig, not sure if you were looking for something different.

http://pastebin.mozilla.org/1806037
Comment 10 Masatoshi Kimura [:emk] 2012-09-05 13:11:23 PDT
Using INTERNET_PER_CONN_FLAGS_UI may solve the issue.
http://stackoverflow.com/questions/2871510/unable-to-query-proxy-automatically-detect-settings-on-windows-7
If it doesn't help, I propose reverting bug 563169 and bug 771115.
Comment 11 (dormant account) 2012-09-05 13:20:37 PDT
I believe this is a duplicate of bug 769764
Comment 12 Patrick McManus [:mcmanus] PTO until Sep 6 2012-09-05 13:24:51 PDT
(In reply to Taras Glek (:taras) from comment #11)
> I believe this is a duplicate of bug 769764

not at 10 minutes its not :( ... moving that to another thread isn't going to help.
Comment 13 Patrick McManus [:mcmanus] PTO until Sep 6 2012-09-05 13:25:24 PDT
but to the extent that some delay is inherent, yes 769764 is the answer.
Comment 14 Alex Keybl [:akeybl] 2012-09-05 13:28:07 PDT
(In reply to Masatoshi Kimura [:emk] from comment #10)
> Using INTERNET_PER_CONN_FLAGS_UI may solve the issue.
> http://stackoverflow.com/questions/2871510/unable-to-query-proxy-
> automatically-detect-settings-on-windows-7
> If it doesn't help, I propose reverting bug 563169 and bug 771115.

Can we spin up a try build for jaws to try out?
Comment 16 Jared Wein [:jaws] (please needinfo? me) 2012-09-06 11:39:59 PDT
I am no longer at my parent's home and can't reproduce this on Nightly 9-06 with Use System Settings.

I think it is a combination of the Use System Settings and the network that is at my parent's home. Unfortunately I won't be back at their home again within the next couple months.
Comment 17 Masatoshi Kimura [:emk] 2012-09-11 08:50:46 PDT
Created attachment 660099 [details] [diff] [review]
Backout patch

Due to inability of testing, I propose backout.
Comment 18 Masatoshi Kimura [:emk] 2012-09-11 08:51:21 PDT
Created attachment 660100 [details] [diff] [review]
Backout patch (for aurora)
Comment 19 Masatoshi Kimura [:emk] 2012-09-11 08:51:50 PDT
Created attachment 660101 [details] [diff] [review]
Backout patch (for beta)
Comment 20 Vladimir Vukicevic [:vlad] [:vladv] 2012-09-11 08:54:35 PDT
I thought I commented in the other bug, or maybe via email -- my dad tested the builds, and they fixed the problem for him.
Comment 21 Masatoshi Kimura [:emk] 2012-09-11 09:00:47 PDT
Created attachment 660104 [details] [diff] [review]
Use INTERNET_PER_CONN_FLAGS_UI if available

Hm, then let's try this.
Comment 22 Jim Mathies [:jimm] 2012-09-11 09:38:30 PDT
(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #20)
> I thought I commented in the other bug, or maybe via email -- my dad tested
> the builds, and they fixed the problem for him.

I'm confused, which build did your dad test Vlad? I don't see try builds posted here with the INTERNET_PER_CONN_FLAGS_UI option, so I'm not sure why we expect that to fix the problem. Plus Jared can't test per comment 16.

Maybe a backout, at least on beta, is the right next step until we can figure out how to reproduce the hanging problem and use that to test potential fixes.
Comment 23 Vladimir Vukicevic [:vlad] [:vladv] 2012-09-11 09:56:05 PDT
He tested the builds in https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/VYV03354@nifty.ne.jp-10c9d086524a/try-win32/ referenced in comment #15.
Comment 24 Jim Mathies [:jimm] 2012-09-11 10:07:45 PDT
Comment on attachment 660104 [details] [diff] [review]
Use INTERNET_PER_CONN_FLAGS_UI if available

Ok!
Comment 25 Patrick McManus [:mcmanus] PTO until Sep 6 2012-09-11 12:20:24 PDT
it sounds like (from http://msdn.microsoft.com/en-us/library/windows/desktop/aa385145%28v=vs.85%29.aspx) that INTERNET_PER_CONN_FLAGS_UI is only avail on >= windows 7.

Shouldn't we fallback to the fast (but somewhat incomplete) old registry code if *_UI isn't available rather than the version that is causing this problem?

I'm all for trying to make this work when UI is available.
Comment 26 Masatoshi Kimura [:emk] 2012-09-11 17:28:14 PDT
If the system doesn't support SmartWPAD, the hung should not occur. Are there any reports from <=Vista users?
@vlad: What version of Windows your father is using?
Comment 27 Vladimir Vukicevic [:vlad] [:vladv] 2012-09-11 17:37:08 PDT
Windows 7
Comment 28 Patrick McManus [:mcmanus] PTO until Sep 6 2012-09-11 17:39:03 PDT
josh and taras had thousands of chrome hangs.. I don't know if they have a OS breakdown
Comment 29 Vladan Djeric (:vladan) 2012-09-12 03:14:09 PDT
(In reply to Patrick McManus [:mcmanus] from comment #28)
> josh and taras had thousands of chrome hangs.. I don't know if they have a
> OS breakdown

98% of hang reports involving ReadInternetOption or IsWpadEnabledForConnectedNetworks are from Windows 7, but we do have 472 reports of hangs with the following stacks from Windows Vista machines:

IsWpadEnabledForConnectedNetworks(int *) (in WININET.dll)
 -> NETWORK_MANAGER::GetWpadDecisionForRegKeyId(HKEY__ *,NETWORK_MANAGER::WPAD_NETWORK_INFO *) (in WININET.dll)
 -> ICSTRING::Strnicmp(char const *,unsigned long) (in WININET.dll)
 -> QueryPerConnOptions(void *,INTERNET_PER_CONN_OPTION_LISTA *) (in WININET.dll)
 -> ReadInternetOption (in xul.dll)
 -> nsWindowsSystemProxySettings::GetProxyForURI(nsIURI *,nsACString_internal &) (in xul.dll)
 -> nsProtocolProxyService::Resolve_Internal(nsIURI *,nsProtocolInfo const &,unsigned int,bool *,nsIProxyInfo * *) (in xul.dll)
 -> nsProtocolProxyService::Resolve(nsIURI *,unsigned int,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::LookupProxyInfo(nsIURI *,nsIURI *,unsigned int,nsCString *,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::NewChannelFromURIWithProxyFlags(nsIURI *,nsIURI *,unsigned int,nsIChannel * *) (in xul.dll)

and

ReadInternetOption (in xul.dll)
 -> nsWindowsSystemProxySettings::GetPACURI(nsACString_internal &) (in xul.dll)
 -> nsProtocolProxyService::Resolve_Internal(nsIURI *,nsProtocolInfo const &,unsigned int,bool *,nsIProxyInfo * *) (in xul.dll)
 -> nsProtocolProxyService::Resolve(nsIURI *,unsigned int,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::LookupProxyInfo(nsIURI *,nsIURI *,unsigned int,nsCString *,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::NewChannelFromURIWithProxyFlags(nsIURI *,nsIURI *,unsigned int,nsIChannel * *) (in xul.dll)
 -> nsIOService::NewChannelFromURI(nsIURI *,nsIChannel * *) (in xul.dll)
 -> NS_NewChannel(nsIChannel * *,nsIURI *,nsIIOService *,nsILoadGroup *,nsIInterfaceRequestor *,unsigned int,nsIChannelPolicy *) (in xul.dll)
 -> nsXMLHttpRequest::Open(nsACString_internal const &,nsACString_internal const &,bool,mozilla::dom::Optional<nsAString_internal> const &,mozilla::dom::Optional<nsAString_internal> const &) (in xul.dll)
 -> nsXMLHttpRequest::Open(nsAString_internal const &,nsAString_internal const &,bool,mozilla::dom::Optional<nsAString_internal> const &,mozilla::dom::Optional<nsAString_internal> const &,mozilla::ErrorResult &) (in xul.dll)


There are also ~200 hangs on Windows XP containing frames like these:

InternetGetConnectedStateExW (in WININET.dll)
 -> ReadInternetOption (in xul.dll)
 -> nsWindowsSystemProxySettings::GetPACURI(nsACString_internal &) (in xul.dll)
 -> nsProtocolProxyService::Resolve_Internal(nsIURI *,nsProtocolInfo const &,unsigned int,bool *,nsIProxyInfo * *) (in xul.dll)
 -> nsProtocolProxyService::Resolve(nsIURI *,unsigned int,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::LookupProxyInfo(nsIURI *,nsIURI *,unsigned int,nsCString *,nsIProxyInfo * *) (in xul.dll)
 -> nsIOService::NewChannelFromURIWithProxyFlags(nsIURI *,nsIURI *,unsigned int,nsIChannel * *) (in xul.dll)
 -> nsIOService::NewChannelFromURI(nsIURI *,nsIChannel * *) (in xul.dll)
 -> NS_NewChannel(nsIChannel * *,nsIURI *,nsIIOService *,nsILoadGroup *,nsIInterfaceRequestor *,unsigned int,nsIChannelPolicy *) (in xul.dll)
 -> nsXMLHttpRequest::Open(nsACString_internal const &,nsACString_internal const &,bool,mozilla::dom::Optional<nsAString_internal> const &,mozilla::dom::Optional<nsAString_internal> const &) (in xul.dll)
Comment 30 Masatoshi Kimura [:emk] 2012-09-12 15:41:43 PDT
I'll attach a followup patch replacing fallback back to registry reading.
Comment 31 Masatoshi Kimura [:emk] 2012-09-12 15:44:47 PDT
Comment on attachment 660101 [details] [diff] [review]
Backout patch (for beta)

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 563169
User impact if declined: Unusable (>10 minutes) hang
Testing completed (on m-c, etc.): backout
Risk to taking this patch (and alternatives if risky): backout
String or UUID changes made by this patch: no

Simple backout would be safer for branches.
Comment 32 Ryan VanderMeulen [:RyanVM] 2012-09-12 19:25:57 PDT
Comment on attachment 660104 [details] [diff] [review]
Use INTERNET_PER_CONN_FLAGS_UI if available

Green on Try (I ran tests too just to be safe).
https://tbpl.mozilla.org/?tree=Try&rev=03f048437c71

https://hg.mozilla.org/integration/mozilla-inbound/rev/a09d350021f0
Comment 33 Masatoshi Kimura [:emk] 2012-09-13 02:59:42 PDT
Branch patches also need to land.
Comment 34 Ed Morley [:emorley] 2012-09-13 13:08:05 PDT
https://hg.mozilla.org/mozilla-central/rev/a09d350021f0
Comment 37 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-09-19 10:30:59 PDT
Jared, are you able to reproduce these hangs anymore on your laptop? This should be fixed with today's Aurora and Nightly builds. I'm not sure if it made the uplift to 16b4 in time.
Comment 38 Alex Keybl [:akeybl] 2012-09-19 11:28:17 PDT
This is in b4.
Comment 39 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2012-10-16 15:50:01 PDT
Flagging [qa-] since we don't have a reproducible testcase. Jared, can you please test this with Firefox 16 and 17 on your laptop to see if has been resolved?
Comment 40 Jared Wein [:jaws] (please needinfo? me) 2012-10-16 16:05:01 PDT
I think Vlad is in a better position than myself to confirm this.
Comment 41 Vladimir Vukicevic [:vlad] [:vladv] 2012-10-16 17:15:30 PDT
I haven't heard about any further issues in 16, though I'll double check.
Comment 42 Jared Wein [:jaws] (please needinfo? me) 2016-02-18 11:11:03 PST
*** Bug 792024 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.