1869403 - incredibly slow page load with "use system proxy" enabled without a system proxy

Status: VERIFIED FIXED

(Core :: Networking: Proxy, defect, P1)

Description

[[:hoernchen]]

Attachment: comaprison.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0

Steps to reproduce:

I am not aware of any recently changed settings, but suddenly pages literally took 20+ seconds to load. I've tracked this down to the "use system proxy" setting, i'm running w11 2h23, and there is no system proxy configured.
Setting this to "no proxy" fixes the performance issues. This happens with new, pristine ff profiles (which default to "use system proxy"), and with the previous 120.0.0 release, too.

See attached pictures, only difference is "no proxy" vs "use system proxy", 1.37 minutes (!) compared to 2.5 seconds.
It looks like the connections are getting serialized instead of happening in parallel?

Expected results:

I expected that a fresh profile which defaults to "use system proxy" setting would work as expected...

Comment 1

[[:hoernchen]]

Attachment: w11noproxy.png

literally no proxy configured...

Comment 2

[BMO Automation]

Moving bug to Core/Networking: Proxy.

Comment 3

[Kershaw Chang [:kershaw]]

Hi Reporter,

Could you try to get a http log when system proxy is used?
Please try to create a log file and send the file to [e-mail].

Thanks.

Comment 4

[[:hoernchen]]

(mail sent)

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

NI myself to do a reproduction test

Comment 6

[Kershaw Chang [:kershaw]]

For some reason, the proxy resolution becomes really slow. As the log below shows, it took 4 seconds.

2024-02-26 12:02:07.372000 UTC - [Parent 28936: Main Thread]: D/nsHttp nsHttpChannel::ResolveProxy [this=1b7d063fc00]
2024-02-26 12:02:11.330000 UTC - [Parent 28936: Main Thread]: D/nsHttp nsHttpChannel::OnProxyAvailable [this=1b7d063fc00 pi=0 status=0 mStatus=0]

Comment 7

[erosman [:erosman]]

Out of curiosity, is there a web-server or any application that listens to HTTP requests on your system?
That might be seen as a proxy server, resulting in the request being forward to it, and only after it times out, the request is sent directly.

Comment 8

[[:hoernchen]]

Nope, nothing as far as I can tell, I checked netstat -anbo and disabled Steam, the AMD driver services, and everything else. Only dnscache and rpc services are left, and those can't be disabled. It is not realated to specific network adapter settings either, I gave a usb ethernet dongle a try, same issue. It's apparently not network related in general, my other windows devices using the same network do not have this problem. The issue still exists with the current FF version, and it survived a repair "in place" upgrade of windows.
It's a bit unfortunate that I can't "just build Firefox real quick" to debug this myself...

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Is this still happening?

If so, do you have some antivirus installed (other than MS Defender)?

Comment 10

[[:hoernchen]]

I just gave a new profile a try with FF 139.0.1 (64-bit), yes, it is still happening, no, I have never used anything else/more than the built in windows defender.

Comment 11

[Sunil Mayya]

Let me check this.

Comment 12

[Sunil Mayya]

Attachment: ProxyResolutionLogs

Comment 13

[Sunil Mayya]

Looks like the initial proxy resolution speed is fine. However, as we squeze lot of requests the resolution time increases upto 7 seconds!

2024-02-26 12:02:02.670000 UTC - [Parent 28936: Main Thread]: D/nsHttp nsHttpChannel::ResolveProxy [this=1b7e33a0400]
....
158 2024-02-26 12:02:09.720000 UTC - [Parent 28936: Main Thread]: D/nsHttp nsHttpChannel::OnProxyAvailable [this=1b7e33a0400 pi=0 status=0 mStatus=0]

I am not sure if this is due to the system proxy or anything that we do. I suspect the former, however we would need some profiler logs for this to confirm. I will try to reproduce this on the windows machine as well.

Dear reporter,
Could you please capture the profiler output related to the issue and non-issue case. It would be useful incase I cant reproduce this.

Comment 14

[[:hoernchen]]

Slow profile (sytem proxy): https://share.firefox.dev/45lfvdQ
Fast profile (no proxy): https://share.firefox.dev/4oGmiWZ

Turns out loading the profiler getting started page is another great example ;)

Comment 15

[Valentin Gosu [:valentin]]

This sounds a lot like bug 1937367.

Could you check if setting network.proxy.enable_wpad_over_dhcp to false in about:config fixes the issue?

Comment 16

[[:hoernchen]]

Nope, does not change anything: https://share.firefox.dev/3HqOPPL - arstechnica.com with enable_wpad_over_dhcp=false, a fairly impressive 40s delay until anything happens.
Another observation: the slow loading version does not terminate properly, FF keeps running instead of quitting for ~30s.

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

Can you try this with chrome?

Chrome only supports system proxys (no manual config). They may only check for proxies on startup or somehow hook into notifications if the system settings for proxies change.

Thanks

Comment 18

[[:hoernchen]]

Oh I already am using Chrome at times, absolutely no issues, neither right now nor during the past few years. Microsoft Chrome (Edge) is working fine, too.

Comment 19

[Randell Jesup [:jesup] (needinfo me)]

My thought is that this can be resolved or at least seriously mitigated by rate-limiting ResolveProxy requests. We don't need to check it 100 times a second or even every second, if we've previously gotten a failure to get proxy info.
We should rate-limit requests to every 5, 10, 30, 60? seconds
Valentin - how common would we fail to have an mProxyInfo? https://searchfox.org/firefox-main/source/netwerk/protocol/http/nsHttpChannel.cpp#7150-7158 implies it's very common

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

The comment goes back at least to the point where we made proxy resolution async, 13 years ago. So I have to assume we're doing this A Lot. Or the comment is (mostly) wrong

Comment 21

[Valentin Gosu [:valentin]]

So, in bug 1937367 we saw the issue was that after resume from suspend the call to DhcpRequestParams would hang.
As mentioned in comment 16, that doesn't seem to be the issue.
My guess is that the calls to ReadInternetOption nsWindowsSystemProxySettings::GetProxyForURI are slow, for "windows reasons". I assume that's why it's blocking shutdown too.

But Randell is probably right. As far as I can tell there's no reason to keep calling ReadInternetOption with the same parameters.
I think it should be possible to register a listener for WM_SETTINGCHANGE here when lparam equals "Software\Microsoft\Windows\CurrentVersion\Internet Settings" we should InternetSetOption(NULL, INTERNET_OPTION_REFRESH, NULL, 0); and refresh the proxy info.

Comment 22

[Kershaw Chang [:kershaw]]

Attachment: Bug 1869403 - Refactor nsWindowsSystemProxySettings to use WindowsInternetFunctionsWrapper, r=#necko

Comment 23

[Kershaw Chang [:kershaw]]

Attachment: Bug 1869403 - Optimize Windows proxy detection by observing network link and registry changes, r=#necko

Depends on D284114

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/a38cb07e3c4a
https://hg.mozilla.org/integration/autoland/rev/2dbc8d0bcd8e
Refactor nsWindowsSystemProxySettings to use WindowsInternetFunctionsWrapper, r=necko-reviewers,jesup
https://github.com/mozilla-firefox/firefox/commit/99fc6b53e423
https://hg.mozilla.org/integration/autoland/rev/09ca2ebc7075
Optimize Windows proxy detection by observing network link and registry changes, r=necko-reviewers,jesup

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/083fa44bd5b5
https://hg.mozilla.org/integration/autoland/rev/7eece62e944f
Revert "Bug 1869403 - Optimize Windows proxy detection by observing network link and registry changes, r=necko-reviewers,jesup" for causing leakcheck failures

Comment 26

[Sandor Molnar[:smolnar]]

Backed out for causing leakcheck failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/7eece62e944fbc08480cb3674ef1bff0cda88721

Push with failures

Failure log -> TEST-UNEXPECTED-FAIL | leakcheck | default 628 bytes leaked (CondVar, Mutex, NetworkLinkObserver, ThreadEventTarget, ThreadTargetSink, ...)

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/641dee910f88
https://hg.mozilla.org/integration/autoland/rev/bd58e2c19cf6
Refactor nsWindowsSystemProxySettings to use WindowsInternetFunctionsWrapper, r=necko-reviewers,jesup
https://github.com/mozilla-firefox/firefox/commit/fd129a8908de
https://hg.mozilla.org/integration/autoland/rev/31608319f0d3
Optimize Windows proxy detection by observing network link and registry changes, r=necko-reviewers,jesup

Comment 28

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/bd58e2c19cf6
https://hg.mozilla.org/mozilla-central/rev/31608319f0d3

Comment 29

[Kershaw Chang [:kershaw]]

Hi Reporter,

Since we recently fixed this issue, could you try downloading Firefox Nightly and see if you can still reproduce it?

Thanks.

Comment 30

[[:hoernchen]]

At least with 150.a1 (2026-02-28) this appears to be fixed, there is no obvious difference between the proxy settings, the network inspector tab timings all look pretty similar with no massive delays. Thanks!

Comment 31

[Adrian Florinescu [:aflorinescu]]

Testing completed on Win10 (22H2 ( OS Build 19045.6466) and Win11 (25H2 Build 26200.7840) with Firefox Nightly 150.0a1, test execution details here.

Comment 32

[Adrian Florinescu [:aflorinescu]]

Beta 150 Spot check completed on Windows 10 and Windows 11, results available here.