1779005 - Broken since Firefox 102.0: no instant fallback to direct connection when proxy became unreachable while runtime

Status: VERIFIED FIXED

(Core :: Networking, defect, P1)

Description

[Joel Burtsche]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

1. Set proxy settings to "Automatic Proxy configuration URL"
2. Set proxy configuration URL to "http://intranethost/proxy.pac" (use reachable URL)
3. Save changes
4. Access any website
5. Switch to network where proxy.pac and there defined proxys are not reachable
6. Reload website or access any website

Actual results:

Firefox shows the error "Unable to find the proxy server."
If you reload the page the error repeats mostly but not strictly!
If you restart Firefox you can access any website with direct connection.

Expected results:

Until Version 101 when proxy is not reachable anymore, Firefox instantly uses direct connection.

Btw: The in bug 1207798 described settings network.proxy.failover_direct or network.proxy.fallback_to_direct have no impact on this, not in 102 nor in 101!

Comment 1

[Joel Burtsche]

I guess this behavior is an impact of changes related to bug 1770123.

Comment 3

[Thomas De Schampheleire]

I experience the same problem in ESR with same reproduction scenario.
The first ESR version with this problem is 91.11.0esr. Version 91.10.0esr was still OK.

Comment 4

[Thomas De Schampheleire]

Comparing 91.10.0esr with 91.11.0esr via [1], there seems to be one PAC related change:
https://hg.mozilla.org/releases/mozilla-esr91/rev/d517719645b2c4f29711564b6681cd2fa6c96cc6

This change was made for bug 1770123, which was already suspected by OP in comment 1.

[1] https://hg.mozilla.org/releases/mozilla-esr91/graph/7ac9b20c412b4641bd732e789a3be532fe3d4c43

Comment 5

[Kershaw Chang [:kershaw]]

Add this to our priority queue, since this is a recent regression.

Comment 6

[Kershaw Chang [:kershaw]]

I think this is what happened:

1. A request to load pac was created before network change.
2. We detected the network change, so proxy service tried to reload pac.
3. Another pac load was created here, and we dispatched a runnable to start loading later.
4. Before nsPACMan::StartLoading got called, nsPACMan::OnStreamComplete was called first, since the first pac request could fail because of network change. We set mLoader to null here, but mLoader was not the initial one anymore.
5. nsPACMan::StartLoading was called, but failed because of no loader.
6. The error code we used is NS_ERROR_ABORT, not NS_ERROR_NOT_AVAILABLE, so we didn't switch to use DIRECT connection.

Comment 7

[Kershaw Chang [:kershaw]]

Attachment: Bug 1779005 - Avoid setting |mLoader| to null if we already start a new PAC load, r=#necko

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4e3d5210feb2
Avoid setting |mLoader| to null if we already start a new PAC load, r=necko-reviewers,valentin

Comment 9

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/4e3d5210feb2

Comment 10

[Thomas De Schampheleire]

Thanks for working on this.

Will the fix also make its way into the ESR stream?

Comment 11

[Kershaw Chang [:kershaw]]

(In reply to Thomas De Schampheleire from comment #10)

Thanks for working on this.

Will the fix also make its way into the ESR stream?

Could you verify if the latest nightly works for you first?
If yes, I'll uplift this to ESR.
Thanks.

Comment 12

[Joel Burtsche]

(In reply to Kershaw Chang [:kershaw] from comment #11)

Could you verify if the latest nightly works for you first?

Since pushed 2 hours ago, it will be first available in the upcoming nightly.

Comment 13

[Thomas De Schampheleire]

Yes, I will verify it on the next nightly and report back. Thanks.

Comment 14

[Thomas De Schampheleire]

I tested with the latest nightly at this moment (https://ftp.mozilla.org/pub/firefox/nightly/2022/08/2022-08-12-09-37-14-mozilla-central/) but I'm sorry to say that I see no change in behavior. I confirmed based on the .txt file with the mercurial revision, that the proposed fix is part of the build. And to double-check my test procedure I did the test again with 91.10.0esr (where it works).

As I mentioned earlier, I'm basically using the same test procedure as OP, but in more detail:

• download firefox from Mozilla FTP
• extract to local directory
• while connected to VPN, open 'firefox-bin' from this directory, as follows:
  ./firefox-bin --safe-mode --ProfileManager --preferences
• In the profile manager, create a new profile and select it
• When firefox opens, in the Preferences window, immediately search for 'Updates' and disable auto-update.
• Also in the Preferences window, go to Network settings, and set the Proxy Autoconfiguration script. Save this window.
• Open one or more external websites, like a google query, wikipedia, ... This should work fine and pass via the proxy.
• Disable the VPN
• In Firefox, in a new tab (not sure if this matters), open more websites. In 91.10.0esr or older, this works immediately, the switchover to 'DIRECT' access is transparent. In 91.11.0esr or newer, this does not work. And after some attempts, the browser reports it cannot contact the proxy server.

Comment 15

[Kershaw Chang [:kershaw]]

Could you try to reproduce again and also capture a http log?
Please also add proxy:5 to MOZ_LOG environment variable. Thanks.

Comment 16

[Thomas De Schampheleire]

Attachment: log-filtered.tar.gz

Please find attached the requested log.

After startup, I first did a google query 'test' while VPN was still connected.
Then I disabled the VPN connection, and made a google query 'test after disabling VPN'. This showed the error, but surprisingly a reload did work.
Then I opened another tab, did a new google query, and it failed again. New reloads continued to show the error that proxy server cannot be found.

Comment 17

[Joel Burtsche]

(In reply to Thomas De Schampheleire from comment #16)

Then I disabled the VPN connection, and made a google query 'test after disabling VPN'. This showed the error, but surprisingly a reload did work.
Then I opened another tab, did a new google query, and it failed again. New reloads continued to show the error that proxy server cannot be found.

I confirm that is exactly my experience, including the spontaneous connect in the first seconds after disabling VPN.

Comment 18

[Kershaw Chang [:kershaw]]

Thanks for the log.
I'll reopen this bug and continually work on this.

Comment 19

[Kershaw Chang [:kershaw]]

Attachment: Bug 1779005 - Use mLoadFailureCount as an indicator for PAC laod, r=#necko

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a9cbfd43cf72
Use mLoadFailureCount as an indicator for PAC laod, r=necko-reviewers,valentin

Comment 21

[Joel Burtsche]

While waiting for the fix to be available in Nightly, I suggest those affected the following

Workaround:

1. Select the Windows Start button, then select Settings > Network & Internet > Proxy.
2. Turn on Use setup script.
3. In the Script address box, enter the script address, then select Save.
4. Select Firefox Settings > Network Settings > Settings...
5. Select Use system proxy settings, then select OK.

Comment 22

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/a9cbfd43cf72

Comment 23

[Joel Burtsche]

Using Nightly 105.0a1 (2022-08-21) (64-bit) 20220821185924 it now works for my setup. Switching back and forth from (V)PN with proxy to public internet without proxy now works without any felt latency and without any connection errors.
I cannot provide any further information on efficiency, hopefully it doesn't check proxy availability on every connection attempt.

Comment 24

[Thomas De Schampheleire]

I also confirm that with latest nightly (2022-08-22) this issue is resolved.

Thanks a lot for working on it!

Please also consider fixing in the ESR stream.

Comment 25

[Kershaw Chang [:kershaw]]

(In reply to Thomas De Schampheleire from comment #24)

I also confirm that with latest nightly (2022-08-22) this issue is resolved.

Thanks a lot for working on it!

Please also consider fixing in the ESR stream.

Thanks for confirming.
I'll request an uplift to ESR.

Comment 26

[Kershaw Chang [:kershaw]]

Comment on attachment 9290314 [details]
Bug 1779005 - Use mLoadFailureCount as an indicator for PAC laod, r=#necko

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: As described in the STR of this bug, Firefox still tries to connect to proxy after network change.
• User impact if declined: They need to restart Firefox or change their proxy settings.
• Fix Landed on Version: 105
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): There is an automatic test for this and this change is also verified by users.

Comment 27

[Joel Burtsche]

(In reply to Thomas De Schampheleire from comment #3)

The first ESR version with this problem is 91.11.0esr. Version 91.10.0esr was still OK.

Have we already missed the boat for ESR 91?

Comment 28

[Ryan VanderMeulen [:RyanVM]]

(In reply to Joel Burtsche from comment #27)

Have we already missed the boat for ESR 91?

Yes. The final planned release (91.13) shipped today before it EOLs next month.

Comment 29

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9289429 [details]
Bug 1779005 - Avoid setting |mLoader| to null if we already start a new PAC load, r=#necko

Approved for 102.3esr.

Comment 30

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/d4c10f0ca367
https://hg.mozilla.org/releases/mozilla-esr102/rev/fb7f2596831b