1722202 - Firefox won't use system proxy or autodetect proxy macos

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[office]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36

Steps to reproduce:

macOS 11.4 and firefox 90.0.2

I have a squid proxy setup, and I have WPAD configured using both DHCP and DNS.

I try setting the proxy settings in firefox to "Autodetect proxy settings for this network" or I also try "Use system proxy settings" and neither are working I have no internet.

If I manually enter the address and port of the squid proxy, it works, so there is an issue with how firefox autodetects proxy settings, and also it is not using the proxy set by macos.

It should be noted that other browsers and applications on the device are detecting and using the proxy just fine:

Brave Browser
Discord
Chrome Browser
Safari Browser

All the above are automatically detecting the squid proxy and work without issue, just firefox has issue.

Actual results:

The internet does not work because firefox does not autodetect the squid proxy, nor does it use the system proxy that macos is using.

Expected results:

firefox should auto detect the proxy through WPAD, or else it should use the macOS system proxy when macOS is setting it, and firefox is instructed to do so.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Kershaw Chang [:kershaw]]

Could you try to visit a simple website (e.g., example.com) with proxy set to Autodetect proxy settings for this network and also capture a http log? Please also add proxy:5 to MOZ_LOG env. Thanks.

Comment 3

[office]

Hi,

I'm really struggling to find out how I can add the proxy:5 to MOZ_LOG on macos. I've been googling and googling and coming up with nothing!

Comment 4

[office]

Attachment: logs.7z

Logs attached as requested

Comment 5

[office]

Ah I found how! logs are attached thx

Comment 6

[office]

Firefox isn't trying to resolve WPAD from DNS over HTTPS or something silly like that is it?

Comment 7

[office]

Did the logs help?

Comment 8

[Valentin Gosu [:valentin] (he/him)]

DHCP proxy autoconfig is not implemented for OSX. See bug 1476899.
I'm assuming the WPAD is served from http://wpad/wpad.dat, yes?
Could you check if the proxy works when you set network.http.http3.enabled to false in about:config ?

Comment 9

[office]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #8)

DHCP proxy autoconfig is not implemented for OSX. See bug 1476899.
I'm assuming the WPAD is served from http://wpad/wpad.dat, yes?
Could you check if the proxy works when you set network.http.http3.enabled to false in about:config ?

Hello,

Yes indeed http://wpad/wpad.dat, http://wpad/proxy.pac and http://wpad/wpad.da are all available.

I am also serving WPAD over DNS and so if macOS Firefox is not using DHCP options does it use DNS?

I will change that config setting and test for you now.

Comment 10

[office]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #8)

DHCP proxy autoconfig is not implemented for OSX. See bug 1476899.
I'm assuming the WPAD is served from http://wpad/wpad.dat, yes?
Could you check if the proxy works when you set network.http.http3.enabled to false in about:config ?

Hello, setting network.http.http3.enabled to false has made no difference, the issue remains.

Comment 11

[office]

If Auto-detect and use system proxy settings do not work, perhaps they should be removed from the GUI? And it be officially mentioned that firefox is unable to autoconfigure a proxy on macos, and that literally any other browser should be used instead of firefox for this outcome.

Comment 12

[Valentin Gosu [:valentin] (he/him)]

(In reply to office from comment #9)

I am also serving WPAD over DNS and so if macOS Firefox is not using DHCP options does it use DNS?

Loading http://wpad/wpad.dat is WPAD over DNS. That should work. It's not clear exactly why it doesn't work for you.
Could you submit the logs again, but this time environment variables? https://firefox-source-docs.mozilla.org/networking/http/logging.html#mac-os-x

export MOZ_LOG=timestamp,rotate:200,nsHttp:5,cache2:5,nsSocketTransport:5,nsHostResolver:5,cookie:5,proxy:5
export MOZ_LOG_FILE=~/Desktop/log.txt
cd /Applications/Firefox.app/Contents/MacOS
./firefox-bin

Also check the browser console (Cmd+Shift+J) and see if there are errors about the proxy there.
Thanks!

Comment 13

[office]

Hello,

From the logs, it kind of looks like firefox is trying to force WPAD to be over HTTPS?

2021-08-06 06:34:34.058114 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpChannel::OnProxyAvailable [this=11f4b9000 pi=0 status=0 mStatus=0]
2021-08-06 06:34:34.058122 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpChannel::BeginConnect [this=11f4b9000]
2021-08-06 06:34:34.058130 UTC - [Parent 41463: Main Thread]: V/nsHttp host=wpad port=-1
2021-08-06 06:34:34.058137 UTC - [Parent 41463: Main Thread]: V/nsHttp uri=https://wpad/wpad.dat
2021-08-06 06:34:34.058149 UTC - [Parent 41463: Main Thread]: V/nsHttp Init nsHttpConnectionInfo @12aff2e30
2021-08-06 06:34:34.058161 UTC - [Parent 41463: Main Thread]: D/nsHttp AltSvcCache::LookupMapping 11e29a9e0 https:wpad:443:.::3
2021-08-06 06:34:34.058170 UTC - [Parent 41463: Main Thread]: D/nsHttp AltSvcCache::LookupMapping 11e29a9e0 MISS
2021-08-06 06:34:34.058178 UTC - [Parent 41463: Main Thread]: D/nsHttp AltSvcCache::GetAltServiceMapping 11e29a9e0 key=https:wpad:443:.::3 existing=0 validated=0 ttl=0
2021-08-06 06:34:34.058185 UTC - [Parent 41463: Main Thread]: D/nsHttp AltSvcCache::LookupMapping 11e29a9e0 https:wpad:443:.::.
2021-08-06 06:34:34.058193 UTC - [Parent 41463: Main Thread]: D/nsHttp AltSvcCache::LookupMapping 11e29a9e0 MISS
2021-08-06 06:34:34.058201 UTC - [Parent 41463: Main Thread]: D/nsHttp AltSvcCache::GetAltServiceMapping 11e29a9e0 key=https:wpad:443:.::. existing=0 validated=0 ttl=0
2021-08-06 06:34:34.058208 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpChannel 11f4b9000 Using default connection info
2021-08-06 06:34:34.058219 UTC - [Parent 41463: Main Thread]: D/nsHttp nsHttpChannelAuthProvider::AddAuthorizationHeaders? [this=12c281b50 channel=11f4b9608]
2021-08-06 06:34:34.058228 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpAuthCache::GetAuthEntryForPath 11b818050 [path=/]
2021-08-06 06:34:34.058236 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpAuthCache::LookupAuthNode 11b818050 key=':https://wpad:-1' found node=0
2021-08-06 06:34:34.058246 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpChannel::MaybeStartDNSPrefetch [this=11f4b9000, strategy=1] prefetching, refresh requested
2021-08-06 06:34:34.058260 UTC - [Parent 41463: Main Thread]: D/nsHostResolver Resolving host [wpad]<> - bypassing cache type 0. [this=11f8b38d0]
2021-08-06 06:34:34.058271 UTC - [Parent 41463: Main Thread]: D/nsHostResolver No usable record in cache for host [wpad] type 0.
2021-08-06 06:34:34.058278 UTC - [Parent 41463: Main Thread]: D/nsHostResolver NameLookup host:wpad af:0
2021-08-06 06:34:34.058289 UTC - [Parent 41463: Main Thread]: D/nsHostResolver NameLookup: wpad effectiveTRRmode: 1 flags: 11
2021-08-06 06:34:34.058324 UTC - [Parent 41463: Main Thread]: D/nsHostResolver TRR service not enabled - off or disabled
2021-08-06 06:34:34.058337 UTC - [Parent 41463: Main Thread]: D/nsHostResolver NativeLookup host:wpad af:0
2021-08-06 06:34:34.058351 UTC - [Parent 41463: Main Thread]: D/nsHostResolver DNS thread counters: total=1 any-live=0 idle=1 pending=1
2021-08-06 06:34:34.058361 UTC - [Parent 41463: Main Thread]: D/nsHostResolver DNS lookup for host [wpad] blocking pending 'getaddrinfo' or trr query: callback [11b8c3340]
2021-08-06 06:34:34.058371 UTC - [Parent 41463: Main Thread]: V/nsHttp nsHttpChannel::PrepareToConnect [this=11f4b9000]
2021-08-06 06:34:34.058854 UTC - [Parent 41463: Main Thread]: W/cookie ===== COOKIE NOT SENT =====
2021-08-06 06:34:34.058872 UTC - [Parent 41463: Main Thread]: W/cookie request URL: https://wpad/wpad.dat
2021-08-06 06:34:34.058884 UTC - [Parent 41463: Main Thread]: W/cookie current time: Fri Aug 06 06:34:34 2021 GMT
2021-08-06 06:34:34.058892 UTC - [Parent 41463: Main Thread]: W/cookie rejected because cookies are disabled
2021-08-06 06:34:34.058899 UTC - [Parent 41463: Main Thread]: W/cookie
2021-08-06 06:34:34.058909 UTC - [Parent 41463: Main Thread]: V/nsHttp HttpBaseChannel::SetRequestHeader [this=11f4b9000 header="Cookie" value="" merge=0]
2021-08-06 06:34:34.058920 UTC - [Parent 41463: Main Thread]: D/nsHttp nsHttpHandler::NotifyObservers [chan=11f4b9040 event="http-on-modify-request"]
2021-08-06 06:34:34.059028 UTC - [Parent 41463: Main Thread]: V/nsHttp HttpBaseChannel::SetRequestHeader [this=11f4b9000 header="Sec-Fetch-Dest"

I can see that it is fetching WPAD over DNS however it is trying to access https://wpad/wpad.dat which is incorrect, I am serving WPAD over HTTP and port 80, I do not serve WPAD over HTTPS and 443 how can I get a valid certificate for WPAD that is crazyness!

I will attach the full log, but I think looking at that it is clear the problem is firefox trying to use https:443 to access WPAD domain.....

Comment 14

[office]

Attachment: log.txt.moz_log.0.cpgz

crazyness

Comment 15

[Valentin Gosu [:valentin] (he/him)]

Ah, it's possible this could be related to the HTTPS-only mode.
I think I know how to fix it. I'll have a custom Firefox build for you momentarily.
Thanks!

Comment 16

[office]

Hello yes,

I thought exact same and so i disabled HTTPS-only-mode and I can see that it is working now.

So HTTPS-only-mode is definitely trying to force DNS WPAD over HTTPS which is a bug and it should not be doing that!!!!

Comment 17

[Valentin Gosu [:valentin] (he/him)]

Thank you for confirming and for reporting this bug. I'm not sure why no one else has reported it so far. I assume it's because WPAD is not very commonly used due to it being insecure. In any case, this is a regression and needs to be fixed.
Cheers!

Comment 18

[office]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #17)

Thank you for confirming and for reporting this bug. I'm not sure why no one else has reported it so far. I assume it's because WPAD is not very commonly used due to it being insecure. In any case, this is a regression and needs to be fixed.
Cheers!

WPAD is a bit insecure but I know that IE/Edge is chopping off any query strings and only passing the TLD to the script to mitigate the security issues. Not sure what/if any other browser is doing to mitigate.

Comment 19

[office]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #17)

Thank you for confirming and for reporting this bug. I'm not sure why no one else has reported it so far. I assume it's because WPAD is not very commonly used due to it being insecure. In any case, this is a regression and needs to be fixed.
Cheers!

Possibly also HTTPS-only-mode I think is not default? I think I had to set that from memory? But yea also I guess WPAD is mostly for places where you have roaming devices, can't really hard set the proxy or they go home and their internet doesn't work x_x

Comment 20

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1722202 - Exempt WPAD requests from HTTPS-only mode upgrades r=ckerschb

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7d18403bbc82
Exempt WPAD requests from HTTPS-only mode upgrades r=necko-reviewers,dragana

Comment 22

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/7d18403bbc82