1547977 - Frequent "Unable to connect" when using automatic proxy configuration URL

Status: RESOLVED FIXED

(Core :: Networking, defect, P2)

Description

[milan.kardos2]

Attachment: troubleshooting information

Build ID:20190429215338
Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

the issue is back again - we are using PAC file for network configuration and after some time the external addresses stop working due to timeout. opening the network configuration and pressing the reload button near the PAC file location box will restore connectivity.

Comment 1

[Junior [inactive]]

Could you attach the log and append the MOZ_LOG environment variable with proxy:5
See instructions here: https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

If it takes times to reproduce, add rotate:200 to limit the log size to 200MB

Comment 2

[milan.kardos2]

i will add log on thursday.

Comment 3

[Honza Bambas (:mayhemer)]

I'll leave the ni on you until the logs are provided, thanks.

Comment 4

[milan.kardos2]

Attachment: log with proxy:5

Comment 5

[Junior [inactive]]

(In reply to milan.kardos2 from comment #4)

TRR cancels lots of http channels, which generates noises.
I'd like to ask some questions to ease the investigation:
(a) What url you failed to connect in this log?
(b) You reloaded the PAC file about 3 minutes before the end of log, and it works well after. Is that right?

Thanks!

Comment 6

[milan.kardos2]

a) f.e https://www.sme.sk/, https://dennikn.sk/, https://gulas.sme.sk/vsetko/, http://forums.mozillazine.org/viewforum.php?f=23 (now really not sure which one during logging time ...)
b) yes after PAC reload it works fine for some time ...

Comment 7

[Junior [inactive]]

2019-05-02 06:28:07.803000 UTC - [Parent 18444: ProxyResolution]: D/proxy Use proxy from PAC:
2019-05-02 06:28:07.803000 UTC - [Parent 18444: Main Thread]: D/proxy pac thread callback did not provide information 0

No proxy indicated in the last minute you reset the PAC file.
Which way did you set the pac file, about:preferences/os system setting/extension?

Comment 8

[milan.kardos2]

PAC was set via Options/General/Network Settings/Settings dialog

Comment 9

[Valentin Gosu [:valentin] (he/him)]

Honza, can you take a look at the logs since Junior is ooo, and maybe make a decision how we should proceed? Thanks!

Comment 10

[Honza Bambas (:mayhemer)]

Some of the responses (failing with 804b000e) simply don't get a proxy (as if the script was failing to evaluate). It's not true that reload of the wpad script helps. I can see hangouts.google.com requests that fail but also start working long before the request for the wpad (reload of it) is made.

Any errors in the browser console? Shift-Ctrl-J. You can filter for PAC.

Can you provide about:support page content? I want to eliminate any webext influence, because the proxy resolution results are filtered.

Note that the code is quite under-logged; w/o good logging it will be hard to figure this out if web extensions are ruled out. I think only way forward is to add more logging and provide a test build to run and try to reproduce.

Comment 11

[Alex Manz]

Hey, I'm also experiencing this bug.

Similar to initial reporter, my company is WPAD to configure proxy usage. After a while all external connections will stop working as the proxy is not resolved anymore. After creating and checking the log file I see the same behavior: suddenly the ProxyResolution is not providing the proxy anymore.

Checking my browser console I see a huge number of these:
PAC Execution Error: uncaught exception: out of memory []

Somehow I have the suspicion that this is closely related to the ProxyResolution being unresponsive after some time (or better after some connections)

For me the most convenient way to reproduce this bug is using google maps with satellite images. Simply scrolling around and loading a lot of images will rather sooner than later result in losing the proxy configuration.
To rule out any possible influence by addons and whatnot, I was testing it with a pretty pure Firefox 68.0.1 (64bit) with only SAML-tracer installed. (I previously suspected our proxy to be the culprit)

Let me know if the need for further information arises :)

Comment 12

[Alex Manz]

Aww crap. It seems that our PAC file is part of the problem. In it a pretty huge array is initialized to store all internal IP addresses that should work without the proxy. Somehow it seems that this allocated memory will not be freed after the script ran. If I null the array myself before the appropriate proxy is returned everything works as intended.

Maybe it is still worth looking into this issue, since I am not sure why the browser is not freeing up this memory itself?

Comment 13

[milan.kardos2]

Our PAC file is huge too (33 KB approx 800 lines), the same behaviour, even when checked system proxy usage in options.

Comment 14

[Honza Bambas (:mayhemer)]

Interesting! So it looks like a memory leak or not aggressive enough garbage collection when we are done with processing the PAC script on the parent process.

Luke, do you have any ideas how to diagnose this further or who to ask?

Leaving as a Networking issue for now.

Comment 15

[Luke Wagner [:luke]]

Hmm, maybe the GC options for the PAC's JSRuntime are not set correctly. Jon do you have any ideas?

Comment 16

[Jan de Mooij [:jandem]]

The PAC runtime uses a default heap size of 4 MB, you could try setting network.proxy.autoconfig_extra_jscontext_heap_size to some big value...

Comment 17

[milan.kardos2]

I set mentioned setting to 65535. Not sure if it is correct value, but now for 3 hrs no issue.
Is this heap visible via some telemetry? Found only MEMORY_JS_GC_HEAP ...

Comment 18

[milan.kardos2]

after 6 hrs i need to reload a PAC ...

Comment 19

[milan.kardos2]

after setting the mentioned preference to 68157440 it is working fine, no need of PAC reload whole day ...

Comment 20

[milan.kardos2]

fyi, with preference network.proxy.autoconfig_extra_jscontext_heap_size set to 68157440 i dont need anymore to reload the PAC file.

Comment 21

[milan.kardos2]

(In reply to milan.kardos2 from comment #20)

fyi, with preference network.proxy.autoconfig_extra_jscontext_heap_size set to 68157440 i dont need anymore to reload the PAC file.

tested on latest nightly builds

Comment 22

[Jan de Mooij [:jandem]]

(In reply to milan.kardos2 from comment #17)

I set mentioned setting to 65535. Not sure if it is correct value, but now for 3 hrs no issue.

(In reply to milan.kardos2 from comment #20)

fyi, with preference network.proxy.autoconfig_extra_jscontext_heap_size set to 68157440 i dont need anymore to reload the PAC file.

That makes sense - the value is in bytes so 65535 gives you only 64 KB extra but with 68157440 you get 65 MB which is significant.

I'd suggest increasing from 4 MB to 64-128 MB or so. The runtime has self-hosted code too so it's expected this got worse over time.

Comment 23

[Jon Coppeard (:jonco)]

Thanks Jan for answering this.

Comment 24

[Jan de Mooij [:jandem]]

Attachment: Bug 1547977 - Use JS::DefaultHeapMaxBytes instead of 4 MB for PAC runtime. r?jonco!,mayhemer!

DefaultHeapMaxBytes is currently 32 MB so it should give us a bit more breathing
room.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/151d38734b7f
Use JS::DefaultHeapMaxBytes instead of 4 MB for PAC runtime. r=jonco,mayhemer

Comment 26

[Jan de Mooij [:jandem]]

Comment on attachment 9086970 [details]
Bug 1547977 - Use JS::DefaultHeapMaxBytes instead of 4 MB for PAC runtime. r?jonco!,mayhemer!

Beta/Release Uplift Approval Request

• User impact if declined: Problems when using JS files for proxy configuration (PAC).
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This just increases the maximum GC heap size for the PAC thread from 4 MB to 32 MB so I think it's worth taking on beta and esr68.
• String changes made/needed: None

Comment 28

[Julien Cristau [:jcristau]]

Comment on attachment 9086970 [details]
Bug 1547977 - Use JS::DefaultHeapMaxBytes instead of 4 MB for PAC runtime. r?jonco!,mayhemer!

bigger heap for proxy config, approved for 69.0b16 and 68.1esr

Comment 29

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/de19545f3c19

Comment 30

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/151d38734b7f

Comment 31

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 32

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/releases/mozilla-beta/rev/f272644e18fa