1245210 - PAC proxy fails after switching from home network to work network, requires manual reload

Status: RESOLVED WORKSFORME

(Core :: Networking, defect)

Description

[Steve Chessin]

Attachment: unable2connect.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:43.0) Gecko/20100101 Firefox/43.0
Build ID: 20160105164030

Steps to reproduce:

(This showed up in Firefox 44.0. I've switched back to 43.0.4)
I launch FF44.0 at work, connected to the internet through a firewall. I browse the office intranet and the external internet with no problem. At the end of the day I put my MBP to sleep and take it home. At home I connect to the internet via AT&T U-verse and browse the internet with no problem. The next day I put the MBP to sleep, take it back to work, and connect to the office network. Now every attempt to get to a website (internal or external) results in "Unable to Connect". (See attachment.) I have to Quit FF and restart it for it to go to websites again.


Actual results:

I get "Unable to connect" messages.


Expected results:

I should have been able to go to websites.

Comment 1

[Steve Chessin]

Just FYI, 43.0.4 does not have this problem (which is why I've switched back to it, as it's a pain to restart FF and put all the windows I keep open into the spaces I want them in).

Comment 2

[Daniel Stenberg [:bagder]]

Thanks for the detailed report. Would you be able to do some HTTP logging (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging) for me when you run such a scenario? (I would imagine that "nsIOService:5" and "nsNotifyAddr:5" logging would be enough in a first run to see what Firefox thinks there.

It is mysterious to me that you can take Firefox in one direction of network-change but not back again!

Are you using a proxy in any of those networks?

When you find yourself in that stupid error state, can you connect to *any* site? Like if you enter a completely not-visited-before URL in the address bar, does that fail too?

When you find yourself in that stupid error state, can you try to click yourself "offline" (menu File->Work offline) and back again and see if that changes anything?

Comment 3

[Steve Chessin]

(In reply to Daniel Stenberg [:bagder] from comment #2)
> Thanks for the detailed report. Would you be able to do some HTTP logging
> (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging)
> for me when you run such a scenario? (I would imagine that "nsIOService:5"
> and "nsNotifyAddr:5" logging would be enough in a first run to see what
> Firefox thinks there.'

I can do that. I note that the instructions assume the default shell whereas I use csh, but I'm familiar enough with setenv to make it work. I also note that the instructions say to first Quit Firefox, and then "Copy and paste each line from Figure 4 into the Terminal window" (and similarly for the other OSes). It's hard to copy from a page that is no longer displayed because you quit the browser :-). The instructions should have the user open the Terminal window (or equivalent) first, then copy and paste each of the lines except the last one into the window, pressing Return after each one, then copy the last command but don't press return, then quit Firefox, and then (once it has completely quit) press return.

Alternatively, tell them to open the page in some other browser (Safari or IE or Chrome), or copy the lines to a text file, or print the page to a PDF file, etc., and copy and paste from there.

> It is mysterious to me that you can take Firefox in one direction of
> network-change but not back again!

It is mysterious to me, too.

> Are you using a proxy in any of those networks?

At work, yes. At home, I don't think so. (Is that the problem? When the network connection changes you don't attempt to re-read the proxy file? My Preferences->Advanced->Network->Connection is "Automatic proxy configuration URL" with a URL that is valid only when I'm connected to the internal network.)

> When you find yourself in that stupid error state, can you connect to *any*
> site? Like if you enter a completely not-visited-before URL in the address
> bar, does that fail too?

I'll have to try that.

> When you find yourself in that stupid error state, can you try to click
> yourself "offline" (menu File->Work offline) and back again and see if that
> changes anything?

I thought I tried that, but I'll try it again.

Comment 4

[Daniel Stenberg [:bagder]]

(In reply to Steve Chessin from comment #3)

> > Are you using a proxy in any of those networks?
> 
> At work, yes. At home, I don't think so. (Is that the problem? When the
> network connection changes you don't attempt to re-read the proxy file? My
> Preferences->Advanced->Network->Connection is "Automatic proxy configuration
> URL" with a URL that is valid only when I'm connected to the internal
> network.)

I suspect the use of a proxy could be related or required to trigger this issue, yes.

When you switch back to your work network again the next time and end up in this error state, maybe you can try to go into your network settings and update that as a means to force Firefox to do an update, and see if that changes anything.

Comment 5

[Daniel Stenberg [:bagder]]

(In reply to Daniel Stenberg [:bagder] from comment #2)

> (I would imagine that "nsIOService:5"
> and "nsNotifyAddr:5" logging would be enough in a first run to see what
> Firefox thinks there.

Better make that "timestamp,nsIOService:5,nsNotifyAddr:5,proxy:5" and we might get some clues about the proxy decisions as well.

Comment 6

[Steve Chessin]

(In reply to Steve Chessin from comment #3)
> (In reply to Daniel Stenberg [:bagder] from comment #2)
> > Thanks for the detailed report. Would you be able to do some HTTP logging
> > (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging)
> > for me when you run such a scenario? (I would imagine that "nsIOService:5"
> > and "nsNotifyAddr:5" logging would be enough in a first run to see what
> > Firefox thinks there.'
> 
> I can do that.

Okay, I did that. I'll attach the file in my next comment (if FF will let me; it's 650MB in size).

This is how I got that file:
At home, on my home WiFi, I quit Firefox, went to my terminal window, and did this:
% setenv NSPR_LOG_MODULES timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5
% setenv NSPR_LOG_FILE ~/Desktop/log.txt
% cd /Applications/Firefox44.0.0.app/Contents/MacOS
% ./firefox-bin &

I did some work from home, closed the MBP, took my car to the dealer for a repair, while at the dealership used their WiFi network to do some work, when the loaner car arrived closed the MBP and drove to work, and connected the MBP to the ethernet at work so I could use its intranet.

While at work, I disconnected the ethernet and went to the "guest" WiFi to access the internet. I then needed to print something so I plugged the cable back in, did the printing, and unplugged the cable. That is at the point that the browser went into "stupid error state".

> > When you find yourself in that stupid error state, can you connect to *any*
> > site? Like if you enter a completely not-visited-before URL in the address
> > bar, does that fail too?
> 
> I'll have to try that.

I tried that, and it failed.

> > When you find yourself in that stupid error state, can you try to click
> > yourself "offline" (menu File->Work offline) and back again and see if that
> > changes anything?
> 
> I thought I tried that, but I'll try it again.

I tried that, and that didn't help either.

Comment 7

[Steve Chessin]

(In reply to Steve Chessin from comment #6)
> (In reply to Steve Chessin from comment #3)
> > (In reply to Daniel Stenberg [:bagder] from comment #2)
> > > Thanks for the detailed report. Would you be able to do some HTTP logging
> > > (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging)
> > > for me when you run such a scenario? (I would imagine that "nsIOService:5"
> > > and "nsNotifyAddr:5" logging would be enough in a first run to see what
> > > Firefox thinks there.'
> > 
> > I can do that.
> 
> Okay, I did that. I'll attach the file in my next comment (if FF will let
> me; it's 650MB in size).

Well, Mozilla won't let me. It says:

"The file you are trying to attach is 665173 kilobytes (KB) in size. Attachments cannot be more than 10240 KB.
We recommend that you store your attachment elsewhere and then paste the URL to this file on the attachment creation page in the appropriate text field."

Do you have an "elsewhere" where I can store the attachment? Or should I just upload the last 10MB of it? Or split it into 65 10MB pieces and upload them? Please advise.

Comment 8

[Steve Chessin]

(In reply to Daniel Stenberg [:bagder] from comment #5)
> (In reply to Daniel Stenberg [:bagder] from comment #2)
> 
> > (I would imagine that "nsIOService:5"
> > and "nsNotifyAddr:5" logging would be enough in a first run to see what
> > Firefox thinks there.
> 
> Better make that "timestamp,nsIOService:5,nsNotifyAddr:5,proxy:5" and we
> might get some clues about the proxy decisions as well.

I'll do that in my next test. Should I include nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5 as well?

Comment 9

[Daniel Stenberg [:bagder]]

(In reply to Steve Chessin from comment #7)

> Do you have an "elsewhere" where I can store the attachment? Or should I
> just upload the last 10MB of it? Or split it into 65 10MB pieces and upload
> them? Please advise.

I emailed you privately about the huge log and how to host it. We really should have a public solution for hosting large log submissions...

> Should I include nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5 as well?

The more the merrier in general, I was just trying to cut off a few modules to make the log file a bit smaller in the end. I'll also work more on trying to reproduce this case in my local setup.

Comment 10

[Steve Chessin]

(In reply to Daniel Stenberg [:bagder] from comment #9)
> (In reply to Steve Chessin from comment #7)
> 
> > Do you have an "elsewhere" where I can store the attachment? Or should I
> > just upload the last 10MB of it? Or split it into 65 10MB pieces and upload
> > them? Please advise.
> 
> I emailed you privately about the huge log and how to host it. We really
> should have a public solution for hosting large log submissions...

I emailed you privately as to where you can find it.

(In reply to Daniel Stenberg [:bagder] from comment #5)
> (In reply to Daniel Stenberg [:bagder] from comment #2)
> 
> > (I would imagine that "nsIOService:5"
> > and "nsNotifyAddr:5" logging would be enough in a first run to see what
> > Firefox thinks there.
> 
> Better make that "timestamp,nsIOService:5,nsNotifyAddr:5,proxy:5" and we
> might get some clues about the proxy decisions as well.

I ran that test (I didn't add the others). I'll upload the zipped log file (it's only 36KB), as well as a description of what I did, and a transcript of the shell session (it produced a lot of "Set intl.ime.nstextinput.enable to true in about:config to fix input." messages on the terminal).

Comment 11

[Steve Chessin]

Attachment: Zipped log file of FF44.0 with "timestamp,nsIOService:5,nsNotifyAddr:5,proxy:5" set in environment.

Comment 12

[Steve Chessin]

Attachment: What I did to get the log.2.txt logfile.

What was interesting was I could surf the corporate network (intranet), but attempts to go outside (for example, to bugzilla.mozilla.org) did not work.

Comment 13

[Steve Chessin]

Attachment: Transcript of shell session.

Note the messages from firefox-bin. I did not get any messages when I used "timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5".

Comment 14

[Daniel Stenberg [:bagder]]

(In reply to Steve Chessin from comment #12)

> What was interesting was I could surf the corporate network (intranet), but
> attempts to go outside (for example, to bugzilla.mozilla.org) did not work.

Oh but that's indeed valuable feedback! My guess: that's because the PAC reload failed somehow and your browser normally uses that PAC and get a proxy returned to be used for the outside servers while you access your intranet servers DIRECT (== without a proxy).

If you end up in this state and then go to the network preferences (Preferences->Advanced->Network tab->settings button), and just toggle/untoggle something - to make Firefox reload/reset the config - does that bring the functionality back?

Comment 15

[Steve Chessin]

(In reply to Daniel Stenberg [:bagder] from comment #14)
> (In reply to Steve Chessin from comment #12)
> 
> > What was interesting was I could surf the corporate network (intranet), but
> > attempts to go outside (for example, to bugzilla.mozilla.org) did not work.
> 
> Oh but that's indeed valuable feedback! My guess: that's because the PAC
> reload failed somehow and your browser normally uses that PAC and get a
> proxy returned to be used for the outside servers while you access your
> intranet servers DIRECT (== without a proxy).
> 
> If you end up in this state and then go to the network preferences
> (Preferences->Advanced->Network tab->settings button), and just
> toggle/untoggle something - to make Firefox reload/reset the config - does
> that bring the functionality back?

I'll try that in the next experiment. What value for NSPR_LOG_MODULES would you like me to use? And should I continue to test FF 44.0, or should I try FF 44.0.2 which just got released?

Comment 16

[Steve Chessin]

(In reply to Daniel Stenberg [:bagder] from comment #14)
> (In reply to Steve Chessin from comment #12)
> 
> > What was interesting was I could surf the corporate network (intranet), but
> > attempts to go outside (for example, to bugzilla.mozilla.org) did not work.
> 
> Oh but that's indeed valuable feedback! My guess: that's because the PAC
> reload failed somehow and your browser normally uses that PAC and get a
> proxy returned to be used for the outside servers while you access your
> intranet servers DIRECT (== without a proxy).
> 
> If you end up in this state and then go to the network preferences
> (Preferences->Advanced->Network tab->settings button), and just
> toggle/untoggle something - to make Firefox reload/reset the config - does
> that bring the functionality back?

That worked! I'll upload the files related to this run.

Comment 17

[Steve Chessin]

Attachment: Test 3: Zipped log file of FF44.0 with "timestamp,nsIOService:5,nsNotifyAddr:5,proxy:5" set in environment.

Comment 18

[Steve Chessin]

Attachment: Test 3: What I did to get the log.3.txt logfile.

Comment 19

[Steve Chessin]

Attachment: Test 3: Transcript of shell session.

Comment 20

[Daniel Stenberg [:bagder]]

(In reply to Steve Chessin from comment #16)

> That worked! I'll upload the files related to this run.

Excellent! Thanks a lot for that test and for the new log. I now have a much more narrow set of tests to run locally to see if I can force-trigger this and work on a patch.

Comment 21

[Steve Chessin]

I upgraded to FF 44.0.2 and it has the same problem (and the same workaround).

Comment 23

[Steve Chessin]

My default setting is "Automatic Proxy Configuration URL:" with a URL that is only valid on the internal office network. Just clicking on "Reload" seems to always fix the problem, whether it occurs when I connect to the internal office network or when I disconnect from it.

Comment 24

[Steve Chessin]

Problem still exists with FF 45.0.1. Workaround still works.

Comment 25

[Daniel Stenberg [:bagder]]

Debugging wonky network situations[1] when using PAC/system proxy with Firefox
==============================================================================

I've recently added additional logging to the proxy module in Firefox [2], so
please try the following with a Firefox build that contains this addition
[3]. The idea is that with the added logging we should get more information
about what exactly is happening and failing in your particular case.

Enable HTTP logging as described here:

  https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

The minimum amount of logging would probably include:

  export NSPR_LOG_MODULES=timestamp,proxy:5,nsNotifyAddr:5
  export NSPR_LOG_FILE=httpproxylog.txt

"proxy" is for the PAC and proxy logic to get logged and "nsNotifyAddr" is for
the "link monitor" logs that inform about detected network changes.

With this logging enabled, try to reproduce the problem you've reported and
then attach the produced HTTP log file to the corresponding bug report for the
bug you're working on.

[1] = this includes laptops coming back from sleep/hibernation etc as that is
basically a network change for Firefox internals.

[2] = Bug 1260407 landed in
https://hg.mozilla.org/mozilla-central/rev/4a7ab6d8c379

[3] = Firefox nightly as of April 1st (no joke) and later includes this
change.

Comment 26

[Steve Chessin]

(In reply to Daniel Stenberg [:bagder] from comment #25)
[...]
> I've recently added additional logging to the proxy module in Firefox [2], so
> please try the following with a Firefox build that contains this addition
> [3]. The idea is that with the added logging we should get more information
> about what exactly is happening and failing in your particular case.
[...]
> [3] = Firefox nightly as of April 1st (no joke) and later includes this
> change.

Please remind me how to find the nightly builds.

Thanks,
--Steve

Comment 27

[Loic]

(In reply to Steve Chessin from comment #26)
> 
> Please remind me how to find the nightly builds.
> 
> Thanks,
> --Steve

For each day of the month, they are in the mozilla-central repository.
For April 1st e.g: http://ftp.mozilla.org/pub/firefox/nightly/2016/04/2016-04-01-03-02-16-mozilla-central/

Comment 28

[Steve Chessin]

I've downloaded firefox-48.0a1.en-US.mac.dmg and will try it over the weekend or early next week. I'll upload the log file to this bug report when I have it.

Comment 29

[Steve Chessin]

I took Nightly out for a spin and noticed it was filling up my Terminal window with lines like these:
wot_warning.hide: failed with TypeError: content.getElementById is not a function

(I use WOT.) Is that an incompatibility or is that message usually suppressed?

The log file (I'll upload it) is full of lines like these:
2016-04-03 04:43:12.911718 UTC - [Main Thread]: D/proxy pac not available, use DIRECT
2016-04-03 04:43:12.911751 UTC - [Main Thread]: D/proxy pac not available, use DIRECT
2016-04-03 04:43:12.913942 UTC - [Main Thread]: D/proxy pac not available, use DIRECT
2016-04-03 04:43:12.914158 UTC - [Main Thread]: D/proxy pac not available, use DIRECT
2016-04-03 04:43:12.914311 UTC - [Main Thread]: D/proxy pac not available, use DIRECT

There are dozens per second. Is that expected?

I'll launch it again when I go to work on Monday, as that is where I can get the problem to easily manifest itself, changing from the internal wired network to the external WiFi one.

Comment 30

[Steve Chessin]

Attachment: Transcript and logs of test run with FirefoxNightly48.0a1.en-US.app

Comment 31

[Steve Chessin]

I'm running FirefoxNightly48.0a1.en right now, and I noticed that it disrespects my desire to use a 2-page per page print layout. Do I report that separately and if so using what Product, Component, and Version?

Comment 32

[Daniel Stenberg [:bagder]]

Product: core
Component: Printing:output 

I think.

Comment 33

[Steve Chessin]

Thanks. Another issue: It doesn't recognize the two-finger scroll on the track pad. What do I use to report that?

Comment 34

[Steve Chessin]

(In reply to Steve Chessin from comment #31)
> I'm running FirefoxNightly48.0a1.en right now, and I noticed that it
> disrespects my desire to use a 2-page per page print layout. Do I report
> that separately and if so using what Product, Component, and Version?

I filed Bug 1262682 - "Nightly 48.0a1 (2016-04-01) ignores print dialogue selections" on this. I let the Product and Component go to the default (Firefox/Untriaged.)

(In reply to Steve Chessin from comment #33)
> Thanks. Another issue: It doesn't recognize the two-finger scroll on the
> track pad. What do I use to report that?

I filed "Bug 1262683 - Nightly 48.0a1 (2016-04-01) ignores two-finger scroll on the track pad" on this, also letting go to the default categorization.

Comment 35

[Steve Chessin]

(In reply to Steve Chessin from comment #29)
[...]
> I'll launch it again when I go to work on Monday, as that is where I can get
> the problem to easily manifest itself, changing from the internal wired
> network to the external WiFi one.

I've had multiple transitions from home network to internal wired network to external WiFi network and back again during the week and the problem has not manifested itself. I guess it's been "inadvertently fixed" in some build between FF 45.0.1 and Nightly 48.0a1 (2016-04-01). I'm going to Quit Nightly, go back to FF 45.0.1, and zip up all the log files and upload them.

Comment 36

[Daniel Stenberg [:bagder]]

> the problem has not manifested itself

I guess that's good in the sense that not experiencing the problem is good.

However, the log file you provided did show Firefox not being able to load the PAC from the given URL and when it fails that, it switches to DIRECT (ie it skips the proxy). In a network that requires the use of a proxy, going DIRECT is usually just going to fail accessing the requested URL...

If that problem can be repeated, we would need to investigate exactly what prevents it from getting the PAC. A log enabled Firefox can be told to log a lot of http details that could help for this, just be aware that enabling say "nsHttp:5" (probably in addition to proxy and nsNotifyAddr) will add significant amounts of data to the log file. You probably want to limit the time you have that logging enabled to keep the log file no larger than it needs to be.

Comment 37

[Daniel Stenberg [:bagder]]

This problem might be fixed in 48+ and I've not managed to reproduce or figure out the culprit. Putting it into the backlog.

Comment 38

[Laurens Vets]

I noticed I can still trigger a related issue.

Environment:
- Firefox 50.0.2
- Using a proxy ("Use system proxy settings")
- Using Cisco AnyConnect VPN 3.1.14018

When the VPN disconnects and reconnects, Firefox is not able to show any pages. I usually do the following to fix it:
1. Options->Advanced->Network Settings->Select "No proxy"->Press "OK"
2. Select "Use system proxy settings" again
3. Firefox works again...

What can I do to help troubleshoot this?