1892995 - Repeated 'Restart Required' popups even when on latest version with Proofpoint

Status: REOPENED

(Core :: IPC, defect)

Description

[fabian-mozilla]

Attachment: Screenshot 2024-04-23 at 17.21.49.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0

Steps to reproduce:

Repeatedly, at seemingly random times throughout the day Firefox opens a 'Restart Required' popup even if the latest version is installed. The popups keep coming after restarts. I have attached a screenshot of the popup and version window. There doesn't appear to be any relevant message in the Browser Console.

Firefox 125.0.2
macOS 14.4.1 Arm

Actual results:

Many 'Restart Required' popups.

Expected results:

At most one 'Restart Required' popup.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Widget: Cocoa' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Nipun Shukla]

Hi, thanks for the report! Can you please send over the following update logs?

Gets logs for the updater tool in Firefox
1. Navigate to about:config.
2. Set app.update.log to true.
3. Open the Browser Console either with the hotkey Control+Shift+J (Command+Shift+J on macOS), or via Hamburger Menu->More Tools->Browser Console
4. In the Filter textbox at the top, enter AUS:SVC to filter out everything except the update messages.
5. Navigate to the "Update" section of about:preferences. It should automatically check for an update.
6. Once the update check has completed, copy the messages out of the Browser Console and attach them to this bug.

Get logs from updater binary that actually applies the updates
1. Navigate to about:support
2. Find the "Update Folder" entry and click "Open Folder".
3. Open the updates directory.
4. Inside, you should find files named last-update.log and backup-update.log. Attach these files to this bug.

I'm also going to NI bytesized as they would know the most about this specific issue.

Comment 3

[fabian-mozilla]

Attachment: backup-update.log

Comment 4

[fabian-mozilla]

Attachment: console.log

Comment 5

[fabian-mozilla]

Attachment: last-update.log

Comment 6

[fabian-mozilla]

Thanks for the quick response. I've attached the three files. They look uninformative to my eye. I'll check them again once I get the next popup.

Comment 7

[Robin Steuber (she/her) [:bytesized]]

In general, this happens when Firefox has been updated by something while it was running. There are two known ways that this can happen:

1. Launch another Firefox instance with another profile which applies a downloaded update, breaking the already-running instance(s).
2. External software updates Firefox while it is running. This can include package management software, but can also includes virus protection and enterprise management software.

Do either of those seem like they could be happening in your situation?

Comment 8

[fabian-mozilla]

Situation 1 doesn't apply. I'm the sole user of this machine and use only one profile. Also, there weren't any updates to download as I'm already on the latest version.

There could potentially be enterprise software interfering. I installed Firefox by downloading directly from Mozilla and not via any package management but I know that we have filescanners, firewalls. etc. What leaves me baffled is that I get the popup despite being on the latest version.

Comment 9

[Robin Steuber (she/her) [:bytesized]]

(In reply to fabian-mozilla from comment #8)

What leaves me baffled is that I get the popup despite being on the latest version.

That is surprising. When this happens next, could you check some information and report back? First, navigate to about:support (navigation to about: urls should continue to work, even when Firefox is in this state). What does the "Build ID" row say? Now open the installation directory (about:support lists the "Application Binary" path, if you can't remember where you installed it). Locate and open the file application.ini. There should be a row that says BuildID=.... What is that build id? Does it match the one listed in about:support?

After restarting the browser to get rid of the "Restart Required" page, please navigate back to about:support and click the "Show Update History" button. Does it list an update as having been installed around the time that "Restart Required" was shown?

Comment 10

[fabian-mozilla]

Here is the data you asked for:

about:support
Build ID: 20240419144423

application.ini
BuildID=20240419144423

The last actual update to 125.0.2 happened on the 23rd. I just keep getting these popups in a private(?) window at seemingly random times. Also there is nothing useful in the browser console when the popup comes.

Comment 11

[Robin Steuber (she/her) [:bytesized]]

Huh, how about that. It sounds like you aren't being updated improperly after all. I guess the "Restart Required" page is being shown incorrectly. In that case, I think this bug needs to be redirected to a different component.

Hopefully this component/needinfo will get the attention of the right people to help with this.

Comment 12

[:gerard-majax]

This would suggest something is crashing early? Anything in about:crashes ? We check against platform.ini https://searchfox.org/mozilla-central/rev/279e588a6d5fcb2f652fc26345ddb2a3b4d58b0e/dom/base/nsFrameLoader.cpp#3662 can you triple check its content?

Comment 13

[:gerard-majax]

(In reply to fabian-mozilla from comment #8)

There could potentially be enterprise software interfering. I installed Firefox by downloading directly from Mozilla and not via any package management but I know that we have filescanners, firewalls. etc. What leaves me baffled is that I get the popup despite being on the latest version.

Could one of those be somehow blocking our reading of platform.ini ?

Comment 14

[fabian-mozilla]

There is nothing in about:crashes. Also the console seems to be surprisingly silent. Only the following four lines stood out to me:

09:06:19.578 AUS:SVC UpdateService.canUsuallyCheckForUpdates - able to check for updates
09:06:50.002 AUS:SVC UpdateManager:_loadXMLFileIntoArray - XML file does not exist. path: /Users/fklotzl/Library/Caches/Mozilla/updates/Applications/Firefox/active-update.xml
09:06:50.002 AUS:SVC UpdateManager:UpdateManager - Initialized downloadingUpdate to null
09:06:50.002 AUS:SVC UpdateManager:UpdateManager - Initialized readyUpdate to null

The platform.ini is fine and contains the build id as written by the latest update:

$ ll /Applications/Firefox.app//Contents/Resources/platform.ini
-rw-rw-r--@ 1 fklotzl admin 167 23 Apr 16:26 /Applications/Firefox.app//Contents/Resources/platform.ini
$ cat /Applications/Firefox.app//Contents/Resources/platform.ini
[Build]
BuildID=20240419144423
Milestone=125.0.2
SourceRepository=https://hg.mozilla.org/releases/mozilla-release
SourceStamp=c5ee44e4135571bec3220340242f9189c59ca5ba

Comment 15

[:gerard-majax]

What about those enterprise software? No OOM situation either? Does it still reproduces after rebooting for example?

Comment 16

[:gerard-majax]

Also, does it means you cannot absolutely use the browser ? Or are you able to get some content process to work ? What does about:processes looks like if you can get to that? Are you able to get about:support ?

Comment 17

[fabian-mozilla]

Mostly I can work fine. Just every few hours I get a popup claiming I should restart when clearly nothing was installed.

Does it still reproduces after rebooting for example?

I restarted my laptop. Will let you know when I get the next popup.

Comment 18

[fabian-mozilla]

Attachment: Screenshot 2024-04-30 at 09.26.56.png

Opening my laptop this morning I again was greeted with the request to restart.

Comment 19

[fabian-mozilla]

Here is the only information in the console at that time. Looks boring:

09:27:00.570 AUS:SVC CheckerService:checkForUpdates - checkType: 2
09:27:00.570 AUS:SVC CheckerService:checkForUpdates - Making new check request for check id 2.
09:27:00.571 AUS:SVC CheckerService:getUpdateURL - checkType: 2
09:27:00.572 AUS:SVC CheckerService:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/125.0.3/20240425211020/Darwin_aarch64-gcc3/en-GB/release/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A32768/default/default/update.xml?force=1
09:27:00.572 AUS:SVC CheckerService:#updateCheck - sending request to: https://aus5.mozilla.org/update/6/Firefox/125.0.3/20240425211020/Darwin_aarch64-gcc3/en-GB/release/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A32768/default/default/update.xml?force=1
09:27:00.681 AUS:SVC CheckerService:#updateCheck - request got 'load' event
09:27:00.681 AUS:SVC CheckerService:#updateCheck - request completed downloading document
09:27:00.681 AUS:SVC CheckerService:#updateCheck - number of updates available: 0

If there are no updates available there shouldn't be a popup, right?

Comment 20

[:gerard-majax]

(In reply to fabian-mozilla from comment #19)

Here is the only information in the console at that time. Looks boring:

09:27:00.570 AUS:SVC CheckerService:checkForUpdates - checkType: 2
09:27:00.570 AUS:SVC CheckerService:checkForUpdates - Making new check request for check id 2.
09:27:00.571 AUS:SVC CheckerService:getUpdateURL - checkType: 2
09:27:00.572 AUS:SVC CheckerService:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/125.0.3/20240425211020/Darwin_aarch64-gcc3/en-GB/release/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A32768/default/default/update.xml?force=1
09:27:00.572 AUS:SVC CheckerService:#updateCheck - sending request to: https://aus5.mozilla.org/update/6/Firefox/125.0.3/20240425211020/Darwin_aarch64-gcc3/en-GB/release/Darwin%252023.4.0/ISET%3ANEON%2CMEM%3A32768/default/default/update.xml?force=1
09:27:00.681 AUS:SVC CheckerService:#updateCheck - request got 'load' event
09:27:00.681 AUS:SVC CheckerService:#updateCheck - request completed downloading document
09:27:00.681 AUS:SVC CheckerService:#updateCheck - number of updates available: 0

If there are no updates available there shouldn't be a popup, right?

Could you please reply to all the other questions above ?

Comment 21

[fabian-mozilla]

Apologies, here are more detailed answers. Thanks a lot for your support and patience.

What about those enterprise software?

Will ask our support.

No OOM situation either?

No, there is gigs of memory available at any one time.

Does it still reproduces after rebooting for example?

Yes.

Also, does it means you cannot absolutely use the browser?

Firefox works alright. It just opens a new window once in a while. I googled for a while but didn't find a way to resolve that myself, hence the ticket.

Or are you able to get some content process to work? What does about:processes looks like if you can get to that?

I'll attach a screenshot.

Are you able to get about:support?

Yes. Anything specific you are interested in?

Comment 22

[fabian-mozilla]

Attachment: processes.jpg

Comment 23

[:gerard-majax]

(In reply to fabian-mozilla from comment #21)

Are you able to get about:support?

Yes. Anything specific you are interested in?

Not really, I think the whole would maybe give hints?

Comment 24

[fabian-mozilla]

Attachment: about-support.json

Comment 25

[:gerard-majax]

i think we need to find a clear way to disambiguate what happens: are we crashing early enough, and if so what kind of information can macOS provide us? are we getting killed?

Comment 26

[:gerard-majax]

It looks like you could set OSCrashReporter within Firefox' plist via the defaults [https://ss64.com/mac/defaults.html] tool, so we allow you to see the macOS crash reporter. Hopefully, you could correlate the about:restartrequired content being displayed with macOS crash reporter.

Comment 27

[fabian-mozilla]

Ok, I wasn't sure which value to use so I just set OSCrashReport to 1. Let's see what happens.

defaults write ~/Library/Preferences/org.mozilla.firefox OSCrashReporter 1

Comment 28

[fabian-mozilla]

I'm not sure that crash reporter did anything but I have some news nonetheless. Recently our VPN client updated to Cisco Secure Client and now I reproducibly get a few popups whenever I change network. Turning VPN on or off, changing from wifi to wired all trigger the popups. Smells like the network connection is lost for a short moment triggering the popups?

Comment 29

[:gerard-majax]

(In reply to fabian-mozilla from comment #28)

I'm not sure that crash reporter did anything but I have some news nonetheless. Recently our VPN client updated to Cisco Secure Client and now I reproducibly get a few popups whenever I change network. Turning VPN on or off, changing from wifi to wired all trigger the popups. Smells like the network connection is lost for a short moment triggering the popups?

How much reliable is it? Do you have a way to disable / uninstall Cisco Secure Client, temporarily (making sure it does not intercept anything / load kernel modules) ? Would the update of this VPN client be consistent with the issue starting ?

Comment 30

[:gerard-majax]

Kershaw, I saw (by pure luck, i have to admit) that bug 1853497 was still there. Could it be related to the present issue ?

Comment 31

[Kershaw Chang [:kershaw]]

(In reply to :gerard-majax from comment #30)

Kershaw, I saw (by pure luck, i have to admit) that bug 1853497 was still there. Could it be related to the present issue ?

Probably not, because the patch in bug 1853497 was backed out again.

(In reply to fabian-mozilla from comment #28)

I'm not sure that crash reporter did anything but I have some news nonetheless. Recently our VPN client updated to Cisco Secure Client and now I reproducibly get a few popups whenever I change network. Turning VPN on or off, changing from wifi to wired all trigger the popups. Smells like the network connection is lost for a short moment triggering the popups?

Reporter, could you try to record a http log while turning VPN on and off? Could you also attach some screenshots for the popups you see?
Thanks.

Comment 32

[:gerard-majax]

(In reply to Kershaw Chang [:kershaw] from comment #31)

(In reply to :gerard-majax from comment #30)

Kershaw, I saw (by pure luck, i have to admit) that bug 1853497 was still there. Could it be related to the present issue ?

Probably not, because the patch in bug 1853497 was backed out again.

Yes, well, maybe not exactly the same bug, but there seems to be similarities?

(In reply to fabian-mozilla from comment #28)

I'm not sure that crash reporter did anything but I have some news nonetheless. Recently our VPN client updated to Cisco Secure Client and now I reproducibly get a few popups whenever I change network. Turning VPN on or off, changing from wifi to wired all trigger the popups. Smells like the network connection is lost for a short moment triggering the popups?

Reporter, could you try to record a http log while turning VPN on and off? Could you also attach some screenshots for the popups you see?
Thanks.

I think the "popup" is the screenshot shared in the first comment, i.e., about:restartrequired

Comment 33

[fabian-mozilla]

Attachment: log.txt-main.87968.moz_log.gz

I've attached a HTTP log. These days going from wired to wlan or vice-versa is enough to trigger the popup. For the log I started logging, unplugged the cable, popup came and stopped logging.
Hope this helps.

Comment 34

[:gerard-majax]

(In reply to fabian-mozilla from comment #33)

Created attachment 9402330 [details]
log.txt-main.87968.moz_log.gz

I've attached a HTTP log. These days going from wired to wlan or vice-versa is enough to trigger the popup. For the log I started logging, unplugged the cable, popup came and stopped logging.
Hope this helps.

Just to make it clear, you switched networks with or without your VPN client still active ?

Comment 35

[fabian-mozilla]

Attachment: vpn.log.gz

Apologies for the slow reply. Above log contains one network switch without vpn.

To make things a bit easier I have added another log with multiple network changes. Each one triggered a restartrequired page. The changes were:

wifi, vpn off -> wired, vpn off -> wired, vpn on -> wired, vpn off -> wifi vpn off

For some reason those steps triggered five restartrequired pages. So what ever we are looking for should be in the vpn.log four or five times.

Comment 36

[:gerard-majax]

Do you reproduce without the vpn active or even installed ?

Comment 37

[fabian-mozilla]

I can try but this is a work computer and I have limited control.

Comment 38

[:gerard-majax]

Of course, but we need to know if it's related to that specific Cisco client or not. People managing your system are aware of that issue ?

Comment 39

[Kershaw Chang [:kershaw]]

(In reply to fabian-mozilla from comment #35)

Created attachment 9403029 [details]
vpn.log.gz

Apologies for the slow reply. Above log contains one network switch without vpn.

To make things a bit easier I have added another log with multiple network changes. Each one triggered a restartrequired page. The changes were:

wifi, vpn off -> wired, vpn off -> wired, vpn on -> wired, vpn off -> wifi vpn off

For some reason those steps triggered five restartrequired pages. So what ever we are looking for should be in the vpn.log four or five times.

Thanks for the log.
However, I am not able to find the reason why the restartrequired page shows from the log. I've seen two http requests to push.services.mozilla.com failed, but I don't think that's the reason.

Maybe this is related to bug 1884349 that I am currently working on, so I might ask you to test again when that bug fixed.

Comment 40

[fabian-mozilla]

Sure, let me know if there are more experiments to run. I'm also working with our IT department to circle in on the issue.

Comment 41

[fabian-mozilla]

Our IT has been looking into this issue. Apparently some external tool called proofpoint is messing with firefox. I'll close this ticket as invalid. Thanks for all the support!

Comment 42

[:gerard-majax]

We are interested in knowing more about that, can you share details? Can be in private if required.

Comment 43

[fabian-mozilla]

Sure, I'll forward your request to our IT support.

Comment 44

[:gerard-majax]

Gentle ping, have you got more details you can share with us?

Comment 45

[fabian-mozilla]

The issue is with an internal threats management software called proofpoint. The current workaround by IT security is to turn off the websecurity feature of proofpoint. As this is not a proper fix I have raised a new ticket with IT to keep digging and give you more feedback once the cause is found. Apologies for the wait.

Comment 48

[fabian-mozilla]

Unfortunately no progress to report. The current "fix" is still to deactivate the ProofPoint web security.

Comment 49

[:gerard-majax]

(In reply to fabian-mozilla from comment #48)

Unfortunately no progress to report. The current "fix" is still to deactivate the ProofPoint web security.

Thanks for keeping us uptodate. As you can see there are now other reports where ProofPoint is also suspected, so maybe we can make progress on our side as well.

Comment 50

[fabian-mozilla]

As you can see there are now other reports …

I strongly suspect that the author of https://bugzilla.mozilla.org/show_bug.cgi?id=1906961 and me work for the same organization. I tried to contact them via our internal channels but at the moment their status is shown as out of office. Hopefully we will know more once they are back from vacation.

Comment 51

[Yannis Juglaret [:yannis]]

I tried to reach a Proofpoint Web Security developer about this issue, I hope we can collaborate.

Comment 52

[Luke Georgalas]

(In reply to Yannis Juglaret [:yannis] from comment #51)

I tried to reach a Proofpoint Web Security developer about this issue, I hope we can collaborate.

@yannis Have you heard anything back?

Comment 53

[Luke Georgalas]

I had my IT support contact Proofpoint regarding this issue. They responded with the following:

This is a known limitation with FireFox. Other browser such as Chrome or Edge can update the Proxy settings dynamically when needed (update to Traffic Steering Rule or stop/start of WS proxy). Firefox cannot do this, to apply the change necessary to work the FireFox browser needs to restart.

Sounds like a potential feature that could be introduced to Firefox that Proofpoint would be able to take advantage of?

Comment 54

[Robin Steuber (she/her) [:bytesized]]

Oh. I very much understood that we were failing to start new processes because of Proofpoint. Now it sounds more like this software is just requesting that we open a tab with about:restartrequired. I am not really happy with external software being able to do that. That's not appropriate. It gives the user the incorrect impression that Firefox has been updated. It gives the incorrect impression that Firefox cannot just continue to be used (since that's what happens when we show this page normally). It is most likely possible for the about:restartrequired tab to be restored with session restore, which is horrible and ugly.

I am strongly inclined to say that we should prevent external software from just requesting that we load this URL.

Comment 55

[Luke Georgalas]

I don't disagree with you. However, I have run into cases where Proofpoint did actually need to change the proxy settings, and I wasn't able to connect to any sites in Firefox until I restarted Firefox -- I got an error that specifically called out not being about to connect to the proxy server. (Is that what you meant when you said "failing to start new processes because of Proofpoint"?)

Proofpoint did also say they would attempt to reduce the frequency at which they request a restart. I have a feeling they attempt to set the proxy server in response to any number of system events, including anything that might alter your network settings, like plugging/unplugging from a dock. I also have a feeling they don't check the old proxy settings before pushing new proxy settings, then claiming you need a restart. I'm hoping they figure out a way to only request a restart when the proxy settings actually change, because I've only run into the above problem a handful of times over the couple months since I had Proofpoint enabled on my PC.

Comment 56

[:gerard-majax]

Thanks, maybe there's something we can do here...

Comment 57

[:gerard-majax]

Yannis, just in case

Comment 58

[Robin Steuber (she/her) [:bytesized]]

(In reply to Luke Georgalas from comment #55)

I got an error that specifically called out not being about to connect to the proxy server. (Is that what you meant when you said "failing to start new processes because of Proofpoint"?)

No, that sounds like something different. And Proofpoint is absolutely welcome to show their own UI asking you to restart Firefox. Or open their own website that explains this in Firefox. But I am much less keen on them asking Firefox to open a page that says that Firefox was updated when it absolutely was not.

---

@mkaply Comment 53 contains what looks like an enterprise feature request. Is this one we already know about? If you'd like me to file a bug for it or something, please let me know.

Comment 59

[Mike Kaply [:mkaply]]

We don't currently support dynamic policy settings. It's something we've talked about, but it's basically a rearchitecting of how Firefox (and our policies) work.

I'm curious why the proxy is changing this much?

There might be a way to just have the proxy policy dynamically update on some interval...

Comment 60

[Luke Georgalas]

(In reply to Mike Kaply [:mkaply] from comment #59)

We don't currently support dynamic policy settings. It's something we've talked about, but it's basically a rearchitecting of how Firefox (and our policies) work.

I'm curious why the proxy is changing this much?

There might be a way to just have the proxy policy dynamically update on some interval...

The times I get these requests to restart Firefox is after I've done something that affects my network configuration. For example, when I plug my laptop into my dock, it changes from wireless to wired network, and vice versa. Today I'm WFH, so I connected to our VPN, so I got the popup. I actually only rarely WFH, so today might be the first day I've connected to my VPN since I installed Proofpoint (or rather had it installed for me). I encountered the problem again just now. After I connected the VPN, the restart request appeared, and when I tried to navigate to a site, I got an error about incorrect proxy settings. After restarting Firefox, I was able to browse again.

Comment 61

[Mike Kaply [:mkaply]]

Thank you. I'll see what I can do.

Comment 62

[Haik Aftandilian [:haik]]

(In reply to Robin Steuber (she/her) [:bytesized] from comment #54)

It is most likely possible for the about:restartrequired tab to be restored with session restore, which is horrible and ugly.

Yeah, testing confirms that is the case. With session restore enabled, after opening a tab to about:restartrequired and clicking the 'Restart Firefox' button, we continually display the about:restartrequired page so a user would have to know to ignore it at some point.

Comment 63

[Peter Van der Beken [:peterv]]

(In reply to Haik Aftandilian [:haik] from comment #62)

(In reply to Robin Steuber (she/her) [:bytesized] from comment #54)

It is most likely possible for the about:restartrequired tab to be restored with session restore, which is horrible and ugly.

Yeah, testing confirms that is the case. With session restore enabled, after opening a tab to about:restartrequired and clicking the 'Restart Firefox' button, we continually display the about:restartrequired page so a user would have to know to ignore it at some point.

That's probably bug 1681829.

Comment 64

[fabian-mozilla]

Quick update, proofpoint has update their agent so that it requires less restarts and opens fewer annoying windows. Still not perfect, but I'm happy to accept it.

Comment 65

[fabian-mozilla]

Closing as invalid as Firefox is not the perpetrator.

Comment 66

[:gerard-majax]

I'd still like we track that and maybe find some way to avoid that kind of disruption

Comment 67

[Luke Georgalas]

Is further discussion about dynamic policy settings either not going to happen or is beyond the scope of this bug?

Comment 68

[:gerard-majax]

(In reply to Luke Georgalas from comment #67)

Is further discussion about dynamic policy settings either not going to happen or is beyond the scope of this bug?

I guess mkaply would know

Comment 69

[Mike Kaply [:mkaply]]

I'd love to at some point, but it's not on our radar.

It's out of scope for this bug though.

I'll open a new bug.

Comment 70

[Mike Kaply [:mkaply]]

I opened https://bugzilla.mozilla.org/show_bug.cgi?id=1944127

Comment 71

[:gerard-majax]

Attachment: Bug 1892995 - Block about:restartrequired from being restored during SessionRestore r?#sessionstore-reviewers!