1749996 - Session Restore (even when disabled) appears randomly after launching Firefox

Status: VERIFIED FIXED

(Firefox :: Session Restore, defect, P1)

Description

[MasterHarp]

Attachment: FF 96 Screen Shot

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0

Steps to reproduce:

This issue did not occur with 95.xx. After update to 96 FF acts like it crashed when it terminated mormally.

Actual results:

Receive the screen attached.

Expected results:

FF should start nomarlly.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox::Session Restore' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[MasterHarp]

Just updated to 96.0.1. Issue has topped occurring. Issue can be closed.

Comment 3

[MasterHarp]

Sorry spoke too fast. Problem not fixed. Now intermittent.

Comment 4

[Alin Ilea, Desktop Test Engineering]

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0

Reproduced the issue in Fx96 but it seems to be intermittent so it's hard to say if it's also reproducible in Beta 97 and Nightly 98, I didn't manage to find clear steps to reproduce it.

Thanks for your report.

Comment 6

[Alin Ilea, Desktop Test Engineering]

Attachment: 2022-01-21_09h38_49.png

Reproduced today in Nightly 98 and Beta 97 as well.

Comment 7

[Pablo]

sorry alice i missed this one and i opened bug 1752208

I am able to reproduce 100% with the steps below on one of my windows10 machines. On another win10 machine i have no issues at all, the only difference i can think of is that one of the machines with the problem has the antivirus (cisco amp)

i am able to reproduce it everywhere, latest nightly, beta and release.

You just have to:
Open firefox and create a new profile, and launch it.
then close firefox and launch again with the same profile,
then close firefox and launch again with the same profile
then close firefox launch again with the same profile.

in some cases it took 4 launches and in some cases 3 launches for the restore screen to appear. but i can get it 100% of the times with those steps

Comment 10

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

I can confirm this issue on my work's PC on new clean and fresh profile with Firefox 96.0.2.

Comment 12

[MasterHarp]

Nice to know I was not imaging this issue. Or it was just me. This is my first time reporting a bug.

This problem/issue started after updating to FF96. I am on the release cycle and keep my machine updated. Prior to FF96 this only appeared when FF really crashed. I know this will sound strange. Something happened/changed between 95 and 96. Currently at 96.0.2.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:dao, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 14

[Sylvestre Ledru [:Sylvestre]]

[Tracking Requested - why for this release]:
A new regression?

Comment 15

[MasterHarp]

Does this mean it will be fixed in 97 and later? Just not 96?

Comment 16

[alexios]

as the OP of this bug suggested i create a new profile, i did create a new profile and for the last 3-4 days i haven't encountered the issue of FF bringing up the session restore page.
could it be that somehow the update "corrupts" the files in the profile folder and tricks FF to enable the session restore feature?

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Kashav, is this something you might be interested in looking at?

Comment 18

[Daniel Bodea [:danibodea]]

I have managed to reproduce this using the steps in duplicate bug 1752208 on a system with Win10 and Cisco AMP installed, while it does not reproduce on another system that does not have Cisco AMP. It may as well be the cause.

Unfortunately, this issue does NOT reproduce with mozregression builds (neither with "normal " restarts from about:profiles, nor with "retry commands in mozregression, with or without the profile argument (argument to use the same profile).
This being said, I've tried my best to manually find a regression range and the result I found is that the bug was introduced with the second build of Nightly v96.0a1 from 25th of November (2021-11-25); BUild ID: 20211125214104.

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

:danibodea, since this bug is a regression, could you fill (if possible) the regressed_by field?
For more information, please visit auto_nag documentation.

Comment 20

[Pablo]

Hi , this is what i got

would this help?

26:34.19 INFO: Narrowed integration regression window from [8aafc062, 785d636b] (3 builds) to [8aafc062, f54285c0] (2 builds) (~1 steps left)
26:34.19 INFO: No more integration revisions, bisection finished.
26:34.19 INFO: Last good revision: 8aafc0622ff9ef46ff2d1d2885961205fd905368
26:34.19 INFO: First bad revision: f54285c09367394da49160ee8970f4e1b66c5f5b

26:34.19 INFO: Pushlog:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=8aafc0622ff9ef46ff2d1d2885961205fd905368&tochange=f54285c09367394da49160ee8970f4e1b66c5f5b

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Mathew, do you have cycles to look at this?

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Also, can people who are affected by this confirm which (if any) antivirus software they have installed?

Comment 24

[:Gijs (he/him)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #23)

Also, can people who are affected by this confirm which (if any) antivirus software they have installed?

Also also, can people who can reproduce check about:crashes after reproducing, and link any crash reports they find there from the 3 close+reopen sessions they have with Firefox before this appears? I'm wondering if we're crashing (silently) on shutdown while trying to write the session store recovery file.

Comment 25

[Pablo]

Me and Dani bodea have Cisco AMP for endpoints ( i have installed v7.3.15.20174)
and in about crashes there is nothing new, i only have an old crash from june 2021

Comment 26

[NewDino]

I am using Avast Free antivirus as well as built in Windows security..

Comment 27

[MasterHarp]

I am using WebRoot. Even after 96.0.3 this issue is still intermittent. Mostly fails but occasional "works."

Comment 28

[alexios]

eset smart security premium 15.0.23

Comment 29

[:Gijs (he/him)]

(In reply to Pablo from comment #7)

i am able to reproduce it everywhere, latest nightly, beta and release.

You just have to:
Open firefox and create a new profile, and launch it.
then close firefox and launch again with the same profile,
then close firefox and launch again with the same profile
then close firefox launch again with the same profile.

in some cases it took 4 launches and in some cases 3 launches for the restore screen to appear. but i can get it 100% of the times with those steps

I tried this with "AVAST One" installed on a Win7 VM (per comment 26), and I cannot reproduce. Are you using the profile manager with every launch, or no? How are you launching Firefox? Can you check the contents (list of files) of the profile directory's sessionstore-backups folder before each launch, and tell me what the contents look like on each launch, especially the one where it breaks?

If you change sessionstore.max_resumed_crashes in about:config to, say, 5, does the issue become harder to reproduce?

Comment 30

[Pablo]

Hi Gijs

these are all the answers:

-Antivirus is installed on a win10 machine
-Im always use profile manager.
-The way i do it is:

1. Unzip firefox zip on desktop enter the firefox folder lets say "Fx98" folder on the desktop
2. I right click on the exe file and i select "create shortcut"
3. I edit the shortcut file by right click on it, and select PROPERTIES
4. Then edit target to have "-p": \nightly\fx98\firefox\firefox.exe -p
5. then i launch shortcut, and i create a new profile.

-For the other question i have attached sessionstore-backups.zip
1st launch and close, then copied the folder and zipped it : first launch shows no restore
2dn launch and close, then copied the folder and zipped it : Second launch shows no restore
3rd launch and close, then copied the folder and zipped it : Third launch shows restore screen

-For the last question about sessionstore.max_resumed_crashes to 5
YES, harder to reproduce, it took about 7 launches for the restore screen to appear.

Comment 31

[Pablo]

Attachment: sessionstore-backups.zip

Comment 32

[:Gijs (he/him)]

(In reply to Pablo from comment #30)

Hi Gijs

these are all the answers:

-Antivirus is installed on a win10 machine
-Im always use profile manager.
-The way i do it is:

1. Unzip firefox zip on desktop enter the firefox folder lets say "Fx98" folder on the desktop
2. I right click on the exe file and i select "create shortcut"
3. I edit the shortcut file by right click on it, and select PROPERTIES
4. Then edit target to have "-p": \nightly\fx98\firefox\firefox.exe -p
5. then i launch shortcut, and i create a new profile.

-For the other question i have attached sessionstore-backups.zip
1st launch and close, then copied the folder and zipped it : first launch shows no restore
2dn launch and close, then copied the folder and zipped it : Second launch shows no restore
3rd launch and close, then copied the folder and zipped it : Third launch shows restore screen

-For the last question about sessionstore.max_resumed_crashes to 5
YES, harder to reproduce, it took about 7 launches for the restore screen to appear.

Thanks, this is really helpful.

I've confirmed that session store thinks at least one of these sessions did not complete shutdown successfully (ie not without crashing). It turns out we use a separate file for that, sessionCheckpoints.json, directly in the profile dir. Can you retrace those steps and include the contents after that file on each run? Also, can you clarify in the naming of the directories, are those files after the Nth run or before the Nth run? :-)
(I assume "after" but just making sure)

For the benefit of engineering following along: we must be hitting https://searchfox.org/mozilla-central/rev/fe800a7fd291db3c4b3e498cfe12ef2097662290/browser/components/sessionstore/SessionStore.jsm#818-824 and showing about:sessionrestore. This can only happen if we're incrementing _recentCrashes, which itself comes from the session restore file. To increment that, willRestoreAsCrashed() must be returning true, which checks sessionType, which in turn means we must be hitting this case where we're missing a checkpoint in the checkpoints file for "sessionstore-final-state-write-complete". Those checkpoints get written by crashmonitor to sessionCheckpoints.json based on observer topics.

Barret, it seems the CrashMonitor adds a shutdown blocker via IOUtils. Are we sure that's guaranteed to happen after these topics have all come in? Because I guess if we're stopping the crash monitor before we send out the "sessionstore-final-state-write-complete" that could potentially explain why we're not including that in the checkpoints file and then think we've crashed, or something. That's just a guess, but it's hard to be sure of much without being able to repro locally. Sadly there's no decent logging inside session restore and/or the crash monitor code. :-(

Comment 33

[Pablo]

Sure Gijs

attached is sessionstore-backups2.zip

Those files on the zip are a copy AFTER i Launch+Close Firefox.
I included sessionCheckpoints.json for each run also

let me know if you need something else, i think this is what you wanted.

Comment 34

[Pablo]

Attachment: sessionstore-backups2.zip

Comment 35

[Mathew Hodson]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #22)

Mathew, do you have cycles to look at this?

I won't be able to work on this. Maybe this issue is similar to bug 1744794 and bug 1740301.

Comment 36

[Ulli Conrad]

Sophos AV Enterprise runs on the affected machine

Comment 37

[Alice0775 White]

I can reproduce the issue with the str comment#7(but after the 14th restart) Nightly98.0a1(20220203003951) windows10 Home x64 21H2 + Windows Defender.

Comment 38

[:Gijs (he/him)]

(In reply to Pablo from comment #33)

Sure Gijs

attached is sessionstore-backups2.zip

Those files on the zip are a copy AFTER i Launch+Close Firefox.
I included sessionCheckpoints.json for each run also

let me know if you need something else, i think this is what you wanted.

Thanks, yes. All of these lack the sessionstore final state write info, which explains why we think we've crashed. The next question is why that information is missing, ie why we either wrote those files before that notification went out, or stopped listening to the notification, or never sent it, or what. I don't have a good answer for that at the moment. I'm hoping Barret can help.

While we're trying to get to the bottom of this and fix it properly, I think that for people who are running into this, there are two options for a temporary workaround:

• setting browser.sessionstore.max_resumed_crashes to some very high number. The downside there is that if you get into a crash loop (ie a site causes Firefox to crash, and restarting we load the same site again and crash again, etc.) it's harder to get out of. With e10s and fission, this should really not happen anymore (ie we might crash a content process but not the parent process), but of course, never say never...
• turn off browser.sessionstore.resume_from_crash in about:config altogether. The downside to doing that is that then if Firefox actually crashes, we won't automatically restore your session. I think in that case it'll still be available from History > Restore Previous session (seems to work with a quick check with about:crashparent, anyhow) but obviously your mileage may vary, use at your own risk, etc... I only mention it because I imagine it's frustrating if this happens a lot.

I'd also be curious if someone could confirm if enabling automatic session restore (ie check "Open previous windows and tabs" in Firefox's settings) makes the problem go away.

Comment 39

[Pablo]

i enabled "Open previous windows and tabs" in settings and then launched Firefox latest nightly 98 many times and i did not see the issue again.

Comment 40

[Beth Rennie [:beth] (she/her)]

We presently don't have an ordering guarantee that "sessionstore-final-state-write-complete" will be sent before we hit the shutdown blocker in CrashMonitor.jsm. This is because "sessionstore-final-state-write-complete" is triggered off AsyncShutdown.profileBeforeChange. AsyncShutdown.profileBeforeChange will guarantee that IOUtils.profileBeforeChange completes before it does, but that is it.

We can add some new ordering guarantees by adding a new blocker/barrier and synchronizing around that.

Comment 41

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1749996 - Ensure CrashMonitor writes sessionstore final checkpoint before IOUtils shuts down r?Gijs

Comment 42

[Atila Butkovits]

Backed out for causing failures at IOUtils.

Backout link: https://hg.mozilla.org/integration/autoland/rev/bf8468870df6f2a41eadb3b5a24e9db79d9a5c3f

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=AR99WKFARaiJ44FMnBFcOg.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=d2951e5c732e538677c059135c4904bf4933f099

Failure log:
https://treeherder.mozilla.org/logviewer?job_id=366745825&repo=autoland&lineNumber=2946
https://treeherder.mozilla.org/logviewer?job_id=366746181&repo=autoland&lineNumber=1934

Comment 43

[u608768]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #17)

Kashav, is this something you might be interested in looking at?

Looks like Gijs and Barret figured it out.

Comment 44

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/ecb12fc81f7a

Comment 45

[Pablo]

tried https://hg.mozilla.org/mozilla-central/rev/ecb12fc81f7a
on windows10 restarting many times and i did not see the restore screen anymore.

Comment 46

[Ryan VanderMeulen [:RyanVM]]

This fix is also on the Beta channel now if people are able to test it there.

Comment 47

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

I'm pretty sure that it fixed issue in my case, but report back from more people will be needed to confirm this for sure.

Comment 48

[Pablo]

I tested with windows 10 64bit (has antivirus Cisco AMP for endpoints ) using Firefox Beta 98.0b3 (64-bit) and i am not able to reproduce it anymore

but yea, it would be nice to receive feedback from other users that have other antivirus installed on their machine to see if it is fixed to other antivirus.

Comment 49

[MasterHarp]

Just installed 97 with WebRoot. As of this writing the problem has not reoccurred. Will continue to test.

Comment 50

[:Gijs (he/him)]

(In reply to MasterHarp from comment #49)

Just installed 97 with WebRoot. As of this writing the problem has not reoccurred. Will continue to test.

To be clear, the fix in this bug is not included in 97 - as noted in comment #46, it is in the beta channel (Firefox 98, https://beta.mozilla.org ).

Comment 51

[MasterHarp]

Presently not doing beta. When will 98 be available?

Comment 52

[Pablo]

Verified fix on Beta 98.0b5 (64-bit) using windows10 and Cisco AMP

Comment 53

[Beth Rennie [:beth] (she/her)]

Comment on attachment 9262188 [details]
Bug 1749996 - Ensure CrashMonitor writes sessionstore final checkpoint before IOUtils shuts down r?Gijs

Beta/Release Uplift Approval Request

• User impact if declined: Release users may see a dialog that we crashed and lost their tabs when we in fact did neither.
• Is this code covered by automated tests?: Unknown
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This fixes a race condition to be deterministic.
• String changes made/needed:

Comment 54

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9262188 [details]
Bug 1749996 - Ensure CrashMonitor writes sessionstore final checkpoint before IOUtils shuts down r?Gijs

This has gotten some bake time and QA verification. Given the number of dupes, let's go ahead and give it a shot with 97. Approved for 97.0.1.

Comment 55

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/0f0ba6e8029d

Comment 56

[Ryan VanderMeulen [:RyanVM]]

Added to the 97.0.1 release notes.

Fixed an issue causing users to see the Restore Session screen unexpectedly when starting Firefox

Comment 57

[Monica Chiorean]

I was able to reproduce the issue on Win10 with Cisco AMP using NB 96.0a1 (20211129215324).
Verified as fixed on Win10 with Cisco AMP using 97.0.1 (20220216172458).

Comment 58

[MasterHarp]

As the reporter of this bug I post these comments. Thanks to all those who helped find this elusive bug. 97.0.1 appears to finally quash it.