2029240 - Local Firefox backup fails every time

Status: RESOLVED FIXED

(Firefox :: Profile Backup, defect, P1)

Description

[l33t_h4xx0r]

Attachment: Firefox Backup Settings.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:149.0) Gecko/20100101 Firefox/149.0

Steps to reproduce:

I set up Firefox some weeks ago to create local backups in the folder "D:\Backup\Firefox\Restore Firefox" and it worked (last time working March 11th), as I saw that there was a back file in the configured folder and the UI showed it was successful.

Actual results:

Today I noticed that it's no longer working, the UI says "Hmm, there was a problem backing up. Try again in a few minutes. Lean more" and the configured backup folder is empty.
When I click on "Backup now" no new backup is created, the task just runs a few seconds and then nothing.
I already dis-/enabled the backup with no success, also a different folder (e.g. the default Documents folder "C:\Users\<Username>\Documents\Restore Firefox") doesn't work.
The NTFS permissions for the backup folder didn't change since it was working last time.
There were no significant changes on my OS (Windows 10 build 10.0.19045.7058) since then, March 2026 Windows updates got installed on March 10th and since then just minor updates of other software.
Firefox got updated to v149 on March 24th, but the issue started with v148.0.2 (installed on March 10th).

Expected results:

Firefox backup should work.

Comment 1

[l33t_h4xx0r]

Attachment: about:support raw data from affected profile

Comment 2

[l33t_h4xx0r]

Attachment: Firefox new profile no backup file after initial backup set up

Comment 3

[l33t_h4xx0r]

Attachment: Firefox new profile manual backup successful

Comment 4

[l33t_h4xx0r]

I can't reproduce in a new profile, about:support information from the offending profile is attached.

But I noticed a faulty behaviour after enabling the backup, there was no file generated although the UI told me so.
When I manually click on "Backup now" it is generated (see screenshots attached).

Comment 5

[l33t_h4xx0r]

I tested the troubleshoot mode and clear startup cache from about:support with the affected profile, none worked.

Comment 6

[:Gijs (he/him)]

Thanks for the report! Could you check the value of browser.backup.errorCode in about:config in the affected profile?

Also, are there any errors in the browser console (you can open it with ctrl-shift-j; this is different from the "regular" devtools!).

Harshit: I was hoping this would be in about:support but it's not. Can we file a separate bug to fix that, adding any prefs relevant to knowing/debugging backup (without revealing PII like paths/directories/filenames)?

Comment 7

[l33t_h4xx0r]

browser.backup.errorCode is 9 and yes there are errors, see the attached console log export.
I haven't touched these stale snapshot items in my user profile in case you want some more infos from them, but I can confirm that they exist.

Comment 8

[l33t_h4xx0r]

Attachment: console-export-2026-4-13_22-55-10.log

Comment 9

[:Gijs (he/him)]

(In reply to l33t_h4xx0r from comment #7)
Thanks a lot for the super quick response.

browser.backup.errorCode is 9 and yes there are errors, see the attached console log export.

Well, funny thing, that's "unknown error" so that's... not as helpful as it could have been.

However! The log is very helpful. It has a bunch of:

Exception: OperationError: Could not remove `[xpconnect wrapped nsIFile]': could not parse path (NS_ERROR_FILE_UNRECOGNIZED_PATH) BackupService.sys.mjs:2027:27

which points to this code.

It takes a bit of staring, but actually, the problem is kind of obvious once you really read what the thing is saying. We're calling IOUtils.remove() with an nsIFile instance, instead of a path.

This code has been here for 6 months. I am kind of confused how this wasn't caught at all so far. Do we not normally run this? It can't ever succeed...

(as an aside, this code is also confused because it's getting an nsIFile instance async but then calling the (synchronous IO!) directoryEntries property on it - it should be using the async IOUtils.getChildren() if it's trying to avoid sync IO)

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1985141

Comment 11

[Harshit Sohaney [:hsohaney]]

Filed a bug to add to about:support!

Comment 12

[David Parks [:handyman]]

(In reply to :Gijs (he/him) from comment #9)

It takes a bit of staring, but actually, the problem is kind of obvious once you really read what the thing is saying. We're calling IOUtils.remove() with an nsIFile instance, instead of a path.

This code has been here for 6 months. I am kind of confused how this wasn't caught at all so far. Do we not normally run this? It can't ever succeed...

(as an aside, this code is also confused because it's getting an nsIFile instance async but then calling the (synchronous IO!) directoryEntries property on it - it should be using the async IOUtils.getChildren() if it's trying to avoid sync IO)

We have a ton of tests for a bunch of conditions but this case is one that is not checked properly. We actually have tests that show that we are robust to this error, up to the limit set by browser.backup.max-num-unremovable-staging-items, which is what happened here. The removal is a make-up attempt to fix something that has previously gone wrong. Staging folders (e.g. staging-0, staging-1, ... in the log) don't normally survive the backup process -- they get renamed. So both points in comment 9 are valid (at one point it's missing .path and is not async), but we probably haven't seen this because it's an unusual failure case.

Comment 13

[Stephen Thompson [:sthompson]]

I can see how I missed that since I probably would not have picked up on the path string vs. nsIFile argument usage.

Normally, the last step of backup creation is removing the snapshot folder's subdirectory that was used to stage the backup, so this cleanup code was already intended as a backstop in case a snapshot couldn't be cleaned up previously.

The test on the cleanup code patch is validating that there's an error when trying to delete those folders. The test code holds an open file in each folder to simulate a case where a folder can't be deleted. Locally, I changed the code to just seed normal, empty, deleteable folders into the snapshots folder, and the test still passed. The test asserts that createBackup rejected with a message matching /Failed to remove/. I can see the Failed to remove stale snapshot item /var/folders/6n/qwvj4q7x3dzf6r_mmpdvjgxw0000gn/T/firefox/xpcshellprofile/profileDir/backups/snapshots/test-1. Exception: OperationError: Could not remove [xpconnect wrapped nsIFile]: could not parse path (NS_ERROR_FILE_UNRECOGNIZED_PATH) error from the test, but I believe the test was trying to catch the message like Failed to remove 6 items from C:\Users\Raerka\AppData\Roaming\Mozilla\Firefox\Profiles\ae02vs94.default\backups\snapshots from the logs in comment 8. The test is passing since the BackupService error message is using the same "Failed to remove" sentence prefix as the PathUtils::InitFileWithPath failure.

I think the code and test need fixing. I did the simple fix for the code. I tried to update the test, but it's seeming like the recursive IOUtils.remove is able to remove the folders with the open file in it despite the non-recursive IOUtils.remove on the folders throwing the expected exception.

Comment 14

[Stephen Thompson [:sthompson]]

Attachment: WIP: Bug 2029240 - backup staging folder cleanup never happens

Comment 15

[David Parks [:handyman]]

l33t_h4xx0r, can you think of a reason the [...]\backups\snapshots\ temp folder path found in your log would have changed permissions? That's not something that should normally happen. If you can remove the contents of that folder (it won't harm anything), then your automatic backups should resume, if they can write to that spot. The problem in the log seems to be that Firefox had a few backups that failed (for some unknown reason) and the code to clean up after that is buggy, and Firefox is giving up instead of making a bigger mess. So you can delete those backups and we can fix the cleanup process but the original failure is the failed backups that were accumulating, and there is no information available about that. OTOH, the problem with that assumption is that it doesn't match with the fact that you are using a new profile, so the accumulated backups in the old profile shouldn't be relevant. Since your log certainly shows some accumulated backups, I'm thinking maybe they were from earlier experiments you ran with the new profile, and they are covering up more significant failures.

If you don't know why the backups to that folder would fail, then you can try collecting a profile, which might point to something. To do that, you would:

1. Create yet another new profile. Don't enable auto backups yet.
2. (All in the new profile) Go to about:config . Dismiss the warning. Enter browser.backup.scheduled.minimum-time-between-backups-seconds in the search field and change the number from 86400 (24 hours) to 30 (30 seconds).
3. Go to about:profiling . Under "Profiler Settings", choose "Debug". Ignore the rest.
4. Go to about:logging . Change "New log modules:" to "timestamp,sync,BackupUIParent:5". Capitalization is important.
5. Finally, we're ready to capture a profile of the failure. Go to preferences and search for backup. Don't start it yet.
6. Press Ctrl + Shift + 1 to start the profiler.
7. Now turn on the auto backups. Wait for two backup attempts to happen -- the first one starts immediately so you shouldn't be waiting more than a half a minute or so.
8. Pres Ctrl + Shift + 2 to stop the profiler and generate a profile.
9. In the upper-right of the profile is a button to "Upload local profile". Press that to send it to Mozilla and paste the URL it generates into a comment here.

Comment 16

[David Parks [:handyman]]

(In reply to Stephen Thompson [:sthompson] from comment #13)

[...] I believe the test was trying to catch the message like Failed to remove 6 items from C:\Users\Raerka\AppData\Roaming\Mozilla\Firefox\Profiles\ae02vs94.default\backups\snapshots from the logs in comment 8. The test is passing since the BackupService error message is using the same "Failed to remove" sentence prefix as the PathUtils::InitFileWithPath failure.

Ahh, a partial string match error. :sigh: Good catch.

Comment 17

[l33t_h4xx0r]

Attachment: Content of snapshots folder

Comment 18

[l33t_h4xx0r]

(In reply to David Parks [:handyman] from comment #15)

I checked the snapshots folder and its contents, all files and folders have the usual inherited NTFS permissions and none are read-only (see the attached "snapshots folder files metadata.txt"). Some of them are named as corrupt and from 2017 though.
And I can confirm that by only deleting the "staging-3" folder the backup was successful again, however staging-0, staging-1 and staging-2 didn't get cleaned up in this process if it should have been, they're still present in the snapshots folder.

OTOH, the problem with that assumption is that it doesn't match with the fact that you are using a new profile, so the accumulated backups in the old profile shouldn't be relevant. Since your log certainly shows some accumulated backups, I'm thinking maybe they were from earlier experiments you ran with the new profile, and they are covering up more significant failures.

I haven't used a new/second/different profile until right before I opened this bug, so all failed backup attempts are/were from my original one with id ae02vs94.default .

I'll do the backup test with a new profile you requested shortly.

Comment 19

[l33t_h4xx0r]

Here is the profiler result: https://share.firefox.dev/3Qpw96W

For some reason the profiler settings got changed from debug to networking after I stopped it, so I'm not sure if it contains the requested data.
And although I set browser.backup.scheduled.minimum-time-between-backups-seconds to 30, it took 3minutes to create a backup.

But the backup was successful every time, "...\Profiles\vi3g10lx.backuptest\backups\snapshots" is empty and the backup file under "D:\Backup\Firefox\Restore Firefox" is present.

Comment 20

[l33t_h4xx0r]

I think I found the root cause of this bug, simply close Firefox during the backup process and I guess that's how all the staging folder got created in my profile.
I was able to generate enough staging folders that the error from the initial screenshot is popping up again.

Comment 21

[David Parks [:handyman]]

Awesome investigation, thanks! It would be a pretty big coincidence that the browser kept closing during the backup process, but it's certainly possible. The fixes already in the works for this bug would resolve that situation -- they would remove the stale staging folders, so later backups would be able to work. The only potential hiccup would be if there was a reason you were not able to remove staging-0, 1, and 2, but I suspect you meant that you decided not to remove them, not that you couldn't. (If you can't then Firefox can't either.) Thanks again for the report.

Comment 22

[l33t_h4xx0r]

I thought of this possibility because the creation times of the staging folders seemed to coincide with the times when Windows rebooted and shut down.
Yeah I tested if a successful backup would clean up the staging folders, but apparently it doesn't (yet).

Well thanks too for developing a fix.

Comment 23

[David Parks [:handyman]]

Attachment: Bug 2029240: Part 1 - Fix FxBackup removal of stale staging items r=sthompson!

There was a missing .path reference in #prepareStagingFolder that caused
files to not actually get deleted. It was also doing some file access on the
main thread. This fixes those. We had tests that were supposed to check that
functionality but this little bit was papers over for testing, so it wasn't
caught. This patch adds a test that would catch the failure.

This patch also updates the regex in the test to match a more specific error
code, but this is just for clarity in the old test and is unrelated to this
bug.

Comment 24

[David Parks [:handyman]]

Attachment: Bug 2029240: Part 2 - Limit FxBackup file system error tests to Windows r=sthompson!

Except for Windows, supported OSs allow unlinking opened files, so this case
does not fail there.

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1985141

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/7e482a302ee4
https://hg.mozilla.org/integration/autoland/rev/ebb8562a2e0d
Part 1 - Fix FxBackup removal of stale staging items r=sthompson
https://github.com/mozilla-firefox/firefox/commit/cfa56a6ee298
https://hg.mozilla.org/integration/autoland/rev/f28b39825da7
Part 2 - Limit FxBackup file system error tests to Windows r=sthompson

Comment 28

[amarc]

https://hg.mozilla.org/mozilla-central/rev/ebb8562a2e0d
https://hg.mozilla.org/mozilla-central/rev/f28b39825da7

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:handyman, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox151 to wontfix.

For more information, please visit BugBot documentation.

Comment 30

[David Parks [:handyman]]

Attachment: Bug 2029240: Part 1 - Fix FxBackup removal of stale staging items

There was a missing .path reference in #prepareStagingFolder that caused
files to not actually get deleted. It was also doing some file access on the
main thread. This fixes those. We had tests that were supposed to check that
functionality but this little bit was papers over for testing, so it wasn't
caught. This patch adds a test that would catch the failure.

This patch also updates the regex in the test to match a more specific error
code, but this is just for clarity in the old test and is unrelated to this
bug.

Original Revision: https://phabricator.services.mozilla.com/D294550

Comment 31

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: FxBackup is trying to find some failure causes. This is believed to be related.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: The behavior is already broken. The code fix was simple.
• String changes made/needed?: N/A
• Is Android affected?: no

Comment 32

[David Parks [:handyman]]

Attachment: Bug 2029240: Part 2 - Limit FxBackup file system error tests to Windows

Except for Windows, supported OSs allow unlinking opened files, so this case
does not fail there.

Original Revision: https://phabricator.services.mozilla.com/D294551

Comment 33

[David Parks [:handyman]]

@

Comment 34

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/11a6ed4ebdd3
https://hg.mozilla.org/releases/mozilla-beta/rev/5ce51f19a28c
https://github.com/mozilla-firefox/firefox/commit/22467e56d7df
https://hg.mozilla.org/releases/mozilla-beta/rev/1c7c3c12f418

Comment 35

[Denys [:denyshon]]

Hey David, would we want to uplift this to release as well?

Comment 36

[David Parks [:handyman]]

I'm not really involved in the behind the scenes here, so I don't know what folks have in mind. Harshit?

Comment 37

[Harshit Sohaney [:hsohaney]]

yes please! if that's possible

Comment 38

[David Parks [:handyman]]

Attachment: Bug 2029240: Part 1 - Fix FxBackup removal of stale staging items

There was a missing .path reference in #prepareStagingFolder that caused
files to not actually get deleted. It was also doing some file access on the
main thread. This fixes those. We had tests that were supposed to check that
functionality but this little bit was papers over for testing, so it wasn't
caught. This patch adds a test that would catch the failure.

This patch also updates the regex in the test to match a more specific error
code, but this is just for clarity in the old test and is unrelated to this
bug.

Original Revision: https://phabricator.services.mozilla.com/D294550

Comment 39

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined/Reason for urgency: FxBackup is trying to find some failure causes. This is believed to be related.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: The behavior is already broken. The code fix was simple.
  String changes made/needed?: N/A
• String changes made/needed?: N/A
• Is Android affected?: no

Comment 40

[David Parks [:handyman]]

Attachment: Bug 2029240: Part 2 - Limit FxBackup file system error tests to Windows

Except for Windows, supported OSs allow unlinking opened files, so this case
does not fail there.

Original Revision: https://phabricator.services.mozilla.com/D294551

Comment 41

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/43a644ce9449
https://hg.mozilla.org/releases/mozilla-release/rev/b7e044b5741f
https://github.com/mozilla-firefox/firefox/commit/4d9d465921f2
https://hg.mozilla.org/releases/mozilla-release/rev/6e1592174b74

Comment 42

[Bogdan Maris, Desktop Test Engineering]

I tried reproducing this issue in order to verify the fix but unfortunately I was unable to do so.

Do you know of a way I can reproduce or simulate the error so I can check if this is fixed now?

Comment 43

[l33t_h4xx0r]

I can confirm that the issue is fixed, the "..\Firefox\Profiles\ae02vs94.default\backups\snapshots" folder gets cleaned up when starting a new backup now and it successfully creates the backup file.

@Bogdan force close Firefox during profile backup so that stale files and folders are in above path. Before the patch they didn't get removed and failed the backup task.