Open Bug 1789326 Opened 3 years ago Updated 1 years ago

All folders missing from all accounts after upgrade from 91.13 to 102.2.1, caused by dead symlinked calendar-data

Categories

(Thunderbird :: Folder and Message Lists, defect)

Product:
Thunderbird
Component:
Folder and Message Lists
Version:
Thunderbird 102
Type:
defect

Tracking

(Not tracked)

Status:
UNCONFIRMED

People

(Reporter: brad, Unassigned)

References

Details

(Whiteboard: [workaround: comment 19,21][datalossy])

Attachments

(1 file)

errorconsole-102.4.2.jpg
276.31 KB, image/jpeg
Details

Steps to reproduce:

Upgraded from 91.13

Actual results:

All folders in folder pane missing. Also, all QuickFolder links broken. Downgrading back to 91.13 resolves problem. Reproducible again and again.

Expected results:

All folders visible

If you installed to another folder you get a new profile. You can use the old one by starting with the profile manager (thunderbird.exe -p), or through Help | More Troubleshooting information | about:profiles

Was that the case? If not, can you reproduce with Help | Troubleshoot mode?

Not the case. I would need guidance for Help|Troubleshoot mode

If the main menu isn't showing already, hold Alt, then choose Help and go from there.

There have been a few reports of this occuring in the Thunderbird Support Forum. Depending upon other settings and info, three possible solutions have been discovered.

  1. In some cases, Thunderbird acts like it has no mail accounts and asks you to create a new account. The 'profile' folders show the accounts and mbox files exist. There are no accounts listed in the 'Account Settings' and the 'prefs.js' file has a low size. There are other 'prefs.js' files with a number and renaming 'prefs.js' to swap to using the one with highest number got accounts and therefore folders back.

  2. Others found Accounts all ok and they could send, nothing displayed in Folder Pane. A solution was found. Please rename files so they can be used for diagnosis, not delete them.

  • Access profile name folder
  • Exit Thunderbird and rename four files:
    • folderCache.json
    • folderTree.json
    • session.json
    • xulstore.json

  1. There are also cases where folder pane was completely empty, accounts were still in Account Settings and they could send. Performing fixes 1 and 2 did not work.
    The user in this case found that deleting just one account and then adding it back, got all the mail accounts and folders displaying again.

  2. In your case you have an addon involved, so there is a potential additional fix.
    re : Also, all QuickFolder links broken.
    As you use an addon which is related to Folders.
    Remove the addon and restart Thunderbird.
    If all is ok, then the addon is the cause.
    Access Quickfolders addon webpage and download up to date version.

If you need more detailed help, then please ask the question in the Thunderbird Support Forum.
Have you asked a question regarding this in the Thunderbird Support forum?
If yes, please supply a link to the question.

Flags: needinfo?(brad)

Magnus
We are getting a lot of reports of missing folders, accounts, emails.
Just a couple :
https://support.mozilla.org/en-US/questions/1388501
https://support.mozilla.org/en-US/questions/1387443

It sounds like a mix of add-ons doing/having done/stored something that turns out non functional, and some cases of likely file corruption for whatever reason. I don't have much of a solution for either. Perhaps Troubleshoot mode should be extended to take care of the files you mention...

(In reply to Anje from comment #4)

There have been a few reports of this occuring in the Thunderbird Support Forum. Depending upon other settings and info, three possible solutions have been discovered.

  1. In some cases, Thunderbird acts like it has no mail accounts and asks you to create a new account. The 'profile' folders show the accounts and mbox files exist. There are no accounts listed in the 'Account Settings' and the 'prefs.js' file has a low size. There are other 'prefs.js' files with a number and renaming 'prefs.js' to swap to using the one with highest number got accounts and therefore folders back.

  2. Others found Accounts all ok and they could send, nothing displayed in Folder Pane. A solution was found. Please rename files so they can be used for diagnosis, not delete them.

  • Access profile name folder
  • Exit Thunderbird and rename four files:
    • folderCache.json
    • folderTree.json
    • session.json
    • xulstore.json
      ...

Number 2 above solved the issue!

Flags: needinfo?(brad)

I backed up those files yesterday. Would you like the old ones and the newly ones? Note that folderTree.json was not re-created.

Yes please.

We previously had similar symptoms in Bug 1785812 - The mail folders and emails weren't shown in Thunderbird Daily 105.0a1 (2022-08-18)

That was a temporary regression. Cause wouldn't be the same.

fyi I have edited comment 4 and 7 to include "Please rename files so they can be used for diagnosis, not delete them."

(In reply to Anje from comment #4)

There have been a few reports of this occuring in the Thunderbird Support Forum. Depending upon other settings and info, three possible solutions have been discovered.

  1. In some cases, Thunderbird acts like it has no mail accounts and asks you to create a new account. The 'profile' folders show the accounts and mbox files exist. There are no accounts listed in the 'Account Settings' and the 'prefs.js' file has a low size. There are other 'prefs.js' files with a number and renaming 'prefs.js' to swap to using the one with highest number got accounts and therefore folders back.

  2. Others found Accounts all ok and they could send, nothing displayed in Folder Pane. A solution was found. Please rename files so they can be used for diagnosis, not delete them.

  • Access profile name folder
  • Exit Thunderbird and rename four files:
    • folderCache.json
    • folderTree.json
    • session.json
    • xulstore.json

  1. There are also cases where folder pane was completely empty, accounts were still in Account Settings and they could send. Performing fixes 1 and 2 did not work.
    The user in this case found that deleting just one account and then adding it back, got all the mail accounts and folders displaying again.

  2. In your case you have an addon involved, so there is a potential additional fix.
    re : Also, all QuickFolder links broken.
    As you use an addon which is related to Folders.
    Remove the addon and restart Thunderbird.
    If all is ok, then the addon is the cause.
    Access Quickfolders addon webpage and download up to date version.

I tried point 2, point 3 with no luck. I do not have any addon installed.
I have 4 email accounts; if I remove one and recreate it, folders and emails reappear.

But when I close TB and reopen it, I still get all blank (no folders, no email, but accounts are present).

I also tried to follow suggestions of [Bug 1785812] (https://bugzilla.mozilla.org/show_bug.cgi?id=1785812) regarding Troubleshoot Mode, but still no success.

I also tried to uninstall TB, make a new installation (Thunderbird 102.3.0 on Debian Testing) with totally new profile.
But when I close it and reopen, I got the same issue: no folders and no emails at all.

No success after trying all approaches in Comment 4.

I can observe that Thunderbird is still receiving and filtering emails, which filters are moving to other folders on the mail servers.

Everything else works, just not the Home page display and UI interactions.

Feedback from Support Forum: We are seeing quite a few of these cases and it is not OS specific. The fix of deleting various profile files seems to help some people, but there are still those where this is not a fix. In some instances, people create a new profile, add mail account and if using pop account of Local Folders as storage move across the mbox files.
For those who canot get a fix by deleting specific files, I'm trying to find out if there are some cases where an addon is involved. I'll post back info if I can get people to provide info on what addon extensions or themes they are using.

@Anje These are the themes and addons I had installed at the time of the upgrade and initial problem (all removed during troubleshooting):

DeepDark for Thunderbird
FileLink Provider for Dropbox
Emoji
Deutsch (de) Language Pack
Monterail Full Dark 2 for TB>68
Send Later
English United States Dictionary
Signature Switch
Mailbox Alert (Version 0.29.1 for 91.0 - 91.*)

Possibly also relevant that two of my (30) Message Filters called Mailbox Alert.

It seems there is more than one problem which on the surface exhibits the same non display of accounts, folders and emails.

Some people find deleting these files work:

folderCache.json
folderTree.json
session.json
xulstore.json

Others - particulary those who used 'Unified' folders - need to delete:

folderCache.json
folderTree.json
virtualFolders.dat
session.json
xulstore.json
Click on 'Mail' folder and delete 'smart mailboxes' folder

Several people tried deleting the various files and had no success, but one has provided what they see in the error console. It seems to suggest an issue with the calendar.

Error Console I see the following:

Calendar: Can't create calendar for 27483b60-ac4a-42ed-a194-daa68f1c9d79 (storage, moz-storage-calendar://): [Exception... "Component returned failure code: 0x8052000b (NS_ERROR_FILE_CORRUPTED) [mozIStorageService.openDatabase]" nsresult: "0x8052000b (NS_ERROR_FILE_CORRUPTED)" location: "JS frame :: resource:///modules/calendar/CalStorageDatabase.jsm :: connect :: line 67" data: no] CalCalendarManager.jsm:536 NS_ERROR_FILE_CORRUPTED: Component returned failure code: 0x8052000b (NS_ERROR_FILE_CORRUPTED) [mozIStorageService.openDatabase] CalStorageDatabase.jsm:67 NS_ERROR_FILE_CORRUPTED: Component returned failure code: 0x8052000b (NS_ERROR_FILE_CORRUPTED) [mozIStorageService.openDatabase] 2 CalStorageDatabase.jsm:67 Uncaught (in promise)

I'm chasing people for more info - I'll update this comment if I can find more information.

I'm chasing people for more info - I'll update this comment if I can find more information.

Anything new?

Flags: needinfo?(anjeyelf)

Feedback:
Discovered another reason for failure to show 'Folder Pane' contents.

'Calendar' - 'local.sqlite' file corrupted.
Person called 'tony.albrecht0' who had the 'Error Console' info as described in comment 19 followed my instructions to access the profile, open 'calendar-data' and replace the 'local.sqlite' file with a file found in the 'backup' folder. I had previously mentioned to then start Thunderbird using 'run as administrator'. This fixed and got Thunderbird working again displaying everything in Folder Pane.
ref: https://support.mozilla.org/en-US/questions/1391870#answer-1540868

Same instructions have now helped another person.
https://support.mozilla.org/en-US/questions/1390813#answer-1545312

Original instructions are still working for some people. (If it fails then check 'Error console' to see if 'Calendar' issue is present.)
Person called 'woodchucking'
Deleted :

Flags: needinfo?(anjeyelf)

Failure to show 'Folder Pane' contents and Error Console showed issue with Calendar.
The method to access the profile, exit Thunderbird, open 'calendar-data' and replace the 'local.sqlite' file with a file found in the 'backup' folder, solved issue in bug 1797147

It seems a some files may have been incorrectly updated because sometimes just deleting or using a backup file (in the cases with calendar issues ) fixes the situation.
But there are many cases when it is necessary to perform file deletions AND also start Thunderbird using 'Run as Administrator' which suggests a change in the permissions occured. When Thunderbird updated, maybe the permissions were not inherited, so the computer would not allow some files or folders in Thunderbird to run in the computer User Account.

Users have computer User Account set as 'Administrator'.

(In reply to Brad Lanier from comment #0)

Did you check the Local Directory paths under account server settings? I had folders disappear after upgrading from 91.13.1 to 102.4. bug 1800710
In my case I suspect the problem was due to a symlink in the paths.

(In reply to Jim Conyngham from comment #24)

(In reply to Brad Lanier from comment #0)

Did you check the Local Directory paths under account server settings? I had folders disappear after upgrading from 91.13.1 to 102.4. https://bugzilla.mozilla.org/show_bug.cgi?id=1800710
In my case I suspect the problem was due to a symlink in the paths.

That bug mentions :

In the “Folders” tree view in Thunderbird there were root entries for those, but they had no folder children under them and could not be expanded.'

Are you saying you could see the mail account names in the Folder Pane but no folders? This bug sees nothing in Folder Pane.
I believe your case it is a different situation and it would not be fixed by the methods which are fixing this particular bug.

After deleting all the advised files - still problem.

folderCache.json
folderTree.json
virtualFolders.dat
session.json
xulstore.json
also deleted virtualFolders-1.dat

User posted Error Console information.

Ensured Thunderbird was installed ok.
Repeated the deletion of files.
This time advised to start Thunderbird 'Run as Administrator' and all fixed.
https://support.mozilla.org/en-US/questions/1388962

The error console may show what files could not be accessed as permissions not inherited.

So apparently there are situations (non-admin install?) where "UpdRootD" can't be read. We shouldn't end up here
https://searchfox.org/mozilla-central/rev/d7d2cc647772de15c4c5aa47f74d25d0e379e404/toolkit/modules/UpdateUtils.sys.mjs#855 ... but I don't know if that is causing the problem at hand. Could cause other issues... Kirk, something to improve there?

The other error points to an exception at https://searchfox.org/comm-central/rev/3cf2b87f0268f9b7e797bf875b816ed12d2f9c98/mail/base/content/specialTabs.js#1098 so something with this profile is confused about whether it has updater or not. Maybe due to the above.

Flags: needinfo?(bytesized)

To me, this error looks very similar to what we encountered in Bug 1756450. It's a long bug, but Comment 63 and Comment 30 do a reasonable job of summarizing what was going on there.

Someone that is affected by this could try the diagnostic icacls commands that I gave in Comment 49 and see if their permissions are screwed up the same way. We never were able to determine how those permissions got to be like that.

In that bug, I ultimately fixed the two main update interfaces so that they would show an error message in this situation. But it is still the case that, if we can't create the update directory, we can't initialize nsIUpdateManager. It would be good to do something about this, but I'm not entirely sure what the right thing to do is. We could catch the errors internally. But pretty much no update components will be able to do anything meaningful without an update directory so we would only succeed in constructing interfaces that would be nearly useless. Ideally, I would like Firefox to be a bit more proactive about surfacing these problems to the user. I filed Bug 1787310 to do this, but unfortunately it isn't a high priority for our team right now.

(In reply to Magnus Melin [:mkmelin] from comment #27)

... but I don't know if that is causing the problem at hand.

I can't say for sure here. But if an error being thrown from update code could cause this problem, then it's certainly possible. I don't know if I'd recommend doing this outside of a VM, but I tested out my UI fixes for this issue by changing the permissions on C:\ProgramData to prevent regular users from creating directories in it and then renaming the existing update directory. Might be worth trying to see if doing that reproduces the other symptoms of this problem.

Flags: needinfo?(bytesized)
Severity: -- → S2
Whiteboard: [datalossy]
See Also: → 1797147

With help from the information in this ticket (thanks!), I managed to solve the problem.

After seeing the calendar-related errors in the error console (similar to, though not exactly the same as, those mentioned in comment 19, I investigated. I'm not using the calendar, but until a few years ago I did, and I had symlinked calendar-data to somewhere else, and it was now a dead link. I removed that link and started Thunderbird (still 91). It created a new calendar-data directory alright and stopped showing related errors in the error console. I then proceeded to upgrade Thunderbird and everything seems to have worked fine.

Just to be clear, this wasn't any trivial disruption. A few months ago, when it automatically updated Thunderbird and everything disappeared, I had to manually reinstall 91, find out how to stop it from automatically updating, and bring back my destroyed profile from backup (although I have everything on IMAP, recreating these accounts would mean I'd lose my message filters and who knows what other stuff [maybe years of spam training?]). I found this ticket and kept an eye on it. Today I felt like giving it a careful fresh try. For less technical users this can be a great pain. I admit that it was my being technical that triggered this bug in the first place. Still, it's a critical data loss bug.

Whiteboard: [datalossy] → [workaround: comment 19,21][datalossy]
Summary: All folders missing from all accounts after upgrade from 91.13 to 102.2.1 → All folders missing from all accounts after upgrade from 91.13 to 102.2.1, caused by dead symlinked calendar-data

(In reply to Magnus Melin [:mkmelin] from comment #27)

So apparently there are situations (non-admin install?) where "UpdRootD" can't be read. We shouldn't end up here
https://searchfox.org/mozilla-central/rev/d7d2cc647772de15c4c5aa47f74d25d0e379e404/toolkit/modules/UpdateUtils.sys.mjs#855 ... but I don't know if that is causing the problem at hand. Could cause other issues... Kirk, something to improve there?

The other error points to an exception at https://searchfox.org/comm-central/rev/3cf2b87f0268f9b7e797bf875b816ed12d2f9c98/mail/base/content/specialTabs.js#1098 so something with this profile is confused about whether it has updater or not. Maybe due to the above.

These are conclusions from Brad's data?

Flags: needinfo?(mkmelin+mozilla)
Summary: All folders missing from all accounts after upgrade from 91.13 to 102.2.1, caused by dead symlinked calendar-data → All folders missing from all accounts after upgrade from 91.13 to 102.2.1

Yes.

Flags: needinfo?(mkmelin+mozilla)
Summary: All folders missing from all accounts after upgrade from 91.13 to 102.2.1 → All folders missing from all accounts after upgrade from 91.13 to 102.2.1, caused by dead symlinked calendar-data
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: