Open Bug 1997705 Opened 7 months ago Updated 3 months ago

TB botches IMAP folder move and delete operations, creates 'tombstone folders', maybe-dataloss

Categories

(MailNews Core :: Networking: IMAP, defect)

Thunderbird 144
defect

Tracking

(Not tracked)

UNCONFIRMED

People

(Reporter: aliledudiable, Unassigned)

References

Details

Attachments

(4 files, 1 obsolete file)

I have an Outlook365 account (as part of some organization). I work with folders a lot: create them, move into them, then finally move them to some archive folder.

I've noticed severe problems in TB's behavior vis-a-vis folders:

  • Often, when I rename a folder - I see the new-name folder with its contents, but I also see another, gray and italicized, folder with the old name. It appears empty, it cannot be deleted, and it cannot be moved. I call this a tombstone folder.
  • Some folders after clearing the Trash folder, show up as tombstones in the Trash, complete with all of their subfolders - again, all appearing empty except for subfolders.
  • Clearing the trash when it holds tombstone folders initially seems to work, but when I close and restart Thunderbird - the tombstones reappear. again.

Notes:

  • The tombstone folders do not appear to exist on the server - when using the Outlook web interface, I don't see them.
  • I tried playing with folder "subscription"; some tombstone folders were subscribed-to, or had subscribed-to subfolders, but unsubscribing did not remove them, nor help otherwise.
  • I don't remember how old this phenomenon is. I only remember it from the past year or so, but I might have just not done all of these folder-move-games in the past.

The most problematic part is that, in some cases, due to moving-and-renaming, I can find myself in a situation where a tombstone on TB's side hides an actual folder, with messages, on the IMAP server. That is why I mentioned a dataloss risk.

On outlook imap account, I just now tried to rename a 2nd level folder f5/tb-election to f5/tb-electron and it seemed to work (and left no tombstone or ghost folder). Then when I tried to open "tb-electron" I get a popup saying:

The current operation on 'tb-electron' did not succeed. 
The mail server for account user@outlook.com responded: 
f5/tb-electron doesn't exist.

In IMAP:4 log, the rename steps all look good but I see when tb-electron is imap SELECTed, outlooks says it doesn't exist. I then went to another non-outlook account and did a similar rename with no problems. Then went back to f5/tb-electron and it opens OK.
So looks like outlook has some imap problems but don't know if this is what is causing the reported problem.

I think you will see a "ghost" folder when the folder has the "no select" property. I'm not sure why exchange imap would leave around renamed folders marked "no select". It would be good to know exactly what exchange is reporting for ghost folders in its LIST response. If you see this again and if the gray folders stick around after restart, it would be good to record an IMAP:4 log that will show the folder discovery activity during startup and we can see how exchange is LISTing these "tombstone" folders.
Here's instruction for recording the IMAP:4 log: https://wiki.mozilla.org/MailNews:Logging

Another outlook/exchange imap issue:
Create a folder a/b/c
You get two ghost folder a and b and a normal folder c. You can't copy messages to a or b which is expected (they are "no select").
Rename c to cc works OK.
Now delete folder a to Trash (which renames it). Outlook complains that the resulting rename command has bad parameter.
Doing the same thing on two other imap servers works fine (my ISP's server openwave and dovecot for tb testing).

This reminds me of bug 1856536. But this (bug 1997705) doesn't seem to be regressed by that.
Reporter a*****?, if you find a way to consistently reproduce the bug, please let me know. And, to repeat, the startup IMAP:4 log showing the folder list responses with gray/italic folders might help.

Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
Attached image No-Select create.jpg

Do not have a outlook account to reproduce.
But in gmail webmail account I can deliberately create a 'no-select'.

I created a folder/label with the name 'Testing/Sub Test' as Nested under '[Gmail]/'
Please note 'Testing' at this point is not a label in it's own right.
You may find Thunderbird auto removes the first 'Testing' by design and you may see the 'Sub Test' as a top level folder.

Therefore:
Create a folder/label with the name 'Testing/' (note the forward slash) as Nested under '[Gmail]/'
Create a folder/label with the name 'Testing/Sub Test' as Nested under 'Testing/'
Perform a new Subscribe.
I do wish it was possible to add another image at this point - but I do not see how. So I've ahd to add it at the bottom again.

Attached image No-Select subscribe.jpg (obsolete) —

Shows how the label called 'Testing/Sub Test' displays the 'Testing' part as a 'no-select' folder because 'Testing' does not exist as a folder/label in it's own right.

Note what it says in the 'Subscribe' list: - - sorry wrong image - you need to be able to see 'Testing > Testing > Sub Test -I uploaded it but it did not actually replace in same comment .

In Thunderbird:

  • rename the no-select folder as 'Testing me'.
  • So in Folder Pane: Testing me contains 'Sub Test'
  • In Account Settings > Synchronisation & Storage - click on 'Advanced'

Actual REsults:
Shows original 'Testing' no select folder
'Testing' contains a 'no-select' subfolder called 'Testing me' which contains the normal subfolder called 'Sub Test'

In my profile folders gmail imap account folder you see:

  • 'Testing.msf' file
  • 'Testing.sbd' folder

The 'Testing.sbd' folder contains 'Testing me.sbd' folder, 'Sub Test.msf' and 'Testing me.msf'
The 'Testing me.sbd' folder contains 'Sub Test.msf'

Note: At no point does this 'Testing me' appear in gmail webmail account - nothing alters in that account.

Restart Thunderbird and note the changes.
Actual Results

  • The 'Testing me' no select folder has disappeared - as it was never displayed in webmail I'm not surprised, although some people might find that odd in itself.
  • 'Sub Test' becomes a top level folder

In my profile folders gmail imap account folder you see:

  • 'Testing.msf' file
  • 'Testing.sbd' folder
  • The 'Testing.sbd' folder now contains 'Sub Test' mbox file and 'Sub Test.msf'
  • The 'Testing me.msf' and 'Testing me.sbd' folder has disappeared.

In Account Settings > Synchronisation & Storage - click on 'Advanced'

  • There is the original 'Testing' no select folder which contains the normal subfolder called 'Sub Test'

Thunderbird auto hides/removes the top level 'Testing' no select file by design, so forcing 'Sub Test to appear as a top level folder.

Benefit:
In my case, to rename a 'no-select' folder and then restart Thunderbird actually removes it completely from the Folder Pane.

No-Select folders are not shown as no-select in webmail, but you might detect them if a subfolder still has it as first part of name eg: 'Testing/Sub Test' or if folder is still set as subfolder to an original folder subsequently renamed.

Attached image gmail-noselect.jpg
Attachment #9524168 - Attachment is obsolete: true

Comment on attachment 9524194 [details]
gmail-noselect.jpg

As you can see - Thunderbird auto hides the first 'Testing' no-select, it's the subfolder called 'Testing' which displays in Folder Pane as a no-select and it's that one which gets edited to 'Testing me' - and after restart it's killed off thus forcing 'Sub Test' to be a top level folder.

Attached image gmail-labels.jpg

This shows 'Create new label called 'Testing/' as sub to '[Gmail]/'
Create label called 'Testing/Sub Test' as sub to 'Testing/'

This creates the same a/b/c issue - Both of the 'Testing/' are no-select, but as Thunderbird auto removes the first you only actually see the second in Folder Pane.

(In reply to gene smith from comment #3)

Another outlook/exchange imap issue:
Create a folder a/b/c
You get two ghost folder a and b and a normal folder c. You can't copy messages to a or b which is expected (they are "no select").

I see the same - using gmail.

Now delete folder 'a' to Trash (which renames it).

I cannot delete my equivalent of 'a' - the top level 'no-select' folder because Thunderbird is not actually displaying it.
In folder Pane, I only see 'b' no-select with normal sub folder 'c'. This I discovered when I renamed 'b' aka 'Testing' to 'Testing me' which subsequently was killed/deleted 'b' aka 'Testing me' upon restart of Thunderbird.

In Thunderbird
Set 'IMAP server directory : [Gmail]'
user_pref("mail.server.server1.server_sub_directory", "[Gmail]");

I do use this to hide the [Gmail] folder. But I do notice the 'INBOX' no-select folder is also hidden which is desirable as I only want to see Inbox subfolders under the real 'Inbox' folder. So I presume the same code is not displaying the first 'Testing' no select folder as well.
Maybe set up in bug 1818592

In tb I can create a gmail folder aa/bb/cc but I only see bb and cc in tb, even after restart. I think tb is "hiding" aa for the same reason [gmail] is hidden. Gmail reports them as subscribed but non-existent. From IMAP:4 log:

* LIST (\HasChildren \NonExistent \Subscribed) "/" "[Gmail]"
* LIST (\HasChildren \NonExistent \Subscribed) "/" "aa"
* LIST (\HasChildren \NonExistent \Subscribed) "/" "aa/bb
* LIST (\HasNoChildren \Subscribed) "/" "aa/bb/cc"

I see the same LIST response after tb restarts.

I can rename bb to bbb and then rename cc to ccc.
I can't rename aa since it is "non-existent" per gmail LIST and doesn't display in tb (again, just like [gmail]), but not sure why bbb displays.
Still not seeing any residual "ghost/tombstone" folders hanging around after the rename, or on tb restart or after deleting bbb to Trash. And deleting bbb to Trash moves whole "aa/bbb/ccc" folder tree to Trash (but aa remains invisible in Trash).

Note: Unlike Anje's tests, I didn't do anything with gmail's webmail.

(In reply to gene smith from comment #2)

The current operation on 'tb-electron' did not succeed. 
The mail server for account user@outlook.com responded: 
f5/tb-electron doesn't exist.

This sometimes happens to me as well, but it hasn't happened to me with most of my tombstone folders. That is, I did not get any error message about them at any time.

If you see this again and if the gray folders stick around after restart,

They stick around even after I unsubscribe, shut down TB, delete the folder and .msf files, and restart! :-(

it would be good to record an IMAP:4 log

is there some script for anonymizing these logs? I would not like to publish the names and subjects of those emails, many of them are sensitive.

Two of my tombstone folders are named "Archives" and a subfolder of that named "2024".

Here are some lines from my IMAP:4 .moz_log file which match the word Archives:

2025-11-07 19:32:12.442684 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LIST (\HasChildren) "/" "Deleted Items/Archives"
2025-11-07 19:32:12.442687 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LIST (\HasNoChildren) "/" "Deleted Items/Archives/2024"
2025-11-07 19:32:13.077803 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LSUB (\HasNoChildren) "/" Archives/2024
2025-11-07 19:32:13.572828 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LSUB (\HasNoChildren) "/" Trash/Archives
2025-11-07 19:32:13.572854 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LSUB (\HasNoChildren) "/" Trash/Archives/2024
2025-11-07 19:32:50.577668 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LIST (\HasChildren) "/" "Deleted Items/Archives"
2025-11-07 19:32:50.577673 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LIST (\HasNoChildren) "/" "Deleted Items/Archives/2024"
2025-11-07 19:32:50.752157 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LSUB (\HasNoChildren) "/" Archives/2024
2025-11-07 19:32:50.785036 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:A:CreateNewLineFromSocket: * LSUB (\HasNoChildren) "/" Trash/Archives

I also looked for the word "select" in the logs and could not find anything like "no select" or "noselect". The closest thing was this line:

025-11-07 19:32:10.130109 UTC - [Parent 9732: IMAP]: I/IMAP 7fcb425f4800:outlook.office365.com:NA:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+

With a further search of bugzilla, I found the extremely-long titled bug 885162:

"Nonexistent IMAP folders listed not just in the folder pane but also in a refreshed subscribe window without check box for (un)subscribe (After "LIST (subscribed)" support by Bug 495318, there is no way in Tb to unsubscribe \NonExistent \Subscribed Mbox) "

which seems to be about the same tombstone-folder phenomenon.

  • LSUB (\HasNoChildren) "/" Archives/2024
  • LSUB (\HasNoChildren) "/" Trash/Archives
  • LSUB (\HasNoChildren) "/" Trash/Archives/2024

It looks like it's saying these folders are subscribed. But there is no similar LIST response for these folders at all in the log (based on what you have provided in comment 14). Since folders can be subscribed and not really exist, this is probably what you could expect to see in that case.
Do you see the above folders in your right-click | "subscribe..." listing?

Are all your folders subscribed as shown in right-click | "subscribe..." dialog?
Do you see grey/italic/ghost folders in subscribe... dialog?

With a further search of bugzilla, I found the extremely-long titled bug 885162:

Yes, Mr. WADA liked making the titles long. :)

It looks like when you renamed folders somehow the unsubscribe step failed or just never happened for unknown reasons. This left original folders non-existent but still subscribed so they still show up gray and can't be deleted. I'll try to simulate this by forcing a failure on the unsubscribe step after rename by tweaking the code. I'll let you know what I find.

One more thing: If all your folders are subscribed, you might try un-checking "Show only subscribed folders" under Server Setting | Advanced...
I don't think this will change anything but it might. Probably a tb restart should be done after changing this.
If you have explicitly unsubscribed folders and a lot of them, this may cause your startup time to take longer.

I forced tb to skip the unsubscribe step on the rename action. It doesn't seem to cause tombstone folders on the mail folder list (I used a cyrus imap account). However, I do see them in the right-click -> subscribe... dialog. In the dialog, there was no box to subscribe or unsubscribe the gray folders orphaned there after rename, so there is no direct way to get rid of them.

Here's a possible indirect workaround to get rid of your grayed out folder(s). This worked for me to get rid of the gray orphaned folders in subscribe dialog. It might work in main folder list too:

If you have just a single grayed out folder in the main folder list or in subscribe, try re-creating the folder. It may (should?) come back into not-gray existence. Then delete it to trash again to permanently delete it.

If you have a tree of folders grayed out, try creating them separately, e.g.,

a <-- normal
  b  <-- gray
    c  <--gray

Select "a" and then create "b", if "b" comes back not-gray, select "b" and then create c. Then delete to trash what you don't want.
Or if folder "a" is actually Trash, just empty trash after they are all normal again.

(In reply to gene smith from comment #12)

In tb I can create a gmail folder aa/bb/cc but I only see bb and cc in tb, even after restart. I think tb is "hiding" aa for the same reason [gmail] is hidden. Gmail reports them as subscribed but non-existent.

I agree, that is what is happening.
My Subscribe list says:
Testing (no select) in gmail webmail this is 'Testing/' - it auto hidden by Thunderbird like the [Gmail]/ and the INBOX/
---Testing (this is the one shown as a no select in Folder Pane and no means of subscribing, but can be renamed)
-----Sub Test (normal folder)

In Account SEttings > Synchronisation & Storage
Click on 'Advanced'
Items for Offline Use
You see the same listed and all three will have the checkbox option to 'download'
Testing (no select)
---Testing (no select)
-----Sub Test (normal folder)

From IMAP:4 log:
I've just copy pasted anything that mentioned the relevant folders.
Edited the first part with timestamps so it shows XLIST and LIST followed by other general info on same folders.

  • XLIST (\HasChildren \Noselect) "/" "[Gmail]/Testing"

  • XLIST (\HasChildren \Noselect) "/" "[Gmail]/Testing/Testing"
    2025-11-12 15:43:04.690000 UTC - [Parent 6192: Main Thread]: D/IMAP proposed url = [Gmail]/Testing/Testing/Sub Test folder for connection has To Wait = false can run = false

  • LIST (\HasChildren \NonExistent \Subscribed) "/" "[Gmail]/Testing"

  • LIST (\HasChildren \NonExistent \Subscribed) "/" "[Gmail]/Testing/Testing"

  • LIST (\HasNoChildren \Subscribed) "/" "[Gmail]/Testing/Testing/Sub Test"

2025-11-12 15:43:05.622000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:SelectMailbox: got m_imapMailFolderSinkSelected
2025-11-12 15:43:05.622000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:SendData: 45 getquotaroot "[Gmail]/Testing/Testing/Sub Test"
2025-11-12 15:43:05.663000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:CreateNewLineFromSocket: * QUOTAROOT "[Gmail]/Testing/Testing/Sub Test" ""
2025-11-12 15:43:05.667000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:CreateNewLineFromSocket: * QUOTA "" (STORAGE 0 15728640)
2025-11-12 15:43:05.671000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:CreateNewLineFromSocket: 45 OK Success
2025-11-12 15:43:05.675000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:SendData: 46 UID fetch 1:* (FLAGS)
2025-11-12 15:43:05.722000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:CreateNewLineFromSocket: * 1 FETCH (UID 1 FLAGS (NonJunk \Seen))
2025-11-12 15:43:05.722000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:CreateNewLineFromSocket: 46 OK Success
2025-11-12 15:43:05.732000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:ImapThreadMainLoop: idlePending set
2025-11-12 15:43:07.747000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:SendData: 47 IDLE
2025-11-12 15:43:07.811000 UTC - [Parent 6192: IMAP]: I/IMAP 1377e1df800:imap.gmail.com:S-[Gmail]/Testing/Testing/Sub Test:CreateNewLineFromSocket: + idling

(In reply to gene smith from comment #16)

It looks like it's saying these folders are subscribed. But there is no similar LIST response for these folders at all in the log (based on what you have provided in comment 14).

That is indeed the case.

Since folders can be subscribed and not really exist, this is probably what you could expect to see in that case.

I would expect folders which don't exist to not show in the folder tree (but only in the tree of folder subscriptions).

Do you see the above folders in your right-click | "subscribe..." listing?
Are all your folders subscribed as shown in right-click | "subscribe..." dialog?
Do you see grey/italic/ghost folders in subscribe... dialog?

I see these folders in the Subscribe... dialog, and they are in gray color and italics. If I try to Unsubscribe from them - it doesn't "take"; when I reopen the dialog again, without even a restart, I am again subscribed to them all.

I will say that for I am not subscribed to all folders along the path. For example, I appear to be subscribed to Archives/2024 but not to Archives itself, which doesn't even have a checkbox for subscription. Not sure if that's intended behavior or not.

I'll let you know what I find.

Thank you! TBH, I was surprised this has problem has not come up more often before. I mean, Outlook365 is a common platform and I can't be the only person using many folders.

One more thing: If all your folders are subscribed, you might try un-checking "Show only subscribed folders" under Server Setting | Advanced...
I don't think this will change anything but it might. Probably a tb restart should be done after changing this.

Tried it, no effect :-(

If you have explicitly unsubscribed folders and a lot of them, this may cause your startup time to take longer.

TBH2, I could never keep my mind wrapped around the idea of folder "subscription". This bug, and the failure-to-unsubscribe, just increases my apprehension of it.


PS and a little off-topic: My address has not been getting email updates for a while. The Bugzilla page about this claims it gets the following error:

Your mail server said: (failed) Amazon SES did not send the message to this address because it is on the suppression list for your account. For more information about removing addresses from the suppression list, see the Amazon SES Developer Guide at https://docs.aws.amazon.com/ses/latest/DeveloperGuide/sending-email-suppression-list.html<br>

I've written the bmo-mods address about this a while ago, but have not received a response.

reporter a*, Did you try the workaround I suggested in comment 17?

I was surprised this has problem has not come up more often before. I mean, Outlook365 is a common platform and I can't be the only person using many folders.

I have a test account on outlook365 (only the free, formerly "hotmail", platform) and I don't see that it is doing anything strange.
About how many are "many folders"? I've been working on an issue with users having folders numbering into the 1000s. But more of a performance at startup problem not ghost folders appearing.
Have you found a way to reproduce the problem at will?

(In reply to gene smith from comment #17)

If you have just a single grayed out folder in the main folder list or in subscribe, try re-creating the folder.

Thunderbird refuses to create a folder with that name; the 'Create Folder' button is grayed-out.

(In reply to gene smith from comment #20)

reporter a*, Did you try the workaround I suggested in comment 17?

Not until just now, sorry. But - it did not work I'm afraid.

I have a test account on outlook365 (only the free, formerly "hotmail", platform) and I don't see that it is doing anything strange.
About how many are "many folders"?

About 350 or so.

Have you found a way to reproduce the problem at will?

Actually, no. But perhaps I can find some time to create an MS mail account and play with the folders for a while to see if I can trigger this.

See Also: → 2007534
See Also: → 885162
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: