1776727 - compact fails with "just mark it as deleted" in 102RC1 (IMAP)

Status: RESOLVED FIXED

(Thunderbird :: Folder and Message Lists, defect, P1)

Description

[Eric Moore]

I'm using 102RC1 with Win11 Pro. I prefer to set "when I delete a message" to "just mark it as deleted". That draws a line through a deleted message in the folder listing, but still lets me read it using the message pane. When I'm done going through my new mail and have reviewed anything I deleted I normally right click on the folder and use "compact" in the context menu to get rid of the deleted messages, but sometimes use file -> compact folders instead.

I also have mail.imap.expunge_option set to 2 to automatically expunge deleted messages when there are 20. I currently have less than 20 deleted messages so that has no effect.

1. With my Fastmail IMAP account I can right click on a message and choose "compact" from the context menu. That works as expected. However, with 102RC1 file -> compact folders has no effect on the currently selected folder or any other folders in the account. It used to.

2. With my Gmail , Yahoo and Comcast IMAP accounts. "compact" is missing from the context menu" and file -> compact folders has no effect. I'm testing this with a folder that has a deleted message with the line drawn through it.

3. "just mark it as deleted" does not work correctly with my Outlook IMAP account. If I delete a message I see a line briefly drawn though it, the line disappears and about 1 second later the message disappears. If I right click on a folder there is no "compact" in the context menu. Note that I do not have a Trash folder for this account.

After restarting Thunderbird "just mark it as deleted" now works as expected in the Outlook IMAP account but "compact" is still missing from the context menu and file -> compact folders still has no effect. I checked the other accounts weren't effected.

4. Clean up ("expunge") inbox on exit still seems to work.

Expected results: The IMAP accounts for different email providers should have consistent behavior when trying to compact folders when "just mark it as deleted" is set.

I set mailnews.imap.loglevel to All to try to enable IMAP logging based on https://wiki.mozilla.org/MailNews:Logging . It has no effect. Is that the correct setting? I had previously set mailnews.smtp.loglevel to All and that worked fine.

I also tested Daily (103.0a1 (2022-06-26) (64-bit)) using a different profile that has just a Fastmail IMAP account. Same symptoms except I couldn't open the error console a second time unless I restarted Thunderbird. No problem opening the Developer Toolbox multiple times.

Comment 1

[Wayne Mery (:wsmwk)]

https://wiki.mozilla.org/MailNews:Logging should be correct
I would think this must be a recent regression. Can you find the regression range with https://mozilla.github.io/mozregression?

Comment 2

[gene smith]

I set mailnews.imap.loglevel to All to try to enable IMAP logging based on https://wiki.mozilla.org/MailNews:Logging . It has no effect. Is that the correct setting? I had previously set mailnews.smtp.loglevel to All and that worked fine.

For imap you still need to set env vars MOZ_LOG and MOZ_LOG_FILE as described at https://wiki.mozilla.org/MailNews:Logging since the prefs are not yet supported for IMAP. Set MOZ_LOG to IMAP:5,sync,timestamp and MOZ_LOG_FILE to where ever you want the log to be put.

My preliminary tests with latest self-built daily on this seem to still work but there is no longer a red X by the deleted message. Not sure if that's intentional. I'll test some more with several servers.

Comment 3

[gene smith]

Looks like outlook behaves like default gmail imap server. When you mark a message as deleted, the server expunges it. However TB doesn't remove it from the list but just leaves it there, which seems wrong. If you go back to it and delete it again, tb sends "uid store" to mark it deleted again (surprisingly, with no error from server) then it draws the line through it and it can be undeleted even though supposedly expunged from the server.

But I still see the compact right-click menu item on all the folders for all my accounts.

Actually, I just remembered, the "successful" uid store on a non-existent (expunged) message is ok as pointed out by another tb user at item 2 here: bug 1770811 comment 4, specifically at https://tools.ietf.org/html/rfc3501#section-6.4.8

Generally it is recommended for servers that auto-expunge messages that are marked deleted (gmail and also outlook) that "just mark it as deleted" not be used, but use the delete to trash or delete immediately option. However, it still seem like tb is ignoring the expunge response since the message is never removed from the list and sometimes it has the line drawn through it and seems to be undeleteable per the right-click menu on the crossed out message. The message should immediately disappear if the server expunges it regardless of the delete options settings.

Just tried File | Compact Folders and I see no activity in the imap log. (I have never tried this before.) But when I compact a single folder with right click menu, I see an imap expunge sent to that folder as expected. Marked a message as deleted on my default (well behaved) account and tried File | Compact Folders and it didn't remove the crossed-out message. Might be a regression of this: bug 1756520 ?

I haven't tried running 102rc in windows so there might really be missing "compact" item there.
But just to double check with reporter, are you sure you are right-clicking on folder and not the message? Only the folder will have the compact right click menu item.

Comment 4

[gene smith]

Side note: While testing this bug I also saw bug 1770811. The fix for it is not in 102rc apparently. Not sure what I did to cause it but it's related to messages getting deleted while being fetched.

Edit: For this to happen is pretty unusual. I never saw it again with 102rc again until I reduced the chunk size down to 16000 and the chunk grow size down to 0. This increased the probability that a message being fetched and marked deleted during the fetch and then expunged by server (outlook) causes the endless empty fetches due fetching a now non-existent message. Bug 1770811 stops the endless fetches.

Comment 5

[gene smith]

Reporter Eric, Are any of your accounts configured to use Maildir offline storage format? I don't have any accounts set up with that right now, but if I recall correctly there is no "compact" right-click menu item for folders using maildir. (FWIW, I think there should be.)

Comment 6

[gene smith]

I tried outlook and yahoo imap accounts on windows 10 and see no differences to speak of. Also, it doesn't look to me like there is really any functional change from 91 release except for the non-functioning File | Compact Folders thing. I see the right-click "Compact" item for every folder and it causes the imap expunge to be sent.

Other than File | Compact Folders, using Outlook, Yahoo and Gmail with "Just mark as deleted" is a not a new issue with the workaround to use "Delete to Trash" or "Delete Immediately". Other than for Gmail, there is no way to change their server's behavior, that I can find, to be more in the spirit of the imap RFC (i.e., don't auto-expunge messages marked as deleted!). And even with Gmail, disabling expunge on marked deleted causes other issues if I recall.

Comment 7

[ISHIKAWA, Chiaki]

(In reply to gene smith from comment #5)

Reporter Eric, Are any of your accounts configured to use Maildir offline storage format? I don't have any accounts set up with that right now, but if I recall correctly there is no "compact" right-click menu item for folders using maildir. (FWIW, I think there should be.)

My question here is tangential to the issue in this bugzilla, but what purpose will "compact" serve if TB is using maildir?

Will it, sort of, clean up any inconsistency in the TB's internal DB after a crash or something?
Or maybe there is some implication in IMAP server usage as discussed here. Maybe it signals the server to expunge the "marked as deleted" messages? If so, it may be necessary even in the case of "maildir" being used.
Currently I am strictly POP3-only user because of the past IMAP server bug issues. So I am not well versed with IMAP interaction.

Comment 8

[Eric Moore]

Thanks for the info about how to enable imap logging.

I haven't tried running 102rc in windows so there might really be missing "compact" item there.
But just to double check with reporter, are you sure you are right-clicking on folder and not the message? Only the folder will have the compact right click menu item.

On the folder.

Are any of your accounts configured to use Maildir offline storage format? I don't have any accounts set up with that right now, but if I recall correctly there is no "compact" right-click menu item for folders using maildir. (FWIW, I think there should be.)

I'd completely forgotten that when I replaced my PC a couple of months ago and created a new profile that I used different storage formats for many of the accounts. Sorry about that. I'm used to using mbox files in the IMAP accounts, occasionally replacing the Gmail IMAP account to use maildir for a little while. Currently Fastmail is using mbox and the rest are using maildir. I've now overwritten 102RC1 (what I tested with) with 102.

What I remember (with my prior profile) was that compact was omitted from the context menu for IMAP accounts using maildir, but file -> compact folders was still enabled and useful as despite the misnaming it sent a IMAP expunge command. It seemed to work regardless of whether I had offline folders disabled (usually also had global search/indexing disabled if I did that) or used mbox or maildir.

Generally it is recommended for servers that auto-expunge messages that are marked deleted (gmail and also outlook) that "just mark it as deleted" not be used, but use the delete to trash or delete immediately option.

Why? How is a user supposed to know this? I'm using mail servers from some of the most popular email providers. I would expect that if Thunderbird was unable to support "just mark it as deleted" that I'd get a warning if I tried to select it.

From my biased viewpoint I expect any setting of "when I delete a message" to work regardless of whether offline synching is enabled or not, and whether mbox or maildir is used. i.e. what to do about maildir and mbox is an issue for the synchronization that I should be oblivious of.

My preliminary tests with latest self-built daily on this seem to still work but there is no longer a red X by the deleted message.

I don't see the red X in any of my IMAP accounts using the released version of 102 or with a Fastmail IMAP mbox based account in a different profile using 104.0a1 (2022-06-29) (64-bit). The latter just draws a line through the entry in the folder listing.

I re-installed 91.10.0 in a different directory and it shows the red X. Its using the same profile as 102 (with --allow-downgrade added to the shortcut) and file -> compact folders does remove any messages marked for deletion in the Gmail IMAP maildir based account. There is no "compact" in the context menu in its folders.

I just deleted a message in the Outlook IMAP maildir based account and it immediately disappeared. I double checked its "just mark it as deleted". I deleted 3 more messages in the inbox and they disappeared too. I then deleted a message in its Archive folder and it remained (red X, line drawn through folder listing). I deleted the message below it and it briefly displayed the red X and line, but then it went back to normal. Deleting it again adds the red X and line again, but this time it sticks.

Do you want me to try to capture that flakiness with 91.10.0 / Outlook / maildir / just mark it as deleted in a IMAP log?

Comment 9

[Eric Moore]

You mentioned problems with email providers that auto-expunge. However, auto-expunge is disabled in my Gmail webmail settings. I believe I did that years ago back when the webmail GUI also had low level options such as whether the server should retain messages despite delete or expunge commands from a email client.

I logged into gmail webmail and looked at the "forwarding and POP/IMAP" tab settings.

• POP and IMAP is enabled .
• When I mark a message in IMAP as deleted: is set to Auto-Expunge off - Wait for the client to update the server.
• When a message is marked as deleted and expunged from the last visible IMAP folder: is set to Immediately delete the message forever
• Folder size limits is set to Do not limit the number of messages in an IMAP folder (default)

I logged into outlook.com webmail and looked at the mail settings

• POP is disabled.
• I couldn't find anything to manage expunge

Looking at various threads on Microsoft support forums my impression is that the Outlook email client lets you set whether to automatically purge deleted messages or not, and has options such as "purge items when switching folders". I see threads that mention stuff like "manual purging is on the folder ribbon - you can get to the account setting option from the button to set up automatic purging or do it through Account settings, Select IMAP account, Change button, More options."

Comment 10

[gene smith]

Do you want me to try to capture that flakiness with 91.10.0 / Outlook / maildir / just mark it as deleted in a IMAP log?

Probably not. I can set this up and test it for myself. One other thing I noticed was the the "professional/paid" version of outlook (outlook365) doesn't seem to auto-expunge messages that are marked deleted, unlike free outlook. (Magnus M., head tb developer, provided me a test account on outlook365.)
Also, I couldn't find any setting at outlook or outlook365 that control auto-expunge either.

So the "Compact Folders" items works (expunges the maildir imap accounts) with version 91. That definitely points to a regression. Ben C. did a lot of work in the "compacting" area so I will hit him with an NI on this.

Also, I really think there needs to be a "Compact" right-click on folder menu items for maildir too. It should probably imap expunge the folder/mailbox and delete any maildir message files for deleted messages, more or less exactly like mbox does.

Not sure about the missing RED X. It just might not be available in the latest widget toolkits being used by the UI. (I'm not a UI guy.)

As for TB not behaving always right for imap servers that auto-expunge, that's not really anything new with 102. I'll look into it and see if I can figure out something.

Comment 11

[Ben Campbell]

https://bugzilla.mozilla.org/show_bug.cgi?id=1130277#c3(In reply to gene smith from comment #10)

Also, I really think there needs to be a "Compact" right-click on folder menu items for maildir too. It should probably imap expunge the folder/mailbox and delete any maildir message files for deleted messages, more or less exactly like mbox does.

Yes - "compact" is a bit of an overloaded term these days, covering both mbox compaction and server-side expunge.
On the GUI side, maybe it'd be worth coming up with a better terminology?
I think bug 1130277 speaks to that confusion... (see comment 3 there - it suggests the text should adapt: "Compact folder" for mbox, "Expunge deleted mails" for IMAP+maildir... )

The underlying code seems a bit confused too - I remember from Bug 1756520 that the IMAP expunge is still a lot more tangled up with mbox compaction than I'm comfortable with. I did try hard not to mess with any of the IMAP expunge in Bug 1756520, but it's obviously regressed and needs fixing...

Comment 12

[gene smith]

Hi Ben, Maybe you can explain something. If I understand it right, "compact" on an imap folder stored as mbox expunges (permanently deletes) messages on the server marked deleted. It also removes the body text for these same expunged messages from the mbox file, making the mbox file smaller. If you compact a maildir folder you would also expunge the marked-deleted messages on the server. Wouldn't you also want to delete the corresponding maildir files for these expunged messages? Am I missing something here?

Edit: Actually, when I think about it more, "compact" for a folder should just trigger the imap expunge command. Then the imap expunge responses that result should cause the mbox shrinkage or maildir file deletions. So disk usage reduction will be independent mbox or maildir format.

Comment 13

[Ben Campbell]

(In reply to gene smith from comment #12)

Hi Ben, Maybe you can explain something. If I understand it right, "compact" on an imap folder stored as mbox expunges (permanently deletes) messages on the server marked deleted. It also removes the body text for these same expunged messages from the mbox file, making the mbox file smaller. If you compact a maildir folder you would also expunge the marked-deleted messages on the server. Wouldn't you also want to delete the corresponding maildir files for these expunged messages? Am I missing something here?

When you delete messages from an mbox, it merely sets the 'Expunged' flag (in the nsMsgDBHdr and by in-place modification of the 'X-Mozilla-Status' header in the mbox). A later "Compact" will go through and squeeze those gaps from the mbox file (plus whatever server commands need to be issued).

For maildir, deleting a message causes the local message file to be deleted immediately. So the "Compact" only needs to perform the server-side operations.

Comment 14

[gene smith]

For maildir, deleting a message causes the local message file to be deleted immediately. So the "Compact" only needs to perform the server-side operations.

From the test I just did on a maildir enable account, if you delete a message with "just mark it as deleted" set, the number of maildir files in "cur" doesn't change at all. This is with a well-behaved server that doesn't auto-expunge messages marked deleted. When I run another tb instance set to mbox for the same account and "compact" the folder, the imap idle response on the maildir instance produces an EXPUNGE response. Only after the EXPUNGE response is parsed by the maildir instance do I see the number of maildir "cur" files go down by one.

So it seems like a maildir file in "cur" is only removed after the message deletion is confirmed with an imap EXPUNGE response. To me this would make sense since you want to keep the offline storage for the message around until the message is expunged at the server for mbox or maildir.

If you configure the default delete method (move to trash) the EXPUNGE response occurs almost immediately in response to the MOVE (to Trash) command. So in this case, which I haven't tested with maildir, EXPUNGE response will quickly trigger the removal of the maildir "cur" file. But if the server doesn't support MOVE (fairly rare I think) and only support COPY, the message will still just be marked deleted, no EXPUNGE response occurs and the maildir "cur" file will remain until the folder is expunged using a future/proposed "compact" menu item.

Comment 15

[gene smith]

Eric wrote from comment 9:

You mentioned problems with email providers that auto-expunge. However, auto-expunge is disabled in my Gmail webmail settings. I believe I did that years ago back when the webmail GUI also had low level options such as whether the server should retain messages despite delete or expunge commands from a email client.

If setting gmail webmail to NOT auto-expunge works for you, then there is no problem. I don't remember exactly what the "problems" were but may have had something to do with when messages get moved to trash. I just prefer leaving the gmail imap setting at webmail to default.

I wrote:

Generally it is recommended for servers that auto-expunge messages that are marked deleted (gmail and also outlook) that "just mark it as deleted" not be used, but use the delete to trash or delete immediately option.

Eric wrote from comment 8:

Why? How is a user supposed to know this? I'm using mail servers from some of the most popular email providers. I would expect that if Thunderbird was unable to support "just mark it as deleted" that I'd get a warning if I tried to select it.

Yes, I see the auto-expunge with "popular" yahoo, aol, outlook and default gmail. I don't see any way to work around this strange server behavior which is outside the spirit of the imap RFC 3501 (although technically it is allowed). The also popular and standards respecting servers like dovecot, cyrus and courier don't automatically expunge messages that are marked deleted. So with these servers, when you delete a message it remains marked out but still present until the folder or the individual message is explicitly expunged.

So I guess the "warning" when you use one of these auto-expunging servers with "just mark it as deleted" is that the cross-out may come and go or the message may vanish unexpectedly (which varies depending on when the server sends the expunge response).

I'm not sure where or how to warn a user about this. I guess the "assumption" is that since "just mark as deleted" is a non-default setting that the user understand that it might not work completely as expected in some situations. Or maybe the description in server setting could be somewhat enhanced:

When I delete a message:
() Move to this folder [folder picker]
() Just mark it as deleted (Not suitable for servers that auto-expunge, e.g., yahoo, outlook, default gmail etc)
() Remove it immediately

Comment 16

[Eric Moore]

Enhancing the description in the server settings is probably the best way to warn them because there is less that could go wrong. Most users don't know what auto-expunge means but I suspect those who might actually think about changing "When I delete a message" would at least interpret that as the behavior is going to be server dependent and what they should find out more about before selecting it.

Comment 17

[Ben Campbell]

Attachment: Bug 1776727 - Have nsImapMailFolder::CompactAll() also start Expunge on affected folders. r=gds,sancus

Comment 18

[Ben Campbell]

I think I've figured it out:
On mbox accounts, you get a "Compact folder" context menu item.
On maildir accounts you don't have the context menuitem (because maildir doesn't require compaction).
But you can still go through the file menu "Compact Folders" item, which you'd think would do basically the same thing. It doesn't. Of course it doesn't.
The file menu "Compact All" does perform mbox compaction on the affected folders, but it doesn't kick off the corresponding server-side Expunge operations.
So it's not a case of maildir not working here, but more a case of maildir forcing you to use the broken operation (file menu "Compact All" didn't work for me with mbox either).

This patch causes expunges to be kicked off for all affected folders, and seems to work for me.

Comment 19

[gene smith]

On maildir accounts you don't have the context menuitem (because maildir doesn't require compaction).

I still think imap maildir accounts needs something in the folder-specific context menu to kick off an expunge (maybe called "Expunge" and not "Compact") since the "Compact Folders" top level menu item iterates through all your folders and you might not want to expunge all your Crossed-out/Marked-as-Delete items in all folders.

However, it looks like your fix does get back the lost functionality.

I'll look closer at it and test it out tomorrow.
P/S: You mention "Compact All" above. I don't see that in the menus so I suppose you mean "Compact Folders".

Comment 20

[gene smith]

Patch seems to work OK for imap/mbox but I need to look closer and make sure the mbox file is getting changed OK.

With maildir, I tested a folder with 40 messages. I see 40 *.eml files in the "cur" directory. I then mark a single message deleted and the line is draw through it. I then do the "Compact Folders" from File menu and I see the imap expunge happen for every folder in the selected account. (I thought it would occur for all folders in all accounts, but I guess my assumption was wrong.) The untagged expunge response for the crossed-out message causes the number of *.eml file in "cur" to go down to 39 and the message is gone from the list. So it looks good so far.

Now I mark another message deleted and it gets crossed out and I do the "Compact Folders" again. Imap expunge happens again and the message disappears from the list, but the number of *.eml files stays at 39. Then if I do another "Compact Folders" without marking anything deleted, something happens that causes all the messages to be re-fetched (downloaded) and now I have almost twice as many *.eml files in "cur" with the new ones having a "-1" at the end of the name. The number of files with -1 at end is the correct number 38.
Repairing the folder brings the number of files in cur back to the correct number 38 and the file names don't end with -1.

Comment 21

[Ben Campbell]

(In reply to gene smith from comment #19)

I still think imap maildir accounts needs something in the folder-specific context menu to kick off an expunge (maybe called "Expunge" and not "Compact") since the "Compact Folders" top level menu item iterates through all your folders and you might not want to expunge all your Crossed-out/Marked-as-Delete items in all folders.

Yup, definitely something is needed. But I'd break it out into a separate bug once we're done with this one. Too much GUI there for me :-)
(string localisation mainly, also I think it's worth stepping back, wearing a UX hat and seeing if there's a better way to present it. IMAP expunge and mbox compaction are separate ops, only bound together like this because they've both got a vague garbage-collect-my-folder feel to them...)

P/S: You mention "Compact All" above. I don't see that in the menus so I suppose you mean "Compact Folders".

Sorry - yes, should've been "Compact Folders".

Comment 22

[Ben Campbell]

(In reply to gene smith from comment #20)

Patch seems to work OK for imap/mbox but I need to look closer and make sure the mbox file is getting changed OK.

Definitely seems to be some issues with cleaning up deleted messages - even with mbox, and even with the default "move message to [Trash]" delete behaviour. Even compaction doesn't seem to remove a well-and-truly deleted message from the mbox. "Repair Folder" does the trick.
I suspect the same issue is causing the maildir wierdness described in comment #20.
Definitely a regression from 91.

But I think D150923 (Comment #17) still stands on it's own - It adds EXPUNGEing to "Compact folders", and is not directly related to this.

Comment 24

[Ben Campbell]

(In reply to Ben Campbell from comment #22)

Definitely seems to be some issues with cleaning up deleted messages - even with mbox, and even with the default "move message to [Trash]" delete behaviour. Even compaction doesn't seem to remove a well-and-truly deleted message from the mbox. "Repair Folder" does the trick.

OK, I've done some more thorough testing on this and I'm going to recant most of what I wrote :-)
With mbox:
"Move to trash" and "remove immediately" settings do act as you'd expect: the mbox file is left as-is, and the next folder compact will actually delete the removed messages. I think I was getting confused because the X-Mozilla-Status flags which modified to reflect the deletion in local folders are not modified in IMAP folders (confusingly, some imap operations (repair folder) insert X-Mozilla- headers while others (compaction) strip them out).
With the "Mark as deleted" setting, there is a little glitch: If you "delete" a message, it stays in the mbox file. When you do a "compact folder", the message is not removed from the mbox file. However, running "compact folder" a second time does the trick. I'm pretty sure that this is just the first compaction starting before the IMAP expunge operation has returned and deleted the message from the msgDB. So the compaction thinks the message still exists. By the time of the second compaction, the message has gone, and the compaction removes it from the mbox as expected. I'll write this up as a separate bug, but I don't think it's too big a deal.
I'm not actually sure this was any different in 91 after all.

The maildir weirdness is another issue. I've got some strong suspicions about this (the way compaction is kicked off is a bit bonkers - different methods for different folder types, and I've an idea it's actually doing a compaction on a maildir store!). Again, not sure it's actually a regression, but this one definitely should be fixed! Looking into it now.

Comment 25

[gene smith]

For me, it seems like your latest patch worked OK when compacting with messages crossed-out (marked as deleted). I saw the full message removed from the mbox after compacting and didn't have to do it again. I'll re-test that.

But with maildir, using the latest patch, what I see happening is, after a folder repair back to a good state, the folder is still passed into the nsMsgFolderCompaction.cpp code. This causes msgHdr->SetStringProperty("storeToken", storeToken); to occur. But for maildir, it seems "storeToken" is the filename in "cur" for the message; but for mbox it's m_startOfNewMsg which corrupts the stored maildir filename. So in nsMsgMaildirStore.cpp where "storeToken" is read, you get a totally wrong filename and in DeleteMessages() you get a log message "DeleteMessages - file does not exist !!".

I can fix this by not calling CompactFolders() in nsImapMailFolders.cpp. If only expunge is done there (at both places) and no compact, the maidir file gets deleted since "storeToken" still contains the correct filename and the response to expunge causes the offline store to be "compacted" for mbox and the file deleted for maildir.

Comment 26

[Ben Campbell]

(In reply to gene smith from comment #25)

But with maildir, using the latest patch, what I see happening is, after a folder repair back to a good state, the folder is still passed into the nsMsgFolderCompaction.cpp code. This causes msgHdr->SetStringProperty("storeToken", storeToken); to occur. But for maildir, it seems "storeToken" is the filename in "cur" for the message; but for mbox it's m_startOfNewMsg which corrupts the stored maildir filename. So in nsMsgMaildirStore.cpp where "storeToken" is read, you get a totally wrong filename and in DeleteMessages() you get a log message "DeleteMessages - file does not exist !!".

Ahh, yep. I think this is Bug 1683714, not a recent regression. And your diagnosis seems spot-on - it's attempting to compact a maildir store(!) and so all the storeTokens get blatted. I've got a patch - I'll post it over at Bug 1683714.

Comment 27

[gene smith]

For me, it seems like your latest patch worked OK when compacting with messages crossed-out (marked as deleted). I saw the full message removed from the mbox after compacting and didn't have to do it again. I'll re-test that.

Yes, I see what you see. The expunged/deleted message isn't removed from mbox until the 2nd "compact". Probably been like that for a while and not due to any recent changes. I tested this on a fairly large mbox >20M. I'm curious if it happens on smaller ones since I thought it was OK when I tested it a few days ago, maybe on a small mbox, not sure.

Anyhow, the patch here is OK (restores File | Compact Folders functionality for mbox but more useful when combined with the fix at bug 1683714 (fixes failure to remove maildir file(s) on File | Compact Folders).

Comment 28

[Wayne Mery (:wsmwk)]

Any idea if these newer bug reports being to address Bug 1260698 ?? - Compact folders - shows wrong estimated and compacted size. Compacts too often. totalExpungedBytes number is wrong

Comment 29

[Ben Campbell]

(In reply to Wayne Mery (:wsmwk) from comment #28)

Any idea if these newer bug reports being to address Bug 1260698 ?? - Compact folders - shows wrong estimated and compacted size. Compacts too often. totalExpungedBytes number is wrong

From a skim of the Bug 1260698 history I can't really tell either way, sorry :-(
Certainly possible, but even if this issue is responsible I'd bet there were other factors in play too.

Comment 30

[gene smith]

This hasn't made it to daily yet. Still waiting for sancus review? (also waiting for bug 1683714 review which kind of goes with this -- see comment 27).

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/ddac1b3f82cb
Have nsImapMailFolder::CompactAll() also start Expunge on affected folders. r=gds,sancus

Comment 34

[Wayne Mery (:wsmwk)]

Good to uplift to esr?

Comment 35

[Ben Campbell]

I'm happy for uplift.
I've got a more robust (but less uplift-friendly) solution lined up in Bug 1782374, but this patch is pretty small and papers over the worst of the issue.

Comment 36

[Wayne Mery (:wsmwk)]

Comment on attachment 9283962 [details]
Bug 1776727 - Have nsImapMailFolder::CompactAll() also start Expunge on affected folders. r=gds,sancus

[Triage Comment]
Approved for esr102

(In reply to Ben Campbell from comment #35)

I'm happy for uplift.
I've got a more robust (but less uplift-friendly) solution lined up in Bug 1782374, but this patch is pretty small and papers over the worst of the issue.

Comment 37

[Rob Lemley [:rjl]]

Thunderbird 102.2.1:
https://hg.mozilla.org/releases/comm-esr102/rev/f0403cf8465b