399475 - mail.imap.expunge_after_delete=true not honored when imap delete model != "Move it to this folder"

Status: ASSIGNED

(Thunderbird :: Preferences, defect)

Description

[Matt]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7
Build Identifier: version 2.0.0.6 (20070728)

In some cases of server setting for IMAP "When I delete a message" drop-down box, deleting a message will not result in an expunge action.

In the case of "Move it to the Trash folder" the source message is removed.  The is because a "Move" operation is performed, resulting in the expunge of the source folder.  This is expected behavior.

In the case of "Mark it as deleted" the source message is marked as deleted, but not expunged.  This is expected behavior.

In the case of "Remove it immediately" the message is marked as deleted and removed from the message window, but NOT expunged.  If the setting is changed back to "Mark it as deleted" the message is re-appear as it was never removed.

From my testing, "Remove it immediately" ONLY changes the display behavior.  The folder/IMAP behavior is identical to "Mark it as deleted".

This is true regardless of the state of "mail.imap.expunge_after_delete"

Reproducible: Always

Steps to Reproduce:
1. Set IMAP account Server Settings to "Remove it immediately"
2. Delete a message.
 - Message is not deleted -
3. Set IMAP account Server Settings to "Mark is as deleted"
4. Refresh your view.
5. Message reappears.  Was never deleted.
Actual Results:  
Messages are not removed.  No EXPUNGE command sent.

Expected Results:  
EXPUNGE should be sent to the server.  Message should be removed.

This is incompatible with large-scale hosting environments.  IMAP users regularly call technical support for mailboxes over-quota, only to find thousands of "deleted" messages filling their boxes.

Comment 1

[Matt]

Adding myself to CC list.

Comment 2

[Matt]

Looking at nsImapProtocol.cpp, the Expunge() happens at gExpungeThreshold which is currently a hard-coded value.  The criteria for that Expunge does not check for gExpungeAfterDelete. (line 3651)

There is probably a more graceful way to do it, but perhaps if gExpungeAfterDelete is true, and Expunge(); should happen as long as there are deleted items, gExpungeThreshold is not reached and user settings are not "Move to Trash"  which should cause the enforcement only under "Remove Immediately"

        if ((m_flagState->GetNumberOfDeletedMessages() >= 20/* gExpungeThreshold */)  
                 && !GetShowDeletedMessages() && m_imapAction != nsIImapUrl::nsImapLiteSelectFolder)
        {
          Expunge();
	    }
        else if (gExpungeAfterDelete && !GetDeleteIsMoveToTrash())
          Expunge();
      }

Comment 3

[Magnus Melin [:mkmelin]]

That pref is only used when you have the delete setting set to "Move it to this
folder:" (xref penelope bug 359281)

Comment 4

[Matt]

I see.  It looks like the discussion is leaning toward dual-use of the setting.  

In my code suggestion, "Remove it immediately" would trigger Expunge() if the gExpungeThreshold is reached.  This leaves the current behavior of hiding deleted messages as the expected behavior while the system cleans up after itself quietly.

To implement gracefully, gExpungeThreshold should be tunable.  It is currently a hard-coded value.

Comment 5

[Nikolay Shopik]

blocks bug 370509 ?

Comment 6

[WADA:World Anti-bad-Duping Agency]

Bug 423953, which is put in Blocks: of Bug 370509, sounds to be dup of this bug.

Comment 8

[Tilman Schmidt]

Since "my" Bug 423953 has been merged as duplicate into this one, let me explain what "expected behaviour" is IMHO:

If I do "remove immediately" (shift/Del), the message should be gone once and for all and not reappear (whether marked as deleted or not) if I open the same folder again (a) from the same Thunderbird installation or (b) from a different IMAP client. This seems to imply that shift/Del under the "When I delete a message: Remove it immediately" setting must translate into Delete+Expunge. Even more so since Thunderbird is *still* unable to hide IMAP deleted messages from view.

Comment 9

[Matt]

This bug needs to be distilled for clarification.

I disagree that bug 423953 (https://bugzilla.mozilla.org/show_bug.cgi?id=423953) is a duplicate of this bug.  423953 states the Expunge() is not called after shift+delete when the behavior selected is "mark as deleted".  I would call bug 423953 at best expected behavior and at worst a limitation of IMAP, not a bug, as Expunge() occurs on a folder and not on a message.  Calling Expunge() after a shift+delete would delete all messages marked "deleted".

+++

Separately, shift+delete under "remove immediately" is essentially a different manifestation of bug, 399475 (this bug).  The bug states the under "remove immediately" Expunge() is not called.  This occurs regardless of the state of the shift key.  My understanding of "remove immediately" is an implied shift+delete for all delete operations where the expected behavior is a call to  Expunge() following the delete operation.

Also noted is Expunge() will be called when the number of deleted messages exceeds 20, honoring a hard-coded, arbitrary value contained in gExpungeThresholdthat and ignoring the registry setting gExpungeAfterDelete.

Comment 10

[Nikolay Shopik]

Matt, 
There is UIDPLUS extension which is improve behavior of expunge and not only, look for RFC4315 to get details.

Comment 11

[Matt]

So again, bug 423953 is not a duplicate of this bug.  Bug 423953 is more an RFE that shift+delete implement UIDPLUS over IMAP connections.

Bug 399475 (this bug) remains an issue of Expunge() not called correctly when prefs set IMAP delete behavior to "remove immediately".

Comment 14

[ktos.obcy]

Adding myself to CC list.

Also: still happening with Thunderbird 11 under Windows 7 x64

Comment 15

[sasha1024]

5 years...

Is thunderbird still maintained?

Comment 16

[monperrus]

This bug is still valid and poses a problem when using GMail.

When using Gmail, the only sensible delete strategy is "Move to folder -> Trash". Otherwise, with "Mark as deleted" or "Remove immediately", the message is only unlabeled and stays forever in "All mail", wasting space resources. See https://support.google.com/mail/thread/184282501/e-mails-not-moved-to-trash-no-matter-imap-setting?hl=en

With "Move to Trash", the problem is that the deleted messages are not visible in TB after deletion (good) but are not unlabeled until a compact/expunge is done. As a result, deleted messages in TB still appear in Gmail Web UI and Android app, until manual compacting or restart.

This happens despite "mail.imap.expunge_after_delete = true"

For the record, the problem also happens for the following other configurations:
mail.imap.expunge_option = 1 (bug 655357)
and
mail.imap.expunge_option = 2 / mail.imap.expunge_threshold_number = 1

Comment 17

[monperrus]

Found a server-side workaround to auto expunge with gmail.

Gmail provides a server-side option:

When I mark a message in IMAP as deleted:
Auto-Expunge on - Immediately update the server. (default)
Auto-Expunge off - Wait for the client to update the server.

With "Auto-Expunge on", all emails are well expunged.

With "Auto-Expunge off", one suffers from all client-side expunge bugs.

Comment 18

[monperrus]

Looking at the code, it's consistent with this bug report, expunge is only called in

case nsIImapUrl::nsImapOnlineMove
case nsIImapUrl::nsImapOnlineToOfflineMove

It should also called in case nsIImapUrl::nsImapDeleteMsg.

Comment 19

[monperrus]

Attachment: Fix missing IMAP expunge (bug 399475)

If expunge_after_delete == true, an expunge command should be issued when deleting single messages

Fix https://bugzilla.mozilla.org/show_bug.cgi?id=399475

Comment 20

[gene smith]

Attachment: WIP: Bug 399475 - Allow shift-del for gmail to expunge from ALL folders with default settings (TB and gmail.com).

This was suggested here: https://groups.google.com/g/mozilla.support.thunderbird/c/9T3VGSvPJgM
What it does is when you shift-delete a message in a gmail folder, it moves the message to
Trash folder, auto-expunging the message from source folder. Then the new message in Trash is
marked deleted. Messages marked deleted in Trash are not auto-expunged by gmail and remain in Trash.
So the proposed code then does an UID expunge on the message. Assuming the deleted message has
not been copied to any other "visible in TB" folder, this causes the message to be expunged
from "All Mail" as well, so the message is completed removed "forever".
This also proposes some other changes that I need to think more about. But the main change
described above occurs under "case nsIImapUrl::nsImapAddMsgFlags:".

Comment 21

[monperrus]

Great progress.

Updated the diff in https://phabricator.services.mozilla.com/D218927 to change nsImapAddMsgFlags (Gene, do you get email notifications from Phabricator?)

Comment 22

[gene smith]

Something seems to be wrong with the email notification from phab. On the 12th it sent me this and I haven't seen anything from it since:

Due to an internal issue (it's been automatically reported!), we weren't able to fetch the details on the action that happened to this revision. You should view the revision on Phabricator to ensure that you're not missing anything important.

So just manually reloading the page to see what's changed.

Comment 23

[gene smith]

From comment 20:

So the proposed code then does an UID expunge on the message. Assuming the deleted message has
not been copied to any other "visible in TB" folder, this causes the message to be expunged
from "All Mail" as well, so the message is completed removed "forever".

I looked again at what gmail does and I see that when a message is just moved to Trash, gmail actually removes it from All Mail (after maybe a minute it appears, looking the All Mail folder at gmail.com). Expunging it in Trash is not needed to trigger the removal from All Mail, but on shift-delete TB warns that the message will be completely gone, so the UID expunge in Trash should still be done, and the user is not expecting to see the message still in Trash.

Another thing I tried in TB was to move a message from gmail TB Inbox to a folder in another TB account. What this does is first read from file/cache or fetch the message from gmail server and do an imap "append" of the message content to the other account/server. Then it just marks the message \deleted in the gmail source folder using the nsImapAddMsgFlags imap action just like shift-delete. So the code I added to that "case" occurs and the same thing occurs: the message is moved to Trash, expunged from the source folder and then expunged in Trash. This causes the message to be permanently removed from gmail.
A problem with this is that "Undo Move Message" (ctrl-z) functionality is broken. Undo tries to remove the \deleted flag from the message in the source folder. But since the message has been moved to trash and expunged in source folder and trash, there is no message to remove the \deleted flag from, so undo is a noop. But even without my added code, undo for gmail in this situation wouldn't work. That's because, with default gmail.com setting, when a message is marked \deleted in the source folder it is auto-expunged. So undo can't just removed the \deleted flag since the message is already expunged. Undo for gmail might be made to work but gmail.com default "auto-expunge" setting would need to be turned off.

Comment 24

[monperrus]

Hi Gene, what do you think of this proposal https://phabricator.services.mozilla.com/differential/changeset/?ref=8269754?

Comment 25

[monperrus]

Hi Gene, how can I help to make progress on merging a patch? Thanks! --Martin

Comment 26

[gene smith]

(In reply to monperrus from comment #25)

Hi Gene, how can I help to make progress on merging a patch? Thanks! --Martin

Sorry for the delay on this. I'll try to get to it today.

Comment 27

[gene smith]

Attachment: WIP: Bug 399475 - Allow shift-del for gmail to expunge from ALL folders with default settings (TB and gmail.com)

This was suggested here: https://groups.google.com/g/mozilla.support.thunderbird/c/9T3VGSvPJgM
What it does is when you shift-delete on selected messages in a gmail folder, it moves the messages to
Trash folder, auto-expunging the messages from source folder. Then the new messages in Trash are
marked deleted. Messages marked deleted in Trash are not auto-expunged by gmail and remain in Trash.
So the proposed code then does UID expunge on the messages moved to Trash. Assuming a deleted message has
not been copied to any other "visible in TB" folder, this causes such messages to be expunged
from "All Mail" as well, so the deleted messages are completed removed "forever".
The main change described above occurs under "case nsIImapUrl::nsImapAddMsgFlags:" in
nsImapProtocol.cpp.
This also merges in the changes proposed by Martin Monperrus in D218927 to allow pref
"expunge_after_delete" to have an effect.

---

Changes and additions to above:
1 Now Shift-del for gmail uid expunging from all folders requires pref "expunge_after_delete" be true.
When expunge_after_delete is true, if UIDPLUS is supported a uid expunge is performed. If UIDPLUS
not supported, a full folder expunge is performed. This prevent functionality changes with default setting.
2 This changes the trash folder discovery so that the trash folder picker always shows the actual
folder being used for trash even if the user has not explicitly chosen a folder, i.e., no longer just
shows "Choose Folder" when a special-use \trash folder has been discovered and is actually in use.
3 This no longer does special-use \trash DiscoveryDone processing only for gmail but now works with any
server that returns \trash special-use in a LIST or XLIST response when setting the server's
"trash_folder_name" pref and flagging a folder as trash.
4 Trash folder picker no longer sets pref "trash_folder_name" to "Trash" when pref not yet set. This
avoids showing possibly incorrect (default) trash folder name in picker and in pref "trash_folder_name"
when trash folder not yet discovered or set.

Observed issues (not related to above changes AFAIK):
1 Sometimes have to close and re-open "Advanced Preferences" or "Account Settings" tab to see changes in
preferences "trash_folder_name" or in trash folder picker for the account.
2 After picking a different trash folder with picker in Account Settings, the "trash can" icon appears
correctly on the new folder but doesn't get removed from the original folder. It does get removed from
original folder after a TB restart.

Comment 28

[monperrus]

Thanks a lot for the progress Gene.

Comment 29

[gene smith]

Martin, I've updated and retested the patch pointed to in comment 27. I don't know if you have a way to build it, so I've made a "try" build based on tonight's daily code here: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=891e22f9148434e69dd1de07aacea719223c12fc
I made it for all platforms in both optimized and debug mode. You can obtain an installer for your platform by clicking the green "B" next to the appropriate platform and optimization level then look down below under "Artifacts and Debugging" to find the installer in the list.

• For windows the installer is target.installer.exe
• For OSX/Mac the installer is target.dmg
• For Linux it is target target.tar.bz2 (not really an installer, just unzip and run "thunderbird" executable inside the thunderbird directory)

Probably the "opt" version is best unless you plan on running a debugger (e.g., gdb) then you would need the "debug" version for your platform.

Comment 30

[monperrus]

Gene, thanks for the build on treeherder.

Good news: The patch works for "normal" IMAP servers, expunge_after_delete is honored (hurray!). It solves both the "Remove it immediately" delete model bug (this bug) and the unsupported "Shift-Delete on IMAP" bug.

The GMAIL case (bug 1759335): The patch works if and only if:

1. The "All mail" folded is not exposed to IMAP on Gmail (show in IMAP == false)
2. AND the Expunge model on Gmail is "Auto-Expunge off - Wait for the client to update the server."
3. AND the Expunge strategy on Gmail is set to "Move the message to the Trash" or "Immediately delete the message forever"

Comment 31

[gene smith]

(In reply to monperrus from comment #30)

Gene, thanks for the build on treeherder.

Good news: The patch works for "normal" IMAP servers, expunge_after_delete is honored (hurray!). It solves both the "Remove it immediately" delete model bug (this bug) and the unsupported "Shift-Delete on IMAP" bug.

That's good.

The GMAIL case (bug 1759335): The patch works if and only if:

1. The "All mail" folded is not exposed to IMAP on Gmail (show in IMAP == false)
2. AND the Expunge model on Gmail is "Auto-Expunge off - Wait for the client to update the server."
3. AND the Expunge strategy on Gmail is set to "Move the message to the Trash" or "Immediately delete the message forever"

To work with default gmail.com imap setting (auto-expunge on) and default TB (move to Trash on delete) you have to set pref expunge_after_delete to false (the default). I was thinking about that after I did made the try build and not sure why I made that condition. So if you could test it again with "expunge_after_delete" false, it should work, at least it does for me.

FYI, here's the IMAP:4 log fragment that shows what happens when I shift-delete a message from gmail:

IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-INBOX:SendData: 36 uid move 71762 "[Gmail]/Trash"
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: * 37054 EXPUNGE
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: * 37053 EXISTS
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: 36 OK [COPYUID 2 71762 20856] (Success)
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-INBOX:SendData: 37 select "[Gmail]/Trash" (CONDSTORE)
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Forwarded $MDNSent $NotPhishing $Phishing $label1 $label2 $label3 $label4 $label5 $notjunk Junk NonJunk _backslash gds0 newtag)
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $Forwarded $MDNSent $NotPhishing $Phishing $label1 $label2 $label3 $label4 $label5 $notjunk Junk NonJunk _backslash gds0 newtag \*)] Flags permitted.
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * OK [UIDVALIDITY 2] UIDs valid.
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 37 EXISTS
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 0 RECENT
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * OK [UIDNEXT 20857] Predicted next UID.
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * OK [HIGHESTMODSEQ 3794908]
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 37 OK [READ-WRITE] [Gmail]/Trash selected. (Success)
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:SelectMailbox: got m_imapMailFolderSinkSelected
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:SendData: 38 uid store 20856 +Flags (\Deleted)
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 37 FETCH (FLAGS (\Seen \Deleted NonJunk) MODSEQ (3794912) UID 20856)
Main Thread]: D/IMAP_CS NotifyMessageFlags(): Store highest MODSEQ=3794912 for folder=INBOX
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 37 FETCH (UID 20856 MODSEQ (3794912) FLAGS (NonJunk \Deleted \Seen))
Main Thread]: D/IMAP_CS NotifyMessageFlags(): Store highest MODSEQ=3794912 for folder=INBOX
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 38 OK Success
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:SendData: 39 uid expunge 20856
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 37 EXPUNGE
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 36 EXISTS
IMAP]: I/IMAP 7f1a657fa100:imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 39 OK Success

Since the "shift-deleted" message is moved to gmail Trash and then marked \deleted in gmail Trash, and since the message only has the "Inbox" label (i.e., was not copied to any other folders), this is enough to expunge it from All Mail. I can see it is gone from All Mail by monitoring All Mail folder at gmail.com (and it is gone from All Mail folder in TB). I do notice there is a bit of delay before it is removed from All Mail by google. Finally I expunge just the deleted message from Trash for good measure and to ensure it is complete gone.

When I make another try build, can you tell me the platform you want? That way I won't have to build them all again.
Thanks!

Comment 32

[monperrus]

Hi Gene,

1. I use the Linux x86_64 build.

2. Gmail case: I confirm that the patch works if expunge_after_delete == false and a trash folder has been set in the main delete strategy screen and all_mail is not exposed to IMAP. (note it does not work out of the box, because the Trash folder is not selected by default as trash folder, hence trashFolderExists == false)

3. Normal Imap server case: I understand that we don't have the change of D218927 anymore, and indeed, Shift-Delete does not expunge for normal servers with expunge-delete == true

Best, --Martin

Comment 33

[gene smith]

(In reply to monperrus from comment #32)

Hi Gene,

1. I use the Linux x86_64 build.

Same here. I haven't found a fix for your item 2 yet so I haven't updated the build.

2. Gmail case: I confirm that the patch works if expunge_after_delete == false and a trash folder has been set in the main delete strategy screen and all_mail is not exposed to IMAP. (note it does not work out of the box, because the Trash folder is not selected by default as trash folder, hence trashFolderExists == false)

Yes, you are (mostly) right. I was testing on an old, existing gmail account and I had tweaked the Trash folder selection for various reasons working on other bugs and had set it back to gmail's Trash [Gmail]/Trash so that worked. Then on another profile I created the same gmail account and the trash folder selection just came up initially as Choose folder. When I tried to shift-delete a message it actually did find a Trash folder but with path of just Trash and the IMAP command to move the message to Trash failed since gmail's trash is not at root level but under [Gmail]. So for me it's failing because the IMAP command failed, not because trashFolderExists is false.

When TB does "folder discovery" at startup using XLIST, gmail returns the info that the trash folder is at path [Gmail]/Trash so I thought TB would see that and use it as the default Trash folder. However, AFAICT, TB actually ignores path returned in the response and doesn't store it. However, what I don't understand is when I do a normal delete (without the shift) the proper destination folder path is sent via imap [Gmail]/Trash. I don't know how the new account in the profile is knowing the correct path so that a normal "delete to trash" works OK.

I wasn't successful in resolving this last night and need to look at it again.

3. Normal Imap server case: I understand that we don't have the change of D218927 anymore, and indeed, Shift-Delete does not expunge for normal servers with expunge-delete == true

I put your changes from D218927 into the patch. So it should work as we want for "normal" servers, I think.

Comment 34

[monperrus]

• Normal case: I open https://phabricator.services.mozilla.com/D220057, download the diff, I can see "if ((msgFlags & kImapMsgDeletedFlag) && gExpungeAfterDelete) Expunge();" at the end of nsImapAddMsgFlags, I tested again, it works. Everything's good.

• Gmail case: good, we're on the same line

Comment 36

[sa212+github]

I'd like to test this on the (now closed) bug 1768322, but I don't have the bandwidth to build from source - can you tell me when this will make it to the esr update channel?

Comment 37

[monperrus]

@sa121 you can test the patch via the builds at https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=891e22f9148434e69dd1de07aacea719223c12fc

@Gene let me know how we can make progress.

Comment 38

[gene smith]

Sorry for the delay in posting back on this. Been trying to decide what to do and how.
I've updated the patch in comment 27. Here's the try build:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=8958a928e7ee5ca073086132c878700aae01b843
I'm seeing at least one relevant unit test failure but I think it is minor and I can fix it. But the build result should be ok to run as is..
Edit: Fixed some typos here and in comment 27.

Comment 39

[monperrus]

Gene, thanks for the last build. tested it, it works as expected.

Comment 40

[monperrus]

Gene, let me know how I can help

Comment 41

[gene smith]

Attachment: Bug 399475 - Allow shift-del for gmail to expunge from ALL folders with default gmail.com settings. r=mkmelin

This was suggested here: https://groups.google.com/g/mozilla.support.thunderbird/c/9T3VGSvPJgM
What it does is when you shift-delete on selected messages in a gmail folder, this moves the messages to
Trash folder and gmail auto-expunges the messages from the source folder. Then the new messages in Trash are
marked deleted by TB. Messages marked deleted in Trash are not auto-expunged by gmail and remain in Trash.
So the proposed code then does UID expunge on the messages moved to Trash. Assuming a deleted message has
not been copied to any other "visible in TB" folder, this causes such messages to be expunged by gmail
from "All Mail" as well, so the deleted messages are completely removed "forever".
The main change described above occurs under "case nsIImapUrl::nsImapAddMsgFlags:" in
nsImapProtocol.cpp.
This also merges in the changes proposed by Martin Monperrus in D218927 to allow pref
"expunge_after_delete" to have more of an effect.

More details:
(1) Shift-del for gmail uid expunging from all folders does require pref "expunge_after_delete" be true.
When expunge_after_delete is true, if UIDPLUS is supported a uid expunge is performed. If UIDPLUS
not supported, a full folder expunge is performed. Requiring this pref to be set true prevents
functionality changes with default TB settings.
(2) This changes the trash folder discovery so that the trash folder picker always shows the actual
folder being used for trash even if the user has not explicitly chosen a folder, i.e., no longer just
shows "Choose Folder" when a special-use \trash folder has been discovered and is actually in use.
(3) This no longer only does special-use \trash DiscoveryDone processing for gmail but now works with any
server that returns \trash special-use in a LIST or XLIST response when setting the server's
"trash_folder_name" pref and flagging a folder as trash.
(4) Trash folder picker no longer sets pref "trash_folder_name" to "Trash" when pref not yet set. This
avoids showing possibly incorrect (default) trash folder name in picker and in pref "trash_folder_name"
when trash folder not yet discovered or set.

Observed issues (not related to above changes AFAIK):
(1) Sometimes have to close and re-open "Advanced Preferences" or "Account Settings" tab to see changes in
preferences "trash_folder_name" or in trash folder picker for the account.
(2) After picking a different trash folder with picker in Account Settings, the "trash can" icon appears
correctly on the new folder but doesn't get removed from the original folder. It does get removed from
original folder after a TB restart.

Comment 42

[gene smith]

(In reply to monperrus from comment #40)

Gene, let me know how I can help

Martin,
I've submitted a review request to Magnus.
I also made a new try build with a few clean-ups and a fix for the unit test failure here:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=41ca1f33f122f853df651d1adc7fb1e32e9fc7fa
So if you could test it again and make sure it works it would be most helpful.
Thanks!

Comment 43

[gene smith]

(In reply to sa212+github from comment #36)

I'd like to test this on the (now closed) bug 1768322, but I don't have the bandwidth to build from source

You can also find the patched "daily" version here for your platform:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=41ca1f33f122f853df651d1adc7fb1e32e9fc7fa

Comment 29 show how to find the "installer" for your platform. Note: All the builds are optimized.

• can you tell me when this will make it to the esr update channel?

First it has to be reviewed and approved. If that happens it will go into daily for a while, then beta and finally ESR. So may be a long time. But if a lot of demand for this (which I doubt) it might be "fast tracked".

Comment 44

[monperrus]

Gene, tested the latest build, it works. Thanks for the progress.

Comment 45

[gene smith]

The patch for this is still pending a review approval and it's been a while since I submitted the patch, so I went back and looked at the issue again.

I got (at least) one thing wrong in the patch description. If the same message is in multiple folders (has multiple gmail labels) when the message is moved or copied to [Gmail]/Trash the message is expunged from all folders where it resides and from "All Mail" and is now only present in [Gmail]/Trash. Before I was thinking that it first had to be deleted/expunged from all other folders it resides in before MOVEing it into Trash to cause it to be removed from "All Mail". So actually when you do a normal delete of a gmail message to Trash with TB, the message will be expunged from ALL folders it resides in, including "All Mail" but, of course, is now present only in [Gmail]/Trash.

Imap UID MOVE and UID COPY to Trash have the same effect where the message is copied to [Gmail]/Trash and is auto-expunged from the source folder. The patch uses UID MOVE but UID COPY to [Gmail]/Trash would work exactly the same.

So instead of doing a shift-delete on a messages, just doing a default delete to Trash will have almost the same effect for gmail (with default gmail.com settings). This will remove all labels for the message (expunge it from all folders it resides in) and will expunge it from All Mail, just leaving a copy in [Gmail]/Trash. So the only extra step needed (compared to shift-delete) is to empty Trash (or just let gmail auto-empty trash after, I think, 30 days).

If a user really wants shift-delete to do what is expected, the patch is still needed. The main issue with shift-delete without the patch is that the message still remains in "All Mail" after being marked deleted and auto-expunged. The only way to remove/expunge a message in All Mail is to put the message in [Gmail]/Trash, which the patch does, and it also marks deleted and expunges the message put into Trash so the message is completely gone. It might be good to modify the warning message that occurs on shift-delete so that it is known that all copies of the message in different folder where it resides will also be permanently deleted (actually, only for gmail). Current message on shift-delete:

This will delete messages immediately, without saving a copy to Trash.
Are you sure you want to continue?

Proposed modified message on shift-delete:

This will immediately delete messages and possibly delete all copies of
the messages in different folders without saving a copy to Trash.
Are you sure you want to continue?

Just an observation: Another way to remove/expunge a message from All Mail is to copy or move the message from All Mail to [Gmail]/Trash. This expunges the message from all its residing folders leaving a copy in Trash. This would be more indirect so the patch doesn't do this. Note: Just marking a message in All Mail as imap \deleted doesn't expunge the message at all.

Comment 46

[monperrus]

Thanks for the update Gene. Agree on the proposed modified message.

Comment 47

[gene smith]

Attachment: Bug 399475 - Allow shift-del for gmail to expunge from ALL folders with default gmail.com settings.

Sorry, this was user error while attempting to request an uplift for another bug.