1857881 - Messages not expunged from INBOX on server

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Frank Gadegast]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/118.0

Steps to reproduce:

Automatic update to Supernova, it works with all versions before Supernova.

Actual results:

Mails are not removed from the INBOX on the server, when I remove them or move them with Thunderbird from the INBOX or into another folder.

The mails are copied to the new folder (or to trash when deleting) on the server, but also stay in the INBOX on the server, so they are still visible for other clients. There is no only a delay, the mails stay in the INBOX even if Thunderbird is terminated.

Expected results:

They should be removed from the servers INBOX.

Comment 1

[Magnus Melin [:mkmelin]]

Perhaps an IMAP log could help. https://wiki.mozilla.org/MailNews:Logging

Comment 2

[Helmut Leininger]

Maybe look at Bug https://bugzilla.mozilla.org/show_bug.cgi?id=1841583, it seems to be the same behaviour

Comment 3

[Frank Gadegast]

No, its not. Local folders have nothing to do with it.

I found this also happening to the Trashfolder now, when deleting mails quickly from it (by pressing the "delete" key again and again). Most emails get removed from the Trashfolder on the server, but some stay undeleted, even when TB does not display them anymore.

Comment 4

[Frank Gadegast]

Just rolled back to TB 102.15.1 and everything is fine again.

Mails get marked on the server as deleted in the moment, I delete them in TB and they get deleted ASAP when I quit TB.
Mails get moved from one folder to another on the server ASAP, when I move them in TB.

Everything fine again.

(I turned automatic updates off ASAP. SuperNova is also ignoring userChrome.css and has a view other quirks I do not have the time to report)

Comment 5

[Magnus Melin [:mkmelin]]

userChrome.css is not ignored. But almost certainly the css rules need to be updated since a lot of the internals changed.

Comment 6

[Alfred Peters [:infofrommozilla]]

This looks more like Bug 1855985.

Comment 7

[gene smith]

(In reply to Frank Gadegast from comment #3)

I found this also happening to the Trashfolder now, when deleting mails quickly from it (by pressing the "delete" key again and again). Most emails get removed from the Trashfolder on the server, but some stay undeleted, even when TB does not display them anymore.

When you press delete key on messages already in Trash folder, TB just marks the message with the \deleted flag. It doesn't permanently delete the message or expunge the folder at the server. So they should not be visible in TB after you hit delete on Trash messages (if they are, that's a bug).
However, depending of if other clients respect or understand the imap \deleted flag, they may or may not still show the messages marked with \deleted flag.
So if you want the Trash messages marked \deleted to be truly removed, you need to also compact the trash folder.

No, its not. Local folders have nothing to do with it.

The title of Bug 1841583 about "Local Folders" is misleading. It actually is similar to what you report, I think.

the mails stay in the INBOX even if Thunderbird is terminated

Are you moving between servers or within the same server? I think you mean within the same server. If so, that would depend on a couple server specific capabilities: MOVE and UIDPLUS. If server supports MOVE, message will be literally moved and auto-expunged by the server. If server only supports UIDPLUS, a move message will be copied, then marked as deleted and then UID EXPUNGED by TB to permanently remove it. If server support neither, message will just be copied and marked \deleted so other clients may still see it depending on how they treat \deleted flag.

Another change was made with 115 to not always do imap CLOSE on folders when shutting down described here: bug 1739833. This may be why the "other clients" are still seeing the messages marked deleted after you shutdown/terminate TB.

Perhaps an IMAP log could help. https://wiki.mozilla.org/MailNews:Logging

If you provide a log, just IMAP:4 will provide a more compact and easier to read file. (The link says to provide IMAP:5 the last time I looked.)

What is your IMAP server type? Also, what are the "other clients" that still see messages you have deleted in TB.

Comment 8

[Frank Gadegast]

When you press delete key on messages already in Trash folder, TB just marks the message with the \deleted flag. It doesn't permanently delete the message or expunge the folder at the server. So they should not be visible in TB after you hit delete on Trash messages (if they are, that's a bug).
However, depending of if other clients respect or understand the imap \deleted flag, they may or may not still show the messages marked with \deleted flag.
So if you want the Trash messages marked \deleted to be truly removed, you need to also compact the trash folder.

Well, Supernova does not mark them as deleted. Never. TB 102.x.x does.
Supernova does not delete them after it terminates (as last connected client). TB 102.x.x does.

Another change was made with 115 to not always do imap CLOSE on folders when shutting down described here: bug 1739833. This may be why the "other clients" are still seeing the messages marked deleted after you shutdown/terminate TB.

Ah, there we might have it. This probably does not work right.

Perhaps an IMAP log could help. https://wiki.mozilla.org/MailNews:Logging
If you provide a log, just IMAP:4 will provide a more compact and easier to read file. (The link says to provide IMAP:5 the last time I looked.)
What is your IMAP server type? Also, what are the "other clients" that still see messages you have deleted in TB.

Well, Im no developer, Im a user ... TB 102 works for me. I have no idea why developer change (and destroy) things, that already worked and need to invent "new features" that break lots of other things or remove older features.

Server is dovecot, current release, other clients are other TBs on Mac or Windows. Checked the folders on the server with mutt.

Comment 9

[gene smith]

(In reply to Frank Gadegast from comment #8)

When you press delete key on messages already in Trash folder, TB just marks the message with the \deleted flag. It doesn't permanently delete the message or expunge the folder at the server. So they should not be visible in TB after you hit delete on Trash messages (if they are, that's a bug).
However, depending of if other clients respect or understand the imap \deleted flag, they may or may not still show the messages marked with \deleted flag.
So if you want the Trash messages marked \deleted to be truly removed, you need to also compact the trash folder.

Well, Supernova does not mark them as deleted. Never. TB 102.x.x does.

By "supernova" I assume you mean 115. Anyhow, pressing delete key for me does mark the message in Trash as deleted (after it tells me incorrectly that it will be permanently deleted). The messages is actually just hidden from view since it's still on the server. I can bring it back with re-do (ctrl-z).

Supernova does not delete them after it terminates (as last connected client). TB 102.x.x does.

Yes that is currently the case, it doesn't expunge folder, but any message "deleted" should remain marked with \deleted flag. But I still don't understand why your "other clients" (other TBs, mutt, etc) are still showing messages marked deleted at the server unless you have them set to "show deleted messages, maybe with a strike-through" as TB can be configured in server settings with "Just mark it as deleted" instead of the default "Move it to this folder [typically Trash]"

Another change was made with 115 to not always do imap CLOSE on folders when shutting down described here: bug 1739833. This may be why the "other clients" are still seeing the messages marked deleted after you shutdown/terminate TB.

Ah, there we might have it. This probably does not work right.

Maybe depends on how you define "right" :).

Perhaps an IMAP log could help. https://wiki.mozilla.org/MailNews:Logging
If you provide a log, just IMAP:4 will provide a more compact and easier to read file. (The link says to provide IMAP:5 the last time I looked.)
What is your IMAP server type? Also, what are the "other clients" that still see messages you have deleted in TB.

I probably don't need a log since I have run my own log going to dovecot server and \delete flags seem to be getting set correctly on messages.

Well, Im no developer, Im a user ... TB 102 works for me. I have no idea why developer change (and destroy) things, that already worked and need to invent "new features" that break lots of other things or remove older features.

That "developer" would be me since I made the change. I may have incorrectly ASSumed that exiting TB without expunging \deleted messages would solve one issue and wouldn't cause other issues for users like you. I've since looked at it some more and thought some more about it and will propose returning to the original behavior -- do imap CLOSE as it was done with 102 and earlier.

Server is dovecot, current release, other clients are other TBs on Mac or Windows. Checked the folders on the server with mutt.

Yes, TB on mac and windows should not see messages on the server marked \deleted provided they are configured with delete method "move message to trash". I don't know how mutt works in that respect. Anyhow, I don't think this is affected by the imap CLOSE issue that I will propose to change back.

Comment 11

[gene smith]

Attachment: Bug 1857881 - Revert imap CLOSE functionality to ESR102 and earlier behavior r=mkmelin

Go back to doing imap CLOSE on folders with marked \deleted message so they are
expunged and thus can't appear as still present in other clients that maybe don't
respect the \deleted flag. This won't affect accounts configured as "just mark
message as deleted" as should be used when doing bulk folder moves between servers.
This effectively reverts most of the the changes to nsImapProtocol.cpp/.h
intoduced in Bug 1828372 but it still allows inhibiting autoexpunge when the
user sets preference mail.imap.expunge_option to 3. It also uses a new name
for imap close function, ImapClose(), to avoid ambiguity with other close
methods in the huge .cpp file.

Comment 12

[gene smith]

Reporter Frank,
I'm still not sure I understand what you are seeing, although I think my proposed "Revert" in comment 11 is valid. I'm just not sure it is the cause of your issue.

Just to be sure, I've tested again without the comment 11 change putting me back to 115 functionality as you are running:

Test 1:
On TB instance 1, I deleted a message in Inbox to Trash, shutdown TB instance 1 and started TB instance 2 (on a different profile) and that message didn't appear in TB instance 2 Inbox. Both profiles are configured to "move messages to trash on delete".
In this case, when I delete the message to Trash, it is actually imap MOVE'd to trash. (The server is dovecot which supports MOVE but not all imap servers do.) Imap MOVE automatically expunges (permanently deletes) the message so shutting down TB afterwards doesn't matter.

Test 2:
On TB instance 1, I move a message from Inbox to a folder on another account/server. Then I shutdown TB instance 1 and start TB instance 2 (again, on a different profile) and that message again didn't appear in TB instance 2 Inbox. In this case, when I move the message to a different imap server, the original message just remains marked with \deleted flag in Inbox and is not expunged, even after TB instance 1 shutdown. The only way I can still see the marked \deleted message on TB instance 2 is to set the delete method to "just mark as deleted" and restart instance 2. Then I see the message crossed out as expected. (Crossed out messages can be expunged by doing a "compact" on the folder.)

Could you try a similar test as these with your "TB instance 2" being the other TBs you are saying you run on Mac and/or Windows?
Thanks.

Comment 13

[Frank Gadegast]

For this scenario, it's enought to work on one server and only use one mailbox, but with several clients (I use TB on a Mac Desktop, TB on a Macbook, sometimes Apple Mail on a MAcbook, TB on Windows, Apple Mail on an iPhone and I can check the mailbox and its folders with "mutt" and by checking the files themself on the server, because it's my root server running dovecot).

I move messages out of the inbox to other folders on this mailbox, after I worked with them. TB 102 moves them straight away, they disappear from the inbox and this is visible for all connected clients straight away and this is even visible for "mutt" directly on the server. TB 115 does not. The mails stay in the inbox, they are not even marked as deleted, they are not moved, they are copied into their destination folders.

If I delete messages, TB is configure to move them to the Trash. Same problem, TB 102 moves them like expected, TB 115 copies them, without marking them as deleted.

It does not even matter, wich client is connected at the same time or if they connect one after each other or if TB 115 is terminated before or after other clients or if accounts are on one server or another. Moved or deleted messages stay in the folder and all other clients still those message when the are connected, when they reconnect, because they are still in the inbox and exist in the servers inbox as files. They are not marked as deleted. Remember, I can always check with "mutt" directly on the server or by checking dovecots files and logs.

(I have several servers, several profiles, quite a view accounts, TB 115 acts wrong on all of them. Moved or deleted messages stay in the old folder and all other clients still see those message. They are never deleted and kind of pile up. Even, if I tell dovecot to expunge all deleted messages and sync all mailboxes. Even, when I terminate all clients and dovecot.)

Comment 14

[Frank Gadegast]

I forgot: TB 115 is the only client that does not show the messages in the inbox after I deleted them with TB 115. So, everything looks ok with TB 115. Until you connect or work with other clients or check on the server ...

Comment 15

[gene smith]

Ok, I think I see what you mean. Please try this simple test with your TBs all configured with server setting "When I delete a message Move it to this folder: [trash folder name]".

1. shutdown TB 115 on all your computers (you don't need to reboot the computers)
2. start TB 115 on computer A
3. delete an email from inbox on computer A dovecot account
4. start TB 115 on computer B
5. Is the email you deleted on computer A present on Computer B dovecot account Inbox?

It sounds like you will say the answer to 5. is "yes, it is still there".

Two more question:

A. Just curious, which computer is your dovecot server running on?
B. Is your Trash folder also on the same dovecot account as the Inbox? Never mind, I forgot that Trash has to be on same server.

Comment 16

[gene smith]

I've tried some more and don't see what you see with 115 using multiple client TBs and a dovecot server. Maybe I needed a few more details:

1. Do you have TB set to get new mail on a timed basis? If so, how many minutes? It's server setting "Check for new messages every X minutes".
2. Have you enabled the server setting "Allow immediate server notification when new messages arrive"? This just enables use of imap IDLE.
3. Do you have imap CONDSTORE enabled? It is enabled with preference mail.server.default.use_condstore in config editor. By default, it is disabled.

The issues I do see is if you have two TB clients looking at the same imap server/account and you delete a message on one TB client it will not immediately go away on the other client if:

The other client is not using imap IDLE, or if it is using IDLE but the imap connection has timed out (typically after 30 minutes inactivity) and there is no timed checked for new mail or the time is set very large so that the connection is not present when the message is deleted by the other client. Therefore, setting the "check for new messages" timer to < 30 minutes, if not already set in this range, might help.

If CONDSTORE is enabled, there's a known TB bug that prevents a message deleted by another client to from going away without a restart. It's described here:
Bug 1747311 comment 3 2nd paragraph.

Just to be (hopefully) clear, none of the above issues are due to the message not getting marked \deleted or not expunged by TB at the server, dovecot or others. They are just display issues that are cleared after, at most, a TB restart. (I think you are saying that even a TB restart doesn't remove the display of messages deleted by the other TB client.)

Comment 17

[gene smith]

If neither of my two previous comments (comment 6 and comment 7) point to anything that resolves this, I really need you to record an IMAP:4 log. To keep it simple, just record the log on one TB instance running 115 and just delete a message to Trash. You should see something like this in the log:

wally.dbnet.lan:S-INBOX:SendData: 94 uid move 720 "Trash"
wally.dbnet.lan:S-INBOX:CreateNewLineFromSocket: * OK [COPYUID 1515999045 720 467707] Moved UIDs.
wally.dbnet.lan:S-INBOX:CreateNewLineFromSocket: * 19 EXPUNGE
wally.dbnet.lan:S-INBOX:CreateNewLineFromSocket: 94 OK Move completed (0.099 + 0.000 + 0.098 secs).

This shows TB 115 requesting dovecot to move message 720 to Trash folder. The other 3 lines are the response by dovecot. You can see that the move was successful and the message UID 720 (with sequence number 19) was expunged.

You can follow the instruction here: https://wiki.mozilla.org/MailNews:Logging#Mac_OS_X
or, you can just run TB from a bash command line like this:

MOZ_LOG=IMAP:4,timestamp,sync  MOZ_LOG_FILE=$HOME/Desktop/tblog  /Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin  -p --allow-downgrade &

I'm assuming that the path to TB executable is correct. If not, you can adjust it. Also, this will put the resulting log file on the Desktop but you can change this to put it and name it whatever you want. The resulting log file will be named tblog.moz_log (with "moz_log" automatically appended to the name you provide).
The parameter -p lets you select a profile if you have more than one and the --allow-downgrade will let you run with "older" versions, e.g., 102, if you want without insisting you create a new profile when you go back to 115, so these are optional but I always include them.

So just run TB via the above command line to record the log and ONLY do a message delete from Inbox into Trash. Then check if the log shows something like what I quoted above. You will see lots of other stuff but you should be able to search the resulting log file for "uid move" and find where the move to trash occurs.
If you don't see a "uid move" then I'm curious to see the whole log to know what happened.

Comment 18

[Wayne Mery (:wsmwk)]

Frank?

Comment 19

[Magnus Melin [:mkmelin]]

Should we get this landed?

Comment 20

[gene smith]

(In reply to Magnus Melin [:mkmelin] from comment #19)

Should we get this landed?

Yes, it should be still OK. But not 100% sure it actually fixes this bug as reported since never heard back from reporter.
Also, changed summary since this has nothing to do with supernova.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/fb54b78ccc74
Revert imap CLOSE functionality to ESR102 and earlier behavior r=mkmelin

Comment 22

[Daniel Darnell [:dandarnell]]

Attachment: Bug 1857881 - Fix formatting issue. r=#thunderbird-reviewers

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/7bf861777a89
Fix formatting issue. r=mkmelin DONTBUILD

Comment 24

[Wayne Mery (:wsmwk)]

Good for esr115?

Comment 25

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #24)

Good for esr115?

Yes, it's fine.

Comment 26

[Wayne Mery (:wsmwk)]

Comment on attachment 9357750 [details]
Bug 1857881 - Revert imap CLOSE functionality to ESR102 and earlier behavior r=mkmelin

[Triage Comment]
Approved for esr115

Comment 27

[Wayne Mery (:wsmwk)]

Comment on attachment 9362748 [details]
Bug 1857881 - Fix formatting issue. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr115

Comment 28

[Rob Lemley [:rjl]]

Thunderbird 115.5.2:
https://hg.mozilla.org/releases/comm-esr115/rev/40bcfc35c6e0
Thunderbird 115.5.2:
https://hg.mozilla.org/releases/comm-esr115/rev/743dc8d5e201