Clean up ("Expunge") Inbox on Exit disabled setting not honoured when not checked - messages are being expunged
Categories
(Thunderbird :: Account Manager, defect)
Tracking
(Not tracked)
People
(Reporter: daves, Unassigned)
References
Details
Attachments
(2 files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0
Steps to reproduce:
I have IMAP message deletion set to 'Just mark it as deleted'. I then have 'Clean up ("Expunge") Inbox on Exit' unchecked. I have two PC's (laptop & desktop). One can be used to monitor what the other does.
Using Windows 10x64. Fastmail IMAP server.
Actual results:
Exiting Thunderbird causes immediate Expunge. This can be seen on the second PC. Messages marked for deletion disappear when Thunderbird is closed.
Expected results:
Closing Thunderbird with the 'Clean up ("Expunge") Inbox on Exit' setting unchecked should not expunge any folder. Nothing should happen.
Comment 1•1 year ago
|
||
TCW does this reproduce for you?
Updated•1 year ago
|
Comment 2•1 year ago
•
|
||
(In reply to Dave from comment #0)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0
Steps to reproduce:
I have IMAP message deletion set to 'Just mark it as deleted'. I then have 'Clean up ("Expunge") Inbox on Exit' unchecked. I have two PC's (laptop & desktop). One can be used to monitor what the other does.
Using Windows 10x64. Fastmail IMAP server.
Actual results:
Exiting Thunderbird causes immediate Expunge. This can be seen on the second PC. Messages marked for deletion disappear when Thunderbird is closed.
Expected results:
Closing Thunderbird with the 'Clean up ("Expunge") Inbox on Exit' setting unchecked should not expunge any folder. Nothing should happen.
I want to make sure I am understanding this clearly as these STR are not super clear for me.
I have IMAP message deletion set to 'Just mark it as deleted'
Where is this setting?
STR:
- In IMAP account Settings > Server Settings, "Clean up ("Expunge") Inbox on Exit" is [ ] / Disabled
- In IMAP account Settings > Server Settings, "Set Empty Trash on Exit" is [ ] / Disabled
'Just mark it as deleted' is on the Server settings page. There is a radio box selection headed 'When I delete a message:'. The default is 'Move it to this folder:'. The second is 'Just mark it as deleted'.
Updated•1 year ago
|
Comment 4•1 year ago
|
||
(In reply to Dave from comment #3)
'Just mark it as deleted' is on the Server settings page. There is a radio box selection headed 'When I delete a message:'. The default is 'Move it to this folder:'. The second is 'Just mark it as deleted'.
Ok, I see it now. I'll see if I have time this weekend to try and repro this on my home setup.
Comment 5•1 year ago
•
|
||
I am confirming this does repro for me on 115.8/115.8.1.
My STR:
- Have two PCs with TB running on them and send yourself a few Test messages to one IMAP account that both PCs share so they are in the Inbox
- Go to the account's Settings > Server Settings > and set "When I delete a message:" to "Just mark it as deleted"
- Set Server Settings > Message Storage to:
[ ] Clean up ("Expunge") Inbox on Exit (a.k.a. Disabled, TB should take no action)
[ ] Empty Trash on Exit (a.k.a. Disabled, TB should take no action) - On one PC, start selecting each test message and press the Delete key. A visible trike-through will appear marking the message as having been deleted
Expected Results:
- The message(s) should stay in the Inbox of both PCs in a state of having been deleted (displayed by strike-through) but not expunged
Actual Results:
- The message(s) stay in the Inbox in a state of having been deleted (displayed by strike-through) on the PC that did the Delete action but is immediately expunged on the secondary PC
Other observations.
- A CTRL-Z action on the PC that did the Delete action doesn't undelete/un-expunge on the secondary PC
- Shutting down TB and then restarting TB on the PC that did the Delete action will briefly show the deleted messages in the Inbox but then they too will disappear within a few seconds
Comment 6•9 months ago
|
||
I am confirming this does repro for me on 115.8/115.8.1
Arthur K.,
Any chance you are doing this on a server that auto-expunges messages that are marked deleted, such as gmail?
I tried this on a dovecot server and do not see a problem. Nothing is expunged while running or at shutdown after "just marking messages as deleted". When TB comes back up, all the "marked deleted" messages are still there and crossed out. IMAP:4 log shows no expunges occurring.
Reporter Dave is using Fastmail server which is cyrus. I just tested with cyrus and and don't see any auto-expunge going on there. So comment 0 doesn't seem like it could occur either.
If anyone can still duplicate this I would be curious to see IMAP:4 level log to see what is really happening at shutdown. Re: https://wiki.mozilla.org/MailNews:Logging. Just recording this on one TB instance on one PC should be sufficient. Just start TB with log recording, mark some message in Inbox deleted and shutdown TB and then attach the log.
I have attached a log file. I am now on 115.11.0 64 bit. The bug still remains. I have deleted some lines and various bits in the file that are my user id or folder names. I hope that has not removed anything needed.
The method for creating the log file in the wiki is out of date in that it refers to the 32 bit folder name. The 64 bit folder name is the same as the XP name.
Comment 9•9 months ago
|
||
Thanks for the log.
I only see one "mark deleted" activity on a single message:
2024-05-20 07:54:13.227000 UTC - [Parent 10492: IMAP]: I/IMAP 222a7ea5d00:mail.messagingengine.com:S-INBOX:SendData: 97 uid store 228273 +Flags (\Seen \Deleted)
and I don't see any mention of "expunge" after that. Not sure if you maybe cut out some relevant lines since the file ends with a partial line:
2024-05-20 07:54:17.166000 UTC - [Parent 10492: IMAP]: I/IMAP 222a7ea560...
The final lines of the log should say things about closing, logging out and leaving the main loop.
Was there anything that contains string "expunge" (ignoring case) that you may have cut out?
The method for creating the log file in the wiki is out of date in that it refers to the 32 bit folder name. The 64 bit folder name is the same as the XP name.
Yes, I think that was written when TB was primarily 32 bit so it should just say "Program Files" without the (x86).
Reporter | ||
Comment 10•9 months ago
|
||
Reporter | ||
Comment 11•9 months ago
|
||
I do not know what happened to the end of that file, but I have attached another. I have deleted everything prior to the (\Seen \Deleted) line. I do not see expunge anywhere in the file.
Comment 12•9 months ago
|
||
Correct, nothing about expunge in new log.
I checked the cyrus and fastmail documentation and it doesn't have an option to "auto-expunge" like gmail, that I can find.
When you start back up and the message that was crossed out is gone, could you record a log on that startup and see if an expunge occurs on startup? That might show something.
There are some preferences in Config Editor that control expunge. When you are set to "just mark as deleted" by default nothing is expunged but that can be changed with non-default settings. So check to make sure these are at default:
mail.imap.expunge_after_delete (default false)
mail.imap.expunge_option (default 0)
mail.imap.expunge_threshold_number (default 20)
FYI, config editor is accessed with a button near the bottom of the General settings page.
If any of these are not default, it may be causing the problem. If not default, set them to default and restart TB and see if it helps. If not, I probably still need to see the log for the next startup.
Reporter | ||
Comment 13•9 months ago
|
||
The expunge definitely happens when Thunderbird exits as it can be seen on a second PC. The config settings are at their defaults. I can record the log anyway tomorrow. I guess it would also be useful to find another email client to see what happens with that.
Comment 14•9 months ago
|
||
(In reply to Dave from comment #13)
The expunge definitely happens when Thunderbird exits as it can be seen on a second PC. The config settings are at their defaults. I can record the log anyway tomorrow. I guess it would also be useful to find another email client to see what happens with that.
I'm wondering if the behavior would repro if you tested in a sandbox with TB Portable?
Comment 15•9 months ago
|
||
(In reply to gene smith from comment #6)
I am confirming this does repro for me on 115.8/115.8.1
Arthur K.,
Any chance you are doing this on a server that auto-expunges messages that are marked deleted, such as gmail?I tried this on a dovecot server and do not see a problem. Nothing is expunged while running or at shutdown after "just marking messages as deleted". When TB comes back up, all the "marked deleted" messages are still there and crossed out. IMAP:4 log shows no expunges occurring.
Reporter Dave is using Fastmail server which is cyrus. I just tested with cyrus and and don't see any auto-expunge going on there. So comment 0 doesn't seem like it could occur either.
If anyone can still duplicate this I would be curious to see IMAP:4 level log to see what is really happening at shutdown. Re: https://wiki.mozilla.org/MailNews:Logging. Just recording this on one TB instance on one PC should be sufficient. Just start TB with log recording, mark some message in Inbox deleted and shutdown TB and then attach the log.
I'm on GMail accounts save for one Exchange. What I did in comment 5 I did to my GMail accounts.
Comment 16•9 months ago
|
||
The expunge definitely happens when Thunderbird exits as it can be seen on a second PC.
I don't think you have said nor have I asked about the 2nd PC. Is it also running TB? If so, is it set up to move deleted messages to Trash?
What I see is my "first" TB instance set up as you describe (mark as deleted) never does an expunge. So I don't think the issue is there and logs won't tell us anything more. However, on my 2nd TB instance accessing the same account but with setting "moved deleted messages to Trash", when it thinks there are more than 20 messages marked deleted, it goes ahead and expunges the folder causing the marked deleted messages on TB 1 to disappear. But I've seen this happen when only 1 message is marked deleted on TB 1, but usually I have to mark maybe 15 or 16.
So here's some things to try on your 2nd PC that I'm assuming is running TB:
Set it to also "just mark as deleted"
and/or set the parameter mail.imap.expunge_option
to 3 (never auto-expunge)
and/or set the parameter mail.imap.expunge_threshold_number
to something a lot bigger than default 20, like maybe 1000
If you make any of these changes, probably good idea to restart TB 2 on 2nd PC.
Any one of these should fix the problem, I think.
But I still need to determine why the 2nd TB sometimes thinks more message are marked deleted than there really are.
Reporter | ||
Comment 17•9 months ago
|
||
Both of my PC's are setup in the same way. I did try changing the two configuration settings to what you have marked above and it made no difference. Using just the one PC, if I mark a message as deleted then close and re-open TB then the message is gone. I had not tried setting Work offline previously, but doing that also causes messages marked for deletion to disappear as soon as work offline is selected.
I tried with MS Outlook which also causes messages to be expunged when closing. I tried changing some settings that looked relevant but the result is the same as with TB. I do not like Outlook as it is a real mess. There could be settings I have missed that I should have changed.
Could this bug be an issue at Fastmail rather than with TB? I could try posting at the forums there or submit a support request.
Comment 18•9 months ago
|
||
Interesting that outlook does the same thing with fastmail. Maybe you could provide me a temporary test account on fastmail if they allow more than one account. I could test it from here. Or maybe Bron (at cyrus/fastrmail) knows what's going on.
I couldn't find any mention of a setting in cyrus or fastmail that does "auto-expunge" when an imap message is marked \deleted.
Reporter | ||
Comment 19•9 months ago
|
||
It looks like the expunge happens when the mail client disconnects. The only settings I know of are for the web interface.
Comment 20•9 months ago
|
||
Maybe you could provide me a temporary test account on fastmail if they allow more than one account.
Never mind. Fast mail allow a free 30 day account that I've set up. Will look at the problem now.
It looks like the expunge happens when the mail client disconnects.
Imap "close" causes expunge. I need to look closer at that.
Comment 21•9 months ago
•
|
||
Ok, I see it too with a trial fastmail account. With just mark it as deleted, TB never sends an imap CLOSE; it just sends imap LOGOUT. Imap CLOSE is supposed to permanently delete any messages marked deleted. But LOGOUT doesn't. So it looks like fastmail is not following the RFC?
That seems strange since doing the same thing with cyrus (and other servers I have, e.g., dovecot) don't expunge messages on shutdown when only LOGOUT is done.
Re: https://datatracker.ietf.org/doc/html/rfc3501#section-6.4.2
Reporter | ||
Comment 22•9 months ago
|
||
Thanks for checking. I have just posted a message at the emaildiscussions Fastmail forum. Hopefully it will be picked up there or here.
http://www.emaildiscussions.com/showthread.php?p=635194#post635194
Comment 23•9 months ago
|
||
IMAP \Deleted and EXPUNGE is a horrid wart upon the protocol and you shouldn't be depending on it. Yes, we delete immediately. Yes, that's allowed by the specs - an IMAP client can't assume it's the only client connected to a mailbox and hence another client can expunge messages - in this case, our JMAP server assumes it can always insta-expunge messages because otherwise the message counts don't work correctly.
Comment 24•9 months ago
•
|
||
Hi Bron, Thanks for the info.
Testing with two TB clients to same fastmail account/server, when I mark \deleted a message on client A, all I see on client B imap log is the same message UID with \deleted flag set. I don't see an EXPUNGE reported on client B for that message until I shutdown client A. (The EXPUNGE is reported in an IDLE response.) So there doesn't seem to be an "insta-expunge" occurring when \deleted flag is set like gmail does.
Since both TB clients are configured to "just mark deleted messages as \deleted", on shutdown TB sends just LOGOUT (not CLOSE) so per https://datatracker.ietf.org/doc/html/rfc3501#section-6.4.2 an expunge should not occur.
Maybe I'm not understanding your answer since I'm not really familiar with what JMAP has to do with this.
Also, FWIW, the distro supplied cyrus version I run (just for testing TB issues) doesn't have this issue (messages marked \deleted are not expunged after LOGOUT) and there is no auto/insta-expunge when a message is marked \deleted.
Comment 25•9 months ago
|
||
Bron,
Probably not important but here's the ID response I'm seeing from fastmail:
Cyrus IMAPD" "version" "3.11.0-alpha0-497-g97f96844c-fm-20240526.001-g97f96844
and here's the ID response from my local cyrus server:
Cyrus IMAPD" "version" "3.4.3-dirty-Debian-3.4.3-3ubuntu0.1
I did another experiment to see if there is really an insta/auto-expunge when a message is marked deleted. I started TB client A and marked message UID 9 as deleted. In the tb imap log I see that this occurs via "UID STORE" command. TB draws a cross-out line through the message as it should.
I then start tb client B and it starts up with message UID 9 also crossed out, so it still "sees" the message, it does not seem to be "auto-expunged" (yet). The initial fetch of flags for inbox shows this untagged response for message uid 9 on client B:
* 6 FETCH (FLAGS (\Deleted \Seen $X-ME-Annot-2) UID 9)
So, to client B, the message is still present, just marked \deleted and \seen.
I then just let client A sit idle for a while running IDLE command with nothing else going on. After about 1 hour, the fastmail server responds to IDLE with
* BYE idle for too long
Then immediately in the client B imap log, I see the untagged IDLE response
* 6 EXPUNGE
where 6 is the sequence number corresponding to message UID 9.
So what this seems to show is there is no "insta/auto-expunge" immediately when a message is marked deleted. But the expunge occurs when the imap connection goes away, e.g., when client initiates the termination via LOGOUT or CLOSE or when the fastmail closes the connection (e.g., when IDLE too long).
I guess you could say the auto-expunge by fastmail is deferred until the connection is shutdown.
Yes, we delete immediately. Yes, that's allowed by the specs
I think it is more that RFC 3501 (and future 9051) doesn't explicitly forbid it but doesn't really address it. Anyhow, I was told the same rationale for auto-expunge by Brandon Long for gmail back in 2017. Here's a link to that discussion (the full imap-protocol list archive is no longer available at washington.edu):
https://marc.info/?l=imap&m=150794397802823&w=2
I haven't tested this on gmail recently, but I suspect it does an immediate auto-expunge right when the deleted flag is stored. I don't think it defers the expunge until the connection is closed.
Comment hidden (offtopic) |
Comment 28•4 months ago
|
||
Closing this as INVALID since this is caused by Fastmail's version of Cyrus IMAP server expunging the folder when imap LOGOUT is sent, which is a violation of RFC 3501 at https://datatracker.ietf.org/doc/html/rfc3501#section-6.4.2, specifically:
The SELECT, EXAMINE, and LOGOUT commands implicitly close the
currently selected mailbox without doing an expunge.
Based on testing with trial Fastmail account, expunge of messages having \deleted flag occurs, as it should, on imap CLOSE. But expunge also occurs on imap LOGOUT and after fastmail server closes a timed-out connection.
Description
•