Open Bug 1644130 Opened 4 years ago Updated 6 months ago

Invisible mails marked as deleted lead to IMAP quota being exceeded

Categories

(Thunderbird :: Folder and Message Lists, defect, P5)

Tracking

(Not tracked)

People

(Reporter: Thunderbird_Mail_DE, Unassigned)

References

Details

Steps to reproduce:

1.) Create an IMAP account in Thunderbird
2.) In Account Manager choose the option to "Mark as deleted" for deleted messages
3.) Delete some Messages - they should be marked as deleted.

Do NOT compact the folder (they should not be expunged at the moment).

4.) In Account Manager change the option from "Mark as deleted" to "Move to Trash folder".

Now in Folder Pane all deleted messages disappear.

The issue with these disappaering messages is, that they are not visible but they fill the IMAP quota (size and/or number). This is a major problem, if many messages are marked as deleted and the number exceeds limitations of your provider (e.g. T-Online has a limit of 1000 messages per folder).

The problem escalates unnoticed if you use multiple IMAP clients (or multiple Thunderbird installations) with different settings. If one client only marks "deleted", the other clients Users cannot see that the mailbox is full.

The problem becomes even worse if the deleted marked messages are not displayed in Webmail. In the case of T-Online, a quota of, for example, 20% is displayed in webmail. Via IMAP, the quota is then 100% (exceeded), although you may only have 10 visible messages in both the webmail and Thunderbird.

Compacting folder would resolve the problem. But there are cases, where the compacting fails (today we had a support request in german forums with 2 undeletable / uncompactable messages of 5000 messages in Trash. Because of not beeing able to compact, all 5000 messages remained in Trash. Due to T-Onlines limit of 1000 messages the whole mailbox was blocked. The user had no chance to find the issue because of the invissible 5000 messages in this trash folder.

This leads to incomprehensible Quota Exceeded messages, which can only be corrected by chance after a long search. The providers Support Team was unable to find these invisible messages.

What I would like to see improved in Thunderbird:
Thunderbird should always display messages marked as deleted via IMAP in the folder pane - even if you select the option "Move to trash when deleting" in the account settings because the messages previously marked "deleted" or marked as deleted in another client are still available . There is no reason why these still existing messages should be hidden!

I don't think we could show the deleted messages like suggested. I guess when changing from "mark-as-deleted" we could ask/remind to expunge at the same time.

Reminding to compact/expunge when changing the option is only for this single user helpfull. It doesn't inform other users / clients about the wasted / used space in this mailbox.

Why do you think, we could not show the deleted messages like suggested? Maybe we could do this as an additional option.

Besides enormous complications, depending on the server users the "moved" messages would show (or not), with no way to users who want the current behaviour to obtain that.

Even if someone (like me) uses the option "Move messages to trash when deleting", they still want to know if there are any messages that have not yet been completely deleted or that still consume quota / storage space.

Where is the technical problem of making the mails marked as deleted always visible - even in the other mode (move to folder xyz)? AFAIK it is possible to move and finally delete these mails marked prior as deleted in the other mode.

(In reply to Alex Ihrig [:Alex_Ihrig] from comment #0)

Steps to reproduce:

1.) Create an IMAP account in Thunderbird
2.) In Account Manager choose the option to "Mark as deleted" for deleted messages
3.) Delete some Messages - they should be marked as deleted.

Do NOT compact the folder (they should not be expunged at the moment).

4.) In Account Manager change the option from "Mark as deleted" to "Move to Trash folder".

Now in Folder Pane all deleted messages disappear.

The issue with these disappaering messages is, that they are not visible but they fill the IMAP quota (size and/or number). This is a major problem, if many messages are marked as deleted and the number exceeds limitations of your provider (e.g. T-Online has a limit of 1000 messages per folder).

Seem like a pretty drastic limit of 1000 messages/folder. Does T-Online also limit the number of folders (mailboxes) you can have and the amount of storage in a folder? FYI, tb latest beta has an improved quota display that will show any and all types of quota (called quotaroots) that the server defines for a folder, e.g., Storage (in Kb), number of messages and number of mailboxes etc. I'd be curious to know if T-online actually shows these quotas. (Well, not sure it's in beta but definitely in daily.)

The problem escalates unnoticed if you use multiple IMAP clients (or multiple Thunderbird installations) with different settings. If one client only marks "deleted", the other clients Users cannot see that the mailbox is full.

The "improved" quota should be visible on any tb client so maybe this will help?

The problem becomes even worse if the deleted marked messages are not displayed in Webmail. In the case of T-Online, a quota of, for example, 20% is displayed in webmail. Via IMAP, the quota is then 100% (exceeded), although you may only have 10 visible messages in both the webmail and Thunderbird.

Does T-Online support imap MOVE or UIDPLUS capability? If imap MOVE is supported, any message deleted and "moved to trash" will actually be expunged from the source folder by the server. If move is not supported but UIDPLUS is supported, messages deleted and "moved to trash" will be "copied to trash" and then UID expunged by tb command from the source folder, so this effectively simulates MOVE. But if neither MOVE or UIDPLUS is supported by the server, the deleted message will by copied to trash and left marked as \deleted in source folder and invisible with "move to trash" selected. If "just mark as deleted" is then selected, the invisible messages appear with the red X and strike-through. (You may need to record an IMAP:5 log to know the supported capabilities: https://wiki.mozilla.org/MailNews:Logging.)

Compacting folder would resolve the problem. But there are cases, where the compacting fails (today we had a support request in german forums with 2 undeletable / uncompactable messages of 5000 messages in Trash. Because of not beeing able to compact, all 5000 messages remained in Trash. Due to T-Onlines limit of 1000 messages the whole mailbox was blocked. The user had no chance to find the issue because of the invissible 5000 messages in this trash folder.

This sounds like a T-Online bug. Do they not send you a warning email or SMS when the (very strict) quota is close to being reached and possibly locking the mailbox?
I'm not clear on how 5000 messages got into Trash and all marked as \deleted and invisible (or maybe just two?). Tb always move/copies to Trash and leaves the \deleted flag off so they are not invisible sitting in Trash. Then when you empty trash the messages are all marked \deleted and then the Trash folder is expunged, permanently deleting them all.

This leads to incomprehensible Quota Exceeded messages, which can only be corrected by chance after a long search. The providers Support Team was unable to find these invisible messages.

Reading again, not sure what you mean by "2 undeletable / uncompactable" in trash. Did tb cause this?

What I would like to see improved in Thunderbird:
Thunderbird should always display messages marked as deleted via IMAP in the folder pane - even if you select the option "Move to trash when deleting" in the account settings because the messages previously marked "deleted" or marked as deleted in another client are still available . There is no reason why these still existing messages should be hidden!

Maybe I'm missing something, but you can just change the delete mode back to "just mark it as deleted" and you will see any invisible messages (marked as \deleted) with the red X and strike-through. But I think you are saying you want to see this always.
P/S: I vaguely remember a request like this before in another bug report but can't locate it.
Edit: This is the same request as bug 1477557.

Seem like a pretty drastic limit of 1000 messages/folder. Does T-Online also limit the number of folders (mailboxes) you can have and the amount of storage in a folder?

T-Online has a limit on the number of messages per folder, the number of folders and the storage space. This limit depends on the tariff booked. Because T-Online is the largest German Internet provider, many users have the associated free mailbox tariff with 1000 messages per folder.

The "improved" quota should be visible on any tb client so maybe this will help?

This would be a great improvement.

Does T-Online support imap MOVE or UIDPLUS capability? If imap MOVE is supported, any message deleted and "moved to trash" will actually be expunged from the source folder by the server. If move is not supported but UIDPLUS is supported, messages deleted and "moved to trash" will be "copied to trash" and then UID expunged by tb command from the source folder, so this effectively simulates MOVE. But if neither MOVE or UIDPLUS is supported by the server, the deleted message will by copied to trash and left marked as \deleted in source folder and invisible with "move to trash" selected. If "just mark as deleted" is then selected, the invisible messages appear with the red X and strike-through. (You may need to record an IMAP:5 log to know the supported capabilities: https://wiki.mozilla.org/MailNews:Logging.)

I can't answer that for sure and I can't test it either because I don't have a T-Online account. I ask in the German support forum whether someone can test it and create an IMAP log.

I'm not clear on how 5000 messages got into Trash and all marked as \deleted and invisible (or maybe just two?). Tb always move/copies to Trash and leaves the \deleted flag off so they are not invisible sitting in Trash. Then when you empty trash the messages are all marked \deleted and then the Trash folder is expunged, permanently deleting them all.

This is a good question that the user concerned will probably not be able to answer with certainty. However, the user had received a warning for 96 % quoata via IMAP in Thunderbird. In the webmail account, however, only about 25 % quota were given (because the invisible, deleted-marked ones were also not visible there). So the warning could not be understood. The user then tentatively deleted lots of emails (to my knowledge, via the "Move to trash" option). Maybe that's why there were almost 5000 deleted-marked mails in the trash, which were still not visible in both Thunderbird and Webmail. The result of this was that the quota was then at 100%. The trash could then not be compacted due to the two "defective" emails. These two messages have probably prevented compacting for 3 months.

Reading again, not sure what you mean by "2 undeletable / uncompactable" in trash. Did tb cause this?

I don't know, what the problem of these 2 "defective" messages is. Invoking the compacting process leads to a neutral network connection error message. But these 2 messages can be moved to other folders. But they are still not completely deletable in the other folder.

Maybe I'm missing something, but you can just change the delete mode back to "just mark it as deleted" and you will see any invisible messages (marked as \deleted) with the red X and strike-through.

But first you have to get the idea to switch the option in Thunderbird and then discover the deleted marked messages. You can't expect that from most "normal" users. Thunderbird used the Account Assistant to guide many users to IMAP accounts. These users would probably have always set up POP accounts, even if the special problem here could have arisen from the use of several (differently configured) IMAP clients. I think we have to help these "naive" users now that they can use IMAP and not fall into traps. One of these common pitfalls is the server quota problem.

But I think you are saying you want to see this always.

IMHO, this would be a good idea and I have no idea, why it would/could be bad.

T-Online (uses Dovecot) IMAP capabilities:

[(null) 2029: Unnamed thread 0x7fa1cedce940]: I/IMAP 0x7fa1cd758000:secureimap.t-online.de:NA:CreateNewLineFromSocket: 2 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SPECIAL-USE METADATA XEXPIRE QUOTA] Logged in

The server supports imap MOVE. Here's what you should see in the imap log when you delete 3 message to Trash:

[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:SendData: 14 uid move 497:499 "Trash"
[(null) 90451: IMAP]: D/IMAP ReadNextLine [stream=0x7f7f36378920 nb=52 needmore=0]
[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:CreateNewLineFromSocket: * OK [COPYUID 1515999045 497:499 8:10] Moved UIDs.
[(null) 90451: IMAP]: D/IMAP ReadNextLine [stream=0x7f7f36378920 nb=13 needmore=0]
[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:CreateNewLineFromSocket: * 5 EXPUNGE
[(null) 90451: IMAP]: D/IMAP ReadNextLine [stream=0x7f7f36378920 nb=13 needmore=0]
[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:CreateNewLineFromSocket: * 4 EXPUNGE
[(null) 90451: IMAP]: D/IMAP ReadNextLine [stream=0x7f7f36378920 nb=13 needmore=0]
[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:CreateNewLineFromSocket: * 3 EXPUNGE
[(null) 90451: IMAP]: D/IMAP ReadNextLine [stream=0x7f7f36378920 nb=12 needmore=0]
[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:CreateNewLineFromSocket: * 0 RECENT
[(null) 90451: IMAP]: D/IMAP ReadNextLine [stream=0x7f7f36378920 nb=52 needmore=0]
[(null) 90451: IMAP]: I/IMAP 0x7f7f3f19e800:mail.wally:S-INBOX:CreateNewLineFromSocket: 14 OK Move completed (0.031 + 0.000 + 0.030 secs).

The above is with my local Dovecot server.

The 3 deleted messages are moved to Trash and then the server reports them as expunged (permanently deleted) from the source folder, Inbox above. So the "deleted" messages should not affect the Inbox quota since they are completely gone and not visible in Inbox at all. Of course, they still need to be emptied from Trash or the total quota usage will not change.

That's okay so far, but it doesn't help the user if for some reason there are still messages that weren't expunged. One reason could apparently be that the messages were previously only marked as deleted with other settings in Thunderbird (or another client).

Yes, if other clients just mark (actually "store" in imap parlance) the \deleted flag on a message it will still be present in the mailbox and uses some quota.

Edit: I mentioned above that this discussion sounded like deja-vu and it is: See bug 1477557. In the last comment of that bug I mentioned that I attempted to implement what the reporter of that bug (and you) requested but ran into a problem with the "undelete" feature that is always present.

See Also: → 1477557

I've done some more looking and experiments using 2 tb clients accessing the same mailbox and to me it appears that if you are in "move to trash" delete mode and you have at least 20 messages in the folder that are marked \deleted, when that folder is selected it will be expunged. So it is unlikely that you will ever build up a huge number of \deleted messages in a folder.

In addition, if a folder is selected when you shut down tb, the selected folder(s) will be expunged when in "move to trash" mode because imap CLOSE command occurs.

Only if you are in "just mark as deleted" mode will the folder never be expunged.

This behavior is with the several pref items containing the word "expunge" at their default settings in config. editor.

This is handled by this piece of code: https://searchfox.org/comm-central/rev/631bd8e8e555d57fb21c3fb18f388516116549d9/mailnews/imap/src/nsImapProtocol.cpp#4201

So displaying messages that are deleted seems pointless since there shouldn't be many of them (unless you operate in "just mark as deleted" mode, which does show the deleted messages crossed out).

So displaying messages that are deleted seems pointless since there shouldn't be many of them

However, this only applies in the event that everything works as intended. I want the invisible messages to be displayed precisely because there are cases where it (your described process) does not work properly. Then it would be helpful, to see these hidden messages.

If it is not possible for technical reasons, then I have to accept it. I just can't imagine what these technical reasons should be.

I am glad and grateful that you all became aware of this bug or RFE so quickly, even if it may not be resolved to my satisfaction.

A bit OT, but a limit of 1000 msgs per folder, what? Who puts up with that? You could be locked out by not reading your mail over the weekend!

However, this only applies in the event that everything works as intended. I want the invisible messages to be displayed precisely because there are cases where it (your described process) does not work properly. Then it would be helpful, to see these hidden messages.

I didn't mean to imply that in certain cases this won't work properly. Maybe you mean this:

Only if you are in "just mark as deleted" mode will the folder never be expunged.

In this case you don't want them expunged. But if you switch back to "move to trash" mode and there are more than 20 marked deleted they will be expunged when you shutdown tb with the folder selected or when you re-select the folder at some point.

Granted, if a folder like INBOX accumulates more than 1000 messages before you access it, the ISP may lock your account and not allow you to access it it all. Therefore you won't be able to select the folder and expunge messages. But being able to display deleted messages won't help with this either since you are already locked out.

A bit OT, but a limit of 1000 msgs per folder, what? Who puts up with that? You could be locked out by not reading your mail over the weekend!

I'm not quite that popular but 1000 does seem like a small limit. But reporter indicates this is a "free" account provided by the ISP. Probably tb account manager should default new tb users for T-Connect to POP3 instead of IMAP to avoid this issue. (Don't know if that's possible.) Apparently to get a reasonable IMAP quota you have to pay a "tariff".

Granted, if a folder like INBOX accumulates more than 1000 messages before you access it, the ISP may lock your account and not allow you to access it it all. Therefore you won't be able to select the folder and expunge messages. But being able to display deleted messages won't help with this either since you are already locked out.

It could help. Knowing about the marked-as-deleted messages led to to the solution in the described support case. I was able to move the messages in chunks from the trash folder to an other folder. From this other folder I was able to mark them deleted and compact/expunge this other folder. Doing this a few times, helped to get rid of all the 5000 messages (except the 2 "defective" messages).

I'm not quite that popular but 1000 does seem like a small limit. But reporter indicates this is a "free" account provided by the ISP.

It's not really "free". It's included when T-Online is your ISP in Germany. There are other existing free T-Online mail account options. The point is, that T-Online is still the main ISP and therefor many people have this mail account with the 1000 messages per folder quota.

This sounds so incredibly rare* that we really shouldn't cater to, when there is an obvious workaround AIUI of enabling Mark as deleted.

  • do I have this right? multiple users accessing the same account, crazy low folder limits, and "something else not working right" (which if true someone - at server or in thunderbird - should fix anyway)

bug 792381 - which offers no compelling rationale - asks for the same thing

Priority: -- → P5
See Also: → 1800813

Does this still happen in version 115?

Flags: needinfo?(bugzilla)

I can't contribute anything more to this. I have not come across any case that caused this problem. But I can't rule it out either.

Flags: needinfo?(bugzilla)
You need to log in before you can comment on or make changes to this bug.