728635 - IMAP Mail deleted then emptied in Thunderbird not emptied in IMAP Web browser

Status: RESOLVED INVALID

(MailNews Core :: Networking: IMAP, defect)

Description

[Derek Williams]

Attachment: Screen Shot 2012-02-19 at 00.22.46.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0) Gecko/20100101 Firefox/11.0
Build ID: 20120215222917

Steps to reproduce:

Deleted mail in Inbox on IMAP server in Thunderbird


Actual results:

The mail was deleted in Thunderbird, but listed with Red X and still visible in IMAP web browser even after Empty Deleted.

Also discrepancy in Folder names - named 'Deleted' in TB (cannot be edited - see image) and 'Trash' in Web IMAP

Even after I manually deleted the mail from the Web IMAP, the deleted mail was still POPed.

Also there is a phantom 'Deleted' folder appearing in an italiised INBOX.

This is definitely created in Thunderbird, and appears on relaunch.  Even after deleting these folders they keep coming back on relaunch


Expected results:

Mail deleted in TB should disappear of IMAP server when Empty Deleted is actioned.

There should not be phantom folders appearing.

Comment 1

[Derek Williams]

Attachment: extra INBOX and Deleted folders creating themselves

These appear automatically after relaunching Thunderbird.  They do not appear in the IMAP Web browser until this relaunch.  After deleting in IMAP Web browser, IMAP Thunderbird and iPhone IMAP Mail, all the folders behave until TB is quit and relaunched, at which point the folders come back again.

Comment 2

[Derek Williams]

Attachment: iPhone Mail showing duplicate folders

the INBOX>Deleted appeared only when I relaunched Thunderbird

Comment 3

[Derek Williams]

Attachment: login specs for IMAP web browser

Comment 4

[Derek Williams]

Attachment: folders viewed in IMAP Web browser

note that the duplicated INBOX>Deleted folders are nowhere to be seen on the Main page

Comment 5

[Derek Williams]

Attachment: duplicate INBOX>Deleted showing in Folder directory

yet these are not visible in this IMAP interface's main page

also visible in iPhone and Thunderbird

appear when TB relaunched even after deleted on every GUI

Comment 6

[Derek Williams]

Attachment: 1. How they're supposed to look in Thunderbird

After I delete the duplicate folders on IMAP web browser and Thunderbird IMAP, the iPhone updates

Comment 7

[Derek Williams]

Attachment: 2. How they're supposed to look in the IMAP Web browser

The duplicates created by Thunderbird had to be separately deleted in the Web IMAP browser, although they never appeared on its home page, only in the Folders directory.

The iPhone Mail however updated straight way as soon as they had been deleted.

Comment 8

[Derek Williams]

Attachment: 3. How they're supposed to look in the iPhone Mail

This is different to the earlier image, in that the duplicate folders deleted from Thunderbird IMAP and the Webmail IMAP auto removed themselves from the iPHone Mail app - i.e. a third deletion was not necessary on the iPhone.

Comment 9

[Derek Williams]

Attachment: 4. Quit then relaunch Thundbird and phantom folders recreate themselves

back to square one

Comment 10

[Derek Williams]

Attachment: 5. webmail IMAP browser after duplicate Deleted folder opened from folder list

Note that there is now a Trash folder and a Deleted folder.

Comment 11

[WADA:World Anti-bad-Duping Agency]

What is set in Server Settings/Advanced?
  IMAP server directory: ?
  namespaces ???
Which folder did you select as trash folder at Server Settings, Move it to this folder?
  "Deleted" folder shown as root-level "Deleted" at folder pane?
What string is set in following prefs entry?
  mail.server.serverN.trash_folder_name  (N is internal server number in Tb)

Comment 12

[David :Bienvenu]

seem like separate bugs getting described - to remove deleted but not expunged messages, you need to right click on the folder and pick compact this folder (or add the compact toolbar button to the toolbar).

Comment 13

[Derek Williams]

The folder selected for deleted messages is the 'Deleted' folder in the IMAP directory.

Mail deleted in webmail gets deleted in Thunderbird and vice versa.  This much is still happening correctly both ways.

However, the problem is that emails deleted in Thunderbird, that are not visible in the InBox on Webmail, and are visible in the Deleted folders of both webmail and IMAP, are still POPing - even if Delete Immediately is chosen, such that the message never makes it to the Deleted folder.

This does appear to be an absurdity of Thunderbird.

I could chose 'empty on exit', but why should I have to?  That deprives me of the right to change me mind and take it out of the trash further down the track.

Surely mail should move to wherever I move it to, and the application should not think it knows better than I do, what's good for me.

If I delete an email, I want it to pass from the Inbox to the Deleted folder.  I do not want to have to delete it twice, and I certainly do not want to have happen again what happened today, where 690 messages I had deleted in Thunderbird via IMAP, and which were not visible in the InBox of either Thunderbird IMAP or the webmail gui, all showed up in the POP Inbox.

If this is the way Thunderbird is supposed to work, then does anyone have a recommendation for an alternative mail program that does what it says 'on the tin'?

Comment 14

[Derek Williams]

posted here, where they seem proud of this fact!

http://kb.mozillazine.org/Deleting_messages_in_IMAP_accounts

Comment 15

[David :Bienvenu]

(In reply to Derek Williams from comment #14)
> posted here, where they seem proud of this fact!
> 
> http://kb.mozillazine.org/Deleting_messages_in_IMAP_accounts

That's the way IMAP works - we didn't design it; we just implement an instance of the protocol.

Comment 16

[Derek Williams]

OK, if it is case that IMAP was designed to behave in such a counter-intuitive fashion, is there any way to configure Thunderbird that achieve do the following:

1. Click an email intended for deletion
2. Press Del or Backspace key to delete the selected email 
3. Move the deleted email from Inbox to Deleted
4. Expunge email from Inbox no matter whether viewed in IMAP or webmail, such that it is not downloaded when Inbox is next POPed?

Comment 17

[Derek Williams]

error:

"achieve do"

should read

"achieves"

Comment 18

[David :Bienvenu]

See the page you referred to earlier: http://kb.mozillazine.org/Deleting_messages_in_IMAP_accounts, the Misc section - use the config editor to set mail.imap.expunge_option to 1. It's up to the imap server how well it keeps connections in sync, but this should help.

Comment 19

[Derek Williams]

"Clean up ("Expunge") Inbox on Exit will compact the Inbox folder when you exit Thunderbird." This I was aware of, however it necessitates Quitting Thunderbird each time I delete messages before running POP.

Is there perhaps some way this can be dealt with by the ISP?

Comment 20

[David :Bienvenu]

(In reply to David :Bienvenu from comment #18)
> See the page you referred to earlier:
> http://kb.mozillazine.org/Deleting_messages_in_IMAP_accounts, the Misc
> section - use the config editor to set mail.imap.expunge_option to 1. It's
> up to the imap server how well it keeps connections in sync, but this should
> help.

If you do what I suggested above, Thunderbird will expunge after every delete.

Comment 21

[Derek Williams]

Attachment: Tools>Account Settings>Server Settings

This is working only if I quit and then relaunch Thunderbird.

I can't see any place to "set mail.imap.expunge_option to 1" other than to select the below option:

Tools>Account Settings>Server Settings>'Clean up ("Expunge")Inbox on Exit'

Is there someplace else to set this?

Attaching image.

Comment 22

[David :Bienvenu]

As I said before, you need to use the config editor to change that pref - http://kb.mozillazine.org/Modify_Thunderbird_settings

Comment 23

[Derek Williams]

Attachment: Tools but no options

There is no Config Editor in my version (10 - Mac) of Thunderbird, at least not that I can find in the path noted. There is no 'Options' inside the Tools menu.

Comment 24

[Derek Williams]

I also opened the prefs.js file, but that doesn't have the word 'expunge' in it.

Comment 25

[David :Bienvenu]

on the mac, it's preferences, not options.

Comment 26

[Derek Williams]

Attachment: duplicate folders

When I set the value to 'true', phantom folders appeared. A fresh INBOX containing an extra Deleted folder.

These folders cannot be deleted, i.e. they do not disappear if Empty Deleted is chosen, nor if File>Delete folder is chosen.

Comment 27

[Derek Williams]

Attachment: Config Editor Expunge setting

Comment 28

[David :Bienvenu]

(In reply to Derek Williams from comment #26)
> Created attachment 606780 [details]
> duplicate folders
> 
> When I set the value to 'true', phantom folders appeared. A fresh INBOX
> containing an extra Deleted folder.

Not sure where you got "true" from the instructions I posted. I suggested setting it to 1.

Comment 29

[Derek Williams]

I have since discovered that if I delete the innermost folder first, I can progressively delete up the chain until all the phantom folders are gone. But as soon as I relaunch Thunberbird, they all come back again.

These phantom folders are not visible in the Webmail host.

Comment 30

[Derek Williams]

There is no '1' option. A double click toggles between True and False.

Comment 31

[Derek Williams]

Attachment: boolean only

Comment 32

[Derek Williams]

Attachment: undeletable preference

I tried rightclick>add Integer '1' but instead of changing from Boolean to Integer, it created a new preference name of '1' which I cannot delete using backspace, Del or Right click.

As the admonition says "here be dragons"!

Comment 33

[Derek Williams]

Attachment: new error on relaunch after folder deletion and pref edit back to default

After I changed the pref config back to False, the folders I deleted have come back, but an error message is saying these cannot be accessed

Comment 34

[David :Bienvenu]

you've been playing with mail.imap.expunge_after_delete whereas I suggested that you change  mail.imap.expunge_option. But setting mail.imap.expunge_after_delete to true should cause us to issue an expunge after every delete, as you wanted, and has absolutely no effect on the folders that we display. Don't know why you had problems, but perhaps you accidentally tweaked some other prefs. I apologize for suggesting that you use the config editor; it can be dangerous, as you say.

Comment 35

[Derek Williams]

OK - I have set the expunge pref option to 1 but kept the Boolean value to True.  Not sure why this can be set twice - does one override the other or do they have to be in step?

I have deleted the account and set up another one anew.

This changed the name of the Deleted folder to Trash in the Tools>Accounts>Servers

Other than adding a pref called '1' I definitely did not do anything else to Config because I used 'expunge' as a filter - thus only those options could possibly have been modified.

That said, how do I delete this '1' pref?

Comment 36

[Derek Williams]

Attachment: pandora's box of folders

since deleting and recreating the account, I have many phantom folders inside other folders

What I would ideally like is just the following:
• Inbox
• Drafts
• Sent (aka 'Sent Messages')
• Trash (aka 'Deleted' aka 'Deleted Items')
• Junk (aka 'Spam')

I would prefer these appear the same in all GUI - be it iPhone, IMAP or webmail.

I would like any action taken on one to be mirrored on all the others.  I had hoped that IMAP might achieve this.

Is this in any way possible?

Comment 37

[David :Bienvenu]

you've messed with your namespaces somehow, or your online server directory. What does the advanced imap server settings button tell you the namespace is set to?

Comment 38

[Derek Williams]

Attachment: INBOX

this appeared all by itself, I certainly never opened this window and typed the word INBOX!

Comment 39

[Derek Williams]

also the mail would not connect via port 993 - I had to revert to the Default offered before it would work.

Previously 993 did work

Comment 40

[Derek Williams]

it is now 143

Comment 41

[Derek Williams]

Attachment: current expunge pref values

Comment 42

[Derek Williams]

Attachment: webmail folders

note it is called Trash on webmail, but Thunderbird renames it as Deleted

Comment 43

[Derek Williams]

Attachment: webmail server settings

Comment 44

[Derek Williams]

cannot now send email at all via IMAP - always goes to wrong outgoing server, not the one set as Default

please compare with bug https://bugzilla.mozilla.org/show_bug.cgi?id=736661

Comment 45

[Derek Williams]

Attachment: duplicate Deleted folder

I managed one by one to dispose of all the duplicated folders by selecting OTHER and then the internal boxes for the IMAP.

However this has not worked for Deleted. I checked which folder deleted mail goes to in both IMAP on Thunderbird and on Round Cube webmail.

In both cases it goes to the same Deleted folder, and this is the one *for sure* I have selected in Options.

When I delete this duplicate Deleted folder, it re-apppears on relaunch of Thunderbird, but has no function at all.

All deleted emails are going to the correct, selected Deleted folder.

Comment 46

[Derek Williams]

BTW FWIW - the great news is:

With the expunge setting at '1', the emails are now deleting off the server when I delete them in Thunderbird.

I will probably get to the bottom of the duplicate folders issue by resetting in consultation with the ISP.

Thanks for all your help and patience.

Comment 47

[Derek Williams]

Attachment: unsubscribe

I experimented with this - and this had the result of removing the duplicate Deleted folder, but I have no idea why that is even there.

Comment 48

[Derek Williams]

this has changed back, even with the setting at 1 the mail deleted in Thunderbird stays on the server and gets downloaded in the hundreds on POP

HELP!

Comment 49

[WADA:World Anti-bad-Duping Agency]

(In reply to Derek Williams from comment #48)
> HELP!

Here, bugzilla.mozilla.org, is for reporting actual Tb's bug(flaw in code of Tb) to developers. I believe forums like following is best place for you.
> http://support.mozillamessaging.com/en-US/home
> https://getsatisfaction.com/mozilla_messaging/
> http://forums.mozillazine.org/viewforum.php?f=39

Comment 50

[Derek Williams]

Thank you for the links, however this started out as a perceived bug, and I was merely continuing the dialog.  I still think the present IMAP implementation is counter-intuitive, because deleted emails disappear from both TB and Webmail, yet are still downloaded by POP.

I will keep those support links in mind though, for future support purposes.

Comment 51

[WADA:World Anti-bad-Duping Agency]

(In reply to Derek Williams from comment #50)
> I still think the present IMAP implementation is counter-intuitive,
> because deleted emails disappear from both TB and Webmail, yet are still downloaded by POP.

Why IMAP Mbox at server can be relevant to POP3 Mbox at server for mail client?
IMAP Mbox at IMAP server and POP3 Mbox at POP3 server are absolutely independent/different Mbox for mail client.
If synchronization between content of IMAP Mbox at IMAP server and content of POP3 Mbox at POP3 server is needed, it should be done at server side who supports all of IMAP Mbox, POP3 Mbox, and Web mail service.
It depends on server side system.
(a) Global mail box DB. All of IMAP Mbox, POP3 Mbox, Web mail accesses Global DB.
    => All are autmatically sync'ed always.
(b) IMAP Mbox and POP3 Mbox is independent. New mail added to Global DB is added to IMAP and/or POP3 Mbox.
(c) Global DB and IMAP Mbox is sync'ed. IMAP Mbox and POP3 Mbox is independent. New mail added to Global DB is added to independent POP3 Mbox. (perhaps your case)
Behavior on Mbox can be controlled by option settings provided via Web mail interface in many cases.

Additional best place for you : Support/Help forum of your mail service provider.

Comment 52

[Derek Williams]

I use POP to archive all email because I do not want to trust outsourcing my emails, which date back to 1996, to an external mail client that could lose the lot (it has happened to friends).  I archive all the email on one machine only, because it is not practical to POP on several different machines.  I back the Thunderbird Library folder up with Time Machine, and on remote sites all over the world.

I operate from very many sites, including overseas travel (webmail), university, and home.  In Thunderbird I always use IMAP until I am ready to download and archive all the email, usually once a week.  Sent mail can be dragged, whereas Inbox is POP'ed.

The fact remains that emails which are deleted in TB disappear from view in both Thunderbird IMAP and in webmail browsers. Yet these deleted, and invisible emails, all download on POP *unless* the IMAP Inbox is compacted first. Setting the Expunge value to '1' seems a bit deep for the average user, however even though I did this, it did not expunge the deleted, invisible emails.

I have of course already raised all this at considerable length with the ISP, and after hours of tinkering, they eventually concluded that the settings in Thunderbird are responsible, hence this ticket being raised here.

I will however pass on your comments to them, just in case they might have missed something.

Comment 53

[WADA:World Anti-bad-Duping Agency]

(In reply to Derek Williams from comment #52)
> I have of course already raised all this at considerable length with the
> ISP, and after hours of tinkering, they eventually concluded that the
> settings in Thunderbird are responsible, hence this ticket being raised here.

Was your reports/questions to your ISP accurate and appropriate? You asked to ISP with your confusions or misunderstandings, didn't you?
Have you checked mail related settings of your Web Mail system which supports both IMAP and POP3 too?
For example, Gmail has following for POP3 access in his Web Mail settings.
 When messages are accessed with POP :
   keep Gmail's copy in the Inbox
   mark Gmail's copy as read
   archive Gmail's copy
   delete Gmail's copy
This is absolutely independent/different from Tb's account settings(mail client side settings).

Comment 54

[Derek Williams]

Thanks for your additional followup, which I will also forward to the ISP.

I gave the ISP full log on rights to my account and they were using Thunderbird and their own webmail client to work with my mail, and they were unable to get the desired result other than by setting to Expunge on Exit in the Tools>Accounts>Server settings.  Unless Thunderbird was Quit, deleted mails remained on the server.

While I did at first have confusion and misunderstanding as regards the minutiae of the settings in the Tools>Accounts and the fact that the Preferences>Config Editor (in different places on Mac and Windows) needed to be used to Expunge deleted mail (which does not work if set to '1'), the ISP clearly understood from the getgo that I want deleted emails to be expunged on the spot, and to be moved holus bolus to the Trash folder, and not to be still in the Inbox at POP time.

Once the ISP have had another look at this, I post back here with their feedback.

Comment 55

[WADA:World Anti-bad-Duping Agency]

For "delete mail" in IMAP.
Try "Server Settigs, When I delete a message: Just mark it as deleted", to unserstand IMAP's concept of "delete a mail".
 - Delete a mail => \Deleted flag is stored
   => Tb shows the deleted mail with strike-thru line at thread pane
 - Compact => expunge command is issued => server removes the mail woth \Deleted
   => Tb removes the mail from thread pane because the mail is removed at server
 - Undelete of mail with strike-thru line via contect menu of the mail
   => \Deleted flag is removed, and strike-thru lie is removed
This is basic delete model of IMAP. There is no need to copy mail to Trash folder just before delete mail with this delete model. So, it's efficient for both IMAP server and mail client.

If expunge is requested at each "delete mail" operation, "undelete" is impossible with this delete model.
So, "Move to Trash" model is provided for user who deeply loves POP3 mail folder like behaviour, to provide a way for "Undelete", and hiden option of "auto expunge per each delete", which you already know, is also provided for such user like you.
This hidden option currently doesn't work if "Just mark it as deleted" and "Remove it immediately", to protect user from accidental delete of mail(known issue and bug for it exists).
Please note that this hidden option is different from "expunge on extit" setting.

Comment 56

[Derek Williams]

This too I will forward to the ISP.

While I can see how the Undelete feature is useful if "Delete immediately" option is selected, I don't believe that is necessary if mail is moved to the Trash, because you can always dig a deleted object back out of the Trash.

Where an object has been moved to the Trash either by dragging or by pressing Del or Backspace, the most intuitive outcome is for the object to be expunged from wherever it was before (e.g. Inbox, Junk or Sent).  I believe that should be the default, regardless of what the IMAP standard is.

If an object in the Trash is itself deleted by pressing Backspace or Del, then it should be erased forever.

Comment 57

[WADA:World Anti-bad-Duping Agency]

(In reply to Derek Williams from comment #56)
> This too I will forward to the ISP.

There is no need to report to ISP any more about "deleted mail by Tb is not immediately expunged without hidden option=1(if one mail is deleted, issue expunge, with Move to Trash model)". This is current implementation(default of the hidden option=20 or 10) and you already know well about the currently available hidden option of Tb and required value for you.
What you should ask to ISP is for following only.
> because deleted emails disappear from both TB and Webmail, yet are still downloaded by POP.

Comment 58

[Magnus Melin [:mkmelin]]

FWIW, sounds like a server bug to me. 
Thunderbird marks the mail as deleted, and the server still including it whey you access the account using pop doesn't really sound reasonable - the server knows the mail is deleted. I don't think IMAP clients usually expunge after every mark-as-deleted action by default.

Comment 59

[Derek Williams]

It falls back to what 'Deletion' actually means.  In the context of a mail application, it simply means 'move the selected item from its current folder to the Trash folder'.

Thunderbird keeps an extra copy, assuming that the user might want to change it back. The 'marked for deletion' status facilitates the Undo.  But to my mind this creates a problem if an email deleted in good faith is not really deleted after all.  Especially when you're talking hundreds of spams and deleted Inbox items that get downloaded even though they're not showing up in Thunderbird (or on the server) as being present.

If some prefer to have the Delete process make a duplicate of everything deleted, then I would at least like the opportunity to say "No, I only want the mail to move 'entirely' from its current location, into the Trash."  This option to 'always expunge' Deleted mail is apparently selectable in theory by going to the Config Editor and setting 'Expunge Deleted Email' value to '1'.  For some reason there is a duplicate of this too, as noted in images above.  However neither option has the effect of immediately expunging the deleted email.

To add to the confusion, the Tools>Accounts>Server settings also offer an expunge option, but you have to Quit Thunderbird each and every time to expunge.  Alternatively, you can right click and Compact folder.

These are extra steps that have to be taken in order to perform the most intuitive option, namely, when deleting an email (i.e. moving it from one folder to another), actually move it without leaving a hidden duplicate behind.  The only case I can see to be made for leaving flagged duplicates behind in the source folder is where the 'Delete immediately' option is selected.  Then you might want to undo it, since there would then be no copy sitting in the Trash.

The fact that there is a Trash folder there in the first place already gives the user the peace of mind to be able to Undelete.  All they have to do is to drag the item back out of the Trash, either back to where they Deleted it from, or someplace else.

I very much doubt whether the majority of users would want all their Trash duplicated and to have it POP after they deleted it.  That's making it harder, not easier for the user in my humble opinion.

Comment 60

[Magnus Melin [:mkmelin]]

(In reply to Derek Williams from comment #59)
> It falls back to what 'Deletion' actually means.  In the context of a mail
> application, it simply means 'move the selected item from its current folder
> to the Trash folder'.

If that is the mail client setting - for the mail client - yes. However, the way IMAP is designed, that is not the case for the server: IMAP expects the client to mark mails as deleted, and then issue expunge commands whenever it finds suitable.  It's not something Thunderbird does for the sake of undo. (I think IMAP was designed that way because expunge can be "expensive" for the server.)

Comment 61

[Wayne Mery (:wsmwk)]

Interesting discussion, but no apparent bug in thunderbird, i.e. working per rfc specs.

Please continue any further questions and discussion in http://groups.google.com/group/mozilla.dev.apps.thunderbird/topics?pli=1

Comment 62

[Derek Williams]

Well since it's apparently been decided that 'Delete Email' won't Delete Emails, it's pointless 'discussing' this any further.

Comment 63

[Derek Williams]

copy of https://bugzilla.mozilla.org/show_bug.cgi?id=747103