427007 - "Undo Move Message" loses message - after move from Gmail IMAP inbox to local folders

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Adam]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.11) Gecko/20061201 Firefox/2.0.0.11 (Ubuntu-feisty)
Build Identifier: 1.5.0.14pre (20071023)

I'm using GMail with IMAP.

I had message in my Inbox on my GMail account, and moved it to a folder in my "Local Folders".

Then I selected "Edit -> Undo Move Message".

The message was gone from both my Inbox and the folder I had moved it to - as far as I could tell, lost completely.

I discovered that I could get it back in the Local Folders folder (but with the Subject line missing) by selecting Edit -> Redo Move Message.


Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[Mark Banner (:standard8)]

Please can you try the latest version of Thunderbird (2.0) available from http://www.getthunderbird.com/

The 1.5 branch is now unsupported, and there have been various bugs fixed that could have already fixed your problem.

Comment 2

[Adam]

Hmm, well the problem is I'm using Ubuntu 7.04, which doesn't seem to want to upgrade me to Thunderbird 2.0 of its own accord.  But if you want to assume it's fixed in the newer version, that's fine.

Comment 3

[Magnus Melin [:mkmelin]]

Does seem to happen on trunk too. I would guess it's something with the gmail imap implementation though. (Works fine for a normal imap server.)

Comment 4

[WADA:World Anti-bad-Duping Agency]

I saw similar report on "Undo delete of Gmail IMAP", but I can't recall bug number.

Move of a mail is combination of "copy of a mail" and "flag \Deleted" when IMAP.
When usual IMAP server, "flag \Deleted" only adds flag of "\Deleted" to the mail, and the mail data is kept until Expunge.
But when Gmail IMAP, "flag \Deleted" means "remove of label", and the "label" of Gmail is mail folder name when Gmail IMAP. i.e. Result of "flag \Deleted" when Gmail IMAP is same as "flag \Deteted then Expunge".
Read Gmail documents listed in Bug 402793 Comment #0, please.

When Gmail IMAP, all mails are kept in [Gmail]/All Mail(all mails have label of [Gmail]/All Mail), so recovery from this "[Gmail]/All Mail" folder looks to be possible, if move of mail. But it seems that delete==mail move to [Gmail]/Trash(standard Trash of Gmail, so cleaned up by retention) is not recoverable in some situations.

Comment 5

[WADA:World Anti-bad-Duping Agency]

"Undo delete" version was Bug 417167.

Comment 6

[Brian Kennelly]

You can disable the Undo & Redo operations in Gimap accounts by setting the "mail.server.server#.canUndoDeleteOnServer" preference to false.  It doesn't solve the problem, but it avoids the data loss.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: IMAP protcol log (Tb trunk, Gmail IMAP, Undo of "move from IMAP folder to IMAP folder")

Attached is IMAP protocol log for next test, with Tb trunk/Gmail IMAP.
 (1) Move a mail from Inbox to My_Trash (both are IMAP folders of same account)
 (2) Undo Move
Although "Undo delete" step is not properly done due to Gmail IMAP's problem,
  Gmail IMAP returns OK to "18 uid store 53 -Flags (\Deleted)",
  even though Gmail IMAP doesn't recover the mail once Gmail label is removed.
  This is problem of Bug 417167.
delete request of moved mail(mail in My_Trash in my test case) was not observed, then loss of the moved mail(mail in My_Trash) was not re-produced.

When move to local mail folder, mail in the local mail folder(move target folder) was deleted by "Undo Move" operation?
If yes, is it re-creatable with Trunk? (Branch only issue or not).

Comment 8

[WADA:World Anti-bad-Duping Agency]

I could re-produce "delete of mail at move target folder" on "Undo Move" when move target folder is local mail folder.
Following flag was set in local mail folder after "Undo Move".
> X-Mozilla-Status: 0009
"Redo Move" after that is probably remove of deleted_flag in X-Mozilla-Status:.

Comment 9

[WADA:World Anti-bad-Duping Agency]

This bug is a variation of phenomena of Bug 417167. Root cause is OK response to "18 uid store 53 -Flags (\Deleted)" by Gmail IMAP even though Gmail IMAP doesn't recover the mail. From mail client view, "Undo Move" has been successfully done by IMAP server, so delete from move target folder(local mail folder in your case) is normal behavior.

One question.
Why not deleted from move target folder when "Undo Move of move from IMAP folder to IMAP folder"?

Comment 10

[Brian Kennelly]

The undo processor is determined by the target folder type.

If the target folder is IMAP, Thunderbird does detect, probably because no untagged FETCH response reported success, that the message was not undeleted from the source folder, so it does not attempt to delete the message from the target folder.  (nsImapUndoTxn.cpp)

If the target folder is local, rather than IMAP, Thunderbird does not detect when the undelete fails in an IMAP source folder, and the target delete is allowed to proceed.  (nsLocalUndoTxn.cpp)

Comment 11

[WADA:World Anti-bad-Duping Agency]

To Brian Kennelly: Thanks again for you explanation.
This bug should be closed as DUP of Bug 417167? Or better to be kept separately?  

Comment 12

[Brian Kennelly]

I believe we have two issues.  

1. Gimap does not support 'undo delete'.  There is nothing that Thunderbird can do about it, except to set the 'canUndoDeleteOnServer' preference to false.  (Bug 417167)

2. Thunderbird does not detect a failed 'undo delete' when undoing a move from an IMAP folder to a local folder.  This is the real problem uncovered by this bug report. 

Comment 13

[Brian Kennelly]

Addendum: this bug is not Gmail specific.  The same problem occurs with any IMAP server that does not support 'undo delete' (I tested it with AOL).  I did not test it, but I expect to see the same problem if the source folder is expunged before the undo.

Comment 14

[WADA:World Anti-bad-Duping Agency]

> The same problem occurs with any IMAP server that does not support 'undo delete'

Questions to confirm situation.

If AOL IMAP & Tb with AOL IMAP support & AOL IMAP account defined via AOL support by Tb, imap.aol.canUndoDeleteOnServer looks to be set to false as default.
> http://users.comlab.ox.ac.uk/ian.collier/Misc/aboutconfig.html
Problem when AOL IMAP account is defined by hand/by usual procedure?

Is there any other IMAP server who doesn't support(or improperly/invalidly supports it like Gmail IMAP) "uid store XX -Flags (\Deleted)" after "store XX Flags (\Deleted)"?

Comment 15

[Brian Kennelly]

It is an AOL IMAP account set up normally (not using the AOL.rdf file).  
I was initially testing the IMAP-to-IMAP moves within the same account.

More careful testing from the AOL account reveals that undo works correctly when moving a message to another account (either IMAP or local), but not within the same account.  (The XAOL capability supports an atomic move operation, which cannot be undone by the normal undelete/delete.)

Gmail supports undelete from the "All Mail", Trash and Spam folders, and moves from those folders to another account can be undone.  However, moving to another Gmail folder cannot be undone correctly.  The original message is retrieved, but the copy is not deleted.  This means that this bug is probably a duplicate of bug 162008 (Gimap does not support UIDPLUS).

Test with a Courier IMAP server (with UIDPLUS) show that undo works in general, but, if the deleted message is expunged before the undo, the original message is not recovered (as expected), but the copied message is still deleted.  (Courier also reports OK to the failed UID STORE.)

As you can see, my original analysis was wrong.  It is still true that Gimap does not, in general, support undo delete, so the other report is unchanged.

This report is modified as follows: 
1. Undo move does not work if the target server is IMAP and does not support UIDPLUS.  This part is a duplicate of bug 162008.

2. Thunderbird does not detect a failure to undelete the source message, and, if the target folder is local or on IMAP with UIDPLUS, the target message will be deleted, resulting in no copies.

Comment 16

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Emre, thoughts?

Comment 17

[Emre Birol]

** Assigning to myself **

Comment 18

[Martin Mokrejs]

I am playing with current trunk version of seamonkey. Deleting a message from a local Inbox (updated through POP3) works but not Undo. At the very first attempt to access Edit/Undo from the menu I got:

WARNING: Key 'key_preferences' of menu item 'Preferences…' could not be found: file /home/mmokrejs/proj/comm-central/src/mozilla/layout/xul/base/src/nsMenuFrame.cpp, line 1055


Several times then I have received

WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80550005: file /home/mmokrejs/proj/comm-central/src/mailnews/local/src/nsLocalUndoTxn.cpp, line 220


I can also confirm the message is not even in Trash folder.

Comment 19

[Navid Zamani]

I’m having this problem too, and boy, loosing messages is really the worst kind of bug there is. I nearly lost a business deal because of it. I found tons of similar bugs in Bugzilla. Looks like there are too many bugs for too few developers…
I wish I could write C++, so  I could help. :/

Comment 20

[Wayne Mery (:wsmwk)]

does this still exist, and is it specific to gmail?

Comment 21

[Navid Zamani]

(In reply to Wayne Mery (:wsmwk) from comment #20)
> does this still exist, and is it specific to gmail?

I just tested it a bit more, and it’s worse than expected. This may become confusing, and it certainly was for me. So please try hard to get past simple confusion, before asking specific questions, as to not further confuse things. :)

————————————————————————————————————————————————————————————

Manually moving and moving back:

For GMail, manually moving a message to the local archive folder results in a duplication, and hence *two* times that same mail in the local folder. Not so for archiving, which works normally.
Manually moving the first duplicate back works as expected. Manually moving the second one back looks like it “overwrites” the first one. (Upon opening the folder after the move, the message shortly goes away and then returns to the list again.)
Manually moving it (whichever that is) to local a second time, doesn’t result in duplication anymore. Moving it back again, works as expected too.

Doing the same for my local server (Courier IMAP) everything works as expected, and the above problem doesn’t occur.
Manually moving it from my local server to GMail, then to a local store, back to GMail, and then back to my local server, works too.

————————————————————————————————————————————————————————————

Manually moving, but using undo to move it back:

With undo, it’s a different story: After manually moving it from GMail to the local archive folder, undoing results in the second of the duplicated message going away from the local archive folder, but never ending up anywhere else (as far as I can tell). It vanishes.
I can not undo the first duplicate, since there obviously is only one undo, which I just used.

Doing the same, but moving it to the local *server* instead works as expected. But undoing that does nothing. I didn’t test the permutations after that.

With my local server, things work as expected.

————————————————————————————————————————————————————————————

Archiving, and manually moving back:

For GMail, archiving works as expected. Moving it back too.
BUT: Archiving it a second time, results in the above message duplication. Manually moving *both* duplicates back, again, results in *one* message in GMail.

With my local server, everything works as expected.

————————————————————————————————————————————————————————————

Archiving, but using undo to move it back:

For GMail, as said, archiving works, BUT the undo here does nothing.

With my local server, everything works as expected.

————————————————————————————————————————————————————————————

What I did not really test, is REDO. (Too many permutations. The above was already too messy for me.) But the two cases where I did it (GMail too), resulted in the mail vanishing.

When the above works, one can re-test redoing.

————————————————————————————————————————————————————————————

I hope this helps debug the problem.

Loss of data is of course really bad.

Comment 22

[arjun]

[Tracking Requested - why for this release]:

Comment 23

[roger]

I have several times recently lost files after a move/undo move. I am currently using TB 45.4.0 with Gmail. I have not been able to find the missing messages, but I have not tried the "redo move message" approach. I will next time it happens, but I am reluctant to lose messages just to verify the problem!

Comment 24

[wrhenshaw99]

I am also experiencing this problem. Why does the Edit > Undo Move Message command even exist for IMAP accounts if it results in messages disappearing? Losing messages is not a good "feature" for an email program.

Comment 25

[toto_23]

I am also experiencing this problem, not with a Gmail-Imap, but with a university Imap-server.
Moving a message to a local folder works
ctrl-z or Edit -> Undo Move Message deletes the local copy, but does not place the original email back into the IMAP folder.

The emails do show up in the INBOX file in my .thunderbird directory (on Ubuntu), but they do not seem to be pushed back to the IMAP server. They hence aren't accessible from the IMAP anymore, and somehow are not shown in Thunderbird anymore, either.

Compacting INBOX then finally removes them from that file (so I guess they are never "undeleted", but have remained in the INBOX file only as artifacts).

In any case, my workaround for the moment is to close Thunderbird, copy the lost emails from the mbox file of the inbox (/ImapMail/.../INBOX) to the mbox file of the folder where I wanted them in the first place (Local\ Folders/.../<target>), delete the <target>.msf and restart Thunderbird to rebuild the msf file.

mbox files are a bit of a mess, suffice to say that emails start with a "From " (space included, at the start of a line - vim: "/^From ") string. Makes parsing a bit easier ;-)

I guess we could simply delete the INBOX.msf and rebuild it (I didn't try if this pushed the undeleted emails back to the IMAP server), but since this folder is much more active I would have to re-delete all the other emails (and my target folder is an archive where I don't delete or work, so the msf-file does not contain any useful information in any case)...

HTH, good luck!

-- t

Comment 26

[Jon Barrilleaux]

Wow! A 13 year old critical bug. For what its worth, archived a message, then tried to undo the move, and the message was lost. Tried to compact folders and restart TB.

I'm using the latest TB version on the latest Window version.

Comment 27

[Adam]

Would the following work as a solution? Instead of relying on "undo delete", which seems to not be supported by all mail protocols, change the "Undo Move Message" operation to copy the message from the original destination back to the original source, then delete it from the original destination. Similarly change "Redo move message".

Comment 29

[Wayne Mery (:wsmwk)]

Do we really need this and bug 467841?

Comment 30

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #29)

Do we really need this and bug 467841?

gmail now appears to support UIDPLUS. I see it in the capabilities when I just now checked.

Comment 32

[amorim.andre.89]

This bug prevented me of using Thunderbird 10 years ago. I gave another try some days ago, using obviously a different computer and a different OS, and the exact same bug is still here. The ability to UNDO is one of the most useful features of any software. Its a shame that a such persistent bug like this will keep preventing me of using one of the otherwise best email clients.

Look, theres another one, this one 13 years old: bug 467841
And yet another one, this 16 years old: bug 323875
And theres one opened by me 2 days ago: bug 1752569

And Im sure there are a ton more threads reporting this same bug.

Why do all email clients support UNDO and Thunderbird has had trouble with it for the last 16 years (at least)? Its time to solve this once and for all, don't you think? 16 years and you still have problems with this?

I see people here discussing if IMAP gmail implementation supports it or not, and stuff like this. Again I ask, why does it work with all email clients and not with Thunderbird?
Its not a problem of IMAP, gmail, ot whatever. Its a problem of Thunderbird. Stop complicating stuff. Its a simple move back operation.

And I will ask a final time: Why does it work with all email clients and not with Thunderbird?

Comment 33

[Wayne Mery (:wsmwk)]

Why does it work with all email clients and not with Thunderbird?

Because Thunderbird hasn't implemented UIDPLUS

(In reply to gene smith from comment #30)

(In reply to Wayne Mery (:wsmwk) from comment #29)

Do we really need this and bug 467841?

gmail now appears to support UIDPLUS. I see it in the capabilities when I just now checked.

It looks like imap-js will provide this, if I correctly understand the library description correctly https://github.com/emailjs/emailjs-imap-client. But imap-js might not come in the next esr. Magnus will have a grasp of this question.

Bugs which mention UIDPLUS or rfc4315 https://mzl.la/3ocFcHp I think the bugs with the most info are this and bug 467841.

Comment 34

[gene smith]

Attachment: undo.diff

I've never really looked into how undo works but just checked it with gmail when moving a message from Inbox to Local Folders. When "moving" the message, tb just copies it to Local Folders and then "marks" the source message as \deleted, so the message should still be present. But gmail, by default, expunges a message marked as \deleted immediately. Tb's "undo" is just to remove the \deleted flag from the moved message and delete the message from move destination (local folders in my case). But the message is already gone so the \deleted flag can't be removed so the message is gone from both folders (except it's still in gmails All Mail "folder" and can be recovered from there). Also, gmail's response to removing the \deleted flag does not indicate a problem even though the message is no long present in the the folder.

A workaround for this is to go into gmail's webmail setting and change "When I mark a messages in imap as deleted" from the default "Auto-expunge on" to "Auto-expunge off". This will make gmail behave more like a typical imap server.

Probably really tb should implement undo by copying (actually IMAP appending) the message back to the original folder and then not deleting the message from move destination until the message is successfully verified to be present again at the original folder. But the simple approach was taken in the undo design since gmail (with it's non-standard or at least non-typical) imap implementation didn't exist at the time.

I personally don't rely on undo functionality, especially when 3rd party remote servers are involved. I would just copy the message back where it belongs manually and then when I'm sure it's back, delete the unwanted message. (I realize that sometimes you don't know where you moved the message to and can't find it. So that might be a reason a reliable undo is really needed.)

Ok, it looked at some code and made one temporary change just as a test (attached). It appears that if tb thinks it is undoing back into an imap4 folder, it uses just the un-store of the \deleted flag to undo the move. If it doesn't detect that the folder to undo back into is imap4, it does a copyfolder() to undo the move. So when I change the code so it doesn't detect imap4, the copyfolder (imap append) back to gmail occurs and the moved message re-appears on undo, without changing gmail's default setup at webmail.

There is another undo case where the move destination folder is also imap (like gmail's trash) and not a local folder. I haven't looked at that or if there are problems there too.

Comment 35

[gene smith]

Because Thunderbird hasn't implemented UIDPLUS

Actually, I added implementation for uidplus when I first started working on imap over 5 years ago. Specifically I implemented the "uid expunge" feature. Not sure what aspects of UIDPLUS are used with undo.

I just tried deleting and moving gmail messages and they come back after undo when moved to other gmail folders. The source folder didn't update immediately after undo (had to move off and then back onto folder so see the un-did message which may be due to some of my ongoing CONDSTORE changes, not sure).

It looks like imap-js will provide this, if I correctly understand the library description correctly https://github.com/emailjs/emailjs-imap-client. But imap-js might not come in the next esr. Magnus will have a grasp of this question.

This is big shock and surprise! Didn't know that what I've been working on for 5 years is slated for discard! I thought only the simpler pop and smtp were moving to JS.

Comment 36

[Wayne Mery (:wsmwk)]

Oops. I assumed because of those open bug reports that uid expunge hadn't been implemented. prior to Bug 1352859 ? - Messages unexpectedly expunged after move when using the mark-as-deleted

pop and smtp, and ldap, were done in js first to prove the concept. imap in js of course has yet to prove itself.

Comment 37

[ulysse68]

There is another undo case where the move destination folder is also imap (like gmail's trash) and not a local folder. I haven't looked at that or if there are problems there too.

I confirm that all my IMAP folders are concerned by the undo problem.

Comment 38

[amorim.andre.89]

I also confirm its not only GMAIL, I am having the problem with an HOTMAIL account.

However, when I drag an email from the Inbox to the Trash, and than from Trash back to the Inbox, everything works fine. And the operation is also correclty synced to the server, whether it is GMAIL, HOTMAIL or anything else.

Theres a problem only when I use the UNDO operation. So why not implementing something like, if I delete an email and then UNDO, simply implement a "move back" operation as if I was drag and dropping the email back to the original folder. Maybe this is an ugly workaround, but should work fine and the users would be happy.

Comment 39

[Magnus Melin [:mkmelin]]

Correct, the plan is to also do IMAP in JavaScript - this is the plan for more-or-less all the c++ backend. Work on imap-js will start in February. It will be behind a pref, and I would say very unlikely to have proven itself before 102 - so the old imap code is would remain until sometime in 2023.
Can't say yet how much of the emailjs-imap-client code would be used, so hard to speculate on what bugs would get fixed by it or not.

Comment 40

[gene smith]

(In reply to amorim.andre.89 from comment #38)

So why not implementing something like, if I delete an email and then UNDO, simply implement a "move back" operation as if I was drag and dropping the email back to the original folder. Maybe this is an ugly workaround, but should work fine and the users would be happy.

That's essentially what my attached diff demonstrates. The "undo" just copies the messages back where it was instead of relying on just removing the \deleted flag and hoping the message comes back. Of course, there are other issues that the diff doesn't take into account.