624082 - IMAP: Moved messages disappears from IMAP move source folder even though "copy to IMAP move target folder" somehow failed when multiple "Move mails to IMAP folder" are requested for same move source IMAP folder

Status: UNCONFIRMED

(MailNews Core :: Networking: IMAP, defect)

Description

[Martin von Gagern]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101215 Gentoo Firefox/3.6.13
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20110107 Lightning/1.0b3pre Thunderbird/3.1.7

Moving a large number of messages to different folders can lead to some of them getting lost: they disappear from the original folder, but don't appear in their target folder. On the imap server, the messages are still in the original folder, but TB won't display them, probably due to some UID mismatch.

Reproducible: Sometimes

Steps to Reproduce:
1. Move a large number of messages from folder A to folder B
2. Quickly after that move remaining messages from folder A to folder C
3. Wait till the busy cursor disappears
Actual Results:  
First bunch of messages visible in B, second bunch not visible in TB at all, but present in A in the maildir store on the server.

Expected Results:  
Messages moved to B and C, or if that shouldn't work for some reason, still visible in A.

Did wireshark the connection after the problem had already occurred. TB didn't ask for any kind of message listing from the server for folder A. It simply displayed it empty, no matter how often I switched folders. Probably because the maximum UID was somehow persistently marked as "seen", "retrieved" or whatever.

Comment 1

[Martin von Gagern]

Renaming A.msf in the Profile/ImapMail/host.name directory makes the messages reappear. That's a good workaround if you notice that there actually are missing messages.

Comment 2

[Wayne Mery (:wsmwk)]

Or in the UI, you can right+click the folder > properties > repair.

what is "a large number"?

Comment 3

[Martin von Gagern]

(In reply to comment #2)
> Or in the UI, you can right+click the folder > properties > repair.

Thanks, hadn't known that. Might be useful!

> what is "a large number"?

700+ to B, then 1100+ to C. By the way, dovecot on localhost is the imap server in this case, and it reported no error to syslog, while they claim to always generate output if anything goes wrong. So I assume the server to be OK here.

After some consideration, I believe that having the messages disappear from their origin folder almost immediately might be a good thing in terms of a responsive UI. But that change in what TB considers the state of that folder should only be made permanent when the mails actually have been moved, i.e. when the move command has been sent and confirmed. From my observation I presume that current behaviour is somewhat different. So if something goes wrong, you can be certain that at least after the next restart of TB you see what's actually there. A discrepancy in the number of messages in some folder might be an indication of a broken msf file as well, and as far as I can see, IMAP provides that number along with the maximum UID.

Comment 4

[Chris Brand]

I can confirm this. I'm running hMailserver and moved about 70 folders from my local folders to the IMAP server. even when moving small folders with just 200 messages a bunch of messages were missing. I tried multiple times, but the messages were missing over and over again. Manually copying *just* the missing messages worked though.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Martin von Gagern from comment #0)
> Steps to Reproduce:
> 1. Move a large number of messages from folder A to folder B
> 2. Quickly after that move remaining messages from folder A to folder C
> 3. Wait till the busy cursor disappears
> Actual Results:  
> First bunch of messages visible in B, second bunch not visible in TB at all,
> but present in A in the maildir store on the server.
(In reply to Martin von Gagern from comment #1)
> Renaming A.msf in the Profile/ImapMail/host.name directory makes the
> messages reappear.

These indicate;
- Tb doesn't store \Deleted flag to disappeared mails.
  (mails which were not copied to move target folder B but not shown at folder A)
- Tb wrongly removes mail's meta data from .msf which was not successfully copied.
  (I assume you use "Move to Trash" model or "Remove immediately" model.)
- Because "renaming A.msf" is similar to "Repair Folders", all active mails in
  folder A at IMAP server(mails without \Deleted flag) is re-fetched from server.

If IMAP delete model of "Just Mark it as Deleted", deleted mail(\Deleted flag is stored) is shown with strike thru line at thread pane by Tb.
To Martin von Gagern (bug opener):
Can you check next when you can reproduce your problem with "Just Mark it as Deleted" model?
(1) Write down UID of all mails in folder A for which you requested "move to B".
    Execute your test with "Just Mark it as Deleted".
    Show "Order Received" column(value is UID of mail if IMAP)
    and check UIDs of mails with strike-thru line at thread pane of folder A.
(2) Check mails in folder A via second Tb instance. 
    Create a new Tb profile(call P2), define the IMAP account only,
    disable Global Indexer and Search, unckeck "Keep messages on this computer..."
    of Synchronization&Disk Space, disable automatic new mail check,
    disable automatic message purge(retention policy etc.),
    select "Just Mark it as Deleted" at Server Settings.
    Start second Tb with -no-remote mode using the new profile.
      thunderbird.exe -no-remote -P "P2"
    Click folder A, Show "Order Received" column.
    Check which mail(which UID) in folder A is shown with strike-thru line by (1)
    but mail of same UID is shown without strike-thru line by (2).

Note:
If "Just Mark it as Deleted" model, you can execute Undelete of any mail of \Deleted flag(shown with strike-thru line) any time via context menu of mail(s) at thread pane, unless Compact is requested(expunge command is issued, and mail of \Deleted flag is erased at server).

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to Chris Brand from comment #4)
> I can confirm this. I'm running hMailserver and moved about 70 folders from
> my local folders to the IMAP server. even when moving small folders with
> just 200 messages a bunch of messages were missing. I tried multiple times,
> but the messages were missing over and over again. Manually copying *just*
> the missing messages worked though.

If your problem is same problem as this bug, why you can manually copy from move source folder after the "move failure"(local mail folder in your case, although this bug is for "move source folder=IMAP folder" case and not-moved mail disappears from move source folder until Repair Folder.

"Move mail from local mail folder to IMAP folder" consists of;
(1) Upload mail data to IMAP server(append) == Copy step from local to IMAP.
(2) Delete the mail from local mail folder.
"Delete" in this context is same as "Shift+Delete" if local mail folder.
So, if a mail is once deleted, user can't see deleted mail any more, unless "flag for deleted status in X-Mozilla-Status: header" is manually reset and Repair Folder" is executed.
Your case is simply a "copy failure" during step (1), isn't it?

Move mails by what kind of operation?
Select mails at thread pane, then Drag&Drop to IMAP folder?
Select mails at thread pane, then "Move To" of context menu?

Comment 7

[WADA:World Anti-bad-Duping Agency]

Martin von Gagern(bug opener), see also bug 522675 and bug 693353, please.
These are for problem like next.
 Tb stores flag \Deleted, even though fetch from IMAP server for "move mail" fails,
 or even though "append to IMAP server for detach/delete" fails.
In your case, \Deleted flag doesn't look stored by Tb, however, similar mechanism is supected.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Chris Brand from comment #4)
> Imoved about 70 folders (snip)
> even when moving small folders (snip)

Move *Folder" instead of "move selected mails"? (this bug is for "move selected mails")

This bug is for problem when multiple requests of "move of selected mails from IMAP folder to IMAP folder of same IMAP account" on single move source folder.
"Move folder" && "Move from local to IMAP" case is better processed in separated bug, because "create folder"+"subscribe folder" is required before mail upload to IMAP server, for the folder and all subfolders.
Chris Brand, open separated bug for your problem, with detailed "Steps to reproduce problem", with attaching IMAP log to the bug(after remove/replace personal information, after remove very many log lines for mail data), please.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Bug 538375 was found for "Move folder or mails" && "Move from local to IMAP" case, in which bug opener says phenomenon was observed with Gmail IMAP and hMailServer.
Chris Brand, see Bug 538375 instead of this bug, please.

Comment 10

[Martin von Gagern]

(In reply to WADA from comment #5)
> To Martin von Gagern (bug opener):
> Can you check next when you can reproduce your problem with "Just Mark it as
> Deleted" model?

Intend to try this out asap, but haven't managed to find the time so far.

Comment 11

[jgehrcke]

Using TB 38.2.0 on Windows. It just happened to me that I moved *one* message from one IMAP folder to another, and the description in this bug report fits perfectly:

- disappears from the original folder
- but does not appear in the target folder
- On the imap server, the message is still in the original folder
- but TB does not display it

An independent IMAP client showed the mail in the original folder, but TB did not.

A repair (right+click the folder > properties > repair) made the message re-appear in the original folder. I then moved it "again", and it worked.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to jgehrcke from comment #11)
> Using TB 38.2.0 on Windows. It just happened to me that I moved *one*
> message from one IMAP folder to another, and the description in this bug
> report fits perfectly:
> 
> - disappears from the original folder
> - but does not appear in the target folder
> - On the imap server, the message is still in the original folder
> - but TB does not display it
 
Do you use MailDir for your IMAP accunt?
I could consistently see such phenomenon in testing for MailDir only.
UI to set msgStore=MailDir was opened by a few Tb versions, but it's hidden again.

Comment 13

[jgehrcke]

(In reply to WADA from comment #12)
  
> Do you use MailDir for your IMAP accunt?
> I could consistently see such phenomenon in testing for MailDir only.
> UI to set msgStore=MailDir was opened by a few Tb versions, but it's hidden
> again.

Where can I see which format I use for a specific account? Probably I don't use MailDir.

I should add that I am using my Thunderbird profile since many years and for the IMAP account in question I regularly move mails between folders via drag'n'drop. Have never observed an issue with that. Today was the first time. I was hugely surprised to not find the mail in the target folder.

I then looked into the activity manager and there was a corresponding entry "Moved 1 message from Unbekannt to Inbox". Also, I found the mail in question via full text search, i.e. it showed up in the search results. However, it could not be opened anymore.

So I knew that it wasn't me who made a mistake, but it was Thunderbird who got confused about that mail. I cannot reproduce this issue anymore, but clearly there was something bogus going on.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to jgehrcke from comment #13)
> Where can I see which format I use for a specific account? Probably I don't use MailDir.

mail.server.server#.storeContractID = @mozilla.org/msgstore/berkeleystore;1
mail.server.server#.storeContractID = @mozilla.org/msgstore/maildirstore;1

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to jgehrcke from comment #13)
> I regularly move mails between folders via drag'n'drop.
> Have never observed an issue with that. Today was the first time.

Drag&Drop is more dangerous than Move in menu, because "Move mails operation" can be canceled by problem in DrogDrop" too. Move in menu is safer thaan Drag&Drop because it's affected by IMAP/IMAP Folder relevant problem only. Safest way is "Do Copy, Then Detete after confirmation of successful Copy" :-) 
   
> I then looked into the activity manager and there was a corresponding entry
> "Moved 1 message from Unbekannt to Inbox".

This indicates;
  For "Move between folders operation", move ended normally, so ""Moved 1 message" was logged,
  or log was written before actual execution of move.
  For IMAP, "copy to target folder" was not successful,
  so "Delete at MoveSoure"=="uid store nn +Flags(\Deleted)" was not executed.
  For "Move messages", move normally ended, so removed msgDbHdr from msgDB(xxx.msf) of MoveSourceFolder.

> Also, I found the mail in question via full text search,
> i.e. it showed up in the search results. However, it could not be opened anymore.

If Full Text Search(Global Search, Ctrl+K), it can be explained.
  Because "uid store nn +Flags(\Deleted)" is not issued, data of mail is not removed by Global Indexer.
If Advanced Search or Quick Search(Ctrl+Shift+K), search is done via messageKey(=UID if imap) of msgDBHdr in msgDB. Because msgDBHdr is removed from msgDB(xxx.msf), search doesn't find the mail. 
Phenomenon itself is same as one I saw in mailDir testing.

I think timing dependent phenomenon(depends on "at what stage, error occurred"), and we can produce phenomenon relatively easily if MailDir is used. MailDir is microscope of IMAP problem :-)

Comment 16

[jgehrcke]

(In reply to WADA from comment #12)

> Do you use MailDir for your IMAP accunt?

I have used the config editor and searched for "maildirstore". No search result. All my accounts are configured to use the berkeleystore.

Comment 17

[jgehrcke]

Thanks for your analysis, Wada. 

I see that "Drag&Drop is more dangerous than Move in menu". However, drag'n'drop is supported, so the goal clearly must be that the user is informed about an error when an error occurs that TB cannot recover from itself. 

I also agree that this is a timing dependent phenomenon. That is, the mechanism in question is subject to a race condition. It should be identified and defused.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to jgehrcke from comment #11)
> I moved *one* message from one IMAP folder to another, (snip)

(1) Move between imap folder of same imap account? (2) Or move between different imap account?
I can't imagine prolem in (1), because move = "uid move xx Target" or "uid copy xx Targrt" + "uid store xx +Flags(\Deleted)" only and move of one mail only.
If (2), move = "fetch body from source server" + "append(upload) to target server" + "uid store xx +Flags(\Deleted) at source server", so error at end of append may produce such problem, because there is problem in error handling of append.
  eg. transfer of all mail data by append completes, but timeout occurs before OK response is returned.
  At server, all mail data is received, so uploaded mail is normally accepted.

Comment 19

[WADA:World Anti-bad-Duping Agency]

FYI.
Bug 538375 is for "Move or Copy from local folder to IMAP folder", so "append, something wrong while appending" is involved.

Comment 20

[Wayne Mery (:wsmwk)]

(might be backend, but imap is at least better than General)