753624 - Copy/Move of imap offline store messages not working for Maildir Lite

Status: RESOLVED DUPLICATE of bug 1113275

(MailNews Core :: Backend, defect)

Description

[Jeff Grossman]

Attachment: Screenshot of empty folder that is created

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/15.0 Firefox/15.0a1
Build ID: 20120509030514

Steps to reproduce:

I installed a Tinderbox build today (1336602750) which includes the fix for bug 748807.  The copying from tmp to cur seems to be working fine now.  But, the message files are not moving between folders if I move or copy a message between folders.


Actual results:

When I deleted a message from my Inbox to my Trash folder, the message file stayed in the Inbox/cur folder instead of going to the Trash/cur folder.  The message showed correctly in the Trash folder in the message list.  If I opened up the message in the Trash folder from within Thunderbird it said it was downloading the message again.  I also noticed there was now an empty folder created in the Trash/cur folder.  I am assuming for the message I moved.  But, the message still stayed in the Inbox/cur folder.


Expected results:

The message file should move from the Inbox/cur folder to the Trash/cur folder it I move that message, or any folder that I move the message to.

Comment 1

[weliot]

Bug still occurs for most recent build of Thunderbird 16.

Comment 2

[Wayne Mery (:wsmwk)]

confirming based on comment 1

Comment 3

[Emre]

I can confirm. There are also some issues with folders appearing and duplicate messages when I move messages between maildir based Local Folders & maildir based IMAP Offline store.

Comment 4

[WADA:World Anti-bad-Duping Agency]

> Bug summary : Copy/Move of imap offline store messages not working for Maildir Lite

What do you call by your "Maildir *Lite*"? What is differnce from "Maildir" used in src file name or module/class/variable in following? 
> http://mxr.mozilla.org/comm-central/source/mailnews/base/src/
>  /mailnews/base/src/nsMailDirProvider.cpp
>  /mailnews/base/src/nsMailDirProvider.h
>  /mailnews/base/src/nsMailDirServiceDefs.h
>  /mailnews/base/test/unit/test_nsMailDirProvider.js
>  /mailnews/local/src/nsMsgMaildirStore.cpp
>  /mailnews/local/src/nsMsgMaildirStore.h
> /mailnews/test/resources/mailDirService.js
> /mailnews/base/src/nsMailDirProvider.cpp
> /mailnews/base/src/nsMailDirProvider.h
> /mailnews/base/src/nsMailDirServiceDefs.h
> /mailnews/local/src/nsMsgMaildirStore.cpp
> /mailnews/local/src/nsMsgMaildirStore.h

(In reply to Jeff Grossman from comment #0)
> Created attachment 622607 [details]
> Screenshot of empty folder that is created
>(snip)
> The copying from tmp to cur seems to be working fine now.
> But, the message files are not moving between folders if I move or copy a message between folders.

Is problem when "move mails from Maildir folder to MailDir folder"?

> Note-1: Maildir folder in this context == mail folder under server with following setting.
> mail.server.serverX.storeContractID = @mozilla.org/msgstore/maildirstore;1

Is non-Maildir account/server/folder involced in your problem?

> Note-2: Non-Maildir folder in this context == mail folder under server with following setting.
> mail.server.serverX.storeContractID = @mozilla.org/msgstore/berkeleystore;1

During test of bug 793524, I could observe "blank directory with name of file for mail in Maildir under cur" when "copy from non-Mildir folder to Mail dir folder".
However, during test of general Maildir, I saw similar "blank directory with name of file for mail in Maildir under cur" after "copy(or move) mails between Maildir folder" in Tb 17.0.2 and latest trunk nightly.

FYI.
Fllowing bugs looks similar phenomenon of "blank directory with name of file for mail in Maildir under cur" after your bug open. Some of them seems phenomenon when "copy/move from non-Mildir folder to Maildir folder", but some others seems phenomenon when "copy/move from Maildir folder to Maildir folder".
> bug 791966 : Move of mails from IMAP mbox folder to local maildir folder results in lost of the mail
> bug 793524 : With maildir storage, mail copy from non-maildir mbox creates empty directories instead of files in "cur"
> bug 816304 : Filters that copy messages crash TB when using maildir
> bug 827048 : Filtered messages lost (POP) going to imap or maildir folder
> bug 852080 : MailDir not working for Local Folders

Comment 5

[Magnus Melin [:mkmelin]]

(In reply to WADA from comment #4)
> What do you call by your "Maildir *Lite*"? What is differnce from "Maildir"

That's the "maildir" we have in the tree. There's at least some difference wrt 
where new mails are kept.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to Jeff Grossman from comment #0)
> Actual results:
> When I deleted a message from my Inbox to my Trash folder, the message file
> stayed in the Inbox/cur folder instead of going to the Trash/cur folder. 
> The message showed correctly in the Trash folder in the message list.  
> If I opened up the message in the Trash folder from within Thunderbird it said it was downloading the message again.
> I also noticed there was now an empty folder created in the Trash/cur folder.

With "Move to Trash model? I think so.
  
> The message showed correctly in the Trash folder in the message list.
> If I opened up the message in the Trash folder from within Thunderbird
> it said it was downloading the message again.

Because od "Delete mail in an IMAP account with IMAP delete model of Move to Trash", "select Inbox, uid xx copy Trash" is requested. Sooner or later, copied mail in Trash i detected and mail data is fetched.downloading ..." is an evidence that "uid xx copy Trash" was executed.

> I also noticed there was now an empty folder created in the Trash/cur folder.

I could see it by "mova a mail from maildirstore/IMAP Offline-Use=On folder to maildirstore/local mail folder(dummt POP3 account)".

> I am assuming for the message I moved. But, the message still stayed in the Inbox/cur folder.

It indicates that "uid xx store +Flag \Deleted" is not requested yet. It's perhaps due to error in Copy step of "Copy + Delete"=="uid copy + uid store \Deleted" in IMAP. 

Phenomenon of "not delete at move source folder" is observed by "mova a mail from maildirstore/IMAP Offline-Use=On folder to maildirstore/local mail folder(dummy POP3 account)".
Because I set IMAP delete model of "Just mark it as deleted", if "uid xx store +Flag \Deleted" is issued, it should be shown with strike-thru line, but mail was still shown as if normal mail data.
However, Offline flag of the *moved* mail was Off even though Offline-Use=On folder, and storeToken=="file name in cur directory" was not set. So, when Work Offline mode, View/Message Source was hidden and CTRL+U showed nothing.

Comment 7

[WADA:World Anti-bad-Duping Agency]

Attachment: Process Monitor log : Move mail from IMAP offline-use=On folder to local mail folder, with maildirstore enabled

Comment 8

[WADA:World Anti-bad-Duping Agency]

Attached log is for;
Move of following mail in Offline-Use-On-02
> \ImapMail\imap.gmail.com\Offline-Use-On-02\cur\1364539917013000
to Offline-Use-Test was requested, and following directory was created.
> \Mail\a.a.a\Inbox.sbd\Offline-Use-Test\cur\1364539917013000

Comment 9

[WADA:World Anti-bad-Duping Agency]

Attachment: Process Monitor log : Move is successfull, although same move test

Process Monitor log for same test : Move mail from IMAP offline-use=On folder to local mail folder, with maildirstore enabled.

Before test, following was done to clean up.
(i)  Delete Offline-Use-On-02.msf
     Delete files in Offline-Use-On-02\cur
     Restart Tb, Repair Folder of Offline-Use-On-02
(ii) Delete files in Offline-Use-Test\cur
     Repair Folder of Offline-Use-Test

Move of following mail in Offline-Use-On-02
> \ImapMail\imap.gmail.com\Offline-Use-On-02\cur\1364545943794000
to Offline-Use-Test was requested, and following FILE is normaly created.
> \Mail\a.a.a\Inbox.sbd\Offline-Use-Test\cur\1364545943794000

Because of same drive on MS Win, Offline-Use-On-02\cur\1364545943794000 looks moved to Offline-Use-Test\cur\1364545943794000 instead of filecopy.
So, at imap.gmail.com\Offline-Use-On-02(IMAP Offline-use=On, with mark it as deleted), different file was generated and storeToken value was changed to  
1364546701825000.
> \ImapMail\imap.gmail.com\Offline-Use-On-02\cur\1364546701825000

Phenomenon in this test is;
(a) "Copy to target" step is normally executed by API for "move file".
(b) "Delete from source" step is somehow not executed as expected.
Problem may be relevant to incident of (b) when maildirstore/IMAP/Offline-use=On.
"Empty directory instead of file" is perhaps phenomenon when "move file via API" fails, because same "empty directory instead of file" is observed by bug 855950 which is for problem in "copy from berkleystore folder to maildirstore/non-IMAP folder" when file in storeToken is not found.

Comment 10

[WADA:World Anti-bad-Duping Agency]

My test procedure was; hold test mails in maildir IMAP A, before test, copy mails to　maildir IMAP B, and moved mails to local maildir C.
During test, I sometimes copied mails to A, deleted from B fr test. And, I was aware of existence of "empty directory of A/cur/xxxxxxxx" in my folder for origial test data...

In my test, phenomena like next occurred.
(i)   empty dir of xxxx is used for mail-x in A. mail-x is copied to B.
      status in mail-x in B is perhaps bad.
      mail-x in B is moved to C => empty directory of C/cur/xxxx
      because of bad status of mail-x in B, delete at B doesn't work.
(ii)  empty dir of xxxx is used for mail-x in A. 
      mail-x in A is moved to C => empty directory of C/cur/xxxx
      because xxxx is directory, delete at A breaks mail's status.
      in my test, mail was shown with strike-thru line after move,
      (Just mark it as deleted)
      so, Tb executed "Delete after copy" step.
      however, after recovery, mail was shown normally,
      so "uid xx store +Flag \Deleted" was not actually issued.
      and, following exception happensedby "Repair Folder".
> Error: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.remove]
> Source File: chrome://messenger/content/folderPane.js
Line: 2095  
(iii) Copy with multiple mail selection" has problem.
      select mail-1 and mail-2 at MboxA(maidirstore, IMAP).
      copy to MboxB of same IMAP account.
      => same subject etc.(mail-2's one) was shown at thread pane.
      different UID was shown, so "copy uid a,b MboxB" is OK.
      correct mail was shown at message pane, so mail data copy is OK.
      when Repair Folder is executed, similar exception occured.
I think "duplicated mail" seen in some bugs is this phenomenon.

After restart Tb, Shift+Delete mails, delete files from Mbox/cur, "Repair Folders", copied mail to MboxA from Offline-Use=Off folder, one by one, and moved a mail to MboxC.
=>
Mail was normally copied to MboxC, and mail disappared from thread pane,
However, when switch to other Mbox and go back to MboxC, mail was shown again. Because Gmail IMAP with auto-expunge=On, if "uid xx store +Flag \Deleted" is issued, mail is not shown because of auto-expunge=On. So, "uid xx store +Flag \Deleted" was not issued by Tb by "Move".
This is same result as comment #9.

I believe major cause of problem in "Move mail(s)" is "Copy", and problem in "multiple mail copy" is main culprit.
I think problem in "Move mail" is one of next, or both.
(a) even though copy phase of "move mail" fails,
    uid xx store +Flag \Deleted is issued.
(b) even though copy phase of "move mail" is successful,
    uid xx store +Flag \Deleted is not issued.
"Move multiple mails" may have problem too.

By the way, if "copy from maildir/offline-use=on to maildir/non-imap" is successful, bug 855954 currently occurs. Be careful in evaluation of test results.

Comment 11

[WADA:World Anti-bad-Duping Agency]

To bug opener and all problem reporters in this bug:

Does you problem occur even when clean IMAP mail folder, clean mail data(if copied, copied from offline-use=off folder, one by one), move single mail only?

Comment 12

[WADA:World Anti-bad-Duping Agency]

Many of "funny Target\cur/nnnnnnnn file(size=0) or empty directory" in "Move mails between maildirstore/local folder or maildirstore/IMA/Offline-Use=On folder" looks cauesed by "Garbage by error in Copy to maildirstore".

Duplicated cur file name use was observed by a simplest "multiple mail copy".
(1) MboxA: maildirstore, IMAP/Offline-Use=Off. "cur" is not used.
    512 mails is held:  mail-1, mail-2, ..., mail-512
                        unique Subject:, unique Message-ID:, is kept.
(2) MboxB: maildirstore, IMAP, Offline-Use=On. cur directory is used.
    Copy all 512 mails in MboxA to MboxB.
    Because MboxA is Offline-use=On, mail data is fetched.
    At thread pane       : All 512 mails is shown.
    In Mbox/cur          : Only 484 files is created.
    Message pane display : mail-491 shows content of mail-456
    mail-491 : messageKey=488
    mail-456 : messageKey=489
    storeToken = 1364617150945000 is used for both mail.
    content of MboxB/cur/1364617150945000 : data of mail-456

Dump data of msgDBHdr of mail-491
>   [messageKey] = 488
>   [statusOffset] = 0
>   [messageOffset] = 0
>   [messageSize] = 938
>   [offlineMessageSize] = 1024
>   [lineCount] = 28
>   [date] = 0
>   [dateInSeconds] = 0
>   [StringProperty_pendingRemoval] = 
>   [flags] = 268435585
>   [flag_Detail] = { FeedMsg = false, IMAPDeleted = false, MDNReportSent = false, Read = true, Replied = false, Marked = false, Expunged = false, HasRe = false, Elided = false, Offline = true, Watched = false, SenderAuthed = false, Partial = false, Queued = false, Forwarded = false, Priorities = false, New = false, Ignored = false, MDNReportNeeded = false, Template = false, Attachment = true, Labels = false, RuntimeOnly = false }
>   [threadId] = 488
>   [threadParent] = 4294967295
>   [messageId] = 1KB-Mail.000491.000001
>   [mime2DecodedSubject] = 1KB-Mail-000491
>   [All_StringProperty] = { flags = 10000081, statusOfset = 0, sender = T-000491-M-000001@f.f.f, recipients = M-000001-T-000491@t.t.t, subject = 1KB-Mail-000491, message-id = 1KB-Mail.000491.000001, date = 0, dateReceived = 0, X-GM-MSGID = 1430885469959438933, X-GM-THRID = 1430885469959438933, X-GM-LABELS = Offline-Use-Off, sender_name = 85|T-000491-M-000001@f.f.f, priority = 1, size = 3aa, keywords = tag-1, threadParent = ffffffff, msgThreadId = 1e8, ProtoThreadFlags = 0, msgOffset = 0, storeToken = 1364617150945000, offlineMsgSize = 400, numLines = 1c, label = 0 }

Dump data of msgDBHdr of mail-456
>   [messageKey] = 489
>   [statusOffset] = 0
>   [messageOffset] = 0
>   [messageSize] = 938
>   [offlineMessageSize] = 1024
>   [lineCount] = 28
>   [date] = 0
>   [dateInSeconds] = 0
>   [StringProperty_pendingRemoval] = 
>   [flags] = 268435585
>   [flag_Detail] = { FeedMsg = false, IMAPDeleted = false, MDNReportSent = false, Read = true, Replied = false, Marked = false, Expunged = false, HasRe = false, Elided = false, Offline = true, Watched = false, SenderAuthed = false, Partial = false, Queued = false, Forwarded = false, Priorities = false, New = false, Ignored = false, MDNReportNeeded = false, Template = false, Attachment = true, Labels = false, RuntimeOnly = false }
>   [threadId] = 489
>   [threadParent] = 4294967295
>   [messageId] = 1KB-Mail.000456.000001
>   [mime2DecodedSubject] = 1KB-Mail-000456
>   [All_StringProperty] = { flags = 10000081, statusOfset = 0, sender = T-000456-M-000001@f.f.f, recipients = M-000001-T-000456@t.t.t, subject = 1KB-Mail-000456, message-id = 1KB-Mail.000456.000001, date = 0, dateReceived = 0, X-GM-MSGID = 1430885437803636784, X-GM-THRID = 1430885437803636784, X-GM-LABELS = Offline-Use-Off, sender_name = 85|T-000456-M-000001@f.f.f, priority = 1, size = 3aa, keywords = tag-1, threadParent = ffffffff, msgThreadId = 1e9, ProtoThreadFlags = 0, msgOffset = 0, storeToken = 1364617150945000, offlineMsgSize = 400, numLines = 1c, label = 0 }

There are at least following garbage by "Copy to maildir".
(A) Empty directory of Mboxname/cur/nnnnnnnn where nnnnnnnn==storeToken
(B) Null file of Mboxname/cur/nnnnnnnn where nnnnnnnn==storeToken
(C) Duplicate use of Mboxname/cur/nnnnnnnn by different messageKey
(D) Other mail's header data use by different UID, different MessageKey
    (storeToke, Mboxname/cur/nnnnnnnn is unique.                    )
    (observed with two mails of same Message-ID:/different Subject: )
    (which is crafted mail for Gmail's duplicate mail detection test)

Because "Move mail from maildirstore to maildirstore" is "move file of Source/cur/storeToken to Target/cur/storeToken in file sytem", (A), (B), (C), produces error in "Move mail".
If error in "move file via API call" occurs, result is unpredictable.

Detecting of (C), (D) in "move mail" code is perhaps difficult, but detecting of (A) and (B) is easy.

Can some one put code to detect (A) and (B) in "move mail" code?
Can some one add "error return code check" step after "move file via API call"?

Comment 13

[WADA:World Anti-bad-Duping Agency]

FYI. I've opened bug 856286 for problem of comment #12.

Comment 14

[WADA:World Anti-bad-Duping Agency]

FYI.

I watched Process Monitor console after end of "move files to Target/cur by move mails" for a while. 

(1) MboxA: maildirstore, IMAP/Offline-Use=Off. "cur" is not used.
    several mails:  mail-1, mail-2, ...
(2) MboxB: maildirstore, IMAP, Offline-Use=On. cur directory is used.
    Copy several mails in MboxA to MboxB.
    Because MboxA is Offline-use=Off, mail data is fetched.
    No problem at this stage.
(3) Move several mails in MboxB to MboxC.
      MboxC: maildirstore, POP3. cur directory is used. MboxC is empty.
    move of MboxB/cur/nnnn to MboxC/cur/nnnn normaly executed.
(4) Activity shown at Process Monitor console.
    At MboxB(move source folder);
    Data is writen to MboxB/tmp/xxxx1, and moved to MboxB/cur/xxxx1,
    Data is writen to MboxB/tmp/xxxx2, and moved to MboxB/cur/xxxx2,
                                    |                             | 
    Data is writen to MboxB/tmp/xxxxN, and moved to MboxB/cur/xxxxN
(5) MboxB(move source folder)
    All mails are shown.
    file name under cur(storeToken) is different from one at step (2).

Phenomenon at move source folder looks;
- Source/cur/xxxx is moved to Target/cur/yyyy by "Move mails"
- "uid xx store +Flags \Deleted" is not issued yet.
- Because Source/cur/xxxx is deleted(moved), re-fetch of message body
  is initiated(perhaps by auto-sync).
- "uid xx fetch body[]" is executed, and written in Source/tmp/yyyy
   where yyyy is newly assigned storeToken.
- After fetch complete, Source/tmp/yyyy is moved to Source/cur/yyyy.

Comment 15

[WADA:World Anti-bad-Duping Agency]

I believe followings are better analyzed in separate bugs, instead of processing all in this bug's summary according to description by bug summary".
(a) Garbage creation in "move Source/cur/nnnn to Target/cur/nnnn" step.
(b-1) No delete action at Source folder after normal end of "move Source/cur/nnnn to Target/cur/nnnn" step.
or
(b-2) Deletion of mail at Source folder even though error occurs in "move Source/cur/nnnn to Target/cur/nnnn" step.

Changing bug summary to orginal to avoid misleading. Sorry for cluttering and spam.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to Jeff Grossman from comment #0)
> Bug summary :Copy/Move of imap offline store messages not working for Maildir Lite

To Jeff Grossman(bug opener): 

(A) For "empty Mbox/cur/nnnnnnnn DIRECTORY" part by Copy/Move in your case.
Is your problem same as known cases written in bug 856387 comment #3?
Or different problem? If different, what is difference in your case?

(B) For phenomenon of "not deleted from move source Mbox" in your case.
Same as bug 856532?
Or you are talking about merely  "non deleted MoveSourceMbox/cur/nnnn file" in bug 771643.

(In reply to weliot from comment #1)
> Bug still occurs for most recent build of Thunderbird 16.
(In reply to Emre from comment #3)
> I can confirm. There are also some issues with folders appearing and
> duplicate messages when I move messages between maildir based Local Folders & maildir based IMAP Offline store.

To weliot and Emre:

Actually same case as original report of this bug == comment #0 == bug opener's case?
Merely similar comment to "I saw Tb's crash too" in bug of "Tb's crash happened" in bugzilla.mozilla.org?

Do you see "same problem in same case as yours" in dependency tree for Meta Bug 845952?
If No, what is difference of your case from bugs for similar symptom in the dependency tree?

Comment 17

[WADA:World Anti-bad-Duping Agency]

To Jeff Grossman(bug opener):

Bugs I opened are for funny phenomena I saw by my simplest check of "copy/move of one, or two, or three mails, betwenn Mbox with enabling MaildirStore for some accounts" after I read your comment #0.
Because "copy or move some mails" is needed for preparation of test, I could meet many funny phenomena due to bug around mail copy/move with MaildirStore :-)
As seen in dependency tree for Meta Bug 845952, "copy/move with MaildirStore" has sufficiently many problems.
And, current meta bug 845952 is not so appropriate for tracking of problems around "copy/move with MaildirStore" only.  
> bug 845952 : finish "maildir" message storage
I think your bug summary is fortunately one of best ones for Meta bug for tracking it.
Can I change this bug, first crisp bug report on "copy/move with MaildirStore", to meta bug for tracking of it?

Comment 18

[Emre]

Dear WADA, it's been sometime since I reported this bug, I can't remember now exactly what happened. But indeed my case was similar to comment #0 as it was happenning when I moved messages around my accounts & folders.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to Emre from comment #18)
> I can't remember now exactly what happened.

I see. Ignore your comment #3 case. In any case, any problem can be observed currently :-)

Lets's focus on comment #0 case.
Because "Delete" case, and it sounds the "Delete"=="Move to trash", and "Inbox/cur" and "Trash/cur" is referred, and IMAP is in bug summary, it's perhaps following case.
- IMAP account,
- IMAP account is msgstore/maildirstore
- IMAP delete model = Move to trash
  trash folder == top level Trash of the IMAP account
- Both Inbox and Trash is Offline-Use=On
- "Delete of mails"(not Mbox) at Inbox is executed,
  and mails are moved to Trash.
So,it can be called next case.
  Move mails from Maildir/IMAP/OfflineUseOn to Maildir/IMAP/OfflineUseOn
  Move mails within same IMAP accont => uid xx copy, instead of append
  Not ordinal move. It's move by explicit "Delete" request.

I don't think it works well, because even copy doesn't look to work...
(1) Two Maildir/IMAP/OfflineUseOn folders of same IMAP account,
    Call MboxA, MboxB.
    Copy several mails from Offline-Use=Off folder of same account,
    in order to avoid unwanted problems, and click mails of both Mbox.
    => MboxA/cur, MboxA/tmp, MboxB/cur, MboxB/tmp are normally created.
    => Mail data is normally held as MboxA/cur/nnnn, MboxB/cur/nnnn.
    Do cleanup of MboxB for test.
     Delete all mails in MboxB, Repair Folder, terminate Tb.
     Delete all files in MboxB/cur, delete MboxB.msf to force refetch.
    Restart Tb, Open MboxB => Clean. No files in MboxB/cur, tmp.
(2) Copy mail(s) from MboxA to MboxB.
    => Copied mails are shown at thread pane of MboxB.
    => No files are created in MboxB/cur.
(3) Click mails in MboxB at thread pane.
    => empty directory named MboxB/cur/nnnn is created for each mail.

At end of step (2), ::Ofline flag of msgDBHdr.flags is already set for copied mail, even though MboxB/cur/nnnnnn(nnnnnnn is storeToken value) is not normally created by Copy.
Because MboxB/cur/nnnnnn pointed by storeToken of stringProperty of msgDBHdr doesn't exist, same phenomenon as bug 856387 occurs.

Could someone see "mail copy between two Maildir/IMAP/OfflineUseOn folders" worked well?

I saw "empty directory by mail copy between two Maildir/IMAP/OfflineUseOn folders of same IMAP account" several times during setup of test mails, I always used "copy mails from Offline-Use=Off folder to Offline-Use=On folder" in test mail setup for my test of other bugs, in order to avoid mis-evaluation of test results. So, I didn't check "mail copy between Maildir/IMAP/OfflineUseOn folders" case utill today.

Comment 20

[WADA:World Anti-bad-Duping Agency]

Phenomenon may be following.
- Because copy between Maildir(Mbox/cur is used), mail copy/move will be
  done by "copy/move of MboxA/cur/nnnn file to MboxB/cur/nnnn",
  (if already used, suffixed and nnnn-1, nnnnn-2, ... is used)
  msgHdr is copied from source mail except messageKey.   
- Because "copy/move source folder" is IMAP,
  "copy/move of MboxA/cur/nnnn file to MboxB/cur/nnnn" is not executed,
  as not executed when "copy/move from IMAP/Offline-Use=Off" folder.
- However, because "::Offline = true" is set, no one will fetch mail
  data from server.

When Berkley/IMAP/Offline-Use=On, manual deletion of offline-store file forces re-fetch from server, because mail data is not found.
If IMAP folder, "Mbox/cur/nnnnnn file is not found" condition should invoke fetch from server.

Comment 21

[WADA:World Anti-bad-Duping Agency]

Current "Copy/Move mail between IMAP/OfflineUseOn folders while Work Online" is almost same as "Copy/Move between IMAP/OfflineUseOn folders while Work Offline" + "Go back to Work Online".
When Save/Copy/Move to IMAP/Offline-Use=On in BerkeleyMilstore, "Copy mail data to offline-store file with faked key" is executed even when Work Online mode, in order to pass mail body data to Gloda as early as possible. (For example, upon "Sent mail copy save to Sent folder")

Followig was observed with "Move mail between Maildir/IMAP/OfflineUseOn folders while Work Offline mode".
(0) MboxB : Highest UID=10, Next UID=11
(1) Copy a mail-x to MboxB by other IMAP client.
    => UID=11 is used by mail-x
(2) Tb, Work Offline mode. for Tb, still Highest UID=10, Next UID=11.
    Move mails from MboxA to MboxB.
    mail-1 : UID(messageKey)=11, ::Offline flag=true, storeToken=nnn1
             MboxB/cur/nnn1 is not created.
    mail-2 : UID(messageKey)=12, ::Offline flag=true, storeToken=nnn2
             MboxB/cur/nnn2 is not created.
    Moved mails are removed from thread pane of MboxA.
(3) Tb, Go Work Online mode.
    mail-x : UID(messageKey)=11, ::Offline flag=true, storeToken=nnnx
             MboxB/cur/nnnx file is normally created.
    mail-1 : UID(messageKey)=12, ::Offline flag=true, storeToken=nnn1
             MboxB/cur/nnn1 is not created.
    mail-2 : UID(messageKey)=13, ::Offline flag=true, storeToken=nnn2
             MboxB/cur/nnn2 is not created.
    Fetch of newly known UID=11 only is normally executed.
    Because Offline flag is copied or set in new msgDBHdr of UID=12/13,
    fetch of UID=12 and UID=13 is not executed
(4) At other IMAP client, with "Just mark it as deleted".
    MboxA : deleted mails is flagged as \Deleted
    MboxB : mail-x/UID=11, mail-1/UID=12, mail-2/UID=13, are shown

I think following is needed.
(1) In Offline mode step of "Copy" or "Copy phase of Move",
    Copy/Move SourceMbox/cur/nnnn to TargetMbox/cur/nnnn even if IMAP,
    because Copy/Move Source folder is Offline-Use=On folder.
(2) In Online mode step of "Copy" or "Copy phase of Move",
    set ::Offline flag in msgDBHdr.flags correctly,
    and, if Move, delete TargetMbox/cur/nnnn in re-sync step.

Note-1:
  "Remove unused mail data for deleted mail in Offline-store file",
  and "Remove old mail data for faked key from Offline-store file",
  is done by "pendingRemoval flag in msgDBHdr.flags" and "Compact"
  in BerkleyStore. "Compact in BerkleyStore" ignores msgDBHdr with
  pendingRemoval=true" and "data not pointed by msgDBHdr" upon copy.
Note-2:
  When UIDPLUS is supported by server, there is no need to use
  "faked key" because UID is known upon copy, so there is no need to
  delete Target/cur/nnnn, and there is no need to fetch mail data again.
Note-3:
  "Delete of Target/cur/nnnn for faked key" can be reduced by message
  header check upon re-sync with server after Copy/Move.

Because "delete of Mbox/cur/nnnn file" is relevant, bug 771643 is relevant to this bug.

Comment 22

[WADA:World Anti-bad-Duping Agency]

Jeff Grossman(bug opener), I've opened meta bug 859011 for problems around "Copy/Move with MaildirStore".
Keep this bug for your specific case of comment #0, please.

Comment 23

[Kent James (:rkent)]

Using the revised description of comment 15 (which focused on copy/move issues only) I can reproduce this bug on esr31, but the bug is gone after I apply the fixes from bug 1113275. So I am going to resolve as dup, which means it should be fixed after that bug lands.