855950 - mail copy from non-maildirstore(berkeleystore/mbox) to maildirstore creates empty directory instead of file in subdirectory of "cur".

Status: RESOLVED DUPLICATE of bug 1017939

(MailNews Core :: Database, defect)

Description

[WADA:World Anti-bad-Duping Agency]

mail copy from non-maidirstore(berkeleystore) to maidirstore creates empty directory instead of file in subdirectory of "cur".

Following bugs looks reports about similar phenomenon of "empty directory instead of file under cur or tmp of maidirstore".
Some of them seem phenomenon when "copy/move from non-maidirstore(berkeleystore) to maidirstore", but some others seem phenomenon when "copy/move from maidirstore to maidirstore", but it's not so clear and it seems both cases involved.
> bug 753624 : Copy/Move of imap offline store messages not working for Maildir Lite
> bug 791966 : Move of mails from IMAP mbox folder to local maildir folder results in lost of the mail
> bug 793524 : With maildir storage, movemail creates empty directories instead of files in "cur"
> bug 816304 : Filters that copy messages crash TB when using maildir
> bug 827048 : Filtered messages lost (POP) going to imap or maildir folder
> bug 852080 : MailDir not working for Local Folders
This bug is for "copy from berkeleystore to maidirstore" specific phenomenon which is 100% reproducible, and is essentially same as bug 793524 which is report on movemail.

[Build ID]
> Tb 17.0.2 on MS Win-XP
> Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Thunderbird/22.0a1

[Steps to reproduce]

(1) prefs.js setting.
- Default (not touched)
> mail.serverDefaultStoreContractID=@mozilla.org/msgstore/berkeleystore;1
- All accounts except specific serverX (not touched)
  (type=none==smart mailboxes, type=imap, type=pop3)
> mail.server.serverN.storeContractID;@mozilla.org/msgstore/berkeleystore;1
- serverX, changed to maildirstore.
  type=pop3, and type=none(Local Folders) is checked
> mail.server.serverX.storeContractID=@mozilla.org/msgstore/maildirstore;1
  All dir/file is deleted from Mail directory, and Tb is restarted.

(2) Copy mail from Mbox of non-maidirstore(berkeleystore)
    to Mbox of non-IMAP maidirstore (type=none, Local Folders).
       Mbox of non-maidirstore(berkeleystore) :
         IMAP Offline-Use=On  folder, storeToken is set
         IMAP Offline-Use=Off folder, storeToken is not set
         IMAP Offline-Use=Off folder, storeToken is set
         Folder of POP3 account,      storeToken is set

(2-A) Source Mbox = berkeleystore(IMAP Offline-Use=On) :
      IMAP Offline-Use=On folder
      Offline flag = On, storeToken is nnnnnnnn(non-ZERO)
      statusOffset=0 because of IMAP folder.
(2-A-1) First copy to maidirstore(Local Folders)
    storeToken of source mail is used for file name(==nnnnnnnn).
    DIRECTORY of "nnnnnnnn" is created under "cur".
    messageKey is normally incremented by 1.
(2-A-2) Second or more copy to maidirstore(Local Folders)
    storeToken of source mail is used for file name(==nnnnnnnn).
    messageKey is normally incremented by 1.
    FILE of "nnnnnnnn-N"(N=1, 2, 3, ...) is created under "cur".
    However, file size was ZERO.

(2-B) Source Mbox = berkeleystore(IMAP Offline-Use=Off) :
      IMAP Offline-Use=Off folder, newly fetched mail
      Offline flag = Off,
      storeToken is not set, because newly fetched mail 
    messageKey is normally incremented by 1.
    File name is generated by Tb.
    Because storeToken of source mail is not set,
    new FILE of "nnnnnnnn" is created under "cur" by each copy,
    and mail data is normally copied.

    Note:
    If new file name already used, problem may occur.
    For example,
      new file name is based on timestamp.
      multiple mails is copied at same time.
      generated name is already used.
      fails to add suffix of "-N" in this case.

(2-C) Source Mbox =berkeleystore(IMAP Offline-Use=Off) :
      IMAP Offline-Use=Off folder, copied mail from other IMAP folder
      Offline flag = Off,
      storeToken=nnnnnnnn is set because of copied mail
(2-C-1) First copy to maidirstore(Local Folders)
    messageKey is normally incremented by 1.
    storeToken of source mail is used for file name(==nnnnnnnn).
    DIRECTORY of "nnnnnnnn" is created under "cur".
(2-C-2) Second or more copy to maidirstore(Local Folders)
    messageKey is normally incremented by 1.
    storeToken of source mail is used for file name(==nnnnnnnn).
    FILE of "nnnnnnnn-N"(N=1, 2, 3, ...) is created under "cur".
    However, file size was ZERO.

(2-D) Source Mbox = berkeleystore(POP3) :
      Offline flag = Off,
      storeToken=nnnnnnnn(==offset) because of local folder
(2-D-1) First copy to maidirstore(Local Folders)
    messageKey is normally incremented by 1.
    Nothing is created under "cur".
(2-D-2) Second copy to maidirstore(Local Folders)
    messageKey is normally incremented by 1.
    DIRECTORY of "nnnnnnnn" is created under "cur".
(2-D-3) Third or more copy to maidirstore(Local Folders)
    messageKey is normally incremented by 1.
    FILE of "nnnnnnnn-N"(N=1, 2, 3, ...) is created under "cur".
    However, file size was always ZERO.

In any case, if Unix Mbox is manually copied to "file of storeToken" under cur directory, mail is normally shown by Tb 17 & Tb 22.

(3) Copy mail from Mbox of maidirstore(IMAP)
    to Mbox of non-IMAP maidirstore (type=none, Local Folders).
    Source Mbox of maidirstore :
      IMAP Offline-Use=On folder
      Offline flag = On, storeToken is normally set (call nnnnnnnn)
      statusOffset=0 because of IMAP folder.
      Copy to maidirstore(Local Folders)
    storeToken of source mail is used for file name.
    mail data is normally copied to nnnnnnnn under "cur".
    messageKey is normally incremented by 1.
    statusOffset=0 is kept as 0.
    However, messageSize of source mail is copied and used
    even though "From - ...", X-Mozilla-Status:/-Status2:, is added.
    So, "cut at messageSize" occurs in mail view and message source,
    even though entire mail data is copied to file of nnnnnnnn.
    When same mail is copied again, file name of nnnnnnnn-1 is used.

msgDBHdr data in .msf is same as one in copy source folder except messageKey.
  messageKey starts from 1, incremented by 1 by each mail copy.
  same messagOffset, same offlineMessageSize, same statusOffset,
  same storeToken if storeToken is set , ...
It looks for me that "all accounts uses maidirstore" is assumed.
- If copy source is maidirstore, storeToken points file as expected,
  then file for mail data can be found and is copied.
- If copy source is not maidirstore, storeToken is messageOffset in Berkley
  Mbox file at copy source, and Tb can't find file for mail data,
  then Tb somehow creates "directory of storeToken" instead of
  "file of storeToken".

[Workaround]

Manual Export/Import of mail data, with forcing "Repair Folder".
(1) Split an existent Berkley Mbox file(Unix Mbox file of multiple mails in a file) to multiple "unix Mbox file of single mail in a file".
(2) Delete excess directory under cur.
(3) Copy all "unix Mbox file of single mail in a file" to Inbox/cur directory. "File name of number" may be needed.
(4) Delete Inbox.msf.
(5) Restart Tb.
Easy way to split.
By Import Export Tools, save all mails in your Berkley or movemail account to .eml files in a directory, append "From -[CRLF]" at top of each file, rename to "number only name".

Comment 1

[WADA:World Anti-bad-Duping Agency]

FYI.
I've opened Bug 855954 for problem of step (3) in comment #0.

Comment 2

[Kent James (:rkent)]

I believe that the changes that I did to nsMsgMaildirStore::GetMsgInputStream in bug 1017939 addressed the cause of this.