793524 - With maildir storage, movemail creates empty directories instead of files in "cur"

Status: RESOLVED DUPLICATE of bug 1017939

(MailNews Core :: Backend, defect)

Description

[Michel Pitermann]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.11 (KHTML, like Gecko) Ubuntu/12.04 Chromium/20.0.1132.47 Chrome/20.0.1132.47 Safari/536.11

Steps to reproduce:

- Installed TB 15.0.1 on Linux Mint 13 from the synaptic manager.
- Ran TB for the first time.
- Asked to open a new account later.
- Replaced "@mozilla.org/msgstore/berkeleystore;1" by "@mozilla.org/msgstore/maildirstore;1" in variable "mail.serverDefaultStoreContractID" from "Edit/Preferences/Advanced/General/Config Editor".
- "File/New/Other Accounts...".  Chose "Unix Mailspool (Movemail)" then Next. Filled my "Your name:" and "Email address:" fields with real values then Next. Filled the "Outgoing server" with a real value and left "Outgoing User Name:" blank then Next. Filled "Account Name" with "localhost" then Next, then Finish.
- Reading email with "localhost" (the only account newly created) at this moment crashed TB.
- In a terminal:
  $ cd ~/.thunderbird/bnt7endi.default/Mail/localhost
  $ rm Drafts Inbox Sent Templates Trash Unsent\ Messages
  $ mkdir Drafts Inbox Sent Templates Trash Unsent\ Messages
  $ thunderbird &
- "Get Mail" using "localhost" account.


Actual results:

- Two emails were read with correct headers but their bodies were empty.
- In a terminal, "ls ~/.thunderbird/bnt7endi.default/Mail/localhost/Inbox/cur" showed two empties directories named "1348431617912395" and "1348431617963137".


Expected results:

- Two emails should have been read with a body each.
- In a terminal, "ls ~/.thunderbird/bnt7endi.default/Mail/localhost/Inbox/cur" should show two files named "1348431617912395" and "1348431617963137" that should contain the two full emails.

In other words, movemail correctly extracted the email headers to show them in the TB windows but it created empty directories instead of email files in Inbox/cur when maildir storage was used.  I only saw this bug when a movemail account was created with maildir storage, i.e., everything worked well with movemail accounts with mbox storage and with POP3 and IMAP accounts irrespective of the storage format used.

Comment 1

[stef]

Happens to me too. Cannot describe exactly the situation, I will collect some repeatable experiences and post them here. Usually happens when I copy several messages at once from an IMAP account to a local folder. Some messages are copied ok, and others produce a correct entry in the folder listing, but when you open the preview pane, the body of the message is missing. In the cur directory I find an empty directory instead of a file (and the name of the directory is the name of the file in the original folder). I am working on KDE, TB 17.0 Beta (I had the issue on 16 too, and moved to 17 beta hoping to find it solved), maildir.

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to stefmorp from comment #1)
> ... I will collect some repeatable experiences and post them here. 

can you also get a stacktrace?
https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report

Comment 3

[stef]

(In reply to Wayne Mery (:wsmwk) from comment #2)
> (In reply to stefmorp from comment #1)
> > ... I will collect some repeatable experiences and post them here. 
> 
> can you also get a stacktrace?
> https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report

Unfortunately not. There is no crash, so I am unable to provide a stacktrace. I have some more evidence of what happens, although not a repeatable pattern. I have observed that with maildir activated when I move messages around:
1) I can't move them from a maildir to a berkeleystore - it simply refuses to move the message
2) If I move a single message from maildir to maildir it works. However, if Thunderbird does not shut down "clean", when I reopen TB the message "seems" to still be in the source directory. If I click on it, nothing shows up in the message pane. If I check the content of the directory, it contains an empty directory with the same name as the file that was moved. In the destination directory everything is clean
3) If I move many messages, sometimes a message or two are not copied to the destination correctly. They show in the listing, but if I click on them, nothing shows up in the message pane. If I check the content of the directory, it contains empty directories instead.

Hope this helps.

Comment 4

[WADA:World Anti-bad-Duping Agency]

maildir only problem?

prefs.js setting
- Default (not touched)
> mail.serverDefaultStoreContractID=@mozilla.org/msgstore/berkeleystore;1
- All accounts except server5 8not touched)
  (type=none Local Folders/smart mailboxes, type=imap, type=pop3)
> mail.server.server6.storeContractID;@mozilla.org/msgstore/berkeleystore;1
- server5, type=pop3, changed to maildir,
  all dir/files is deleted from Mail directory, restarted
> mail.server.server5.storeContractID=@mozilla.org/msgstore/maildirstore;1

By mail copy from Local Folders, POP3 account, IMAP Offline-Use=On folder, IMAP Offline-Use=On folder, empty directory of "name is number" is created under \Inbox\cur directory of server5 account.

msgDBHdr data in .msf is same as copy source except messageKey.
  messageKey starts from 1, incremented by 1 by each mail copy
  same messagOffset, same offlineMessageSize, same statusOffset,
  same storeToken, ...

It looks for me that "all accounts uses maildir" is assumed.
- If copy source is maildir, storeToken points file as expected,
  then file for mail data can be found and is copied.
- If copy source is not maildir, storeToken is messageOffset in Berkley
  Mbox file at copy source, and Tb can't find file for mail data,
  then Tb somehow creates "directory of storeToken" istead of "file of
  storeToken".

Above is seen in both Tb 17.0.2 and Tb 22.0a1(nightly).
When an Berkley Mbox file is manually copied to "file of storeToken" under cur directory, mail is normaly shown by Tb 17 & Tb 22.

Workaround:
Manual Export/Import of mail data, with forcing "Repair Folder".
(1) Split an existent Berkley Mbox file(Unix Mbox file of multiple mails in a file) to "unix Mbox file of single mail in a file".
(2) Delete excess directory under cur.
(3) Copy all "unix Mbox file of single mail in a file" to Inbox/cur directory. "File name of number" may be needed.
(4) Delete Inbox.msf.
(5) Restart Tb.

Easy way to split.
By Import Export Tools, save all mails in your Berkley or movemail account to .eml files in a directory, append "From -[CRLF]" at top of each file, rename to "number only name".

Comment 5

[WADA:World Anti-bad-Duping Agency]

"Confirmed" is for phenomenon specified in bug summary only.
Not for "crash, stackwanted" in Keywords: field and "Severity: critical".

Comment 6

[WADA:World Anti-bad-Duping Agency]

Sorry typo.
  movemail only problem?

Comment 7

[WADA:World Anti-bad-Duping Agency]

There are several kind of source/target of mail copy/move.
  (a) Berkley POP3(and Local Folders), (b) Berkley IMAP Offline-Use=On,
  (c) Berkley IMAP Offline-Use=Off, (d) Berkley movemail,
  (e) Maildir POP3(and Local Folders), (f) Maildir IMAP Offline-Use=On,
  (c) Maildir IMAP Offline-Use=Off, (d) Maildir movemail
Which case is bad in your case?

I guess "copy from any of Berkley to any of Maildir" doesn't work and empty directory is generated under cur of Maildir, and copy from any of Maildir to any of both Maildir and Berkley works well, but I'm not sure.
"Move mail case may be slightly different, because "delete at source" is involved.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to stefmorp from comment #3)
> 3) If I move many messages, sometimes a message or two are not copied to the destination correctly. 
> They show in the listing, but if I click on them, nothing shows up in the message pane.
> If I check the content of the directory, it contains empty directories instead.

Move(copy to target then delete at source) from Maildir to Mildir?
If so, file name is suffixed?

When file name(==storeToken) is already used by oter mail or duplicate copy of same mail, sufix of "-N" which starts from 1 is added to file name. When file name of copied mail has already has suffix, and if the suffixed name already exits, copy of file with new suffix looks to fail.
- abc is used for mail-x of MaildirX.
- mail-x in MaildirX is copied to MaildirY and MaildirZ multiple times.
  abc, abc-1, abc-2, ..., abc-N, ... is created in MaildirY/cur.
  abc, abc-1, abc-2, ..., abc-N, ... is created in MaildirZ/cur.
- If file abc-N for mail-y-n in MaildirY is copied to MaildirZ,
  abc-N is used for mail-z-n in MaildirZ.
  In such situation, Tb looks to fail to generate file with new suffix.
  In such case, message is not shown because pointed ile doesn't exist.

Suffixed file name is used when your problem occured?
Did garbage of directry "abc" or "abc-N" by this bug exist in cur directory?

Comment 9

[WADA:World Anti-bad-Duping Agency]

FYI. 
Bug 753624 was found for pheomenon of "empty directories instead of files in cur, by move/copy between Maildie folders".

Comment 10

[WADA:World Anti-bad-Duping Agency]

Fllowing bugs looks similar phenomenon of "blank directory with name of file for mail in Maildir under cur" after your bug open. Some of them seems phenomenon when "copy/move from non-Mildir folder to Maildir folder", but some others seems phenomenon when "copy/move from Maildir folder to Maildir folder".
> bug 753624 : Copy/Move of imap offline store messages not working for Maildir Lite
> bug 791966 : Move of mails from IMAP mbox folder to local maildir folder results in lost of the mail
> bug 793524 : With maildir storage, movemail creates empty directories instead of files in "cur"
> bug 816304 : Filters that copy messages crash TB when using maildir
> bug 827048 : Filtered messages lost (POP) going to imap or maildir folder
> bug 852080 : MailDir not working for Local Folders

It may be next;
(1) Tb has a hole when something wrong happens in creation of mail data
    file step, and generates directory under "cur"(or "tmp").
(2) If copy/move from non-maidirstore(berkeleystore) to maidirstore,
    problem of (1) always occurs.
(3) If copy/move from maidirstore to maidirstore,
    problem of (1) occurs only when special condition.

Because problem looks to occur always if "copy/move from non-maidirstore(berkeleystore) to maidirstore", analysis of (2) is perhaps easier than analysis of (3).
I'll open separate bug for (2).

And, because this bug has "crash, stackwanted" in Keywords: field and "Severity: critical", change bug summary of this bug back to original, in order to avoid misleading and to keep "movemail" in bug summary.
Sorry for my cluttering and spam.

Comment 11

[WADA:World Anti-bad-Duping Agency]

FYI.
I've opened Bug 855950 for "empty directory instead of file in cur".

Comment 12

[WADA:World Anti-bad-Duping Agency]

This bug is for which problem?
(a) Crash with movemail and with MaildirStore?
(b) Empty Inbox/cur/xxxxxxxx directory with MaildirStore && maildir in following comment?
> but it created empty directories instead of email files in Inbox/cur when maildir storage was used.

Keep "one problem per one bug", please.
As far as set of "crash, stackwanted, criticall" is places in fields of this bug, this bug is never other than (a).

For "Empty Inbox/cur/xxxxxxxx" part.

To Michel Pitermann(bug opener):

As I wrote in bug 856387 comment #3, there are some known "Empty Mbox/cur/nnnn directory" cases.
What is difference of your problem with movemail from known cases written in that bug?
What is movemail specific symptom or problem? 

(In reply to stefmorp from comment #1)
> Happens to me too.

Is your problem actually same case as bug opener's case with movemail?
Do you see same problem as yours in dependency tree for Meta bug 845952?
If No, what is difference of your case from bugs for similar condition/similar symptom?

Comment 13

[Michel Pitermann]

I was requested a crash stack.  I have just tried to reproduce the bug following the procedure written in the bug report but I could not find a way to create a Movemail account in TB 17.0.5 that I presently have in my computer.  How can I create a Movemail account?

Comment 14

[Michel Pitermann]

@WADA.  I cannot answer your questions asked in your comments #12 because I cannot create a Movemail account.  I will try when I know how to do it.  Best regards.

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to Michel Pitermann from comment #14)
> because I cannot create a Movemail account.

You are the "Michel Pitermann" who is bug opener of this bug, who is comment poster of comment #0 in which you said "you created Movemail account and tested with MaildirStore", aren't you?
Or comment #0 is copy of problem report sent to you from someone?

Comment 16

[Michel Pitermann]

Yes, I am the bug opener.  The story is that I wanted to switch from Evolution to Thunderbird.  I could not read even one mail after creating a Movemail account in TB (I need this feature because I often change place and I do not want to depend on Internet connections to consult my previous emails).  I spent more than a full day trying to figure out how to create a Movemail account in Maildir format, to struggle with the bug, and to conclude after asking confirmation in a Thunderbird forum that I was facing a real bug.  I opened this bug report.  At that time, Thunderbird 15 was the current version.

Of course, I could not work with TB, so I sticked to Evolution.  This morning, I came back here, I wanted to help providing a crash stack, but I did not figure out how to create a movemail account in TB 17.  I even wonder whether it is an abandoned feature.  As a consequence, I cannot provide the requested crash stack.

Since I have never used TB except the couple of days I was trying to open a movemail account in maildir format, I cannot answer your questions regarding the similarity between the bug of this report and the copy problem.

So, do you know how to create a movemail account in TB 17?

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to Michel Pitermann from comment #16)
> So, do you know how to create a movemail account in TB 17?

IIRC, Linux version of Tb has "movemail" in selection list of protocol of account definition. In account creation(File/New/Existing Account), Manual Setup, change IMAP/POP3 part to movemail, is possible?
Before test with MaildirStorage, check with standard BerkleyStore first to confirm your movemail account setup is correct.

As for account definition in Tb, define as POP3 account with dummy server such as localhost, change mail.server.server1X.type = pop3 -> movemail, was manual procedure to create movemail account before UI support was implemented. If no UI for movemail, try it after Google search for "Thunderbird movemail", please.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Do you use localized Thunderbird? If yes, bug 787605 may be relevant.

Comment 19

[WADA:World Anti-bad-Duping Agency]

FYI.
"moveail" may be in "Other Accounts..."
> http://ubuntuforums.org/showthread.php?t=1718795
See also document for moveail support in Thunderbird.
> http://www-archive.mozilla.org/mailnews/movemail/index.html

Comment 20

[Michel Pitermann]

OK, I found out why I could not create a movemail account.  The "File/New/Other Accounts..." is not included in the TB menu pulled down by clicking on the upper-right corner of the window.  I did not understand that it was possible to restore the full menu bar using the "Preferences" menu.  It also took me time to figure out that the file read by TB in a movemail account was provided by the MAIL environment variable.  The TB documentation is not helpful.

I removed my "~/.thunderbird" directory, I replicated the process described in comment #0, and everything worked well.  I suppose that this bug was fixed between TB 15 and TB 17.  I am going to check with another Linux version in a couple of days.  I will keep you informed.

Comment 21

[Michel Pitermann]

(In reply to WADA from comment #18)
> Do you use localized Thunderbird? If yes, bug 787605 may be relevant.

No, I use a Linux LMDE 201303 or Linux Mint 13 American version with a French keyboard in France.  Same for TB.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(Off-opic)

(In reply to Michel Pitermann from comment #20)
> The "File/New/Other Accounts..." is not included in the TB menu
> pulled down by clicking on the upper-right corner of the window.
> I did not understand that it was possible to restore the full menu bar using the "Preferences" menu.

You looks a victim of new App-Menu with hideng traditional full Manu Bar by default, what can be called "a change for the worse" for some peoples like me. However, button which is placed under account list box of Account Settings, which is labeled "Account Actions" currently though, still exists, and drop down menu of the botton has still full menu of account creation. This is unchanged, even though label change, structure change etc. were made in it.

> It also took me time to figure out that the file read by TB in a movemail account was provided by the MAIL environment variable.
> The TB documentation is not helpful.

I agree with you.
Even following document, which I could frequently reach by Google Search for "movemail thunderbird", I can't reach quickly, because this document was moved to www-archive.mozilla.org from www.mozilla.org after "movemail support in UI of Account Wizard in Linux".  
> http://www-archive.mozilla.org/mailnews/movemail/index.html
If we can know easily that "change type from pop3 to movemail" is sufficient, I believe that we can easily guess that spool file is obtained from mail related setting in Linux.
I guess other movemail related documents are also archived after support of movemail in account defnition UI.

Comment 23

[Kent James (:rkent)]

I believe that the changes to nsMsgMaildirStore::GetMsgInputStream in bug 1017939 solved this issue. If not, feel free to reopen. I have not tested with movemail specifically.