Closed Bug 852080 Opened 12 years ago Closed 8 years ago

MailDir not working for Local Folders when copying from MBox

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86_64
Windows 7
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: irne.barnard, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: dataloss)

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0 Build ID: 20130307023931 Steps to reproduce: I changed the default data store (i.e. mail.serverDefaultStoreContractID) from the normal mbox version (@mozilla.org/msgstore/berkeleystore;1) to the "new" MailDir type (@mozilla.org/msgstore/maildirstore;1). Then I recreated a normal IMAP account. It seems to work fine downloading / sending. Even copying to / copying from a mbox account seems to work. Then I tried creating a 2nd local folders account. It created fine, allowed me to make new folders at will. I can even copy into it from the other MailDir account. But then I tried to copy one or more message(s) from a mbox account into one of those folders. Actual results: The copy process seems to complete and the subjects are listed in the browser. But when opening the message it is blank. Looking at the actual files in the profile shows them as 0 bytes. Expected results: The body of the message needs to be stored in the file after such copy. And it needs to be visible in the editor / viewer.
Keywords: dataloss
See Also: → 462665
See Also: → 793524
See Also: → 771643
See Also: → 58308
Have you tried to use maildir-lite on the first Local Folders account?
Let me be more clear about what I tried to do: 1) I opened the about:config editor. 2) Searched for defaultstore 3) Changed mail.serverDefaultStoreContractID from the normal "@mozilla.org/msgstore/berkeleystore;1" to the new (since v12) "@mozilla.org/msgstore/maildirstore;1" 4) Created a new account linked to an IMAP server with about 10MB of messages in the Inbox. 5) Subscribed & synced ... all messages seemed to work fine. 6) Tested copying one of those messages (and then multiple msgs) to the existing berkleystore Local Mail account. Worked fine. 7) Tested copying from the Local to the new IMAP (both single & multiple at once). Seemed to work ... but that might be due to it being an IMAP account. 8) Installed the LocalFolder addon, so I could create a 2nd LocalFolder account. 9) Tried copying from the MailDir IMAP account into the 2nd LocalFolder. This worked fine. 10) Tried copying back from the MD-Local to the MD-IMAP, this worked fine. Again, don't know if this is due to IMAP or MD-to-MD. 11) Tried copying from the MBox-Local to the MD-Local. Message header(s) appear correct, but body contains nothing. Empty folders are created in the FS and blank (0 byte sized) files, but no actual data is stored. 12) Tried copying some of the msgs from MD-Local to MBox-Local (msgs which worked in 6 above). Worked fine. So afaict the MD backend works if moving/copying from the MD to the MBox, but not the other way round. Unless the MD is an IMAP account, in which case it seems that the copy actually sends to the server and then the sync simulates the copy. It doesn't work from MBox to MD without some server intervention. Haven't tried through a POP3 account, but it would not be dissimilar from the LocalFolder version - would it? Though it might be something to do with the extension then - will try with a POP server to see if that would work. What are you referring to about he maildir-lite on 1st LocalFolder? Do you mean I should convert it somehow from MBox to MD? No I didn't do that, I created a 2nd LocalFolder through the extension.
Yes, if it is possible to set mail.serverDefaultStoreContractID to maildir before the Local Folders account is created in a new profile. You could probably change the default settings in TB install files. Look for the file all-thunderbird.js (may be zipped in a file omni.jar) and add the pref there with the maildir value.
Then create a new profile with this TB installation.
What is problem of this bug? Following per server setting is not usable? > mail.server.serverX.storeContractID;@mozilla.org/msgstore/maildirstore;1 > mail.server.serverX.storeContractID;@mozilla.org/msgstore/berkeleystore;1 If so, such "per server setting" dosn't function in your environment? Or such "per server setting" is useles for your purpose? What is reason why you use add on for second Local Folders long the way for tesing of Maildir in Tb? Or you are trying to migrate current "standard Local Folders account of berkleystore" to "Local Folders account of maildirstore" utilizig addon? Problem of this bug is that second Local Folders account created by addon doesn't work well when Maildir is used even though standard Local Folders account works well with Maildir?
No longer blocks: 856058
> Bug ummary : MailDir not working for Local Folders What do you call by "Local Folders"? - standard "Local Folders" with standard berklaystore - standard "Local Folders" with enhanced maildirstore - second "Local Folders" created by addon, with enhanced maildirstore What kind of phenomenon with what kind folder do you call by "not working"? - POP3 folder, berklaystore - POP3 folder, maildirstore - IMAP folder, Offline-Use=Off, berklaystore - IMAP folder, Offline-Use=On, berklaystore - IMAP folder, Offline-Use=Off, maildirstore - IMAP folder, Offline-Use=On, maildirstore - Other(News, RSS)
To irneb(bug opener): Answer to my questions, please. Please note that here is bugzilla.mozilla.org to resolve actual bug of Tb==flaw in code of Tb. Please note that here is never support forum nor Customer Help Center.
Flags: needinfo?(irne.barnard)
Sorry for the delay. (In reply to WADA from comment #6) > What do you call by "Local Folders"? My original Local folder is the normal Berkley Store one as per a default (clean) install of TB without changing any settings. The IMAP I created after changing the default storage to use the MailDir-Lite. The 2nd Local Folders was created to test the interoperation between MBox (Berkley) and MailDir-Lite without using IMAP in the mix. The reason I did this is to check if the apparent working between a MBox Local and a MailDir-Lite IMAP was due to the IMAP. What I noticed was copying a msg off the MD-IMAP into the MBox-Local is near immediate, but the other way round (MB-Local to MD-IMAP) took longer and required a sync to complete. So it seemed that what was actually happening is the copy sent the msgs to the IMAP server, then the sync introduced it into the MailDir-Lite store. So rather than create a 2nd account (something like a POP account) I installed the addon so I could create a 2nd Local Folder (but using MailDir-Lite). Thus enabling a test using only the datastore, not any interference from a server. > What kind of phenomenon with what kind folder do you call by "not working"? What I've found not to work is: Copying from a Berkley Local Mail into a MailBox-Like Local Mail. I'm in the process of using a 2nd profile to test this phenomenon without the 2nd Local Mail folder - using a dummy POP account instead. Perhaps the naming should be changed to reflect it more clearly - I used the Local Folder in the name to distinguish it from any server-side interoperation. (In reply to WADA from comment #7) > Please note that here is never support forum nor Customer Help Center. I know that. I was testing MailDir since I would like to rather use it than MBox (mainly due to ease of incremental backup). But before I go ahead and use it directly, I wanted to test if I might be losing data. And when I found that such is actually occurring, I thought it might help to point out my findings.
Flags: needinfo?(irne.barnard)
Summary: MailDir not working for Local Folders → MailDir not working for Local Folders when copying from MBox
(In reply to irneb from comment #8) > without the 2nd Local Mail folder - using a dummy POP account instead. I think it's better, because POP3(type=pop3) and Local Folders(type=none) is same from perspective of "Mail folder", and multiple non-hidden type=none account is never officially supported. There are many combinations in copy/move, as I wrote. (i) First, please describe "which msgstore". As I wrote before, "which msgstore" is known by next setting. > mail.server.serverX.storeContractID;@mozilla.org/msgstore/maildirstore;1 > mail.server.serverX.storeContractID;@mozilla.org/msgstore/berkeleystore;1 So, there is no need to explain "How you created account". (ii) Second, account type(server type). Because Local Folders/POP3 and IMAP is different schema, folder access mechanism is different. (iii) If IMAP, Offline-Use=On or Off. If Offline-Use=Off, offline-store file is not used. So, Mbox\cur directory is not used and Mbox/cur/nnnnnnnn file is never used. If Offline-Use=On, offline-store file is used. So, Mbox\cur directory is used and Mbox/cur/nnnnnnnn file is used. From perspective of offline-store file, it's similar to mail folder file of POP3 and Local Folders. However, because IMAP, communication with IMAP server is always relevant. I reported some bugs for phenomena I saw while checking some combinations. Read bugs listed in dependency tree for meta bug 845952 which is set in Blocks: field of this bug, please. If I understand correctly your report, I think I covered your case in comment #0.
Severity: normal → critical
Component: Untriaged → Database
Product: Thunderbird → MailNews Core
Confirm this problem
Tried it again. This time with a dummy POP account. Using TB 31.1.2 on Windows 7 Pro 64bit. I already had 2 other accounts in this profile, one a POP the other an IMAP as well as a Local. All 3 were created as berkleystore. Let's call them POP-B, IMAP-B and Loc-B. Then I changed the default to use the MailDir Lite instead. Created a new POP account through manually configuring some arb server addresses. Let's call this POP-MD. Next I try to copy a single message from IMAP-B into POP-MD. The display updates, and selecting the new copied message shows the contents. So that works. Next select 3 messages in the IMAP-B account and copy to the POP-MD. The POP-MD shows them in the mail table. Clicking on them to view their contents only works for the first, the others don't appear to have any content copied. Open a file browser to see the actual files. The cur subfolder contains only an empty folder for those messages without content. Only those displaying content in TB have a file inside cur with some size to it. Also opened these files in a text editor - content seems to be correct. This seems as if it's now worse than it was last year. I.e. IMAP-B to POP-MD only works for single messages in one copy, multiple messages in one copy instruction update the table correctly but only the file for the first. Also doing a repair folder deletes these blank messages. Next tried the same from POP-B to POP-MD. Same trouble as I've described before. Copying one single message only updates the table - no content. Copying multiples also only updates the table, no content is copied. Checked the file system again - only empty folders inside the cur folder (no files whatsoever). Tried the same from Loc-B. Exactly the same issue as that from POP-B. Then tried to copy from the POP-MD: First to the IMAP-B, works perfectly both for single and multiple messages. The table is updated properly and the content is there as well - although after sync with the IMAP server. Then to POP-B, only copies singles, even if multiples are selected only the first is copied (even shows as only the first in the table) - those that are copied do have their content so that works now (previously it didn't). The same goes for Loc-B as for POP-B.
(In reply to miken from comment #10) > Confirm this problem What symptom in this bug do you confirm?
(In reply to irneb from comment #11) As seen in Dependency tree for Meta Bug 859011, which is put in Blocks: field of this bug, bugs for specific case is already opened on "move/copy to MailDir account's folder of server type=pop3 or none(Local Folders)", for various "move/copy source folder" type(BerkleyStore or MailDir, pop3/none or imap, if imap, Offline-Store=On or Off). From perspective of "move copy/target folder", Folder of Local Folders is identical to Folder of POP3 aaccount. irneb(bug opener), what is your new findings in this your bug report? Which case is problem what is not reported yet by other bug?
(In reply to WADA from comment #12) > > Confirm this problem > What symptom in this bug do you confirm? I have two accounts in TB: one has mbox storage and another MailDir storage. When I select and copy multiple messages from mbox folder to MailDir folder, the copy process seems to complete and the subjects are listed in the browser. But when I opening any message it is blank. Looking at the actual files in the profile shows them as 0 bytes. If I then make Repair folder operation, I get EMPTY MailDir folder. Another example: I copy multiple messages in .eml format to MailDir folder (using drag-and-drop from WinExplorer). Again copy process seems to complete and all the subjects are listed in the browser. But when I look at MailDir folder in filesystem, I see less than half files (in my case 809 vs 1690). If I then make Repair folder operation, I get 809 messages in MailDir folder instead of 1690 copied.
(In reply to miken from comment #14) > (In reply to WADA from comment #12) > > > Confirm this problem > > What symptom in this bug do you confirm? > I have two accounts in TB: one has mbox storage and another MailDir storage. > When I select and copy multiple messages from mbox folder to MailDir folder, > the copy process seems to complete and the subjects are listed in the browser. > But when I opening any message it is blank. Looking at the actual files in the profile shows them as 0 bytes. > If I then make Repair folder operation, I get EMPTY MailDir folder. > Another example: (snip) >(snip) Question again to you. "What/Which symptom in this bug that was reported by bug opener" do you confirm? By what operatios with what setups, by what phenomenon which you saw, do you confirm what problem reported to this bug? What/Which is your actual "this problem" in this bug report which is referred by your statement of "Confirm this problem"?
(In reply to irneb from comment #11) I attached attachment 8500288 [details] to meta bug 859011 comment #2. irneb, which case in table of section (A) did you cover? Note: If copy/move between IMAP folders, "same server" or "different server" should be cared. If "move by message filter", "Filter before Junk Classification" or "Filter after Junk Classification" is pretty important.
(In reply to WADA from comment #15) The following simply don't work: POP/NULL Berkley --> 1 msg copy --> POP MailDir (no content) POP/NULL Berkley --> 3 msg copy --> POP MailDir (no content) IMAP Berkley Offline --> 3 msg copy --> POP MailDir (only 1st copied, others no content) IMAP Berkley Online --> 3 msg copy --> POP MailDir (only 1st copied, others no content)
(In reply to irneb from comment #17) > The following simply don't work (snip) (Q1) What is difference of your test cases from bugs lisred in Dependency tree for Bug 859011? IIUC, IIRC, I already checked your test cases in other bugs listed in Dependency tree for Bug 859011. I mainly covered "copy/move to pop3/MaildirStore" cases. I covered "copy/move from MaildirStore" cases pretty partially, because I believe that problems in "copy/move to MaildirStore" should be resolved first, before resolving problems in "copy/move from MaildirStore". Unless "MaildirStore of a mail folder" is correctly created, test of "reading from MaildirStore" cases is meaningless. I didn't cover "message filter move" cases, because I don't believe "message filter" can work well even though "simple copy/move doesn't work". (Q2) What is detail of "don't work" in your case? FolderName/cur/nnnnnnnn is created or not, FolderName/cur/nnnnnnnn is file or directory, content of the nnnnnnnn file, FolderName/tmp/nnnnnnnn is created or not, if move, moved mail is deleted at move source folder or not, if "filter move", "Filter before Junk Classification" or "Filter after junk Classification", and so on. (Q3) What is difference of your test results from bugs lisred in Dependency tree for Bug 859011?
(In reply to WADA from comment #18) (A1) I think the closest match is bug 791966 and bug 856519. This might be classed as a duplicate to 856519. (A2) The "don't work" was indicated with the description at the end. E.g. (no content). I've re-done these and in each of these cases where the email message appears in the view table in TB but has no content the following happens in the file structure: A new folder is made under the account's ./cur/ folder, it's empty and there's no file with the same name. Those that do get copied properly get both a folder as well as a file with contents greater than 0kb. (A3) Difference from my test results on similar test from 856519 is that if the source is a BerkleyStore on an IMAP account (either on-line or off for the folder) a single message copy seems to work as expected. A multiple only copies the first message selected, the rest only gets a blank folder in the file structure and the message header is only updated in TB (no content). With any other berkley folder (local or POP) I see the same as listed in 856519.
Suyash, may any of the work you have done on maildir this summer since TB31 have made a difference here?
Flags: needinfo?(syshagarwal)
Ya, this should work now. Just nsCopyService won't work. Otherwise, manual copy should be good [1]. I have a class right now. I'll try it in a few hours. [1] https://wiki.mozilla.org/Thunderbird/Maildir
Flags: needinfo?(syshagarwal)
Ya, this isn't working for me. :/
Well, I think, this should be a part covered under migration and we haven't covered this so far.
Yes, if I recall correctly there were several places in the code where mixed mbox/maildir would cause issues. That is probably the biggest missing area right now in maildir.
This should not be migration as in we forcefully migrate all users' mbox files to maildir. I understand it we will keep support for mbox and it will coexist with maildir for a long time. Maybe new accounts will be created as maildir by default, but old accounts can be kept as mbox. So copying between them must work.
This kind of basic problems in "copy/move from/to MaildirStore" looks already resolved by changes in other bugs. irne(bug opener), can you still reproduce your problem in recent Tb trunk nightly?
(In reply to WADA:World Anti-bad-Duping Agency from comment #26) > This kind of basic problems in "copy/move from/to MaildirStore" looks > already resolved by changes in other bugs. > irne(bug opener), can you still reproduce your problem in recent Tb trunk > nightly? irneb, do you agree?
Flags: needinfo?(irne.barnard)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #27) I've just retested this. It seems to work fine even through the LocalFolder addon and a 2nd MailDir typed Local Folder. Both copying to/from an IMAP (Berkley) account to the 2nd Local Folder (MailDir), as well as copying to/from the 1st Local Folder (Berkley) to the 2nd. It seems to have cleared up for me at least. TB 45.6.0, running from a PortableApps 64GB USB on Win8.1 Pro 64bit. I did notice one hiccup ... after creating the 2nd Local folder I made a new Inbox folder under it. But when copying to it I got an error message about disk space. Though I think it's just an initialization error. Restarting TB showed that the Inbox folder wasn't created, but after that it worked fine. So I'd advise anyone making a new MailDir account / folder set to restart TB before starting to place anything inside it.
Status: UNCONFIRMED → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
Flags: needinfo?(irne.barnard)
You need to log in before you can comment on or make changes to this bug.