Open Bug 845952 (maildirblockers) Opened 9 years ago Updated 9 days ago
[meta] finish "maildir" message storage
This meta bug is to organize the work to finish "maildir" message storage. Blocking are bugs from  and from bug 58308 (which is qmail and won't block this) I include fixed bug 402392 and bug 738651. The end goal is to enable maildir as default, but the meta bug could certainly live on to finish straggling issues after maildir enabled by default. To do list: - Someone to file a bug for "enable maildir as default message store" - check to see if I have missed anything in blockers - remove anything in blockers that doesn't exactly belong  https://bugzilla.mozilla.org/buglist.cgi?list_id=5785777;field0-0-0=short_desc;type0-0-1=substring;field0-0-1=status_whiteboard;resolution=---;query_format=advanced;value0-0-1=maildir;type0-0-0=anywordssubstr;value0-0-0=maildir%20mail-dir;product=MailNews%20Core;product=Thunderbird
Add to the to-do another task: - Tool to optimize mail storage (convert mbox to maildir in existing installations)
"maildir" in bug summary of Bug 534135 is pure "Maildir" used by IMAP server instead of Tb's maildir-like, Maildir Lite, .... Removing from depndency.
No longer depends on: 534135
One day (after the >4GB mbox work) I'd like to try out the maildir-lite support and see if I can polish some of the bugs reported here.
FYI. Quick summary of observed phenomena in some basic/simple test with recent trunk nightly(8Tb 22.0a0). Current maildir store looks; - RETR to LocalMbox/tmp/nnnn, move to LocalMbox/cur/nnnn, works. - fetch body to IMAPMbox/tmp/nnnn, move to IMAPMbox/cur/nnnn, works. - Copy/Move mail(s) is always same as Copy/Move from LocalSource/cur/nnnn to LocalTargetMbox/cur/nnnn. - No care for Copy/Move Source/Target folder == berkleystore No Mbox/cur/nnnn, so misbehave. - No care for Copy/Move Source/Target folder == maildirstore/IMAP - Offline-Use=Off : No offline-store file(Mbox/cur/nnnn) => misbehave - Offline-Use=On : Copy/Move from/to IMAP_maildir is; same Copy/Move as Copy/Move from/to Local_maildir + re-synchronizasion with server because of IMAP, if Move. So, if move from IMAP folder, IMAP_maildir/cur/nnnn is simply moved to maildir_Target_folder/cur/nnnn as done in move from maildir_local_mbox to maildir_local_mbox. then, when IMAP Mbox open after move, mail is fetched again. - It looks that return code, status etc. is not checked many places. So, empty directory of Mbox/cur/nnnn, Mbox/cur/nnnn of file size=0 is easily created in many many cases. - Unique filename generation has holes, so same cur/nnnn file can be used by multiple mails. - If directory for Mbox is suffixed, Mbox name is case sensitive in IMAP server. File name in client file syatem is case insensitive. bc/cur for abc, Abc-1/cur for Abc, ABC-2/cur for ABC, are used. In this case, association between Mbox name and Directory name is easily broken by rename of folder, by unsubscribe/subscribe(due to known bug). Funny phenomena was also observed. - If auto-sync of IMAP account is disabled, <server_name>.sbd is created, and <server_name>.sbd\INBOX, INBOX.msf is created and used, and <server_name>.sbd\INBOX/cur is created and used. Directory/file for sub folders under INBOX is created in <server_name>.sbd\INBOX.sbd. Directory/file for all other folder is normally created under <server_name> directory(and <server_name>.msf is created, as usual). This may be caused by /INBOX/Inbox folder what is intensionally created for bug testing. .
Component: General → Database
Product: Thunderbird → MailNews Core
Hardware: x86 → All
Version: unspecified → Trunk
I've opened meta bug 859011 for many currently known problems around "Copy/Move mails with MaildirStore". So moving such bugs from this bug's dependency tree to that bug, to keep this bug as root meta bug for "followups after MaildirStore implementation".
Depends on: 1143241
Is it possible to have the maildir files have the same filename [or at least the date/time part], sans the flags as they do on the server?
Deleting a folder with Thunderbird will delete the .msf file, but not the actual folder that messages are stored in.
My inbox folder has about 156k emails in it, but the corresponding file system folder has 200k files. How can this be reconciled without downloading all the emails again? It's taken close to 40 hours to download my whole mail store so far and it isn't quite finished yet. The sluggishness and lock ups during this time are painful too. I've had crashes and had to restart Thunderbird, this may be part of the problem. I haven't deleted 40k emails either (from the inbox). The inbox is only one of my folders, some other folders have huge numbers too. All up I should have about 2M emails (not quite), which includes mailing list data amongst other emails going back over 10 years. It would be much better to be able to rsync the Maildir folder and have a process adjust file names and remove tag information from the file name to the lines at the top of each email. Then build the .msf files from the actual content ... so, basically an offline build.
Did you guys even test this? I've initiated a repair on my Inbox and now I have 16K extra files in the OS with what looks like another 120K to go. Total emails in the Inbox (reported by TB) is identical to the server's OS directory, that is around 156K; right now the client's OS directory has 217K of email files. Of course the problems are amplified with large mail storage, but even small test folders exhibit problems.
Andrew, AFAIK no one using maildir has experienced this problem. Please file a separate bug report describing your issue https://bugzilla.mozilla.org/enter_bug.cgi?product=Thunderbird and answering the questions posed here and make it block this bug - because this bug is a meta for overall tracking, not for fixing specific bugs. Thanks (In reply to Andrew McGlashan from comment #8) > My inbox folder has about 156k emails in it, but the corresponding file > system folder has 200k files. How can this be reconciled without > downloading all the emails again? It's taken close to 40 hours to download > my whole mail store so far and it isn't quite finished yet. The > sluggishness and lock ups during this time are painful too. > I've had crashes and had to restart Thunderbird, this may be part of the > problem. I haven't deleted 40k emails either (from the inbox). Please post your crash IDs in the new bug report https://support.mozilla.org/en-US/kb/mozilla-crash-reporter#w_viewing-crash-reports > The inbox is only one of my folders, some other folders have huge numbers > too. All up I should have about 2M emails (not quite), which includes > mailing list data amongst other emails going back over 10 years. > > It would be much better to be able to rsync the Maildir folder and have a > process adjust file names and remove tag information from the file name to > the lines at the top of each email. Then build the .msf files from the > actual content ... so, basically an offline build. I believe imap would not like that. Please direct all future comments to the new bug(s) you file.
Is there a progress on this issue? Thomas
Updating dependency list, from bugs found in the query https://bugzilla.mozilla.org/buglist.cgi?v4=maildir&bug_id=476239%2C%201306254%20845952&bug_id_type=nowords&f1=short_desc&o3=substring&list_id=14305682&f8=blocked&v3=maildir&j2=OR&o1=nowordssubstr&resolution=---&classification=Client%20Software&classification=Components&f4=status_whiteboard&query_format=advanced&f3=short_desc&o4=substring&f2=OP&v8=845952%20859011%20&f7=CP&product=MailNews%20Core&product=Thunderbird&o8=nowordssubstr And adding selected "non-blocking" maildir bugs to "related" list
Summary: finish "maildir" message storage [meta] → [meta] finish "maildir" message storage
You need to log in before you can comment on or make changes to this bug.