Closed Bug 58308 Opened 24 years ago Closed 5 years ago

support qmail's maildir format

Categories

(MailNews Core :: Backend, enhancement)

Product:
MailNews Core
Component:
Backend
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: sspitzer, Unassigned)

References

(
URL
)

Details

(Keywords: helpwanted, Whiteboard: [gs][temporary manual configuration of a 'maildir-like' store is possible per comment 172 and 174])

Attachments

(1 file)

text mockups of folder settings
2.67 KB, text/plain
Details 
thanks to jag for the suggestions.

now that we have movemail support, this should not be that much work.

if a lot of people start asking for this, maybe us hackers will add it in.
Keywords: helpwanted
Priority: P3 → --
I want two different features here:

1) Support a maildir folder as an inbox.

It could possibly be a separate account setup (POP/IMAP/Maildir). I use various
fetchmail, procmail, qmail combinations to get mail into "incoming" maildir
folders. I'd like to just point Mozilla at them to use as the inboxes. Currently
I use mutt as my client.

2) Support maildir as a folder format.

I'm pretty sold on maildir over mbox (that's a personal preference -- I'm not
trying to argue it's better). I'd like to be able to ask Mozilla to store
folders in maildir format. I'd also like to feel comfortable using multiple mail
clients to look over my mail.

Also (hope I'm not overloading this feature request), I'd like to be able to
store those maildir folders anywhere I want, not just in the .mozilla directory.

offtopic, you can pick any mailbox path you like, it's in account settings.

if we're going to support random mailbox formats,
1. we'll need to be able to convert between them
2. it should be configurable per account
personally i'd suggest that we rearrange the copies and folders panel.
Comment on attachment 116326 [details]
text mockups of folder settings

jglick: what do you think about this arrangement for the settings?

the reason not to use radio buttons for the folder type is that the number of
types can't be known and it would make adding additional types very hard.

in an ideal world, xbl widgets could allow the listbox to be converted by a
skin author into a radio set if there are only a few items. as designers of the
content we really should worry about scalability/pluggability instead.
Attachment #116326 - Flags: review?(jglick)
>When sending messages, automatically ---------------------------
>[x] Place a copy in: [Sent on timeless@myrealbox....|v]
>[ ] Bcc [timeless@myrealbox....,           ]

>Drafts and Templates -------------------------------------------
>Keep message drafts in:    [Drafts on timeless@myrealbox....   |v]
>Keep message templates in: [Templates on timeless@myrealbox....|v]
>[ ] Show confirmation dialog when messages are saved


I'd like to see the bcc items combined.
See Bug 178328.
http://bugzilla.mozilla.org/attachment.cgi?id=105110&action=view

As for the Send/Draft/Template combining with their associated "Other", I'd like
to see that as well. I've been told in the past it wasn't do able though.  
While the re-arrangement of the "Copies & Folders" configuration may be a good
idea, it seems like a separate issue from Maildir support. Should that be moved
to another bug perhaps?

Just to make sure I've been clear on the Maildir support, from an end user
perspective, I would like to be able to:

- use qmail/fetchmail/procmail combinations to deliver new mail into Maildir
folders (that's just background, not something Mozilla in involved in)

- use Mozilla to read the mail in those folders

- move mail between Maildir folders from Mozilla

- create new Maildir folders from Mozilla (optional, not a big deal if I can't)

- be able to switch between Mozilla and other mail clients (like Mutt) without
having to worry about them stomping on each other

I'd like to propose a cleanup to this bug's intent. Let's make this bug concern
those issues:

1) Make Mozilla support both maildir and mbox format internally (for the mail
stuff in the profile directory), the setting should be per-folder, not global,
e.g. it should be possible to mix mbox/maildir folders in one account
2) use maildir instead of mbox format by default for new folders

This would probably result in several big winnages:
a) fix bug 172786 (extremely long summary file build times because each message
needs to be parsed in whole to find the start of next message)
b) Boost MailNews performance in general
b) Boost MailNews stability in general

Mbox format should still be supported (it may be better for folders with high
message count, but small average msg size on slower filesystems like ext2)

Dynamic conversion between those formats should be possible:
    in folder properties group of 2 radio buttons:
        Folder format:
          (*) Maildir
          ( ) Mbox


So what you say about that?
Blocks: 172786
Hi,
I'm using Postfix and Fetchmail and so the Maildir-feature would be really
great:) At the moment i still have to use mutt instead of mozilla for "local"
mail. Please add Maildir-support!:)
Hi, I use a combination of fetchmail+procmail+courier-imap to use Maildir style
with Mozilla, but it's use a lot of memory degrading the performance of the system.

If you can enable this "Maildir" feature, it would be nice, will help a lot of
people.
I use fetchmail, procmail, and mutt with the Maildir format and would also like
to see Thunderbird support the Maildir format.
To Comment #6. Just an idea.

Aleksander, how about *BOTH* unox mbox and maildir for single mail folder?
 (x) Hold in maildir instead of unix mbox when mail size >= *NNN* kilo-bytes
 (x) Hold in maildir instead of unix mbox when mail has attachment(s)

Advantage:
 (A-1) Users can enjoy both speed efficiency and space efficiency
Disadvantage:
 (D-1) Many many functions should be re-written (Almost overhaul)
 (D-2) Possibility of many mail loss due to (D-1) until functions are stabilized
 (D-3) Difficulty in manual mail data recovery
There is another problem with the "both" idea:

Incompatible with any other mailer - both MBOX and MAILDIR tools would fail to 
work with it.

First, lets at least get MAILDIR support.
Then, automatic conversion from one to the other.
After that, maybe some form of dual support, if that is still seen as useful.

If you need people asking for Maildir format support for Movemail feature, then
I'm here. This new hacking would be greatly appreciated for the Maildir-style users.
Comment on attachment 116326 [details]
text mockups of folder settings

neil: do you think you could implement the combined widget?
Attachment #116326 - Flags: review?(jglick) → review?(neil.parkwaycc.co.uk)
Comment on attachment 116326 [details]
text mockups of folder settings

Tricky - the "X" Folder on: will automagically create the folder, the Other:
won't.
I'd like the maildir support too because i use
getmail+procmail(+bogofilter)+mutt to read my mail. I'm trying kmail but its
maildir support is different from the procmail. so please add this feature to
thunderbird. thanks

ps: sorry my (eventual) bad english :P

Any more development being done here?  I'd need this feature before I switch
from evolution -- and I don't like evolution. ^ ^
*** Bug 219148 has been marked as a duplicate of this bug. ***
*** Bug 233730 has been marked as a duplicate of this bug. ***
Added CC

I'd also like maildir support so it can read mail from qmail+procmail(+bogofilter)
I want to use maildir sources to interoperate with other mailers and scripts

This means pointing thunderbird/mozilla to a root folder and have it export the
included maildir forders in the mail folder list.

I don't care about magic hidden mail boxes that can not be shared between
clients - I want my folders appear the same in
evolution/mozilla/thunderbird/whatever. Private local storage may be dandy but
it burns you as soon as you change your client.

One should not have to set up a local imap servers just to share mail folders
between mailers.
OS: Linux → All
Hardware: PC → All
Ok,

I believe I found a student to make his thesis on adding maildir support in mozilla.

As Mr. Jonathan Saunders defines things there a 2 separate issues:
1. Support for maildir as an incomming source of mail - as is the case of
movemail. Mail is read from maildir/new, optionaly deleted from maildir/new
and/or moved to maildir/cur and appended to mailbox file at mozila mail backend

2. Mail backend storage format
  2.1 Replacing/modifiyng current mozilla mail storage format to/from mailbox to
maildir.
  2.2 Convert between mailbox and maildir. I think this is not essential, as
people that may want to do that, can do that using third party tools, even
writing their own scripts.


I do believe, that issue number 1 can be done very easily. Issue Number is much
more difficult and essential. I need to discuss this further with the module
maintainer.



I think issue #2.1 is what many of us are hoping for.  Issue #2.2 is not
critical for the technical user but may be a show stopper for the non-technical
user.  This may be a case where switching can not easily be done.

One thing I know is that I would like "From last night" in my EMail message to
remain that and not become ">From last night" :-)
(In reply to comment #22)
> I think issue #2.1 is what many of us are hoping for.  

I knew you gonna say that:).
After a 2 days staring at the mailnews code, I think that changing the backend
from mbox format to anything else, will bring substantial changes to almost
every class in the application. Not sure if we can(are allowed and being able)
do that, first I have to discuss current design with someone familiar with the
codebase.
What really scares me is the IMAP support(have not look at that one yet).

If you're wondering how to implement IMAP keywords (Mozilla's labels) in
Maildir, here's an already thought-out implementation:

http://www.courier-mta.org/imap/?README.imapkeywords.html

Nice if your implementation would be compatible with it.
for future reference....

a point for the maildir format:

Another possible reason to consider the maildir format is interoperatbility with
anti-virus software.  From what I've seen, Norton AntiVirus will interact better
if messages are one per-file.  In particular, I'm thinking of the
"delete/fix/quarantine" operations it can be configured to do on virus detection.

a point against the maildir format (from david):

"The drawbacks of the maildir format are that it wastes a large amount of disk
space, especially on FAT file systems, and it slows down searches that don't use
the db, like body searches, because you have to open each file."
Today's disks are pretty big. Oh yes, FAT users can't really handle them either;)
Another point for maildir format is backups.
If one backs one's local e-mails up using the mbox or similar format, one gets
an enormous change per day.  That is, if one adds a single e-mail which was
received today, one has to backup the whole mailbox, which may be a couple of megas.

    As these backups in our lab cost money, this prevented the backup of the
e-mail directory (the amounts were huge, as the diffs contained the whole
mailbox every day, which resulted in a large amount of data to backup daily).

        With the maildir, one backs up only the new file, rather than the whole
directory in an incremental backup.
Product: MailNews → Core
*** Bug 273674 has been marked as a duplicate of this bug. ***
I would also like to see the Maildir format supported on the client side in
addition to mbox.

As far as implementation is concerned, perhaps look at the way mutt handles both:
- if the mail folder name ends with a slash "mailbox/", then it's maildir
- if it ends with no slash "mailbox" then it's mbox

I'd even be satisfied with that as a solution.  Each mailbox could be a
different format.  Only time conversion would be necessary (from my perspective)
is when moving a message or messages from one folder to another, in which case
you have the mail data (think of as independent of storage format) and you just
call the function to write it in the format appropriate for the target folder
based on a slash at the end (or not).
I would like maildir format support too, becouse antivirs usualy can't disinfact
viruses from one big mail file and remove is better for remove only infected
message.
Hello,

> if a lot of people start asking for this, maybe us hackers will add it in.
I'm one of those asking for it :-D

Thx, bye,
Gismo / Luca
I would like to add another possible benefit of maildir: it's safer. With
maildir, if something goes wrong, only the messages being modified are affected.
With mbox, you could lose the whole folder.
Alo interested by these feature mainly for safety reasons (one file per mail)
I too would be very interested in seeing this capability, for all of the reasons
outlined above.
In reply to comment #25 by seth spitzer:

Maybe the Mozilla 2.0 platform could be the chance to get rid of the mbox
format?!  Today we don't support Win95, so we could do the step to not support
the old FAT format in Mozilla 2.0 platform.

Bug 284170 is a duplicate of this bug.

IMHO it's not worth to support the old FAT on the one hand and to ignore lost
inbox / mbox files on the other hand, if antivirus apps are killing them.

BTW: Ben Goodger, Benjamin Smedberg and others are well known people > they lost
their inbox files because of the 2GB problem at the end of their hollidays....
> Today we don't support Win95
>
> We don't? Since when? In fact, Mozilla Suite runs well on Windows NT 3.51!

http://www.mozilla.org/products/thunderbird/sysreq.html

Best example for hated old core compenents: DOS
*** Bug 284170 has been marked as a duplicate of this bug. ***
Another pros for maildir:
- We do not need folder compacting anymore (people have 500+ MB folders - it can
take a long time).
- Deleting messages is instant (moving to Trash)
- Moving messages between folders is instant.

Maildir is not "QMail" anymore. Lots of mail storage systems can use it.
I would really like to see this feature in future versions.
Just a note: KMail currently supports both flat files ("mbox" format) and
directories ("maildir" format).

Maildir is of course the default, because it is much more efficient (unless the
user only has some couple of dozens of messages, but it's very rare for any kind
of user to have so little mail).
In my situation, Maildir was not a decision, it was the only suitable format. I
use nfs-rooted clients AND servers; mailbox does not work because NFS does not
support correct locking. (No, I don't want to move away from NFS.) Maildir needs
no locking. The only thing preventing me from moving to Mozilla as a mail user
agent is that it does not support Maildir.
lorro@lorro.wigner.bme.hu: that's nice, in the interim, run an imap server that
speaks maildir and have mozmail speak imap to that imap server.

if you're interested in helping teach mozmail about maildir, then let us know
and we'll put you to work (c++ coding experience required, experience building
mozilla  required, lots of time and patience .. required)
Blocks: majorbugs
No longer blocks: majorbugs
*** Bug 300900 has been marked as a duplicate of this bug. ***
Hello,
I'v just been pointed to this bug and would like to share a few comments. :-)

To comment #6: Evolution as of 1.5.2 (?) has per-folder settings for
this, and you can just change the format of any given mailbox folder
from the context menu. Maybe that code can be ripped.


#20: agreed


#27: disagreed, at least from the point of view of a mutt user.
Mutt handles flags on messages in maildirs by renaming them, eg.
messagefile -> messagefile;S if you have read the message. This
is very bad for backups and/or rsync'ing mailboxes over the 'Net,
but very good otherwise. To _efficiently_ backup such things,
one would probably need a specialized archiver that can handle
such flags.

In the same vein, something will probably need to be done in order
to have appropriate conversions from and to the X-Mozilla-Status:
headers that mails seem to have.


#35: The 2GB problem also strikes on NTFS as I've recently experienced
on W2k. OTOH, you can have _A_LOT_ of mail until your 'hcache.db' file
overflows the 2GB limit...


General remarks: 

* Define an API for accessing mailboxes, then create plugins
  to handle these.

* Nested folders, quota etc. is not part of the original Maildir
  specification by DJB, but several ideas on how to work with
  these have been evolving over time. I think that "Maildir++" is
  the current common superset of all of these.
Hello,
I'v just been pointed to this bug and would like to share a few comments. :-)

To comment #6: Evolution as of 1.5.2 (?) has per-folder settings for
this, and you can just change the format of any given mailbox folder
from the context menu. Maybe that code can be ripped.


#20: agreed


#27: disagreed, at least from the point of view of a mutt user.
Mutt handles flags on messages in maildirs by renaming them, eg.
messagefile -> messagefile;S if you have read the message. This
is very bad for backups and/or rsync'ing mailboxes over the 'Net,
but very good otherwise. To _efficiently_ backup such things,
one would probably need a specialized archiver that can handle
such flags.

In the same vein, something will probably need to be done in order
to have appropriate conversions from and to the X-Mozilla-Status:
headers that mails seem to have.


#35: The 2GB problem also strikes on NTFS as I've recently experienced
on W2k. OTOH, you can have _A_LOT_ of mail until your 'hcache.db' file
overflows the 2GB limit...


General remarks: 

* Define an API for accessing mailboxes, then create plugins
  to handle these.

* Nested folders, quota etc. is not part of the original Maildir
  specification by DJB, but several ideas on how to work with
  these have been evolving over time. I think that "Maildir++" is
  the current common superset of all of these.
I would really like to see support for maildir-format. On the one hand there are
the short-comings of mbox - very slow if you have many (!) mails in one folder,
problems with virus-protection, etc. - which come to mind, on the other hand
I've seen the advantages of maildir from Apples mail.app and KDEs KMail.

What's more is that I personally would like to use mail.app and mozilla on the
same mail-base without installing local IMAP-Servers.

(In reply to comment #46)
> I've seen the advantages of maildir from Apples mail.app and KDEs KMail.

Apple made some changes to how they store e-mails with OS X 10.4 (Tiger). All
messages are now individually stored to make them easily identifiable by the
integrated search function (Spotlight). I guess that [and MS Vista] is something
to consider when discussing file formats.
wrt comment #47:

Please don't work for an Apple-only solution, but instead please first get
Maildir going, then perhaps Maildir++.

If mailbox handling could probably stored per-mailbox like in Evolution
(could be part of the personal profile), this would be good. Programs
for converting maildir to mbox and vice versa already exist, and are
simple to create.
cc
*** Bug 321337 has been marked as a duplicate of this bug. ***
Could we change the name of this bug to "support maildir format" ? I use maildir format in several programs right now and none of them are qmail. I added a vote for this also as it's the main hold up that prevents me from switching to thunderbird.
*** Bug 335650 has been marked as a duplicate of this bug. ***
This is an enhancement, isn't it?

yes, it's an enhancement, thx.
Severity: normal → enhancement
maildir support is just what I'm expecting to begin to use thunderbird
Lack of maildir format prevents me from switching to Thunderbird because huge mailbox files often changing are difficult to backup :=(
FWIW, maildir may make enhance the functionality of tagging:  I would like to keep everything in my Inbox, tag it as needed, and then use search folders in place of standard folders.

(The benefit is that, because one message can have multiple tags, it would appear in multiple search folders.  So a message from Bob about the Chicago project can appear in both the "Bob" search folder and "Chicago" search folder.)

mbox can't handle an Inbox that large.  I suspect that maildir could.


My main reason for supporting this "bug" is performance.  Big mailboxes are really a pain to deal, specially on slow file systems, like windows FAT.  Delete a message is a very costly operation, and in these times of spam, delete is the most used function.   ;-)

The main issue is to abandon mbox+index format, and since Maildir is already a standard, why not use it?
does anyone know if one of the reasons maildir suport isn't being worked on is due to character limitations on the win32 platform?

if it is may be it's time to differentiate thunderbird for *nix by including maildir support.
When my company moved from POP3 based server to IMAP one,
I found even a bigger issue. Mails of all of my IMAP folders
are stored in a common folder.
 Earlier, I have created various sub-folders, like "Jan-mails",
"Feb-mails", etc, which had kept my folder size under control.
 But this hack no more works. And my local Inbox mailbox file
has grown to around 1 GB.
 Since I have work offline lot of time, I want to keep
a local copy of all my messages.
  One option is to have an IMAP server running locally on 
my machine, which regularly syncs with actual IMAP server.
But it would be much better, if thunderbird itself
supports maildir format, so that users do not need to
take burden of using various workarounds.
Wrt. comment #57, I'd like to add that some programs use much less memory when working with maildir formats. Programs seem to slurp mailboxes in mbox format completely into RAM while working on them, but retain only the index when working with maildirs. If you then also use something like hcache (as in mutt), you can have HUGE speedups between mbox and maildirs. But please note that maildirs w/o additional helpers like hcache can be quite slow, too, especially on startup. Also, maildir makes the compulsory backup copy of the mbox file unneccessary which roughly doubles the disk space requirement for mbox style mailboxen, thus also using much less disk space while in operation.
(In reply to comment #59)
> does anyone know if one of the reasons maildir suport isn't being worked on is
> due to character limitations on the win32 platform?

What exactly do you mean, 8.3 style crippled file names? Wouldn't these be solved through VFAT, and/or can't you declare 8.3 file systems as officially unsupported (some older Unix variants have them, too)?

In all other cases I've seen so far, the individual messages are stored in file names like _QKC+R2xdFB.hostname (these were generated by procmail) or 1166144441.10233.hostname (these were generated by qmail, timestamp.pid.hostname). I'm sure that other programs come up with yet different naming schemes, but as long as most ASCII characters and sufficiently long names are supported, one should be on the safe side.
(In reply to comment #62)
> (In reply to comment #59)
> > does anyone know if one of the reasons maildir suport isn't being worked on is
> > due to character limitations on the win32 platform?
> 
> What exactly do you mean, 8.3 style crippled file names?

IIRC the standard maildir filename convention uses one or several characters forbidden on windows filesystems (unix is more relaxed). Which does not mean adapting the convention to win32 limitations would be difficult.
(In reply to comment #63)
> IIRC the standard maildir filename convention uses one or several characters
> forbidden on windows filesystems (unix is more relaxed). Which does not mean
> adapting the convention to win32 limitations would be difficult.

For pure storage there is no issue. Per URL, the standard filename for a delivery agent to use is time.pid.host but from then on it is simply treated as an arbitrary unique name with no special meaning ( "other than [ignoring a leading dot] readers should not attempt to parse filenames").

However, a reader is permitted (but not obliged) to rename the file adding a ':<info>' section as a file suffix, and most Maildir-supporting systems use this as the flag storage mechanism (NTFS uses : for file streams). There is, of course, nothing to stop Thunderbird storing its flags somewhere else external to the Maildir itself. Perhaps a similar mechanism using a different character (such as ';' or '_'?) could be used on filesystems where ':' is unusable... are there any other Win32 clients using a Maildir-compatible storage format? If so what do they do? What about a Linux system with an NTFS mount (perhaps accessed over the network from a Windows server)?


Personally I'd be happy with any storage format such that each message is stored in a separate message/rfc822 file. Whether flags are stored in the filename, file attributes, an additional X-Thunderbird-Status: header, or a separate index file is largely irrelevant to me.
(In reply to comment #64)
> (In reply to comment #63)
> > IIRC the standard maildir filename convention uses one or several characters
> > forbidden on windows filesystems (unix is more relaxed). Which does not mean
> > adapting the convention to win32 limitations would be difficult.
> 
> For pure storage there is no issue. Per URL, the standard filename for a
> delivery agent to use is time.pid.host but from then on it is simply treated as
> an arbitrary unique name with no special meaning ( "other than [ignoring a
> leading dot] readers should not attempt to parse filenames").

> Personally I'd be happy with any storage format such that each message is
> stored in a separate message/rfc822 file.

That's not what people ask for there, and you underestimate the level of maildir standardisation since it was first proposed

Under Unix/Linux/BSD/Mac OS thunderbird should read and write pure standard-compliant maildirs, to interoperate with all the Unix mail tools that use this format. That was true when comment #1 was written in 2003, and the number of maildir-using tools has only grown since. A variant only thunderbird understands wouldn't be useless, but close.

That means the maildir backend under win32 must be as close as possible to the canonical unix one to avoid problems. Even if the win32 platform is limited enough there is no other mail tool to interoperate with

(In reply to comment #65)
> Under Unix/Linux/BSD/Mac OS thunderbird should read and write pure
> standard-compliant maildirs, to interoperate with all the Unix mail tools that
> use this format.

Agreed, but my discussion was about what to do in cases where ':' is not a valid filename character. A *minimal* implementation of Maildir format according to URL says that "The reader *may* [...] rename new/unique as cur/unique:info", but does not require it to do so. A Maildir with all of the messages in the new folder would still be a valid Maildir with no change to the spec.

I understand, after a certain amount of searching, that Mutt, and some other tools on Windows use a modified version of the Maildir format. This gives a target for interoperability... looking at the maildir.c file from the source code at http://www.geocities.com/win32mutt/poppet.html it seems that the ':' is replaced with a '-'. According to this message from 2002 http://www.cygwin.com/ml/cygwin/2002-09/msg01009.html Exim allows the separator used to be configured. Thunderbird could do the same, perhaps allowing selections of ':', '-', and any others that seem sensible (perhaps ';', '|', and a few others dependent on OS, filesystem, and what will be inserted by the mail delivery agent if applicable - any character that appears in filenames in tmp/ or new/ would be inappropriate). Obviously ':' would be the default for systems that support it as a filename character.
(In reply to comment #64)
> However, a reader is permitted (but not obliged) to rename the file adding a
> ':<info>' section as a file suffix, and most Maildir-supporting systems use
> this as the flag storage mechanism (NTFS uses : for file streams). There is, of
> course, nothing to stop Thunderbird storing its flags somewhere else external
> to the Maildir itself. Perhaps a similar mechanism using a different character
> (such as ';' or '_'?) could be used on filesystems where ':' is unusable... are
> there any other Win32 clients using a Maildir-compatible storage format? If so
> what do they do? What about a Linux system with an NTFS mount (perhaps accessed
> over the network from a Windows server)?

I guess that this would have to be converted to something else, if anything. But I've got a suggestion, though no code to back it up:

Let's split the maildir handling problem into three parts:

* Define an interface to handle maildirs, and
* define a library that takes care of the specifics with respect to the underlying file system (can this be detected, reliably?).
* Provide a tool that converts between the various maildir storages (ie, converting appendant ':info' parts to something else when going from non-NTFS to NTFS (other problems, eg. forks in Apple file systems?), and vice versa).

This should imho solve most of the problem if users are properly advised to not blindly copy maildirs between NTFS and Unix file systems.

> filename, file attributes, an additional X-Thunderbird-Status: header, or a
> separate index file is largely irrelevant to me.

An included X-Thunderbird-Status: header would not be terribly useful, and the external index file (hcache) in mutt is only used for performance reasons, not functionality, but yes, I think the tool mentioned above could put this additional info into some kind of additional external file. Using SQLlite springs to mind...
(In reply to comment #65)
> That means the maildir backend under win32 must be as close as possible to the
> canonical unix one to avoid problems. Even if the win32 platform is limited
> enough there is no other mail tool to interoperate with

Since Cygwin, win32 is no more a limit to unix tools.  This means mutt, pine, procmail, etc.

I would wonder if it is possible, for example, to compile a Unix Flavored Thunderbird inside Cygwin.  Apart from the performance, why not?  Windows TB with remote X display capabilities.   ;-)

Back to real world, I like the idea of a config to select the mapping for the ':' char in maildir spools.  Other invalid chars exist, but are less important: "\/:*<>|

Also, I just thought of another use for maildir format: remote storage on file servers.  I already use remote storage to mail folders, but mbox format is too slow for any operation.  Using maildir would not only allow faster operations, but also simultaneous use.  This would allow for a shared mailbox, for example, without the need of an IMAP server for file storage.  May not appear very useful now, but corporate world could use it.


Hey, just wanted to also voice my support for this enhancement request.  A
file-per-message format is badly needed these days of large mailboxes (and
therefore large backup bandwidth requirements, large corruption risks, folder
size limitations, etc...).

It seems that no other Windows-compatible email client supports large amounts
of email well these days.  Old versions of Outlook don't support more than 2GB
(per PST, not per folder!) and the newer ones have serious speed issues when
the PST files get that big (and again, that's for ALL folders combined, not PER
folder!)  If Thunderbird could add this sort of ability, it would be alone in
the Windows world with respect to that much-needed capability.  Perhaps a great
opportunity for spreading the use of Thunderbird, especially on win32...

What could be done do further this project?  Seems like talk on this bug has
died.  I would love to help but not sure what I could do to help.  Not strong
in C++, but fairly knowledgeable otherwise...
It's interesting that a justification for this enhancement is the size of the inbox.  HTML-formatted messages can be 3-5 times larger than ASCII-formatted messages with the same text content.  Yet, when I request others to send me only ASCII-formatted messages because of the size (affecting not only disc space but also download bandwidth), they hoot at me for living in the past (I have dial-up). 

Since attachments often exceed the size of their messages, would not implementation of bug #9309 help address the size of the inbox file?  
 
Seems to me like the size in terms of number of messages is as much, if not more, the problem than the size of the mbox files.
(In reply to comment #71)
> Seems to me like the size in terms of number of messages is as much, if not
> more, the problem than the size of the mbox files.

"You're living in the past, we have 64 bit numbers by now." ;-)

Note that size of the mbox has already been an issue for a number of reasons, most of which have already been clearly stated.  Backup, operational performance, corruption danger (mainly in that is has a drastic effect as it affects all messages in the file), I/O growth, and filesystem stability (a file that changes is in danger of loss during a system crash/power outage and since this file always changes it is always in danger)

Now, Mozilla (and other mbox programs) do their best to minimize certain risks, but the risks are there none the less, and as size increases (be it from number of messages or larger messages) the amount of time spent in those operations increases (I/O being the bottleneck, not the CPU performance).

The good thing is that imap solves all of this (and most all imap server implementations I know of use maildir format or a database)

The question I have would be, why not make a "fake" imap server as part of the client?  Then the imap API would just go through the local file system into maildir format.  (Hey, I have a real imap server doing this, but that requires a bit more work than most users have the time and/or skill for)

This way the complexities of the mbox format code would not need to be touched.
7 years, and still no obvious sign of interest by developers (not assigned).

I've added a vote, but really, it appears that the developers don't personally need this, and those who do need it are unable or unwilling to provide development assistance.

All of the reasons for this have been discussed.  The primary reasons remaining would be that backups and antivirus tools having less data to process on changes.

Scanning mbox files with large numbers of messages can be time consuming, but so can maildirs with large numbers of messages.

Local mail users can just use IMAP to localhost.

I personally still use TB, and I create archive dirs every quarter for high-volume or every year for mid-volume folders.  This helps greatly, but I still back up several gigs of unnecessary, duplicate data every week because of mbox vs maildirs.

I'm grateful for the effort that the Mozilla team provides and the product they have created thusfar.  I wish it were within my realm of expertise to even rip and merge code from other sources to help this.

All I can do is ask that a developer with the necessary skills would pick up this enhancement and move it along.  There is a sizable subset of TB's users who would appreciate such work.
Per a comment in bug 367774 dated Jan '07, maildir is due for TB3:

    https://bugzilla.mozilla.org/show_bug.cgi?id=367774#c1

Also, I see comments from various developers saying they're amenable to it. Just search Bugzilla for 'maildir'.
Here's my vote for maildir.... I'm now setting up a separate IMAP retriever/server just to eliminate the dangers inherent in monolithic mbox file structures.  Would it not be possible to create an API layer which would allow either/or (or even other non-mbox/non-maildir) structures if required thus abstracting the problem from within Thunderbird?
What mutt does, if they use a '-' for the flags separator in Maildir for the filenames, is wrong: you need to add the hostname to the filename and a hostname can legally contain a '-' character.

What exim does, letting the user choose this character, is even worse as you end up with a character that will be incompatible with any other E-mail solution. Each and every problem that '-' has in mutt, will exim have too: the user can choose the character so the user can also choose for the defect. So the defect is possible (and software like this should imo not make it possible to make it create a defect mail storage).

A character that I propose to replace the ':' of the filenames of Maildir, to make Maildir possible on Windows's partition types, USB sticks and flash cards, is '!'. You obviously can't use the ':' character as that character is also used to separate the drive letter from the path on Windows itself.
Blocks: 402392
I vote for this mainly because it is realy a pain to to incremental backups or any remote synchronizing of a large mailbox on regular intervals.
Because of this I have to use a local IMAP-server (dovecot) on linux to work around this.
But I did not find a workaround for windows machines by now, since there is no dovecot available for this platform.
Having maildir support in thunderbird would make live much easier and performance would be much better for those storing their mailbox remotely or on any slow medium.
I don't know what the current state of affairs is, but after meeting a number of people who claimed that Thunderbird has destroyed their mailboxen, and that, for that reason, they won't allow Thunderbird into their companies' or authorities' offices, I'd say that this issue is probably really a high-profile issue. All usability aspects aside, losing mail is something that almost no user I'm aware of is prepared to tolerate. Incidentically, I'm also using Thunderbird only for mail that I'm prepared to lose, not for my business mail, but I have clients using it for serious things, and I'd rather not be held responsible for suggesting a mail user agent with such a known defect (using maildir should reduce the chances of losing mail by a very wide margin). Thank you for listening!
from subjective irc support experience I'd say in almost all cases of disappeared mail the problem is prefs corruption or file access problems probably above the mbox level (and in historical versions, profile locking issues), and the folder contents remain intact and recoverable. so while maildir may be a good idea, its impact alone in that regard could turn out limited.
I'm probably too naive, then, but what would prevent you (who "know", contrary to myself) from repairing the problems that cause mail to disappear? Because, whatever the reason might be, losing mail is very annoying to many people, and recovery procedures also don't seem to be widely documented - I'm aware of none or one, at most (corrupted index files?). But implementing maildir (or Maildir++) would also sort of solve the 2gig or 4gig problem that mailboxen in SeaMonkey and Thunderbird currently have, to the best of my knowledge.
OK, now for some good news: from 2008-02-11 I'll have a week to work on this and I really would like to see things moving (at least a little bit). The problem is that I'm not really familiar with mozilla codebase, so I'll welcome any competent tutoring/help possible - e.g. hints which parts of documentation I may skip while preparing for this problem, any consistent overview of the mail/news subsystem (the documentation subproject is still lacking this AFAIK). Please, feel free to contact me by mail.
Not to discourage you, but I'd be very impressed if you pulled off doing this in two weeks... ;)

There is not very much documentation that I know of, but I'd start of reading the code around here: http://mxr.mozilla.org/seamonkey/source/mailnews/local/src/nsParseMailbox.cpp ... and related classes.
QA Contact: esther → backend
I would like to strongly support this - Thunderbird's unreliable storage of emails and the complex recovery procedures are a major inhibitor to use by novices, based on my experience with several non-expert PC users who really struggle when Thunderbird loses their emails (usually a corrupt index file).

For more on this and some related discussion, please see http://lwn.net/Articles/266938/rss - responses to article about Thunderbird 3 planning.
(In reply to comment #54)
> yes, it's an enhancement, thx.

I'd say that this should be a show-stopper, and I fully agree with comment #84 (esp. the linked article).

I also find this issue pressing enough to not have this issue blocked by bug #402392.


My guts feeling is that this dependency mainly helps to delay Maildir support for a few dozen years more. Now that I'd have a long-awaited opportunity to switch a few users away from Outlook, I'm not sure that I can do it in good faith because of this bloody bug. Maybe someone can say what *is* the remaining problem with this, and how to get started (I know nothing about building Mozilla apps, and my C++ coding experience is more than 10 years in the past).
(In reply to comment #54)
> yes, it's an enhancement, thx.

I'd say that this should be a show-stopper, and I fully agree with comment #84 (esp. the linked article).

I also find this issue pressing enough to not have this issue blocked by bug #402392. I highly object to any opaque storage formats that are (almost?) only supported by Mozilla apps. Interoperability and robustness are THE key points for me.

My guts feeling is that this dependency mainly helps to delay Maildir support for a few dozen years more. Now that I'd have a long-awaited opportunity to switch a few users away from Outlook, I'm not sure that I can do it in good faith because of this bloody bug. Maybe someone can say what *is* the remaining problem with this, and how to get started (I know nothing about building Mozilla apps, and my C++ coding experience is more than 10 years in the past).
Dear a_geek,

I'd like to correct a couple of misconceptions apparent in your post(s):

1. This issue is blocked by *no* bugs; that would be indicated in the "Depends on" field.  Bug #402392 depends on *this* one.

2. The current mbox format used by Mozilla *is* a well-documented and standard format, and is most certainly supported by *many* more apps than just Mozilla.  It just happens to have some shortcomings which have become apparent over the years (I won't repeat the comparison here, I'm sure you have read plenty about it in the discussion above and elsewhere on the web).

I'm frankly surprised, given the number of posts you've had in this discussion, that you would make these comments.

Why don't you contact peterph@centrum.cz from comment #82 if you'd like to help?
Hi Shi Sherebin, thanks for your answer.
1: Ok, my fault
2. Yes, mbox *is* a widely supported standard (but so is "Word"), but a very poor one from a technical point of view - imho. I had problems with mbox files in other apps (which I didn't write) before Mozilla even came into existance. My impression is that nowadays, Mozilla apps seem to be amongst a very small, and dwindling, crowd of Free Software apps that still do *not* support Maildir.

I was probably too aggravated when I wrote the post, so I managed to not have the idea to contact Peter, but have just sent him a message.

About which part of my message were you so surprised?
re 2: I completely agree that we want maildir and not mbox, don't get me wrong.  I just think that repeating ourselves in this forum won't do much to advance the cause.  I think it's pretty clear why maildir is better than mbox from what's already written.  The point is, it won't happen unless someone *chooses* to spend the time.  All the power to you for putting your money (/time/effort) where your mouth is and contacting Peter to offer your help.

I'll answer your final question offline later.
Hello folks,

IMHO this whole discussion goes far too short. 
Invidual MUAs should not be tied to several storage formats - those things should be handled by mail servers. So the local-IMAP suggestion goes into the right direction. 

But: I don't think IMAP is the right interface/protocol as an universal mail storage interface. It's too complex to implement and lacks several features, an MUA would need (eg. arbitrary client-made tagging, content searching, direct access to message parts, etc). 

I'm thinking of an synthetic filesystem like Plan9's mailfs, but with more functionality on a very high level (so, eg. message parts, encodings, etc should also be handled by the server).

I've written down a few ideas here:
http://oss-qm.metux.de/index.php/9forge/mailfs-ng

Accessing such synthetic filesystems via 9P can be easily done via userland (eg. via libmixp/libmvfs). 

Once we run evrything through that synthetic filesystem, we don't have to care about OS's filesystem constraints.

To summarize the arguments to the synthentic filesystem solution:

* individual mail storage is the job of the MUA anymore -> reduces code complexity
* as servers run in separate processes, talking via 9P, everything's network agnositic
* multiple access is no problem anymore, since the server coordinates this
* switching to another mail storage is now just a matter of mounting the right server
* synthetic filesystems are quite easy to implement (eg. via libmixpsrv)
* virtual mailboxes (eg. r/w views to certain other information systems like CRM) can be done easily by coding a proper sever
* stacking several distinct storages together to an larger tree, mapping folders, custom views and complex access control can be done entirely via filesystem overlays


I'm currently working on an paper on that issue. I'll post the link once it's finished.


cu
IMHO mozilla products always tended to be selfcontained as much as possible and be able run on many different platforms. Therefore until a reliable functioning servers are available on each platform that the mail app runs on mozilla has to have internal storage engine(s). Also using external storage _only_ would require users to hunt other sw packages.

However (as I quite like this idea) - a binding could be used to allow mozilla access such fs once the mail handling backend is split (see bug #402392 comment #5)
Early versions of the GNOME Evolution email program used an abstraction layer for the local mail storage. This allowed the mail program to present the user with a virtual view of mail storage folders that looked familiar. But, under the hood, the actual mail storage could be done via maildir, mbox, mh, imap, or whatever. What was really cool is that the user could right click on a folder and switch that particular folder from one format to another at will. So I could have my spam folder be mbox format, which made it easy for me to submit the whole folder to spamassassin periodically for learning. And I could have my other folders in maildir format, which was safe, fast, and had no mail or size limits to worry about. 

Sadly, Evolution later dumped this scheme in favor making all local storage in mbox format, imposing the usual limits, slowdowns, and corruption dangers. It was dumped not because of any technical issues but (I think) because of the overall GNOME project's drive to remove user configuration options and choices where ever possible to "simplify" things for dumb users. I've since figured out a less than ideal workaround that allows me to sort of use maildir with the current version of Evolution but it's not pretty. I'd switch to Thunderbird instantly if it provided a flexible local storage system like the Evolution v1.4 era client had.

The code probably still exists somewhere and was GPL'd so maybe it could even be reused as a starting point for a Thunderbird mail storage virtualization layer?
(In reply to comment #91)
> IMHO mozilla products always tended to be selfcontained as much as possible and
> be able run on many different platforms. 

Thats perhaps the main reason, why Mozilla is that fat so hard to maintain. 
Third-party packages like cairo, zlib, etc, should not be bundled into Moz.
(I've already supplied patches to kick them off)

> Therefore until a reliable functioning servers are available on each platform 
> that the mail app runs on mozilla has to have internal storage engine(s). 

Well, I'm working on that :)
I've currently got no win32 system available, but porting should be quite easy.

BTW: I came to this bug through the current FB3 discussion. Don't know how long it will take to finish, but I'm confident to get it finished soon enough.

> Also using external storage _only_ would require users to hunt other sw packages.

Of couse, as it also does with all other imported packages, down to libc.

> However (as I quite like this idea) - a binding could be used to allow mozilla
> access such fs once the mail handling backend is split (see bug #402392 comment
> #5)

Yes, at least this. 
But once I've got my mailfs-ng specified and the reference implementation
running, I'll start implementing as the only message storage for Moz (and other MUAs like mutt).


cu


(In reply to comment #90)
> IMHO this whole discussion goes far too short. 
> Invidual MUAs should not be tied to several storage formats - those things
> should be handled by mail servers. So the local-IMAP suggestion goes into the
> right direction. 

I can think of several reasons to not use local-IMAP, including some potential performance and security problems.

> I'm thinking of an synthetic filesystem like Plan9's mailfs, but with more
> functionality on a very high level (so, eg. message parts, encodings, etc
> should also be handled by the server).

* cringes *

> Accessing such synthetic filesystems via 9P can be easily done via userland
> (eg. via libmixp/libmvfs). 

And how are you proposing to do this through Windows? Windows is a far cry from POSIX and *nix, last I checked; IIRC, there's also hoops to go through when creating filesystem drivers.


> * individual mail storage is the job of the MUA anymore -> reduces code
> complexity

You have to maintain a filesystem, which is MUCH more complex

> * virtual mailboxes (eg. r/w views to certain other information systems like
> CRM) can be done easily by coding a proper sever

Your thesis for this is that it reduces complexity. This makes stuff much more complex.

(In reply to comment #92)
> The code probably still exists somewhere and was GPL'd so maybe it could even
> be reused as a starting point for a Thunderbird mail storage virtualization
> layer?

I'd have to look at the code, but integration is not necessarily easy within Thunderbird.


(In reply to comment #93)
> (In reply to comment #91)
> > IMHO mozilla products always tended to be selfcontained as much as possible and
> > be able run on many different platforms. 
> 
> Thats perhaps the main reason, why Mozilla is that fat so hard to maintain. 
> Third-party packages like cairo, zlib, etc, should not be bundled into Moz.
> (I've already supplied patches to kick them off)

And kill Windows support? That would essentially do the opposite of what mailco is trying to do...

> Well, I'm working on that :)
> I've currently got no win32 system available, but porting should be quite easy.

Porting to win32 if you're coming from Linux is NOT a simple task.

> BTW: I came to this bug through the current FB3 discussion. Don't know how long
> it will take to finish, but I'm confident to get it finished soon enough.

A lot longer than you think.

> > Also using external storage _only_ would require users to hunt other sw packages.
> Of couse, as it also does with all other imported packages, down to libc.

Consider Windows: basic stuff is already provided, but cairo.dll is probably not commonly found on Windows systems, and most users are not going to go through such hoops.

In short, I am personally against what you are proposing...
(In reply to comment #92)
> The code probably still exists somewhere and was GPL'd so maybe it could even
> be reused as a starting point for a Thunderbird mail storage virtualization
> layer?

We should also have a look at mutt. Last time I checked, it's mailbox layer wasn't an separate lib, but concentrated in a few .c files. 

Maybe we should first design an (abstract) interface and then look how to implement it best.

cu
(In reply to comment #94)

> I can think of several reasons to not use local-IMAP, including some potential
> performance and security problems.

Yes, IMAP isn't well suited for this. It's intended for remote use and allows only access to *complete* messages.
My mailfs-ng will allow access to individual parts (even single metainf) through separate files. 

> > Accessing such synthetic filesystems via 9P can be easily done via userland
> > (eg. via libmixp/libmvfs). 
> 
> And how are you proposing to do this through Windows? Windows is a far cry from
> POSIX and *nix, last I checked; 

At least, ANSI-C is available there ;-O
A tricky point is the (AFAIK) missing BSD socket interface, so it needs it's own send/receive stuff (just a few funcs in libmixp to be replaced).

> IIRC, there's also hoops to go through when
> creating filesystem drivers.

NO! You misunderstood me ;-P
The filesystem stuff is *entirely* in userland - both server and client. 
Everything happens through the libmvfs library functions (mvfs_*()), not the OS's filesystem ops. The OS doesn't need to have any FS API at all (as long as we have some way for passing packets between client and server).

> > * individual mail storage is the job of the MUA anymore -> reduces code
> > complexity
> 
> You have to maintain a filesystem, which is MUCH more complex

No, it isnt. libmixpsrv makes it very easy, it even allows (almost-)direct acess to in-memory structures. 

Note that Plan9's filesystem semantics are *much* easier than traditional filesystems. This makes synthentic filesystems very easy and cheap.

> > * virtual mailboxes (eg. r/w views to certain other information systems like
> > CRM) can be done easily by coding a proper sever
> 
> Your thesis for this is that it reduces complexity. This makes stuff much more
> complex.

Only for the case that you're really using such an fileserver. The point is that the user has the choice to either use simple standard server(s) or replace it by his own. Of couse, one could write an IMAP server for his CRM, but thats more complex, since IMAP expects complete RFC2822 messages.

> (In reply to comment #92)
> > The code probably still exists somewhere and was GPL'd so maybe it could even
> > be reused as a starting point for a Thunderbird mail storage virtualization
> > layer?
> 
> I'd have to look at the code, but integration is not necessarily easy within
> Thunderbird.

Why ?

> (In reply to comment #93)
> > (In reply to comment #91)
> > > IMHO mozilla products always tended to be selfcontained as much as possible and
> > > be able run on many different platforms. 
> > 
> > Thats perhaps the main reason, why Mozilla is that fat so hard to maintain. 
> > Third-party packages like cairo, zlib, etc, should not be bundled into Moz.
> > (I've already supplied patches to kick them off)
> 
> And kill Windows support? That would essentially do the opposite of what 
> mailco is trying to do...

Heh ? Where's the problem with just installing these libs on Windows (as it's done on all other platforms) or just ship them within the binary distribution ?

> > Well, I'm working on that :)
> > I've currently got no win32 system available, but porting should be quite easy.
> 
> Porting to win32 if you're coming from Linux is NOT a simple task.

Heavily depends on what an individual application expects. 
Except the packet transciever (which currently uses BSD sockets) everything's fine with ANSI-C. So just the few transceiver funcs have to be ported to winsocks. (should take less than 1h)

> Consider Windows: basic stuff is already provided, but cairo.dll is probably
> not commonly found on Windows systems, and most users are not going to go
> through such hoops.

That's really a matter of the package management. Yes, Windoze package management is bad at best. If you want to have an self-contained binary distro, simply add the binary libs to the package. There's really no need for maintaining an separate copy of the source within the Moz tree.


cu
(In reply to comment #96)
> Yes, IMAP isn't well suited for this. It's intended for remote use and allows
> only access to *complete* messages.
> My mailfs-ng will allow access to individual parts (even single metainf)
> through separate files. 

Only to clarify this, I believe this is not correct. When I last read the IMAPv4 specification to implement my own client for it, I saw that it is possible to fetch single parts and also single headers of a message. Did you confuse this with POP3?

I didn't really follow the technical discussion, but somehow I don't feel comfortable with installing my own local IMAP server only to store e-mails. Also, my experience from Thunderbird is that it's highly unstable and prone to massive data loss when handling large number of messages over IMAP. I tried to put my local folders to my IMAP server and after the first some copied, only empty messages arrived. So I decided that local folders seem to be the best for archiving older messages.

Are mbox, maildir and SQLite so much of a proprietary format? Does it always have to be middle-age human-readable ASCII formats (like IMAP is)? Databases were exactly invented to handle large data efficiently. But that was the other of the several bugs here about mail storage backend...
(In reply to comment #97)

> Only to clarify this, I believe this is not correct. When I last read the
> IMAPv4 specification to implement my own client for it, I saw that it is
> possible to fetch single parts and also single headers of a message. 

I admit, I didn't completely read the full IMAPv4 spec, so I might be mistaken at this point. 

But an MUA would still need much finer granularity (eg. accessing single header fields), certain kinds of views (filters, etc). I don't think IMAP is really suited for this.

AND: the IMAP protocol is quite complex, compared with an simple filesystem structure (which is just an hierachical object representation). 

> Also, my experience from Thunderbird is that it's highly unstable and prone to
> massive data loss when handling large number of messages over IMAP.

IMHO, this also applies to Mozilla's Mail component. It tends to be very lazy in recognizing mailbox changes and sometimes crashes mails. But it could also be an server-side problem.

> Are mbox, maildir and SQLite so much of a proprietary format? Does it always
> have to be middle-age human-readable ASCII formats (like IMAP is)? 

ASCII formats often make lot's of processing jobs (eg. via scripts) much easier. 
So there should be at least an ASCII representation. 

BTW: in case of error, an ASCII mailbox can be repaired manually. If some binary DB file crashes, you're almost lost.


cu
(In reply to comment #98)

> But an MUA would still need much finer granularity (eg. accessing single header
> fields), certain kinds of views (filters, etc). I don't think IMAP is really
> suited for this.

Read not only the actual IMAPv4 RFC, but the other RFCs produced by the Imapext and Lemonade working groups as well. IMAP can grab the value of a header for all messages, and you can specify filters as well with an extension.

In any case, I would like to remind people that this bug has a rather specific scope: supporting handling of the maildir format as defined by qmail (NOT any other directory-based format!). Another bug exists for using SQLite as the mail backend; a third bug exists for having a pluggable mail storage API.

If you want to continue talking about using an internal IMAP server or a filesystem API, create a new bug and discuss there. Keep this bug limited to the topic at hand.
What has been proposed sounds to me like it may be a clean
simple solution which not only provides support for maildir but
also includes an interface which may be leveraged as part of
solutions which provide support for other storage providers.

If within the context of the discussion of the maildir solution,
the developer of the solution is amenable to discussing the
suitability of the interface for other particular storage
backends, I would like imagine that would be encouraged,
regardless of whether or not the interface will eventually be
proposed as part of another solution, such as sqlite support, or
a pluggable mail storage API, without requiring that discussion
be moved to another bug, especially considering this bug is one
of the most voted for, yet has been left languishing for years.

I think many of the last comment authors fail to realize that the mail&news core code has not seen much (or any) real work for many years, while the Thunderbird people are busy with mostly bells and whistles. See my comment here:

http://wiki.mozilla.org/Talk:Thunderbird:Future_of_Thunderbird#Roadmap_suggestion:_Focus_on_the_backend
re comment 92, that's nice, but our codebase isn't compatible with GPL. If you have tri licensed maildir code you're welcome to try integrating it into Mozilla, if you only have GPL code, please don't bother mentioning it.

this bug is for work, not discussion. please don't comment unless you have a patch.
For me one of the strongest arguments for this feature (or choice) is to avoid
loss of email due to crashes. I get many thousands of messages a day, and
sometimes I just don't have time to keep up with archiving them in an orderly
way, especially since there is no automatic archiving function to do it for me.
Several times now my Inbox has gotten so large that Thurderbird crashed,
apparently from lack of sufficient memory, resulting in either a totally
unrecoverable Inbox file, or a damaged file from which only some of the
messages could be recovered. If I set it up for separate files, then the
crashes might be avoided, and if one occurred, I would only lose the last
message. I wonder how many others have had this problem.
+1 to define a generic API for pluggable storages. Maildir would be another user of that API. I could use a Berkeley DB one, for instance.
I've seen, that TB 3.0a1 is about to come out.
So this means, that this feature won't be in Thunderbird 3.0?
TB 3.0a1 very early pre-release, everything you see in it not final
Product: Core → MailNews Core
Dear Developers,

I have filed the bug listed above, that was identified as a duplicate of this bug. As I don't know how many of you use Macs, I just wanted to let you know that solving this bug and implementing a system that stores every mail in a single file would dramatically improve the everyday use of nearly every Mac user out there. Current versions of OS X perform an incremental backup of the entire system - always backing up those huge mail-folders just because of one email - every hour!
If you would want to gain user share on Macs, this would be a key feature to start at (next to implementing AppleScript support).

Thanks to everybody that helps solving this issue.

Best,
Phillip


If it can make things go on, here is my comment asking for maildir in Thunderbird for all improvements already described.

I think we already have lots of reasons for implementing this. The question now is how to do it.

We need an API and to make a plugin. As I read in an early comment, it seems that the API already exist?
One very important limitation about Maildir (and most other standard formats) is that attachments and other binary MIME parts are stored in Base64 (or another) encoding.

This makes it more difficult for metadata extraction engines to extract, for example, Exif data (or XMP) in case of for example image attachments.

Such engines collect this data for search purposes.

For example Tracker, Beagle, Strigi or one of the (other) Nepomuk-based projects.

Having to MIME parse and decode the cache of the E-mails first either requires decoding the parts to /tmp or to use (a lot of) memory for this task. Which doubles I/O compared to storing the attachments in "Save attachment as" format (just decode it).

Keeping it encoded also increases the diskspace and doesn't really add value. Except for a feature like "View source of E-mail". But such a feature can also be implemented by simply re-encoding the decoded-cache-stored parts into their original encoding while/before the screen opens that shows "the source of the E-mail".

MIME part viewers, like thumbnailers and inline image viewers, and inline attached images that are cid-referred in a IMG tag in a text/html MIME part of the E-mail (images in spam E-mail, for example) wont be as difficult to render once they are stored in the cache as decoded, too. Right now those must be decoded in memory before they can be rendered. Then they can just be rendered just like any other local JPEG file on the FS.

So, not sure if Maildir is actually an ideal format anyway.
(In reply to comment #109)
> We need an API and to make a plugin. As I read in an early comment, it seems
> that the API already exist?

Almost the entire difficult with fixing this bug at this point is because there is no real API with which to do this.
(In reply to comment #110)
> One very important limitation about Maildir (and most other standard formats)
> is that attachments and other binary MIME parts are stored in Base64 (or
> another) encoding.

Like mbox, the current Thunderbird storage format.

But this bug is not about how easy the mail storage can be used for other applications. It's primarily about how *safe* Thunderbird can store its data at all. From what I've read, quite a few of us (incl me) already had data loss because of corrupt mbox files that would not have occured with Maildir format.

> Keeping it encoded also increases the diskspace and doesn't really add value.
> Except for a feature like "View source of E-mail". But such a feature can also
> be implemented by simply re-encoding the decoded-cache-stored parts into their
> original encoding while/before the screen opens that shows "the source of the
> E-mail".

Do you believe that all messages are encoded in error-free MIME? I really would like to be able to view the real original source code of the message to find out why something didn't work or how something was actually intended. If I don't have this possibility, I could as well be using Microsoft products that don't show the e-mail source code.

> So, not sure if Maildir is actually an ideal format anyway.

It definitely is, for safety and performance. And it definitely is not any worse than mbox in the regards you're concerning about. So let's first get this Maildir thing done, finally, and then you can worry about doubling the data to make it more desktop search engine firendly.
I'd like to second all of what I can read in comment #112, and would like to add that one could probably lift a good deal of Maildir processing from other applications.
(In reply to comment #110)
> One very important limitation about Maildir (and most other standard formats)
> is that attachments and other binary MIME parts are stored in Base64 (or
> another) encoding.
> 
> This makes it more difficult for metadata extraction engines to extract, for
> example, Exif data (or XMP) in case of for example image attachments.

Apart from obvious security and performance issues connected with indexing engines: email is email is email. Problems connected with users using emails, powerpoint presentations and word documents for storing images and audio files should be mended primarily in education not in software.

> So, not sure if Maildir is actually an ideal format anyway.

For the purpose of indexing, I'd reckon it should do better than mbox. At least you don't have to rescan the whole file when change is made (which is quite often).
(In reply to comment #114)
> email is email is email. Problems connected with users using emails,
> powerpoint presentations and word documents for storing images and audio files
> should be mended primarily in education not in software.

Webpages are webpages are webpages.  Problems with developers wanting to update pages from the server except by using window.location should be solved by education and iframes, not new technology like XMLHttpRequest.

That's an awfully closed minded attitude.  Honestly, I don't know that I've ever met anyone who doesn't use their (offline except in the case of Gmail) email as a storage space.  Correct or not, it's an important use case that is extremely common.

Adding better facilities to find image attachments more quickly, and indexing these better, are definitely not bad things.

Unfortunately, email is the way it is, so storing it differently (not as base64) probably isn't going to work out well, as you say, especially for debugging.  That's not a reason to ignore other needs, though, and and when planning new storage methodologies, it's best to keep these sorts of needs (storing meta data for attachments, for example) in mind.

-[Unknown]
Please add maildir support - it is such an obvious and clear win to have this in addition to mbox. 

please .. pretty after 8 or more years please ... 

Is there any effort in this direction whatsoever ? 

little please ?

thanks
may-be this will finally get some attention when there is real competition like from google-chrome-mail beta ...
Gmail has been real competition since it was released...
I can't understand why there is resistance to developing maildir as an alternative storage format available.  After all Thunderbird is a currently mainstream mail client, and if google does develop a mail client alongside its Chrome browser, then unless Thunderbird matches performance it will lose market share of the mail client "market" surely?
I understand that people are frustrated that this hasn't moved forward much, and it's not exactly clear when we're going to find time to work on it.  That said, Bugzilla is intended to be used for the technical work of resolving bugs.  Using it for lobbying or discussion actually causes real development work to move more slowly.  Recent commenters, please review <https://bugzilla.mozilla.org/page.cgi?id=etiquette.html> before commenting again.  Thanks!
Such a discussion must be allowed!! It is direly important.
I think there have been so many unnecessary changes in Thunderbird 3.
This is what really is wasted time.- implementing unnecessary and undesireable features instead of important ones which keep to be asked for.
That is why I do also personally not use Thunderbird 3.
@Elmar: I am sorry to kick in, and while i find maildir to be "nice to have", I am sure many, many more people use "undesireble and unnecessary" TB3 features like quick fulltext search or the new, nicer and simplified user interface.

Anyway, everybody seems to agree, maildir would be nice, so no need to discuss that. If you want a certain feature quicker, it's better to send a patch than saying you don't use TB3. Telling the developers they just implemented useless things in TB3 probably won't encourage them to improve thunderbird.

Also Dan is right, this bug report with such things is not the right place to discuss the feature and sending out messages to all subscribers of the bug. I know, i just posted "spam" myself in the hope we can get back to technical discussions.
 Think about it as you like but please be open to discuss features here. We do not have any better place for it!
 (- and yes I am always glad to hear about users who conceive things in a similar way than me.).
(In reply to comment #125)
>  Think about it as you like but please be open to discuss features here. We do
> not have any better place for it!

Yes, you have. You have 

http://forums.mozillazine.org/viewforum.php?f=30&start=0

you have the various feature trackers of linux distributions:

http://brainstorm.ubuntu.com/

you can have your own blog, and you could setup a forum, mailing list or google group. In such places you can discuss all you want without doing anything useful but you do not annoy developers that are working and users that want to follow the development (and not pointless complains).

This place is for technical discussions, that means "code" (where is your code?). If you cannot contribute in any manner http://en-us.www.mozillamessaging.com/en-US/about/get-involved/ but you still complain, then you are harming the community. This is neither the place nor the manner. 

I am sending this here because this bug has attracted quite a lot of people that do not seem to realise they are not helping but actually hurting development, and I really wish there was a more clear warning message before each non-developer (or people with the warn-before-comment flag) could post each single comment. But I hope (if you care about Thunderbird at all) that any subsequent discussion is moved to private mail or elsewhere.
1) My coment earlier was a little incendiary - for which I apologize - but 8 years later for a very reasonable suggestion o improve TB enormously leads to some frustration. It would be nice to hear something definite about when this will be included in TB. 

2) mozillazine has nothing to do with mozilla - so it is not relevant. And there is no developer response there best I can.

3) Ditto for ubuntu (or any other non-mozillamessaging blog or whatever. 

4) bugzilla is very appropriate for RFE's ... not just bugs. This is an outstanding RFE

5) Some discussion of pros and cons of an RFE is appropriate.

6) There are many interested parties in this so of course there are a lot of email addresses. So? 

7) TB is the best mail client out there - this request is an attempt to help make it better ...
I plan on implementing a pluggable store interface, so that we can add support for different storage formats like maildir to Thunderbird, either in the core product, or as an extension.  There is more info here - https://wiki.mozilla.org/Thunderbird:Pluggable_Mail_Stores
David 

Thank you for that feedback and positive suggestions on how to actually proceed.

Has there been any progress since last fall ? What kind of resourcing is mozilla messaging attaching to the project? 

Thank you!!
(In reply to comment #127)
> 1) My coment earlier was a little incendiary - for which I apologize - but 8
> years later for a very reasonable suggestion o improve TB enormously leads to
> some frustration. It would be nice to hear something definite about when this
> will be included in TB. 

I hear you.  The apology is appreciated; thank you. 

> 2) mozillazine has nothing to do with mozilla - so it is not relevant. And
> there is no developer response there best I can.
> 
> 3) Ditto for ubuntu (or any other non-mozillamessaging blog or whatever. 

You're quite right that neither of these are entirely suitable.  Fortunately, as of Thunderbird 3, we now have a forum more suitable to less technical discussion and helping users get their needs met: <http://getsatisfaction.com/mozilla_messaging/topics/new>.
(In reply to comment #129)
> 
> Has there been any progress since last fall ? What kind of resourcing is
> mozilla messaging attaching to the project? 
Just little ol' me, unfortunately, but it has risen to the top of my non-bug fix priority list.

I'm hoping to rope in a contributor to do the actual maildir work, after I change the mailnews backend to use the pluggable mailstore interface instead of directly assuming berkeley mailbox format. So if anyone on the cc list is interested in doing that, please let me know!


- David
(In reply to comment #128)
> I plan on implementing a pluggable store interface, so that we can add support
> for different storage formats like maildir to Thunderbird, either in the core
> product, or as an extension.  There is more info here -
> https://wiki.mozilla.org/Thunderbird:Pluggable_Mail_Stores

I'd rather suggest an extended version of upasfs and so let mozilla just
operate on a defined filesystem hierachy.
An intriguing idea.  For the interested, http://lsub.org/magic/man2html/4/upasfs has more info about upasfs.   Given where we are on the process, the important thing you could do there would be to spend some time trying to figure out if the interface David has proposed on the wiki page would map well to implementing that way, and leave any comments on the wiki page.
I'm not sure how that would get us things like sqlite storage, but perhaps I'm missing something.
I was not familiar with upasfs and found very little on details in my casual googling ... 

That said,  upasfs does seem a little dated and Plan 9'ish ...  and was unclear to me anyway, how we might fit maildir under this. I'll let those that understand the details better think about that. Is there a windows implementation? Or OS X ? 

Is Plan 9 used anywhere ?????

I suspect we should stick with mainstream solutions and not wander too far to the periphery.
upasfs per the man page exposes mailboxes as a file-system.  In fact, based on other man pages, it would appear (at least by default) it actually operates on mbox files.

This is not the layer of the mailnews stack that is being discussed for refactoring.  It is conceivable that the lower levels of the upasfs implementation, assuming it supports things other than just mbox, may have routines that could be used as the basis for an implementation of the interface bienvenu is planning *assuming they are compatible with the tri-license*.

However, this has no impact on the actual interface bienvenu is planning.
(In reply to comment #136)
> This is not the layer of the mailnews stack that is being discussed for
> refactoring.  It is conceivable that the lower levels of the upasfs
> implementation, assuming it supports things other than just mbox, may have
> routines that could be used as the basis for an implementation of the interface
> bienvenu is planning 

Right, I was insufficiently clear about my intent here.  What I was trying to say to Enrico (or any other upasfs-interested folks) was that if they wish to investigate that further, it may be worth a few minutes to ensure that there are no impedance mismatches between what upasfs needs and what that interface provides to be sure that such an implementation is actually possible.

I wasn't suggested that we'd want to ship such an implementation as part of the core Thunderbird (though I suppose that's not technically impossible, it doesn't strike me as particularly likely).

> *assuming they are compatible with the tri-license*.

Such compatibility would only be required if we were interested in hosting that implementation in our core tree.

> However, this has no impact on the actual interface bienvenu is planning.

Except, as above, in the case of any impedance mismatches, which might be interesting to know about up front, if someone is sufficiently motivated to work on such a thing.
I would also be very interested by this functionality:
* Backups are far more efficient with maildir
* If something goes wrong, data corruption would only affect (at most) one mail instead of the whole mbox
* With a good FileSystem, there would be no noticeable waste of space (FAT32 days are gone, welcome Btrfs)
Apple Mail V1 used Unix Mbox format(multiple mails in a file, with "From -" as separator) as file for local mail folder.
Apple changed it to "one file per mail"(.emlx file) by Apple Mail V2. Then "Import from Unix Mbox file" of Apple Mail V2 imported only first mail in Unix Mbox file. So, Apple's official recomendation for importing mails was; Use IMAP (upload to IMAP, download from IMAP).

Space inefficiency issue with "one file per mail" doesn't seem critical if recent fast Mac/PC with sufficiently large hard disk.

Problem with "one file per mail":
  - Cost of physical file open upon each mail access.
    (If Unix Mbox, one physical open +  seek/read per a mail)
  - Cost of directory access when many many files in a directory.
    Some mechanism to limit number of files in a directory will be required.
  - Incompatibility with former versions of Tb and other softwares.
Fruit by "one file per mail":
  - Free from hard-to-avoid problems(nearlly imposible to avoid in some cases)
    produced by badly-designed "From -" line of Unix Mbox format.
  - Free from expensive/error-prawn "Compact" of Unix Mbox format file.
    (See dependency tree for bug 498274, please)
  - Easy bypass of file size limitation of 4GB/2GB due to 32 bits integer use
    for offset in Unix Mbox file.
    (for offline-store of IMAP, 4GB/2GB limit was removed by Bug 532323,)
    (by using 64bits integer use for offset in Unix Mbox file)
Biggest issue of "one file per mail":
  - Who will implement it, who can implement it.
    No one could provid design proposal, mock up, prototype, add-on like codes
    etc. yet...
(In reply to comment #131)
> I'm hoping to rope in a contributor to do the actual maildir work, after I
> change the mailnews backend to use the pluggable mailstore interface instead of
> directly assuming berkeley mailbox format.

What is the state of the pluggable mailstore?  Once you get one implemented and I get some free time, I was thinking of taking a crack at maildir.  The interface looks reasonable, though I don't know how TB works well enough to say any thing conclusive.  I'm not sure how you were thinking of creating a new folder with it, but I could be mistaken.
The pluggable mailstore work is plugging along. I'm hoping to get a repo setup today with the work I've done so far. My goal was to get it to be able to parse existing profiles, read mail, create folders, and download new pop3 mail, using the berkeley mailbox store plugin. I've written code for all that, and the first two work, but I haven't tested the last two. And yes, I did have to add a method to create a folder :-)

I've been in contact with someone else who was interested in doing a maildir implementation. Perhaps you could work together, once I get the core code using the pluggable store, and get a repo setup.
(In reply to comment #141)
> I've been in contact with someone else who was interested in doing a maildir
> implementation. 

That would be me, I guess. :)
So, TB3 already creates *.mozmsgs folders with the text form of each email in its own .eml file.  Does that buy us anything?
Assignee: nobody → bienvenu
No longer blocks: 402392
Depends on: 402392
Comment on attachment 116326 [details]
text mockups of folder settings

Canceling review request - this is really a question for ui review (how to chose mail storage formats) and that discussion will probably happen in the pluggable store bug that this bug depends on...I'm not sure I understand the other suggestion about changing the folder picker, tbh - a screen shot might be easier to understand.
Attachment #116326 - Flags: review?(neil)
This bug / feature request is now ten years old! Let's pop a bottle!
I think that maildir support is the only really necessary thing in Thunderbird. With maildir support Thunderbird will not have to use gigabytes of RAM while trying to work with my 3-gigabyte file in mbox format. It will not crash this file.
And you are spend over ten years to implement this feature!
Great work, guys!
Keep it going!
I just typed "aptitude purge thunderbird".
(In reply to comment #147)
> I think that maildir support is the only really necessary thing in Thunderbird.
> With maildir support Thunderbird will not have to use gigabytes of RAM while
> trying to work with my 3-gigabyte file in mbox format. It will not crash this
> file.

Thunderbird does not load the whole mbox file in memory, and maildir support will make no noticeable difference in memory usage for large mailboxes during normal usage.
What makes difference is when you do regular backups of your mails. Instead of having to save each time the whole mbox file which can sometimes be very big, you juste have to save the maildir files corresponding to new messages. It may save a lot of time and preserve your backup support lifetime.
(In reply to comment #149)
> What makes difference is when you do regular backups of your mails. Instead of
> having to save each time the whole mbox file which can sometimes be very big,
> you juste have to save the maildir files corresponding to new messages. It may
> save a lot of time and preserve your backup support lifetime.

There are lots of advantages to maildir, which is why I'm working on it. But memory usage is not one of them.
(In reply to comment #150)
> There are lots of advantages to maildir, which is why I'm working on it. But
> memory usage is not one of them.

David: do you publish some summaries of where you are with the implementation? Blog posts, write-ups?
(In reply to comment #151)
> (In reply to comment #150)
> > There are lots of advantages to maildir, which is why I'm working on it. But
> > memory usage is not one of them.
> 
> David: do you publish some summaries of where you are with the implementation?
> Blog posts, write-ups?

Every week in the status updates.
(In reply to comment #152)
> Every week in the status updates.

For anyone else who is interested, I think this is the relevant link:

http://benjamin.smedbergs.us/weekly-updates.fcgi/user/bienvenu/posts

Looks like the maildir pluggable store is basically finished and David is now working on a migration framework.
(In reply to comment #153)
> (In reply to comment #152)
> > Every week in the status updates.
> 
> 
> Looks like the maildir pluggable store is basically finished and David is now
> working on a migration framework.

It's not basically finished, unfortunately; it's just to a state where all the xpcshell tests pass, but our xpcshell tests are by no means exhaustive. I would say rather that it basically works.
Josh D S Davis: (2010-03-22)

I'm trying creating an account from scratch in TB3 to see if it makes these " .. *.mozmsgs folders with the text form of each email in its own .eml file."  I am not seeing this; it is making *.msf files etc.  I'm poking around and not seeing settings to store anything in .eml format.. am I missing something, or does it not do this?
(In reply to comment #144)

Whoops - meant to post that as reply, below:
https://bugzilla.mozilla.org/show_bug.cgi?id=58308#c155
Whiteboard: [gs]
Current version 7.0.1 has not mailfile support.

It would be really helpful, otherwise one new e-mail causes a change of an 1-4 GB file and that causes a full backup of this file.
If I could choice mailfile the backup would only include the new mail and the index file(s) changes.

Since 11 year we beg for this feature. Please provide it. Thanks!
(In reply to Klaus from comment #157)


Please see bug 402392 where this work is going on.
Just curios: why not just delegating all the mailbox handling to an tiny local imap server, which is just started by mozilla automatically ?
That would move a lot of problems out of the way (including the single-process/-core bottleneck).
Is this RFE merely a specific instance of the more general bug #402392?
(In reply to David E. Ross from comment #160)
> Is this RFE merely a specific instance of the more general bug #402392?

yes, essentially, though the patch in bug 402392 also implements a maildir-like pluggable store.
10 year old bug... it is time to delete me !:)
Are you also too old and thus should get killed?
A Bug doesn't need to be deleted or closed just because it is too old as long as the issues is still there unresolved.

Besides that, it was just days ago that Bug #402392 which this one depends on was closed. So only now this Bug can be addressed anyway.
(In reply to Björn Kautler from comment #163)
> Besides that, it was just days ago that Bug #402392 which this one depends
> on was closed. So only now this Bug can be addressed anyway.

The final commit 097bc36f180d in Bug #402392 contains maildir implementation. Maybe this could be marked as duplicate then?
The folks on 402393 say they've completed the backend but actual user functionality still remains to be done and is what *this* bug is about. So I am still wondering what date/version are users likely to see maildir support?
(In reply to joseph.acct from comment #165)
> The folks on 402393 say they've completed the backend but actual user
> functionality still remains to be done and is what *this* bug is about. So I
> am still wondering what date/version are users likely to see maildir support?

The backend API changes should be complete, there is still more backend work to be done so that users would be able to actually use a different format per account. That work is continuing, but until this stabilises, we won't be able to give any ETAs.
(In reply to Mark Banner (:standard8) from comment #166)

> The backend API changes should be complete, there is still more backend work
> to be done so that users would be able to actually use a different format
> per account. That work is continuing, but until this stabilises, we won't be
> able to give any ETAs.

Actually, it's front end work that needs to be done to allow the user to pick a different storage format per account, or globally. The backend already supports different storage formats per account.

There is backend work to be done to allow the user to migrate from one storage format to an other, which is perhaps what Mark is referring to. But for new accounts, you can use maildir if you like, though it's not well-tested.
I just downloaded earlybird but couldn't find out how to select maildir storage when creating a new account. Any hints would be appreciated! Thanks!
Please read comment 167. There is no user interface for this yet.
thanks, i did read comment 167, but i'm looking for how to do it manually.
Now that is a good question.
David Bienvenu, is it possible to somehow activate the maildir format, editing prefs.js or something?
Or is that considered also part of the missing UI?
You can set mail.serverDefaultStoreContractID = "@mozilla.org/msgstore/maildirstore;1" in about:config to use berkeleydb for new accounts. Default is "@mozilla.org/msgstore/berkeleystore;1", which is mbox.
Just to make sure I got this right.

To make it use "mbox" you configure "berkeleystore"
To make it use "Berkeley-DB" you configure "maildirstore"

So how do you configure "maildir" then? "mboxstore"? o_O
Ouch, got to sleep. "@mozilla.org/msgstore/maildirstore;1" is for maildir, of course.
Thank you! I just set maildirstore in prefs.js for 2 existing accounts (1 pop, 1 imap) after deleting the "Mail" and "IMAPMail" folders with the old mbox data and voila: Mails were redownloaded from the server and it works!
Great!
Whiteboard: [gs] → [gs][temporary manual configuration possible per comment 172 and 174]
there seems to be an issue with Trash folders, ie they are no longer recognized in the account, so deleting folders/messages fails.

str:
1.create feed account.
2.change the pref for the server to maildirstore.
3.restart.
4.no trash folder in folderpane, though there's a file on disk; new subscriptions/folders seem to work fine, just no deleting.
Removal does not seem to work when moving messages between folders, either
Depends on: 738651
My Linux distribution has Thunderbird 10.0.3 and I have that installed. However, it does not appear to support the prefs.js line mentioned above. Which release do I need to be able to test this? Thanks.
You'll need thunderbird12 or greater.
I've got maildir working for a new gmail account as described in Comment 172. However, in all cases only "cur" and "tmp" directories are created (for Inbox, Sent, All Mail, Trash etc); in no case is there a "new" directory, even though there are unread messages. Also, in each case the "tmp" directory contains all the message files except for one which is in the "cur" directory (this is a message with an attached file). I expected to find the unread message in "new", all the read messages in "cur", and probably nothing in "tmp".
My understanding of maildir format is that "new" makes more sense in the content of a mail server, not a mail client thing. "tmp" is for messages we're currently downloading from the server, and "cur" is where they get moved after being downloaded. "new" is for messages that the mail user agent hasn't processed yet, not for "unread" messages.
I tried implementing maildir about a week ago and about a month ago with the same results.  There were a couple of folders that would continue to download the same messages over and over.  Those messages would build up in the tmp folder with different file names but obviously the same message.  It was not creating duplicate messages in the message list though.  The total number of messages in cur and tmp did not agree with the total number of messages in the message list.

I am using IMAP connected to Gmail.  I can try again if there is some debugging information that I can provide.  You will just need to point me to how to get that information to you.
As requested I've entered this as new bug 748807.

I observe exactly the same as described in comment 183. With time there are more and more redundant messages in tmp.

With regard to comment 182, kmail has cur, new and tmp directories. Indeed, that appears to be the common format for maildir, e.g. see :

http://ws.edu.isoc.org/workshops/2005/pre-SANOG-VI/bc/mail/maildir.htm
(In reply to David :Bienvenu from comment #182)
> My understanding of maildir format is that "new" makes more sense in the
> content of a mail server, not a mail client thing.

There is no hard separation between server and client for maildir.

A maildir store can (and often is) be both accessed locally via a GUI mail client, and exported for remote accesses via imap (or imap + maildir)

And the maildir store must not behave differently depending on which method was used to read a particular message.
we're never going to put anything into the "new" directory, so there's no point in creating it. The point of maildir support for TB is to avoid the issues the berkeley mailbox format has, not to conform to the exact maildir "spec".
(In reply to David :Bienvenu from comment #186)
> we're never going to put anything into the "new" directory, so there's no
> point in creating it.

But there is no reason the maildir won't be fed by other processes too

> The point of maildir support for TB is to avoid the
> issues the berkeley mailbox format has, not to conform to the exact maildir "spec".

Not conforming to the spec will create interoperability problems. You can only not conform to the spec if you consider it a private store, and private stores are a PITA for users that use multiple mail clients (sorting mails in one client is not reflected in others, and you quickly forget which store has the message you need).

One of the strengths of maildir is safe access of a common store by multiple processes, it is a lot less interesting if you don't allow it.
I agree with the previous poster. If you're implementing maildir, you need to follow the spec accurately. After waiting 12 years since this feature request was filed, I'd hate to finally get maildir support only to find out it's not actually interoperable with standard maildir programs, or worse yet, that the data becomes corrupt when multiple programs are using the data store.
Count me in the "if you're going to do it, do it right" camp
(In reply to R. Steven Rainwater from comment #188)
> I agree with the previous poster. If you're implementing maildir, you need
> to follow the spec accurately. After waiting 12 years since this feature
> request was filed, I'd hate to finally get maildir support only to find out
> it's not actually interoperable with standard maildir programs, or worse
> yet, that the data becomes corrupt when multiple programs are using the data
> store.

OK, we're not implementing maildir, but the storage mechanism is pluggable, so someone else can implement maildir.
If it's not maildir, what is it? I don't know anything about maildir but why wouldn't we use the same conventions as other programs, assuming it doesn't complicate the code significantly?

So what do other mail *clients* do?
maildir is mainly a server format. Other mail clients use a single file per message format (e.g., apple mail, windows live mail) w/o getting caught up in all the arcana of maildir. I really just want the advantages of the single file per message format. I don't care about having other programs writing into our profile; we went down a giant rat hole in the early days of mozilla mail trying to support that with berkeley mailbox, and I don't want to repeat that mistake. But as I said, other people are free to implement a maildir pluggable store.
If Thunderbird will not be conforming to the maildir specification
like Mutt, KMail and Evolution, I hope there will not be anything
in the Thunderbird UI or documentation which leads to an impression
that conformant behavior is intended.
@David :Bienvenu: Please go and install mutt, which has supported Maildir (almost?) from day one as a local mailstore, which is typically fed by procmail or something alike, then please play around with it a little to get the feeling. Oh, and please also count me in on the wishlist for having those broken maildirs _NOT_ as private TB message stores, but conform to the standard (Maildir++ desired) if you call it Maildir. Otherwise, please call it Maildir-- or something, so one can distinguish it from real Maildirs. While I'm at it, there are also conventions for where to place such Maildirs, and that's not in ~/.mozilla/thunderbird/profile/Mail/account/folder, but in ~/Mail or something alike. It should, over time, become easy to use such (real) Maildirs, and also make proper use of indexing like eg. present in mutt.

The one reason why I currently prefer TB over mutt is that it doesn't crash so often when using IMAP.

http://www.mutt.org/download.html

Mutt appears to be in Macports. Try to get the 1.5.21 version if you can...
trust me, I'm going to call it something else. And I'm completely serious that if someone else wants to implement true maildir, they can. I've done all the hard work of getting the code to use the pluggable stores.
Wow. Looks like I still cannot move to Thunderbird any time soon. I have a *huge* local Maildir store that works with Kmail and Evolution that I do *not* want "privatized" just for one application. I *really* want a desktop agnostic GUI mailer (not Mutt) for Maildir that is not tied to one GUI camp or another. Oh well, guess I am stuck with the "competition" for now. :(
I'm just a very interested lurker on this feature so perhaps my comments/voice are not that important, so take this post with that a disclaimer and caveat.

That said, I've been reading some of the posts regarding the implementation of this with growing surprise, and wish to make the following observations, points and questions:

1) Posts which state that maildir is somehow only a "server" format misses (it appears to me) the point that maildir was typically used on servers **with multiple users**.  So while "the server" would put mail in maildirs, that same "server" would be used by end-users to read their mail.  This use case is no different today, except that we all have our own servers. ;)  So the fact it's used on servers is somewhat irrelevant to whether Thunderbird should be "well behaved" w.r.t. the maildir api/spec IMHO.  Interoperability with other mail software (whether "server" or "client") using the maildir standard is really what everyone that is clamouring for maildir is after I suspect.

2) Can someone (David?) please clarify what real reason exists for not just sticking to the specification?  What problems does sticking to the spec cause and why can't Thunderbird just comply with the spec?  Is there some sort of interface/api/impedance mismatch between Thunderbird's view of the world and the implied maildir view of the world?  If so what exactly is it?  If not, it seems to me that no credible argument can be made to not comply with the maildir standard/spec and in that case I'd agree with other posters: Either maildir id properly implemented in which case one can say Thunderbird supports maildir, or we call this something else and make it clear this is not maildir and won't interoperate with other software using the maildir standard.  In that case we're effectively inventing a new mailbox format using single files (inspired by maildir) and might as well consider what other features one might add to this to distinguish it from crufty old unix maildirs. (and, the title of this feature should probably not be "support qmail's maildir format").
(In reply to Walter Prins from comment #197)
> 2) Can someone (David?) please clarify what real reason exists for not just
> sticking to the specification?  What problems does sticking to the spec
> cause and why can't Thunderbird just comply with the spec?

The primary purpose of the current not-quite-maildir implementation is to provide a file-per-message collection of files. For internal architectural reasons, Thunderbird maintains all of the metadata for files in auxiliary files; changing this metadata to fully conform to the maildir spec(s) would require invasive changes to pretty much the entire Thunderbird codebase and are entirely impractical.

Furthermore, and more importantly, Thunderbird manages local folders under the assumption that no one else is modifying them. If the local folders were to be touched by other people, due to the aforementioned metadata mismatch issue, it is quite likely that users would experience mild to severe forms of dataloss. We do not support this use case and are not likely to support it because it quickly causes lots of issues.

> Either maildir id properly implemented in which case one can say
> Thunderbird supports maildir, or we call this something else and make it
> clear this is not maildir and won't interoperate with other software using
> the maildir standard.

We will be renaming the use of "maildir" internally, which is why I earlier said not-quite-maildir.

To my knowledge, the not-quite-maildir format that Thunderbird uses complies sufficiently well with the spec that it is possible to extract the list and contents of messages in a folder. This is not being guaranteed, but (if it is true and remains true) this would simplify importers. As the vast majority of users do not use Thunderbird concurrently with other mail programs with the same local store, there is no reason to greatly complicate the code to the service the needs of relatively few users. People are welcome, however, to write an extension that does this.
(In reply to Joshua Cranmer [:jcranmer] from comment #198)
> The primary purpose of the current not-quite-maildir implementation is to
> provide a file-per-message collection of files.

I've subscribed to this bug for this very reason, and I think this work would be a good starting point for a future compliant maildir implementation.
(In reply to Joshua Cranmer [:jcranmer] from comment #198)
> Furthermore, and more importantly, Thunderbird manages local folders under
> the assumption that no one else is modifying them. If the local folders were
> to be touched by other people, due to the aforementioned metadata mismatch
> issue, it is quite likely that users would experience mild to severe forms
> of dataloss. We do not support this use case and are not likely to support
> it because it quickly causes lots of issues.

But surely the plugable mailstore API everyone has been talking about would solve this? Isn't the whole point of the API to abstract the details of file, directory, and database handling and leave that up to the plugin code? All it needs to do is keep track of what mailstore corresponds to which internal "mail folder", why should it care whether a given folder's mailstore uses mbox, Maildir, an SQL database, or cuneiform tablets - and why care whether other agents are also accessing it? Does the mailstore plugin API not understand the concept of resource locking at all? Or does the API lack some kind of event notice telling Thunderbird the mailstore has been updated?

Anyone have a link to the source code for this part of the backend?
Joshua, thanks for responding so promptly.  Please allow me to then respond and ask as below:

(In reply to Joshua Cranmer [:jcranmer] from comment #198)
> (In reply to Walter Prins from comment #197)
> > 2) Can someone (David?) please clarify what real reason exists for not just
> > sticking to the specification?  What problems does sticking to the spec
> > cause and why can't Thunderbird just comply with the spec?
> 
> The primary purpose of the current not-quite-maildir implementation is to
> provide a file-per-message collection of files. 

This appears to contradict/conflict both with the title of this ticket as well as the use-case as explained in the first comment.  Where is the primary purpose for this ticket stated/defined that confirms your description above as being the correct purpose and clarifies that in fact this ticket does not have as goal the correct implementation of maildir to enable interoperation with other maildir application such procmail, qmail, fetchmail as mentioned in comment #1? (No offence intended but your comment just seems to me to just fly right in the face of what appears to be a clear original intent on this ticket.)

> For internal architectural
> reasons, Thunderbird maintains all of the metadata for files in auxiliary
> files; changing this metadata to fully conform to the maildir spec(s) would
> require invasive changes to pretty much the entire Thunderbird codebase and
> are entirely impractical.

OK... that is a great pity.   Just to confirm/clarify then, you seem to be saying that it's impossible to implement maildir currently, even with the pluggable storage mechanism,  due to invasive changes being required to the Thunderbird codebase?  Am I interpreting you correctly? What about the comments stating that someone else can implement maildir using an extension and/or via the pluggable storage mechanism?  Won't this also be impossible due to the change requirements mentioned above?

> Furthermore, and more importantly, Thunderbird manages local folders under
> the assumption that no one else is modifying them. If the local folders were
> to be touched by other people, due to the aforementioned metadata mismatch
> issue, it is quite likely that users would experience mild to severe forms
> of dataloss. We do not support this use case and are not likely to support
> it because it quickly causes lots of issues.

Do you mean concurrent access by different applications by the same user or using serially using multiple clients against the same maildir store?  (The former is not typically what a user would do and in any case one can state that as a caveat in the Thunderbird documentation, while the latter should not be an issue as long as on shutdown the correct data/metadata is written to disk.  Either way this doesn't seem to be a strong argument against proper maildir support.)

> > > Either maildir id properly implemented in which case one can say
> > Thunderbird supports maildir, or we call this something else and make it
> > clear this is not maildir and won't interoperate with other software using
> > the maildir standard.
> 
> We will be renaming the use of "maildir" internally, which is why I earlier
> said not-quite-maildir.
> 
> To my knowledge, the not-quite-maildir format that Thunderbird uses complies
> sufficiently well with the spec that it is possible to extract the list and
> contents of messages in a folder. This is not being guaranteed, but (if it
> is true and remains true) this would simplify importers. As the vast
> majority of users do not use Thunderbird concurrently with other mail
> programs with the same local store, 

Well given that Thunderbird doesn't really support local stores that might enable its users to use other mail program against the same store, it's not exactly surprising that the majority of Thunderbird users don't then in fact do this...   (E.g: Users of Thunderbird doesn't currently use multiple mail clients against the same local store because Thunderbird doesn't really support it...   Of course, having proper maildir support might actually change that.  ;) )

> People are welcome, however, to write an extension that does this.

Can you clarify how this comment relates to the codebase requiring invasive changes?  

Finally, to summarise, what I got from your email is that implementing proper maildir support will require invasive changes to the codebase, which makes it impractical for now.  As a result, the goal has implicitly along the way shifted from supporting interoperation with other maildir based programs as originally mentioned in the title of the ticket and early comments (such as procmail, fetchmail, qmail, mutt) to a reduced goal of supporting maildir partially, which still gives the benefit of single file based emails and the ability to import/export more easily to/from this format, but which unfortunately does not provide any level of interoperability with other maildir supporting programs.  Is this correct?

(Please accept my apologies if some of the questions or comments are perhaps naieve or noobish or appear appears impolite -- I have no familiarity with the Thunderbird codebase, and am responding to what appears to be inconsistencies and contradictions in the current path being taken, with a feature that I for one care about.  No doubt these apparent contradictions and inconsistencies are due to misunderstandings on my part, which is why I'm asking all these hopefully clarifying questions. No offence is intended.  With much thanks for reading and best wishes.)
I'd like to chime in. My main interest for the change is the ability to do incremental and reliable backup of the mail store. Full compatibility with the Maildir format could be an added nicety.
Weightin' in: my core requirement is having a mail storage format that can backup easily - without synchronizing gigabytes of data for each new mail. I left Thunderbird for Mail.app for this only reason.

Proper maildir standard support would be a nice thing, but really optionnal for me.
Good work and effort is underway on the pluggable store.
The Pluggable Store is a precursor to any sort of MAILDIR support.
This partial implementation of not-quite-MAILDIR is required for verification of the pluggable store.
Full MAILDIR is not required for testing the pluggable store.
The partial implementation could be further developed into a full implementation, but is not required at this stage.

This is still early stage, pre-release code.  I fail to see any problem with the current state.

Further, note that this bug is not the place for sarcastic comments.  Please keep it civil and constructive.

(In reply to eraccjcktn from comment #196)
> Wow. Looks like I still cannot move to Thunderbird any time soon. I have a
> *huge* local Maildir store that works with Kmail and Evolution that I do
> *not* want "privatized" just for one application. I *really* want a desktop
> agnostic GUI mailer (not Mutt) for Maildir that is not tied to one GUI camp
> or another. Oh well, guess I am stuck with the "competition" for now. :(
As to the pluggable store solving concurrent access issues, not really.  You still need triggers to detect when the underlying storage has been modified, so internal tables, views, indices, etc can be refreshed.  That COULD be added, but isn't a requirement for a pluggable store, nor for maildir support.

Maildir as a private repository is not the same as server-side maildir with client access.

That being said, it's a feature that would be nice to add to the pluggable store - detect message store modification times, and/or subscribe to files/folders at the OS level, and/or periodic rescans, to see if the underlying store has been modified, and then refresh the current view.

The options should be configurable, because a high volume store could receive updates faster than TB's view could be refreshed.
(In reply to Josh D S Davis from comment #204)
...
> Further, note that this bug is not the place for sarcastic comments.  Please
> keep it civil and constructive.
> 
> (In reply to eraccjcktn from comment #196)
> > Wow. Looks like I still cannot move to Thunderbird any time soon. ...

I apologize. There was no sarcasm intended. The idea was to point out what I am looking for with this feature and to express my disappointment with the direction this appears to be taking.
howdy y'all,

would it be useful to split this bug into two bugs? one for real, complete maildir support and another for single-msg-per-file? the 2nd seems to be the way things in this bug are going [and is what i most desire from this]. that would allow tracking real maildir for those who are needing or wanting that while the single-msg-per-file stuff could continue here. 

it would also remove the need for tmp/cur/new folders since the only current function of those seems to be read/unread status - which is handled in the X-Mozilla-Status line of each msg header.

take care,
lee
David has done a lot of work - for that we thank you very much. I think the ability to use maildir-light for all local storage is a huge huge win. 

If people really want to keep mail on a local computer then just run a local imap server - on linux this is completely trivial, and it is client indifferent. The idea of using client private store across clients is really not ideal. Dovecot is trivial to install and run for such a purpose.

TB still uses file store (inbox, copies of all mail for "offline" use etc) and it is here that the maildir-light will be a HUGE win.

I assume that after tutning this on that maildir-light will be the default store for all mail. So if I delete and re-create an account I will have no mbox files anywhere?
Please guys. See comment 161. David Bienvenu already calls the existing format a 'maildir-like'.

There exists only backend implemented (in bug 402392) to allow new storage formats to be added. And he implemented a 'maildir-like' format to test the infrastructure.

This bug here is for implementing STANDARD maildir format (or at least qmail's, I don't know the difference).

So, because there is no standard maildir yet, this bug is still OPEN (otherwise it would be closed or duped as already proposed earlier). Anybody is free to implement it. David just says it is not a necessity for him (or mozilla) for now.

The naming of that 'maildir-like' format will be resolved shortly so that no more comments asking why TB's 'maildir' is not conforming to standards are necessary.

Lee, the single-msg-per-file is already done elsewhere so no need to split this bug. There is no patch here. This bug is for the complete (standard) maildir support.
(In reply to :aceman from comment #209)
> Please guys. See comment 161. David Bienvenu already calls the existing
> format a 'maildir-like'.
> 
> There exists only backend implemented (in bug 402392) to allow new storage
> formats to be added. And he implemented a 'maildir-like' format to test the
> infrastructure.
> 
> This bug here is for implementing STANDARD maildir format (or at least
> qmail's, I don't know the difference).
> 
> So, because there is no standard maildir yet, this bug is still OPEN
> (otherwise it would be closed or duped as already proposed earlier). Anybody
> is free to implement it. David just says it is not a necessity for him (or
> mozilla) for now.
> 
> The naming of that 'maildir-like' format will be resolved shortly so that no
> more comments asking why TB's 'maildir' is not conforming to standards are
> necessary.
> 
> Lee, the single-msg-per-file is already done elsewhere so no need to split
> this bug. There is no patch here. This bug is for the complete (standard)
> maildir support.

Would you happen to have a bug number for the single-msg-per-file setup?  I would like to follow that one also.
(In reply to Jeff Grossman from comment #210)
> Would you happen to have a bug number for the single-msg-per-file setup?  I
> would like to follow that one also.

Per comment 161 it was done as part of bug 402392. That is why you can already enable it via steps in comment 172 and 174.
Whiteboard: [gs][temporary manual configuration possible per comment 172 and 174] → [gs][temporary manual configuration of a 'maildir-like' store is possible per comment 172 and 174]
FWIW, with proper Maildir support, the Mailboxen could be sync'ed to other servers with any IMAP client, not only with rsync or a tape backup program.
@ :aceman : The Maildir format was conceived and pioneered by qmail, and has since evolved to something called Maildir++, which supports quotas and sub-folders (at least, maybe more).
The original Maildir is some/directory/{new,tmp,cur}, and the Maildir++ adds 'maildirsize' for the quota, plus '.Sub.Folder.Capability', but I don't know where the specs are.
(In reply to gene c from comment #208)
> David has done a lot of work - for that we thank you very 
> If people really want to keep mail on a local computer then just run a local
> imap server - on linux this is completely trivial, and it is client
> indifferent. The idea of using client private store across clients is really
> not ideal. Dovecot is trivial to install and run for such a purpose.

Only if you like to watch your mail client spend minutes copying already local files to its own local store every time it is launched.

That's why I completely stopped using thunderbird. For all its UI quirks even going through a local webmail is better than watching thunderbird falling over trying to copy messages and headers to its own store. At least webmail + dovecot is smart enough to use the common maildir store directly. And be instantly available.
Depends on: 756432
(In reply to Joshua Cranmer [:jcranmer] from comment #198)

> The primary purpose of the current not-quite-maildir implementation is to
> provide a file-per-message collection of files. For internal architectural
> reasons, Thunderbird maintains all of the metadata for files in auxiliary
> files; .....

  So the 'metadata' is the result of 'indexing' the maildir files in e.g. Inbox?

> Furthermore, and more importantly, Thunderbird manages local folders under
> the assumption that no one else is modifying them. If the local folders were
> to be touched by other people, due to the aforementioned metadata mismatch
> issue, it is quite likely that users would experience mild to severe forms
> of dataloss. We do not support this use case and are not likely to support
> it because it quickly causes lots of issues.

  Are you saying that if I (manually or with rsync) changed a maildir store (e.g. Inbox) - by adding and/or deleting files - T'Bird would not be able to automatically
re-index them to update the 'metadata'?
  (KMail 4.8.2 does that, beautifully (but sadly its Find Messages is broken...)).

If T'Bird could, indeed, cope well with such adjustments to a maildir store, then I would seriously consider moving from KMail to T'Bird.
    (It has been T'Bird's insistence on Mbox that has deterred me, as I need my email kept in maildir format for the purpose of efficient backup/cloning with rsync.)

If not, then I shall have to stay with KDE 4.5.2's KMail until T'Bird does cope with adjustments to  maildir stores (or until KMail 4.8.2+ repairs its Find Messages!).
(In reply to Maurice from comment #215)
>   Are you saying that if I (manually or with rsync) changed a maildir store
> (e.g. Inbox) - by adding and/or deleting files - T'Bird would not be able to
> automatically
> re-index them to update the 'metadata'?

Yes, that is what is said here and previous comments. Thunderbird "maildir" is not a real maildir. So it does not (yet) have this capability. Maybe you could make it work for the case you describe by deleting the metadata files (.msf files) after you rsync TB's "maildir" store. But then starting TB may be slow, depending on the number of files, because it will need to reindex/rebuild the metadata files by reading all the message files.
(In reply to :aceman from comment #216)
> (In reply to Maurice from comment #215)

> ....Maybe you
> could make it work for the case you describe by deleting the metadata files
> (.msf files) after you rsync TB's "maildir" store. But then starting TB may
> be slow, depending on the number of files, because it will need to
> reindex/rebuild the metadata files by reading all the message files.

   As the changes in a maildir store would usually be minor (say 1%), that might not be a problem.
    KMail 4.8.2 has an option (enabled by default) to re-index the files, and in my experience it works very well and smoothly.
    If T'Bird were to provide a 'delete .msf files' option to facilitate automatic re-indexing, then maildir would become most usable! 

I don't fancy using T'Bird unless it provides something like that, rather than users having to mess around manually deleting what is hopefully the correct objects...

I guess it might take some time for the T'Bird designers' inhibitions w.r.t. to maildir to acquire a taste for the maildir world - that's understandable - and I look forward to being able to move over to T'Bird when the time comes.
(In reply to Jeff Grossman from comment #183)
> I tried implementing maildir about a week ago and about a month ago with the
> same results.  There were a couple of folders that would continue to
> download the same messages over and over.  Those messages would build up in
> the tmp folder with different file names but obviously the same message.  It
> was not creating duplicate messages in the message list though.  The total
> number of messages in cur and tmp did not agree with the total number of
> messages in the message list.
> 
> I am using IMAP connected to Gmail.  I can try again if there is some
> debugging information that I can provide.  You will just need to point me to
> how to get that information to you.

I'm observing the same. Something is wrong here. My tmp folder is growing and growing. It seems TB downloads emails again and again and puts it in the tmp folder every time I repair the folder.

Any reply from the dev regarding this problem would be nice as it seems that not only one person has this problem.
(In reply to mozilla.org from comment #218)
> I'm observing the same. Something is wrong here. My tmp folder is growing
> and growing. It seems TB downloads emails again and again and puts it in the
> tmp folder every time I repair the folder.
> 
> Any reply from the dev regarding this problem would be nice as it seems that
> not only one person has this problem.

It should be fixed in nightly and early bird builds.
(In reply to mozilla.org from comment #218)
> I'm observing the same. Something is wrong here. My tmp folder is growing
> and growing. It seems TB downloads emails again and again and puts it in the
> tmp folder every time I repair the folder.
> 
> Any reply from the dev regarding this problem would be nice as it seems that
> not only one person has this problem.

It was fixed in bug 748807.  Looks like it was fixed in Thunderbird 15.

I am waiting for bug 753624 to be fixed and then I will hopefully switch to Maildir-Lite.
Hello there.

I briefly tested this feature-to-be in a Ubuntu 12.04 x86_64 box with Thunderbird 12.0.1 installed from Ubuntu's 'precise' repository.

I had 3 accounts from the same domain configured as the default mbox configuration. Deleted them both from TB's configuration and the stored files, changed the options and tried to set them up again, as Mailbox-like configuration.

However, I noticed two issues.

(i) Randomly, some messages were trimmed (i.e., some of the last characters weren't stored/retrieved at all). I guess it is not a retrival issue, since I was able to get the same messages afterwards as a whole, by storing in the mbox-like structure.

(ii) I was not able to configure more than 2 accounts from the same domain. When I tried to configure the 3rd one, the message "Error Creating Account. Incoming server already exists." showed up. The domain needs some manual configuration, but I think it shouldn't be an issue in general, since mbox-like handles this.

For now, I got the default mbox setting. I would like to use a Maildir-like structure for my local backups out of the box, since it splits the messages as one-mail-per-file.

I thank you for spending time reading these issues.
(In reply to Alexandre Y. Harano from comment #221)
> Hello there.
> 
> I briefly tested this feature-to-be in a Ubuntu 12.04 x86_64 box with
> Thunderbird 12.0.1 installed from Ubuntu's 'precise' repository.
I'd prefer that you try Thunderbird nightly builds (16.0a builds) since there are ongoing fixes to maildir support there.
> 
> 
> (i) Randomly, some messages were trimmed (i.e., some of the last characters
> weren't stored/retrieved at all). I guess it is not a retrival issue, since
> I was able to get the same messages afterwards as a whole, by storing in the
> mbox-like structure.
That sounds like a missing flush issue. I've fixed some issues like that in nightly builds.
> 
> (ii) I was not able to configure more than 2 accounts from the same domain.
> When I tried to configure the 3rd one, the message "Error Creating Account.
> Incoming server already exists." showed up. The domain needs some manual
> configuration, but I think it shouldn't be an issue in general, since
> mbox-like handles this.

That has nothing to do with maildir vs. mailbox but rather, to do with the account code's fragility in deleting accounts and adding back the same accounts.
(In reply to David :Bienvenu from comment #222)
> (In reply to Alexandre Y. Harano from comment #221)
> > (ii) I was not able to configure more than 2 accounts from the same domain.
> > When I tried to configure the 3rd one, the message "Error Creating Account.
> > Incoming server already exists." showed up. The domain needs some manual
> > configuration, but I think it shouldn't be an issue in general, since
> > mbox-like handles this.
> 
> That has nothing to do with maildir vs. mailbox but rather, to do with the
> account code's fragility in deleting accounts and adding back the same
> accounts.

Even if I reboot before adding the accounts again?

Regarding the other answers, I will try the nightly builds. Thanks for the quick response!
(In reply to Alexandre Y. Harano from comment #223)
> > account code's fragility in deleting accounts and adding back the same
> > accounts.
> 
> Even if I reboot before adding the accounts again?
Restarting TB should be enough to clear its confusion.
I think David talks about bugs like bug 636017.
Although for this sort of testing I would recommend a separate profile anyway.
I changed the default to maildir and created a new profile. Now even the Local Folders have maildir. Then I imported mails from Outlook. The importation summary shows the quantity of messages imported correctly, but when I click on the imported items in Local Folders, no message is shown. If I look at the filesystem, I can see many maildir type messages piled up in the correct folder though.
So why doesn't Thunderbird show these messages imported into a maildir type Local Folders store? Shall I open a new bug for it?
If I import into normal mbox based Local Folder store, then it shows all mails correctly.
Yes please, file a new bug.
I've got a similar problem: Using TB Portable 17.0.2 in Win7 - running from a USB3 HDD. Also have the LocalFolder extension to allow multiple local folders in account settings.

I've changed the setting as per comment 172. Then created a 2nd local folder in the Account Settings dialog. Created an Inbox folder in this new "account". Selected some 14 emails from my main local's inbox and right-click-copied into that new Inbox. The subjects are listed if I open that inbox folder, but clicking an any one of them in the browser simply displays a blank preview (note even showing anything in the header).

So then I opened the physical path in Win Explorer:
G:\PortableApps\ThunderbirdPortable\Data\profile\Mail\MailDir\Inbox\cur

There are several files and folders in there, each named with some digit key (as I understand MailDir this should be something to do with the message ID). Anyhow, all these folders are empty, as are the files (they are each 0 bytes in size).
(In reply to Tomasz Ostrowski from comment #38)
> - We do not need folder compacting anymore

Traditional "Compact" of Unix Mbox format file/Berkley Mbox file is not needed by "one mail per a file" / "one file per a mail" by maildirstore in Tb.
(accurate used architecture name seems "Maildir Lite" instead of "Maildir")

However, "Compact" in IMAP is (a) "issue expunge commad to remove mail with \Deleted flag from server Mbox" + (b) "removal of non-referred mail data in offline-store file".
Even if mail data file is automatically deleted from "cur" directory by "Delete mail", mail data file should be kept when IMAP delete model of "Just mark it as deleted" until actually removed from Mbox at server.
And, even if the "mail of marked as \Dleted flag with Just mark it as deleted model" is automatically removed by Tb's actin of (a) or by "EXPUNGED" notification/response from server, there is no way to request (a) when madirstore is used for IMAP...

(Q1) What is reason why you removed "Compact" menu and disabled(grayed out) "Compact Folders" menu even from IMAP account?
No need of menu for explicit expunge request by user after maildirstore?

(Q2) When IMAP Mbox is changed from Offline-Use=On to Offline-Use=Off, useless "Mbox", "cur" under "Mbox", and obsolete files under the "cur" is not removed even after "delete Mbox.msf and restart Tb and Repair Folder".
When berkleystore, offlie-store file of IMAP Offline-Use=Off folder was deleted at least by Compact.
Is it intentional change in maildirstore from berkleystore?
(In reply to :aceman from comment #209)
> (A) Please guys. See comment 161. David Bienvenu already calls the existing format a 'maildir-like'.
>(snip)
> (B) The naming of that 'maildir-like' format will be resolved shortly so that
> no more comments asking why TB's 'maildir' is not conforming to standards are necessary.
>(snip) 
> (C) This bug is for the complete (standard) maildir support.

(D) System called "Maildir++" also exists.
(E) Bug opener of Bug 753624 calls Tb's the single-msg-per-file system, which seems currently named maildirstore, "Maildir Lite".
> mail.server.serverX.storeContractID = @mozilla.org/msgstore/maildirstore;1
> mail.server.serverX.storeContractID;@mozilla.org/msgstore/berkeleystore;1

What is actual offcial name in Tb?

The 'maildir-like' curretly used in module name is as follows.
Which is officially correct name in Tb?
MailDir? Maildir? mailDir? or maildir?

> http://mxr.mozilla.org/comm-central/source/mailnews/base/src
>   /mailnews/base/src/nsMailDirProvider.cpp
>   /mailnews/base/src/nsMailDirProvider.h
>   /mailnews/base/src/nsMailDirServiceDefs.h
>   /mailnews/base/test/unit/test_nsMailDirProvider.js
>   /mailnews/local/src/nsMsgMaildirStore.cpp
>   /mailnews/local/src/nsMsgMaildirStore.h
>   /mailnews/test/resources/mailDirService.js
>   /mailnews/base/src/nsMailDirProvider.cpp
>   /mailnews/base/src/nsMailDirProvider.h
>   /mailnews/base/src/nsMailDirServiceDefs.h
>   /mailnews/local/src/nsMsgMaildirStore.cpp
>   /mailnews/local/src/nsMsgMaildirStore.h
(In reply to Emre from comment #226)
> I changed the default to maildir and created a new profile. 
> Now even the Local Folders have maildir. Then I imported mails from Outlook.
> The importation summary shows the quantity of messages imported correctly,
> but when I click on the imported items in Local Folders, no message is shown.
> If I look at the filesystem, I can see many maildir type messages piled up in the correct folder though.
> So why doesn't Thunderbird show these messages imported into a maildir type Local Folders store?
>(snip)
> If I import into normal mbox based Local Folder store, then it shows all mails correctly.

If all mail data files are correctly created under <Mbox>/cur directry by Import, it's perhaps essentially same as known bug;
  After import of mail data, re-build index is not invoked by Import,
  or .msf file is not correctly initialized by Import.
  (see bug 478627)
When msgstore/berkeleystore;1 is used, non-intialized .msf produces outdated-msf condition, then internal re-build index is automatically invoked when folder is opened(clicked).
I guess logic around outdated-msf condition in msgstore/maildirstore;1 is different from msgstore/berkeleystore;1.

Is explicit "Repair Folder" of Folder Properties/General a workaround?

> Shall I open a new bug for it?

File name in <Mboxname>/cur directry is pointed by storeToken in StringPropery of msgDBHdr. If it's not set by Import, Tb can't access the file. If so, problem may be different from bug 478627.
Open separate bug for it, with [maildir] in Whiteboard: field, with reference to this bug, please.
No longer blocks: 856058
(In reply to Joshua Cranmer [:jcranmer] from comment #198)
> Furthermore, and more importantly, Thunderbird manages local folders under
> the assumption that no one else is modifying them.

Here we see one fundamental issue of TB: it assumes noone modifying its mailboxes and there's 
no (standard) way to inject messages into TB. This could naturally be implemented as an addon but I couldn't find one.

> If the local folders were
> to be touched by other people, due to the aforementioned metadata mismatch
> issue, it is quite likely that users would experience mild to severe forms
> of dataloss. We do not support this use case and are not likely to support
> it because it quickly causes lots of issues.

And this, I guess, Joshua's very cautious disclaimer.
Actually I do practice appending messages to TB's mbox file, but it requires `Properties -> Fix index` to be done to sync msf with updated mailbox. I guess maildir wont behave differently.
Two obvious flaws of this approach are
1. rebuilding msf being quite slow due to rereading of all messages in the box.
2. this works well only for injecting new messages. I am not sure other operations would not confuse TB to segfaults or such.
Is there a way by now to include a local folder, that is in maildir format? I would like tu use the maildir created by notbit: http://busydoingnothing.co.uk/notbit/
(In reply to alta88 from comment #177)
> there seems to be an issue with Trash folders, ie they are no longer
> recognized in the account, so deleting folders/messages fails.
> 
> str:
> 1.create feed account.
> 2.change the pref for the server to maildirstore.
> 3.restart.
> 4.no trash folder in folderpane, though there's a file on disk; new
> subscriptions/folders seem to work fine, just no deleting.

It turns out the problem is that existing /Trash and /Trash.msf files exist and the delete code fails for maildirstore as it expects /Trash/cur and won't autocorrect for this.  If /Trash and /Trash.msf are deleted, then a proper /Trash directory with /cur will be autocreated upon first delete and things will work.

This is likely true of any local folders which need to be directories with /cur instead, upon changing the contractid to maildirstore.
So what to do with this bug? We're going to ship what we have as our file-per-message implementation, and it is not possible to use it in conjunction with other mail readers. Thunderbird with its message summary file really does not support anyone else adding to the mail folders, and that is not likely to change anytime soon.

So what I'll do is keep this bug open, defining it as a request to support a message storage format that is compatible with third-part mail utilities that support qmail's maildir format. But it does not block our file-per-message implementation, so I am removing that dependency.
No longer blocks: maildirblockers
> ... it is not possible to use it in conjunction with other mail readers. Thunderbird with its message > summary file really does not support anyone else adding to the mail folders

My interest is not to use 'maildir'  Thunderbird with any other email reader, but in place of!

My current email agent is KMail, and it is impervious to sets of emails being added or deleted from a  maildir folder, which I occasionally want to do for housekeeping purposes

In the meantime, KMail's 'find messages' has now recovered to being usable, so I am not so urgently looking for a replacement (e.g. T'bird), and so can stay with KMail pro tem.

But I do like T'bird, and would consider jumping ship if the maildor handling could be generalised as above.
> it is not possible to use it in conjunction with other mail readers.

I've not been following development, and I'm sure it's too late now, but WHY did Thunderbird go and develop their own file-per-message implementation? There is one already out there that works pefectly: it's called Maildir. As Maurice mentions, it is used by KMail, and any other information KMail wants to store about a message, it stores in another location, not in the Maildir directory, not in the message.  In other works, it complies with established standards.

I'm disappointed that Thunderbird chose to go this route, as it will make migrating too/from Thunderbird very difficult, which means I'm that much less likely to try it, as I can't have it corrupting all my existing e-mail.
It *is* maildir, modulo a couple of changes to fit Thunderbird's architecture: see comment 198.
(In reply to Joshua Kugler from comment #237)
*First*
It is using MailDir, in that each file is the RAW email message. I.e. the ANSI 7bit character codes as sent in the email message. The first difference from a "normal" "naive" MailDir format is that TB uses an indexing system:

The trouble with just a normal MailDir format is the same trouble as TB's old MBox format - no indexing, thus very slow to find any one single message. Actually if you were (say) searching for a message with a specific subject, MailDir would be quite a lot slower than MBox since many more file handles would need to be opened and closed just to read each file to find that subject line. In MBox it just needs to search through the one file.

However most (if not all) email clients have some sort of indexing to make such searches much faster, no matter what underlying storage format they use. Even TB's old MBox uses Sqlite database to store indexing to the positions of the messages inside that file - i.e. search through a fast index, find position, jump directly to the message's content.

Now the issue is, since an index exists - does the software still have to go searching through each and every file in the storage folders just to make sure the user didn't go copy/rename/move/delete some file manually? This could slow down the software drastically - effectively nullifying any speedup from the index. IMO something like this should only happen through some manual "refresh" operation - e.g. a "repair-folder".

*Second*
Not sure exactly of TB's File-per-Message idea, or for that matter QMail's or KMail's. But most email clients implementing MailDir do so through a hybrid approach, except some which require a specialized file system to be installed in the OS (i.e. one which is geared to many small files instead of the more normal balanced FS's). The major trouble with MailDir is large numbers of small/short messages. It's usually extremely inefficient on those, so most clients save all messages smaller than a pre-set size to a "common" message file (very similar to the MBox format), only larger messages get to have their own file. See as example of one of these the MIX format: http://en.wikipedia.org/wiki/MIX_(Email)
(In reply to irneb from comment #239)
> (In reply to Joshua Kugler from comment #237)
> *First*
> (snip)

"Indexing" is done in both BerkleyStore and MaildirStore, and "Index" is held in MsgDatabase, and "Index information" is held in msgDBHdr for each mail.
Method to reach mail data.
  BerkleyStore : File name + Offset of mail data. Offset is saved in msgDBHdr => open file + Seek + Read
  MaildirStore  : Directory name + file name in the directory. file name is saved in msgDBHdr => open file under directory + Read(offset=0 always)
There is no difference in "time required to reach mail data"  between BerkleyStore and MaildirStore.
Once reached at top of mail data, there is no difference between BerkleyStore and MaildirStore.

What do you call by your "Indexing"?

"Indexing of word in message body text of each mail"?
If so, it's currently done by Global Search and Indexer(Gloda) in Tb.

> *Second*
> The major trouble with MailDir is large numbers of small/short messages.
> It's usually extremely inefficient on those, so most clients save all messages smaller than a pre-set size to a
> "common" message file (very similar to the MBox format), only larger messages get to have their own file. 

It's very pretty well known issue or difference between (a) "one file per a mail" and (b) "may mails in a file"
For "many small mails", BerkleyStore is still usable.
For "many big mails", MildirStore is already imlemented.
Although "both MaildirStore and BerkleyStore for an Mbox " is not supported, "MaildirStore for Gmail IMAP account" and "BerkleyStore for small pop3 account with small Mbox at server" is supported.

If "1KB mail * 1,000,000 mails", total size = 1GB. I believe BerkleyStore can be used for such case.
If POP3, Mbox can pretty easily be held under "Local Folders with BerkleyStore" or "dummy POP3 account with BerkleyStore".

FYI.
"Apple Mail" was (b) "may mails in a file" when "Apple Mail V1" but Apple changed it to (a) "one file per a mail"(they call .emlx) from "Apple Mail V2", and (a) "one file per a mail" is still used by Apple.
Do you heard criticaal issue of "extremely inefficient on those" due to (a) "one file per a mail" in "Apple Mail V2 or later"?
Like others here I think there should be an option to convert the existing accounts between mbox and maildir.
I started following this "bug" years ago, while trying to recuperate from the kdepim catastrophe.
After some time of energetic waiting, I realized that Tbird is a great program, but it does not have to do everything. Instead of bloating it, complement in with the functions you need, the *nix way.

So, I installed dovecot, kept enjoying Tbird, perfect, never looking back. I recommend.

We have the Thunderbird maildir version now. I don't think we'll be pursuing qmail compatibility.

Assignee: mozilla → nobody
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX

@Magnus: For the benefit of less-informed users, could you summarize - either here or via a link to some documentation:

  1. What's the difference between the Thunderbird and qmail maildir formats?
  2. Are they close enough for it to be possible to just point Thunderbird at a qmail maildir store and have that work?
  3. What tools, if any, are available for converting between the two maildir formats?

AFAIK, the differences is related to how status is stored, and that the storage locations in tb may not be exactly like elsewhere. The actual messages are just the same. So, there is no need for any tools to convert between the two.

Re #2 - likely for reading (didn't try), but that would likely mess up some things for other implementations. Sharing the maildir store between Thunderbird and some other application is explicitly not supported.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: