rename maildir pluggable store to something other than maildir

RESOLVED WONTFIX

Status

MailNews Core
Backend
RESOLVED WONTFIX
6 years ago
3 years ago

People

(Reporter: Bienvenu, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
Because our single file storage isn't "true maildir", we shouldn't be calling it maildir. I'll need to change the contract id, hg rename the source files, and upgrade existing profiles to use the new contract id, because there isn't any "true maildir" pluggable store out there yet. I'm not sure what to call the single file per message store but I'm leaning towards MsgPerFileStore.

Updated

6 years ago
OS: Windows 7 → All
Hardware: x86_64 → All
I like the name.

Updated

5 years ago
Blocks: 845952
"single-msg-per-file" is seen in some bugs.
David looks to call "maildir-like".
Bug opener of Bug 753624 calls "Maildir Lite".
String in prefs.js setting.
> mail.server.serverX.storeContractID = @mozilla.org/msgstore/maildirstore;1
> mail.server.serverX.storeContractID;@mozilla.org/msgstore/berkeleystore;1
MailDir, Maildir, mailDir, are used in module name under http://mxr.mozilla.org/comm-central/source/mailnews.
I'm recently using maildirstore or msgstore/maildirstore, berkeleystore or msgstore/berkeleystore, in bug at B.M.O in order to clearly state  what kind of mail folder.  

Objective is "one file per one message", so FilePerMsg or FileStorePerMsg is accurate expression, but actual implementation is "one message in one Berkley Mbox file" with "one Berkley Mbox file per one message" and with "multiple Berkley Mbox files in one directory"==Message folder in Tb.
I feel "MsgPerFileStore" is pretty good to avoid "true maildir" vs. "faked maildir" war.

Comment 3

3 years ago
(In reply to David :Bienvenu from comment #0)
> Because our single file storage isn't "true maildir", we shouldn't be
> calling it maildir. I'll need to change the contract id, hg rename the
> source files, and upgrade existing profiles to use the new contract id,
> because there isn't any "true maildir" pluggable store out there yet. I'm
> not sure what to call the single file per message store but I'm leaning
> towards MsgPerFileStore.

Is this in your mix?
Assignee: mozilla → nobody
Flags: needinfo?(kent)

Comment 4

3 years ago
Personally don't think this makes any sense. We should just have our maildir be as close to other implementations as possible.
(In reply to Magnus Melin from comment #4)
> We should just have our maildir be as close to other implementations as possible.

From perspective of directory structure, current Thunderbird's @mozilla.org/msgstore/maildirstore;1 is a subset of Maildir++ which is enhanced for subfolder support.
  Difference is ;
  - "<parentname>.sbd" directory is used to hold data of subfolder, instead of ".<subfoldername>" like directory in Maildir++.
  - ".../new" directory is not used because there is no need of ".../new" directory in Tb.
  - There is associated file named "<foldername>.msf" in Tb.
However, from perspective of mail data held under ".../cur" directory(and ".../tmp" ), there is big difference from Maildir/Maildir++.
  (a) File Name : filename doesn't have string for meta data such as download timestamp, server name.
  (b) .../cur/nnnnnnnn file content :
  (b-1) .../cur/nnnnnnnn as IMAP offline store file : 
            (b-1-i)  No "From -" line,
            (b-1-ii)  No escaping of "From " line in mail data by ">" or " ",
            (b-1-iii) No X-Mozilla-Staus: header etc. which is added by Thunderbird
            (b-1-iv) Additional CRLF is perhaps written at end by Tb,
                         even when IMAP serer doesn't send CRLF at end of mail data strean.
           This is same as Maildir/Maildir++ style.
  (b-2) .../cur/nnnnnnnn as local mail folder, newsgroup folder : 
            (b-2-i)   "From -..." line is written, which is Unix Mbox mail separator in BerkleyStore.
            (b-2-ii)   Escaping of "From " line by ">" or " " is done in mail data by Tb. 
            (b-2-iii)  X-Mozilla-Staus: etc. is written by Tb.
            (b-2-iv)  Additional CRLF at end, even when serer doesn't send CRLF at end of mail data stream.
This may be better called "Maildir Lite family".

To avoid confusion by utility/tool developers and users, one of next is needed :
  - Don't call our directory structure/mail data "Maildir".
  - Remove at least (b-2-i)  and (b-2-ii).
This is perhaps main reason why David tried to avoid using term of "Maildir".
If called "Maildir" or something like it, (b-2-iii) is better removed, and it's better moved to .../cur/nnnnnnnn.Status like file.

I don't believe that "implementation change" is needed merely for calling it "Maildir".
"Borrowing a part of directory structure only from Maildir" is wrong?
Is "actually subset of Maildir" or "actually enhanced Maildir" needed?

How about calling "MaildirLike"(change 't' to 'k'), "MaildirFake"(L->F, t->k)? -:)

Comment 6

3 years ago
Since I like naming, I suggest "quaildir", since a quail is a bird and, you know... Thunder_bird_...

(Also it's not as bad for PR as "faildir".)
I vote for "quaildir" if no trademark issue exists, although I prefer "faildir" because it's currently pretty accurate.
How about Component name when shipped="quaildir", Project code name while implemennting="faildir" :-)

Comment 8

3 years ago
I agree with Magnus in comment 4. We have much more important things to do at the moment than rename this type to something else, and we should use that energy to try to make it more closely match the format that maildir tools are expecting. maildir itself has evolved and is a mismash of things anyway, like note Wikipedia "The Maildir standard cannot be implemented without modification on systems that do not accept colons in filenames. This includes Microsoft Windows".
Flags: needinfo?(kent)

Comment 9

3 years ago
I think that's a false dichotomy; "effort" spent on naming things doesn't prevent any work on the actual implementation. And since this is Bugzilla, not a mailing list, bikeshedding on the names only spams people subscribed to the bug. :)
David :Bienvenu wanted to do next in comment #0.
  - change the contract id, 
  - rename the source files,
  - and upgrade existing profiles to use the new contract id,
because there isn't any "true maildir".
I think this is perhaps for:
  - Avoid "true maildir" vs. "faked maildir" war, which may be caused by string of "maildir" in contract id and source file name.
  - Avoid confusion by utility/tool developers and users which may be caused by string of "maildir" in contract id and name.
  - Avoid complaints by utility/tool developers and users which may be produced by confusions due to string of "maildir".

However. current contract id = @mozilla.org/msgstore/maildirstore.
It's called "msgstore/maildirstore" or "maildirstore". It's not called "Maildir".
And opposite is "msgstore/maildirstore"

Following is "Maildir" in WikiPedia.
> http://en.wikipedia.org/wiki/Maildir
> Mail readers
>     Thunderbird <= need references
Following is "Thundetbird" in WikiPedia which is pointed by above link.
> http://en.wikipedia.org/wiki/Mozilla_Thunderbird.
In this document, "maildirstore in Tb" is explained as follows
> File formats 
>     mbox   – Unix mailbox format (one file holding many emails)
>     maildir - known as maildir-lite (one file per email).
>                   Note: not yet stable, as of July 2014 Mozilla advise this format is still "too buggy for normal use"

I believe there is no need to remove string of "maildir" from contract id nor source file name, as far as msgstore/maildirstore is continuously used and "what is msgstore/maildirstore" is described at appropriate place, such as comment of source code, Mozilla Wiki.   
   msgstore/berkleystore :  Unix mbox style file. ... 
                                           Note: if IMAP offline-store file, it's not "pure Unix Mbox format".
                                                     "escape of From line" is not done, because re-parse of offline-store will be never executed.
   msgstore/maildirstore  :  one file per a message.
                                           Similar directory structure to Maildir Lite, Maildir, Maildir++.
                                           Unix mbox format is used for the "one file per a message".
                                           Note: if IMAP offline-store file, it's not "pure Unix Mbox format".
                                                     "escaoe of From line" is not done, because re-parse of offline-store will be never executed.
                                           Note: if IMAP, and if folder is never used for "Offline Use",
                                                     there is no difference between msgstore/maildirstore  and msgstore/berkleystore.
                                                     because offline-store file is not used.  

FYI.
Apple already introduced "one file per a message". Apple changed "Unix Mbox format file" in Apple Mail V1 to "one file per a message" from Apple Mail V2.
Apple's choice was : Call the "one file per a message" "emlx file"(eml file extended), with adding file extension of ".emlx" to each file.
FYI.
If change like next will be made, string of "maildir" can be removed from anywhere in Thunderbird, although it's tough work because "From " line related code should be moved to BerkleStore code and "From " line handling should't be exposed to higher level.
>                       Unix Mbox format file                  One file per a message
>  MsgDB             <FolderName>.msf file              <FolderName>.msf file
>  msgStore file   <FolderName> file                     <FolderName> directory
>                                                                             <FolderName>\nnnnnn.EMLX file <= changed
>                                                                             Don't write "From - ..." line at top of file as this is .eml <= changed
>                                                                             Don't add additional CRLF at end of file as this is .eml <= changed
>                                                                             => almost same as imap offline-store of berkleystore.
>                                                                             => they should be added upon copy to berkleystore/local maol folder,
>                                                                                   as currently done upon copy from IMAP folder to local mail folder.
>  for subfolder    <FolderName>.sbd directory     <FolderName>.sbd directory

By the way, I can't understand why <FolderName>\cur and <FolderName>\tmp are mandtory in Tb even though <FolderName>\new is never needed in Tb.
(In reply to David :Bienvenu from comment #0)
> but I'm leaning towards MsgPerFileStore.

David :Bienvenu, do you still think "change of contract id" is mandatory or needed?
Can we close this bug as WONTFIX?
If contract id will not be changed, are we better to open bug for documentation about berkelystore and maildirstore?

(In reply to Magnus Melin from comment #4)
> We should just have our maildir be as close to other implementations as possible.
(In reply to Kent James (:rkent) from comment #8)
> I agree with Magnus in comment 4.

Magnus Melin and Kent James, should we modify or alter our maidirstore implementation?

I don't believe you both think so, but your comments to this bug sound claim of "maildirstore in Tb should be as close to spec of MailDir as possible".

Comment 13

3 years ago
(In reply to WADA from comment #12)
> Magnus Melin and Kent James, should we modify or alter our maidirstore
> implementation?

I see no technical reason not to.
As for the differences you list in comment 5: case a) should be easy, don't know about case b) but doing those changes should clear up all the bugs we have about From escaping.
(In reply to Magnus Melin from comment #13)
> (In reply to WADA from comment #12)
> > Magnus Melin and Kent James, should we modify or alter our maidirstore implementation?
> I see no technical reason not to.

I'm never opposing modification/improvement of maildirstore, but I believe that objective should not be "in order to get true MailDir in Tb".
I believe we can freely improve our maildirstore, as you say, regardless of MailDir's spec, regardless of existent utities/tools for MailDir, even if we continue to use string of "maildir" in contract id, even if we borrow design from MailDir.
I also think maildirstore is a big chance for us to get rid of "dirty Unix Mbox format design".

Comment 15

3 years ago
I don't really know enough about third-party maildir tools to comment much, but I do know that any tool that manipulates the maildir directories would cause lack of sync with the message summary database - which is effectively dataloss from Thunderbird's perspective.

Actually this bug is about renaming, not about changing the implementation. We should probably discuss that in another bug.

I just don't think the name is important, and would rather not expend any effort changing it. "maildir" implies to people file-per-message so has some justification as a name. Beyond that I don't see the point in rethinking the decision.

Comment 16

3 years ago
(In reply to Kent James (:rkent) from comment #15)
> I just don't think the name is important, and would rather not expend any
> effort changing it. "maildir" implies to people file-per-message so has some
> justification as a name. Beyond that I don't see the point in rethinking the
> decision.

I think the implication is there was not a "thoughtful decision", or even a decision as such but a convienent name, when the name was chosen. Hence the bug report.

The name may or not be important in the context of the source code. But what is certainly important, is the name we use when we tell the public and the press what has been implemented.

Comment 17

3 years ago
"But what is certainly important, is the name we use when we tell the public and the press what has been implemented." So this is a marketing question?

Wikipedia introduces Maildir as:

"The Maildir e-mail format is a common way of storing e-mail messages, where each message is kept in a separate file with a unique name, and each folder is a directory. The local filesystem handles file locking as messages are added, moved and deleted. A major design goal of Maildir is to eliminate program code having to handle locking, which is often difficult."

Our implementation meets this. From a communication perspective, I think it is fine to call it maildir, along with some qualifications that it has been adapted to the specific needs of Thunderbird, and is not designed for interoperability with maildir utilities.

I also think we should keep the name the same in the code, with the understanding that this is currently our partial implementation of maildir, and ideally we would like to move that implementation toward being as complete and interoperable as reasonable given the structure of Thunderbird.

So we want to communicate that we have a file-per-message store, and the best way to do that is to say that we have implemented maildir (qualifying that in notes). Giving it a different name would only serve to confuse users. Very few people are concerned with interoperability with maildir tools.

Comment 18

3 years ago
Both marketing and expectation.  And "not confusing" as you say is the key. If chosen term is not confusing then we're good as far as I am concerned.  I can't speak to the point you make about interoperability.
(In reply to Wayne Mery (:wsmwk) from comment #18)
> I can't speak to the point you make about interoperability.

An example : interoperability with tool like maildir2mbox, mbox2maildir.
   http://manpages.ubuntu.com/manpages/precise/man1/maildir2mbox.1.html
   http://linux.alanstudio.hk/mbox2maildir
maildir2mbox assumes no "From - ... line as mail separator of Unix Mbox format" in Maildir file(/cur, file, /new file), and escapes "From " line in mail dara stream by ">From ".
In Tb's maidirstore for local mail folder,  "From - ... line as mail separator of Unix Mbox format" is written at top of file.
So, if maildir2mbox is used by user for Tb's maildiestore, wrong messeage header of ">From ..." is generated at top of each mail in created Mbox file.
If user find it, the user surely says "Tb's bug!!!".

I believe it's pretty rare case and "removing From line from our /cur file" is not so hard job, but I think that at least "maildirstore of Tb !=== true Maildir" should be stated at somewhere.

Anyway, as far as we continue to use "/cur directory", it looks that we can call our "msgstore/maildirstore" "Maildir family".
To all comment posters : Can we close this bug as WONTFIX?
David : Do you agree on WONTFIX?

Comment 20

3 years ago
So what to do with this bug? At least on the commenters here, I am in a minority who does not want to put effort into renaming this. Yet I think the ship has sailed for Thunderbird 38, and we are using "File per message (maildir)" as the name for this in the UI. We are not going to block shipping this feature based on a decision to rename, so I am removing this from the maildir block list.

I think that in any discussions that we have about this feature publicly, we will call it primarily file per message, but also mention maildir with the caveat that it is not intended to be compatible with third-party tools.
No longer blocks: 845952

Comment 21

3 years ago
I think keep the name, document that it's a slight modification of maildir, fix incompatibilities where possible. This doesn't mean it needs to be usuable from many clients, but obviously nice if e.g. import/export can work with existing tools. It's not like we ever renamed mbox even though tb doesn't use the standard format precisely.
(In reply to Magnus Melin from comment #21)
> but obviously nice if e.g. import/export can work with existing tools.

I think following is better done.
   (a) For Unix tools, remove excess "From -" lne. Essentially, there is no need of it for MaildirStore too, because /cur/xxx file is never Unix Mbox file.
   (b) For ImportExportTools, if required, do something for better co-operation with ImportExportTools.
Are we better to open bug for (a)?

Comment 23

3 years ago
(In reply to WADA from comment #22)
> Are we better to open bug for (a)?

Yes, please open a bug for that if there isn't one already.

Comment 24

3 years ago
I agree. done.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.