197733 - 1.3 mbox newline format mixes CR/LF, breaks compatibility with earlier Mozilla and other apps

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Jeff Mann]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.3) Gecko/20030312
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.3) Gecko/20030312

Similar to, but really the inverse of
http://bugzilla.mozilla.org/show_bug.cgi?id=156614

Apparently Mozilla 1.3 on MacOS X has changed from using old-style Mac CR line
termination to LF in the mbox files. On the one hand this is good because we are
on UNIX, and it provides compatibility with BSD mail and text tools. Apple
Mail.app also uses this format.

However, Mozilla 1.2.1 and earlier have difficulties with this format. Certain
multipart and/or HTML mail messages do not display correctly, the body appears
blank or shows only part of the message, attachments do not show up, etc.,
although there is no error message. This is particularly a problem since many
people still have dual-boot systems. For example, I upgraded to 1.3 on OS X, but
still sometimes need to access my Mozilla profile/mailboxes on OS 9 where 1.3 is
not available (and I assume there are no plans to port it), and now many of my
messages downloaded with 1.3 do not show up properly in 1.2.1. (I also
downgraded back to 1.2.1 since Themes don't work on 1.3; now I have to go into
the terminal and convert my mboxes back to CRs using tr). I can't really think
of a "fix" for this, other than to notify people in the release notes that there
is a new mbox format which is not compatible with earlier Mozilla versions,
Netscape, etc., and if they want to continue using 1.2.1 on OS 9, they should
*not* upgrade to 1.3 on OS X.

Secondly, rather than converting old mboxes to the new newline convention, 1.3
simply appends new LF-terminated messages onto the old CR-terminated mboxes,
creating files with mixed CR - LF conventions in different parts of the file.
This can't be the Right Way to do it! For example, BSD mail will read the mbox,
but only show the LF-terminated messages; the CR-terminated ones just don't show
up, and there is no error message. Nor can text editors deal properly with the
mixed files, even ones like BBEdit that auto-detect newline convention. Also, I
would assume that it would break other things like PERL scripts, conversion
filters, and other utilities one might want to run on mboxes, which could lead
to data loss. In other words, I think Mozilla should use one or the other
convention, and either work or not work, rather than mixing them together and
"sort-of" working with other apps. 

I think perhaps 1.3 should check and convert all old mboxes' CR->LF on opening
them, and people should be warned not to access their profiles with older
versions. Perhaps someone could whip up a little shell script to convert the
mboxes back to CRs in case someone needs to downgrade. Unless someone else has a
better idea...

I've marked this as Critical because I think that the mixed newline conventions
could lead to data loss if used in conjunction with other software.



Reproducible: Always

Steps to Reproduce:
1. Download mail with Mozilla 1.3; then quit.
2. Run Mozilla 1.2.1, read mail
3. Run "mail -f Inbox" in the terminal

Actual Results:  
2 - Mozilla 1.2.1 can't read many of the new messages
3 - BSD mail etc. reads some but not all of the messages

Expected Results:  
2 - warned not to use new mbox format with old software?
3 - created proper mbox with single newline convention

Comment 1

[David :Bienvenu]

not a mail database (.msf file) issue -> Seth - this might also be a dup.

Comment 2

[(not reading, please use [e-mail] instead)]

sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.

Comment 3

[Serge Gautherie (:sgautherie)]

Filter on "Nobody_NScomTLD_20080620"

Comment 4

[Nicolas Owens]

After running THunderbird for many years, I was looking at converting to APple Mail.  I imported all my old messages from Thunderbird, and I believe this issue has prevented most of the old messages (those with attachments/multipart in particular) from importing properly.  Although the message bodies appear in Mail, the attachments appear as long blocks of text.

What's odd/intriguing is that THunderbird is obviously backwards compatible with its own messages -- that is I have no problem looking at these ancient emails with .zip or other attachments in Tbird, but when the mbox files containing them are imported to Mail, the messages are broken.

I am not a coder, just a user, so I am voting for this bug.

-N

Comment 5

[Nicolas Owens]

Running Thunderbird 6.0.2 and Mail 5.1 on Lion 10.7.2.

Comment 6

[Jeff Mann]

A work-around is to manually convert the CR's to NL's, and see if that solves your problem. If not, then this is not the bug you're looking for.

There are many ways to do the conversion, using tr, sed, awk, perl, etc. There's a handy drag-and-drop utility here:
https://code.google.com/p/linebreak/downloads/list
...I haven't tested it but it looks like you'd set it up with "Protection On" and add the .msf extension to the list of files to exclude; convert line breaks to Unix "\n", and check Handle folders - recursively". Then you should be able to just drop your whole Mail folder on it to convert everything. But - do I need to say this? - make a backup first!!

It would still be really nice if this bug was fixed. It's close to ten years old now, and still biting people. Anyone converting mailboxes containing messages prior to 2004 or so is going to have trouble, and most people won't know why. It would be good if Thunderbird would automatically check for and convert CRs to LFs in mbox files on OS X.

Comment 7

[Jeff Mann]

p.s., you'd also want to add ".dat", ".html", and whatever non-mailbox files you might have in your mail folder, to the file extensions to exclude from processing...

Comment 8

[Nicolas Owens]

Does this belong in MIME or is it really backend?  There are many many open bugs over in MIME about multipart messages not rendering, not getting forwarded correctly etc.  THis bug seems to be about the actual storage of the mail in MBOX, versus its manipulation or transmission, but still...

-N