2055384 - Mbox file Content-Length header Mutt interoperability (mboxcl)

Status: UNCONFIRMED

(MailNews Core :: Backend, enhancement)

Description

[Erik Persson]

Steps to reproduce:

Thunderbird generally works well with mbox:es from other tools.
I created a mbox file using mutt containing a Thunderbird start of message marker in the message body, ie From - <datetime>CRLF, immediately after a blank CRLF line. The From line was NOT escaped and From was the first 4 characters on the line. Mutt adds a Content-Length header and is seems like Thunderbird honors this header - the mbox file, the message in question as well as other messages in the mbox file was correctly displayed in Thunderbird. Then I copied the messages to a new Thunderbird "folder" (mbox file). This also worked as expected except that the Content-Length header was neither changed nor removed.

I file this under "defect report" since even though this kind of interoperability is not a supported feature, as far as I know, the resulting mbox file is technically not correct.

Actual results:

Thunderbird did not remove nor change the Content-Length header when escaping a line starting with From in the message body.

Expected results:

I believe the Content-Length header should either be removed or changed to reflect the new Content-Length.

Comment 1

[Wayne Mery (:wsmwk)]

I could be wrong, but I wouldn't expect this to be a new bug/isssue, so setting it to an earlier version

Comment 2

[Max Emig [:maxe]]

We are using what is known as the mboxrd format which does not have a Content-Length header.

Mutt is using mboxcl which is a different variant.

Hence, this is a feature request not a bug.

Comment 3

[Erik Persson]

Hello again!

I know Thunderbird is not using mboxcl or mboxcl2.

I was also wrong when I said that Thunderbird honors the Content-Length message header and can read mboxcl and mboxcl2 mboxes. It seems to have been an edge case of the actual mboxcl2 mbox file I tested with and nothing that could be generalized.

I have now tested more extensively with multiple other mboxcl2 mboxes and Thunderbird does not read them as mboxcl2 mboxes but parses them as mboxo or mboxrd, ie splits one message into two if it encounters:

LF
CRLF
From - <datetime>CRLF
<Some message headers>

or

LF
LF
From - <datetime>LF
<Some message headers>

within a mboxcl2 message.

So Thunderbird does not at all seem to support parsing of mboxcl2 files. I was wrong. However one could argue that the fact that thunderbird leaves an existent Content-Length header in the message is a bug since it makes the mbox internally inconsistent, but, on the other hand, the Content-Length header is not in any RFC, as far as I know.

However it would be nice if Thunderbird supported reading mboxcl2 mboxes and then removed (or changed) the Content-Length header when rewriting them, so an enhancement request.
(However, it would be even better if Thunderbird gets ok support for maildirs that uses one of the maildir formats already in use, instead of implementing a new one with very limited compatibility with known maildir formats.)

Comment 4

[Max Emig [:maxe]]

Could you please describe your usecase end to end, which software in which order?

It might be tangentially related to bug 1802145.

Comment 5

[Erik Persson]

(In reply to Max Emig [:maxe] from comment #4)

Could you please describe your usecase end to end, which software in which order?

It might be tangentially related to bug 1802145.

This will be a bit convoluted and long.
I have used Thunderbird (TB) for about 30 years, since before Mozilla even existed, back when it was Netscape.
For the last approximately 15 years I have also used Mutt.
However, Mutt and TB have never shared any mboxes, and I have never moved or copied any mboxes between the two (aside from testing it, and then only on separate TB profiles). This is relevant to the discussion below.
TB has been the main and "historic" mail storage for emails that I want to preserve.

A couple of weeks ago I needed to sort and move emails in TB.
In the process of sorting and moving emails, TB, for the first time, destroyed mbox files - three mbox files, each containing around 3000 emails, were corrupted.
However this was not reproducible. I tried to find out exactly what caused the problem but I could not isolate the cause and furthermore the corruption occurred while using the somewhat older version of TB supplied by my distribution (and thus I did not report it).
Nevertheless, I needed to recover the mbox files.
In the process of recovering the mbox files I needed, among other things, another mbox reader, and for this I used Mutt.
I don't think that I could have managed to recover the emails, at least not convincingly, without another mbox reader.

However, since I knew that Mutt and TB don't use the same mbox format and I didn't know if TB could parse the mbox format used by Mutt, I needed to make sure that the emails in a mbox file were the same when reading the file with Mutt and with TB.
This would have been far easier if TB could read mboxcl2 files. In addition I needed to write a set of tools to validate the structure of the mbox files, the message count, and verify that some headers were only present at most once per message, etc. If TB could read mboxcl2 files some parts of this could have been avoided.
Since then I have found that there are some mboxcl2 to mboxrd converters (and now I have created one myself), but it would really be nice if TB could read other mbox formats.
And, as I noted yesterday, if an email contains a Content-Length header, TB will preserve it when writing the email to another mbox even if the message body is changed.
This may break mbox compatibility with mbox readers that use the Content-Length header but that otherwise have no problem reading TB mboxrd files.
If the formats don't agree and in this case TB may create a mbox file that is internally inconsistent when viewed with Mutt, then it may become harder to recover TB mbox files if they get corrupted. However, I did not encounter this problem when repairing the mbox files.
(Now I have created a small utility that removes all Content-Length message headers in a mbox file, but using all these tools is a bit messy.)

So in that sense it is related to bug 180214. However it has been many years since I tested movemail and I can't really remember how it worked, but generally TB will read any mbox file that is just dumped into the mail storage (unless it is a mboxcl2 file without escaped From lines in the message body).

So the use case is really:
Mboxcl2 compatibility is valuable when mbox files become corrupted and need to be repaired, but I can't give you the exact workflow and tools used since the process was iterative and a bit of a mess.

There are also multiple other use cases for better mbox compatibility and/or movemail, for example:

• Mutt is an excellent MUA and in some ways better than TB (for example it is far more flexible when it comes to choosing which mboxes/maildirs to use), but in other areas TB is better, for example TB is excellent for html emails - here interoperability would be nice (creating or copying a mutt mbox file to TB to view html emails).
• For reasons not important here I need to rsync over emails from my mail server to my "client" computer. Since Thunderbird has no way of getting the emails other than IMAP/POP/Exchange, unless you want to dump mbox files into the mail storage, and I don't want to run a full IMAP or POP server on the client, whenever I want to fetch emails I need to run courierpop3d as a local user, have a shim in front of it since TB insists on logging in, and then use socat to expose stdin on a port on localhost so TB can connect, be happy and fetch the emails. It would be FAR easier if TB supported movemail.
• When working on projects I generally want to have the emails associated with the project in a mail storage that is also associated with the project. To accomplish this I just use Mutt to save the emails to a mbox or a maildir where I have the other project files. When the project is done, I delete the emails from the main storage. Again, it would be nice to have mbox compatibility with TB so TB could also be used.

There are more or less hacky ways to solve the "problems" above using only TB (for example symlinks in the latter example and the whole setup in the 2nd example) but it would be nice if it were easier and TB could parse mboxcl2 files.