1134690 - Thunderbird prepends greater than symbol or space to any line starting with From: >From (escape character for mbox format), breaking patch attachments

Status: RESOLVED DUPLICATE of bug 121947

(Thunderbird :: Untriaged, defect)

Description

[neilchittenden]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Firefox/35.0
Build ID: 20150125222008

Steps to reproduce:

- Start Thunderbird in safe mode
- Attach a git patch (from 'git format-patch') to a new (plain-text) email
- Email to self
(not sure whether it's outgoing or incoming mail that's causing the issue)


Actual results:

Saving the attached patch, the first 'From' line per commit in the git patch has had a greater than '>' symbol prepended, making the patch corrupted (although easy to fix by replacing '>From' with 'From'). This is also the case on viewing the email message source.


Expected results:

The patch should be saved without having been corrupted.

Comment 1

[neilchittenden]

- Tested sending from Thunderbird to a gmail account, and it appears fine in gmail.
- Also tested sending from Thunderbird and checking mail server webmail to confirm it's not the mail server.

The above tests point to Thunderbird's processing of incoming email as being the issue.

Comment 2

[neilchittenden]

This seems related to the encoding and file extension. The above issue happens when the patch has a .patch extension, but if it is renamed to have a .txt extension it is fine. The difference in encodings in the incoming message source is as follows:

.patch extension:

Content-Type: text/x-patch;
 name="tests_2014-11-27-1.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="tests_2014-11-27-1.patch"

.txt extension:

Content-Type: text/plain; charset=UTF-8;
 name="tests_2014-11-28.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="tests_2014-11-28.txt"

Comment 3

[Thomas D. (:thomas8)]

Neil, thanks for this bug report.

I think there's something in the internal format of TB's mailboxes which adds a first line starting with "From" to delimit messages. Then, all the other lines starting with "From" are "escaped" by the > sign.
Which really looks like a very poor design.

But it's really surprising that it can also affect plaintext attachments, to the extent of altering their content. Shame.

Joshua, can you comment?

Comment 4

[Joshua Cranmer [:jcranmer]]

(In reply to Thomas D. from comment #3)
> I think there's something in the internal format of TB's mailboxes which
> adds a first line starting with "From" to delimit messages. Then, all the
> other lines starting with "From" are "escaped" by the > sign.
> Which really looks like a very poor design.

The mbox file format uses the presence of "From " to determine the delimiter between mail messages. This produces a problem when a body line has that text. The solution was to munge the line to have ">From ", which then raises the question as to what to do if ">From " was originally in the body. This gives rise to mutually-incompatible mbox formats, mboxo (where only From is munged), and mboxrd (where /^>*From / is munged). In the case of mboxo, it is difficult to recommend de->-ifying the From line, as well as difficult to recommend not doing so--in any scenario, you are potentially causing some sort of irreversible munging, which potentially causes issues in scenarios like cryptographic signatures.

Thunderbird's implementation is mboxo for, well, ancient historical reasons. In theory, maildir does not contain the same problems; in practice, we probably share enough code paths that the munging goes on some place. Without a thorough audit of all relevant code, it's hard to say for certain what's going on.

> But it's really surprising that it can also affect plaintext attachments, to
> the extent of altering their content. Shame.

As far as mbox is concerned, an attachment is the part of the body: if it affects the body, it affects the attachments.

In truth, we probably need to consolidate all of our >From bugs into one.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Joshua Cranmer [:jcranmer] from comment #4)
> In truth, we probably need to consolidate all of our >From bugs into one.

FYI.

IIRC, "From " line is escped by ">From " or " From " in BerkleyStore mbox fille for local folder.
And IIRC, in Mozilla MailNews, "> for escaping of From " was removed upon mail viewing.
However, IIRC, after "escape by space of From line" was introduced for format=flowed, "removal of escape char for From line" was lost.
If imap offline-store file, even though Unix Mbox format is used, "From " line is not escaped, because "Reparsing of imap offline-store file" won't be invoked by Tb.

To get rid of "design flaw in Unix Mbox format", we already provided MaildirStore(one file for a mail), althugh MaildirStore support is still under construction.