Last Comment Bug 684508 - Line breaks in internet-cited text break UTF-8 quoted-printable characters
: Line breaks in internet-cited text break UTF-8 quoted-printable characters
Status: RESOLVED FIXED
:
Product: Thunderbird
Classification: Client Software
Component: Message Compose Window (show other bugs)
: 3.0
: x86_64 Windows 7
: -- normal (vote)
: Thunderbird 10.0
Assigned To: Diomidis Spinellis
:
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-03 08:50 PDT by Diomidis Spinellis
Modified: 2011-10-06 02:05 PDT (History)
2 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
The generated reply with the problem (7.31 KB, application/octet-stream)
2011-09-03 08:50 PDT, Diomidis Spinellis
no flags Details
Fix bug by replacing conservative line length limit with the exact value (596 bytes, patch)
2011-09-11 03:58 PDT, Diomidis Spinellis
mozilla: review+
Details | Diff | Splinter Review

Description Diomidis Spinellis 2011-09-03 08:50:38 PDT
Created attachment 558067 [details]
The generated reply with the problem

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.16) Gecko/20101125 Thunderbird/3.0.11

In a non-HTML reply to a message containing long lines with non-ASCII characters the internet-cited (>) line of the original text has a newline splitting UTF-8 quoted-printable characters in two.  This results in two illegal characters appearing in the mail message body.  In Thunderbird these appear with a question mark (?).  When Apple Mail displays such a message it fails to decode its contents as UTF-8 displaying it as gibberish.

Reproducible: Always

Steps to Reproduce:
1. Create a plain-text UTF-8 encoded message containing a line of 1000 non-ASCII characters
(e.g. αααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααα)
2. Send message to yourself
3. Reply to the message
4. Examine the sent message in the sent folder or the Outbox
5. The message will contain lines like the following
> ααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααααα�
�α 

Here is one line of the problematic internet-cited message, with the problem marked with "<------- ERROR"

> =CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=
=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=
=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE=B1=CE  <------- ERROR
=B1=CE=B1
>
Comment 1 Diomidis Spinellis 2011-09-11 03:58:17 PDT
Created attachment 559756 [details] [diff] [review]
Fix bug by replacing conservative line length limit with the exact value

This patch increases LINE_BREAK_MAX from 990 to 1000 - MSG_LINEBREAK_LEN i.e. to 998, which is the correct value.  The previous conservative value would break lines that were correctly crafted during preceding steps to be within the standard-imposed limit of 998 chars (plus CR/LF) by inserting unneeded line breaks.  These could be inserted at the wrong place, for instance breaking a UTF-8 pair in half.  EnsureLineBreaks doesn't have enough knowledge about the body's encoding, as it deals with plain bytes.  Therefore, it should avoid inserting line breaks unless absolutely necessary.
Comment 2 David :Bienvenu 2011-10-04 07:29:16 PDT
Comment on attachment 559756 [details] [diff] [review]
Fix bug by replacing conservative line length limit with the exact value

How common is this? It seems like it would be exceedingly rare.
Comment 3 Diomidis Spinellis 2011-10-04 07:53:06 PDT
(In reply to David :Bienvenu from comment #2)
> How common is this? It seems like it would be exceedingly rare.

I get it relatively often (20% of the time) when replying to email composed in Greek by an Apple MUA.  This MUA merges paragraphs sent in UTF-8 quoted printable flowed format into one long line, which then triggers the bug.
Comment 4 Mark Banner (:standard8) 2011-10-06 02:05:22 PDT
Checked in: http://hg.mozilla.org/comm-central/rev/1b02fbd2f3b0

Note You need to log in before you can comment on or make changes to this bug.