1540943 - Broken message body on forwarding a mail including invalid character

Status: RESOLVED FIXED

(MailNews Core :: Internationalization, defect)

Description

[YUKI "Piro" Hiroshi]

Attachment: sample.eml

Thunderbird cannot forward a message when the message has a quoted-printable encoded body with invalid character.

Steps to reproduce:

1. Open the attached file "example.eml".
2. Choose View => Message Body as => Original HTML or Simple HTML.
3. Then you'll see an email with a message body "にほんご".
   The first character is invalid, rest four character is a term
   meaning "Japanese" in hiragana.
4. Click the "Forward" toolbar button.

Actual result:

The editing message has a forwarded body as: "(B$B$K$[$s$4(B"

Expected result:

The editing message has a forwarded body as: "にほんご"

Environment:

• Ubuntu 16.04LTS, Thunderbird 60.6.1 (64-bit)
• Ubuntu 16.04LTS, Daily 68.0a1 (2019-03-31) (64-bit)

Comment 1

[YUKI "Piro" Hiroshi]

The message has a body "=1B(B=1B=24B=24K=24=5B=24s=244=1B=28B", and the part "=1B(B" is an invalid character.

Comment 2

[YUKI "Piro" Hiroshi]

I've tried with Thunderbird 52.9.1, and it works as expected: the forwarded body has the original body "にほんご" as-is. So this looks a regression.

Comment 3

[Jorg K (CEST = GMT+2)]

Sorry, I'm just going through some old bugs since the beginning of 2019. Yes, in a reply you get what's expected, but when forwarding, you get the raw undecoded (ASCII) text.

If that broke between 52 and 60 it's likely due to the switch to encoding_rs in bug 1363281 which happened during the 56 cycle.

Just for the record: Editing the message as new message (Ctrl+E) shows the same behaviour as Forward since it's a similar code path.

Comment 4

[Jorg K (CEST = GMT+2)]

Henri, here's the code that changed:
https://hg.mozilla.org/comm-central/rev/f558febc1ead57c76ba5eb5af4443a67680f7fa4#l3.173

We're now doing "without replacement", so it will just fail on illegal input. What did the previous code do?

BTW, in this case it's called from here:
https://searchfox.org/comm-central/rev/eb044a1bc9fca62c103b25e77dfccb9adcf8a9c4/mailnews/mime/src/mimedrft.cpp#1408
and that will just pump out ASCII as we saw.

Comment 5

[Jorg K (CEST = GMT+2)]

Attachment: 1540943-fix-bad-chars.patch

This fixes the issue. Somehow in the past in TB 52 nsMsgI18NConvertToUnicode() must have been more tolerant and returned replacement characters for bad input.

Comment 6

[Jorg K (CEST = GMT+2)]

Changing the summary: This has got nothing to do with quoted-printable, or am I missing something?

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/aeca286583a8
Use DecodeWithoutBOMHandling() in nsMsgI18NConvertToUnicode() to handle bad characters better. r=hsivonen

Comment 8

[Jorg K (CEST = GMT+2)]

TB 69 beta 4:
https://hg.mozilla.org/releases/comm-beta/rev/0cdb8a1fb8e8ed67be1fd89708e7d001542c562f

Comment 9

[Jorg K (CEST = GMT+2)]

TB 68.1 ESR:
https://hg.mozilla.org/releases/comm-esr68/rev/653aaac0fd02fcae12f6faac373bba096a91b034

Comment 10

[Jorg K (CEST = GMT+2)]

TB 60.9 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/d2ed16aca2b69a0132a993561ea521b4780a9d9e