1737245 - Mojibake after forwarding message with defective charset

Status: RESOLVED FIXED

(Thunderbird :: Message Compose Window, defect)

Description

[Rachel Martin]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

The attached message is encoded in windows-1250 (central European) and is missing the charset information from the headers. It is displayed correctly, answering also works. Forward or "Edit as new message" (same code path, different to "reply") shows Mojibake, that is UTF-8 replacement characters <?>. It would be desirable to handle that better. In TB 78 it was handled better, however, TB 78 also didn't display the message correctly, instead it displayed some Cyrillic characters.

Comment 1

[Rachel Martin]

Attachment: spanish, no encoding, windows-1250.eml

Alice, could you see where the behaviour changed here. STR:
Import the message into a local folder (drag and drop). Forward.

What you get in TB 78 is different to TB 91. When did that change?

Comment 2

[Rachel Martin]

Sorry, "It is displayed correctly" is not true for TB 91, there the <?> are also displayed. It is only displayed correctly with the patch from bug 1734361 which corrects the regression mentioned in bug 1734361 comment #13.

Comment 3

[Alice0775 White]

(In reply to Rachel Martin from comment #2)

Sorry, "It is displayed correctly" is not true for TB 91, there the <?> are also displayed. It is only displayed correctly with the patch from bug 1734361 which corrects the regression mentioned in bug 1734361 comment #13.

So what should I do?

Comment 4

[Rachel Martin]

So what should I do?

Please find the change of behaviour when forwarding, STR in comment #1. Compare TB 78 to TB 91 to see what the difference. Maybe it's the same as in bug 1734361 comment #9.

Comment 5

[Alice0775 White]

Regression window:
https://hg.mozilla.org/comm-central/pushloghtml?fromchange=ea8e92388dfe64465640b7ee7e471083cf9bcfe2&tochange=a5ec7d54dd63c8c9d1e8ba34e32d5090eb8754a6
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=e581b02b5df54c1cdf194316dba74a5cf65145b6&tochange=68867f327c6267f971e6483443148a4025f94633

Comment 6

[Rachel Martin]

Thanks Alice, much appreciated, very helpful. In general, knowing where the regression came from frequently allows finding a solution quicker.

Comment 7

[Rachel Martin]

https://github.com/Betterbird/thunderbird-patches/blob/main/91/bugs/1737245-forward-charset-detect.patch

Comment 8

[BB]

Attachment: 1737245-forward-charset-detect.patch

To test this, import the attached message, view it (displays fine), then forward it.

Comment 9

[Magnus Melin [:mkmelin]]

Attachment: Bug 1737245 - Add charset detection also for the 'forward/edit as new' codepath. r=#thunderbird-reviewers

Non-test code based on https://github.com/Betterbird/thunderbird-patches/blob/main/91/bugs/1737245-forward-charset-detect.patch

Comment 10

[Magnus Melin [:mkmelin]]

Comment on attachment 9331204 [details] [diff] [review]
1737245-forward-charset-detect.patch

Looks good, I folded it the patch to test that case in phabricator.

Comment 11

[BB]

Thanks for adding the test. The non-test part is based on https://github.com/Betterbird/thunderbird-patches/blob/main/115/bugs/1737245-forward-charset-detect.patch, the 91 version was different.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/7784caaf3422
Add charset detection also for the 'forward/edit as new' codepath. r=leftmostcat