1217161 - Mojibake with Chinese characters in email body (support cp936 encoding as alias of gbk)

Status: RESOLVED FIXED

(MailNews Core :: Internationalization, defect)

Description

[Rachel]

Attachment: thunderbird_mojibake_1c.png

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0
Build ID: 20151006000732

Steps to reproduce:

When receiving email with Chinese text in Thunderbird, it sometimes shows up as garbage/mojibake. 

I'm uncertain the exact cause, but it seems related to messages encoded in cp936.  


Actual results:

On several separate occasions, receiving email that contains Chinese text, the results have been garbage/mojibake when viewed in Thunderbird. 

This is true both for messages received via an IMAP (Yahoo) server and via a POP server. 

With these mojibake messages, the View > Character Encoding menu item is grayed out and unavailable. With other messages, even received from the same senders, the Character Encoding menu item is not grayed out and is easily accessible.

Checking the full headers, all the mojibake messages appear to have been encoded in cp936, which is (according to Wikipedia) "Microsoft's character encoding for simplified Chinese". One message contained Simplified Chinese characters, another contained Traditional Chinese characters, and a third I am uncertain of. All are listed in their headers as cp936. I don't have mojibake problems with messages listed as other encodings such as Big5, GB2312 or UTF-8.

Also, the message I received in Traditional Characters, which is via my IMAP (Yahoo) account, shows correctly in both my webmail (via Firefox) and in my cellphone's email app. It only appears as mojibake in Thunderbird.

I'm using Thunderbird 38.3.0 under Linux Mint 17.1. 

Perhaps this is an error in how Thunderbird handles cp936? And perhaps it's related to this bug https://bugzilla.mozilla.org/show_bug.cgi?id=1176662 ?


Expected results:

Messages should contain readable, non-garbled Chinese.

Comment 1

[Masatoshi Kimura [:emk]]

Encoding Standard does not define "cp936" as an alias of gbk.
https://encoding.spec.whatwg.org/#concept-encoding-get

Thunderbird should handle this alias before using the Core encoding converter.

Comment 2

[Rachel]

I haven't seen any movement on this in the past month or so. Should I be supplying additional information, or...?

Comment 3

[Magnus Melin [:mkmelin]]

Attachment: bug1217161_cp936_support.patch

Looks like it might be commonly used (at least in general, dunno about email), so we could add the alias.

Comment 4

[Jorg K (CEST = GMT+2)]

Any chance to get this one line addition reviewed so we can get this into TB 45. Doesn't look like a very contentious change.

Comment 5

[Kent James (:rkent)]

Comment on attachment 8702994 [details] [diff] [review]
bug1217161_cp936_support.patch

Review of attachment 8702994 [details] [diff] [review]:
-----------------------------------------------------------------

I don't claim to be an expert here, but a little searching convinced me this is probably fine. Let's just try it.

Comment 6

[Jorg K (CEST = GMT+2)]

Comment on attachment 8702994 [details] [diff] [review]
bug1217161_cp936_support.patch

Magnus, I assume that you want to request uplift here, right?
We sort of promised the feature to the reporter of bug 1235294 (see bug 1235294 comment #4).

[Approval Request Comment]
Regression caused by (bug #): Not a regression.
User impact if declined: Unhappy Asian users, cp936 seems to be out in the wild.
Testing completed (on c-c, etc.): I assume Magnus did manual testing. I'll try it once it's landed.
Risk to taking this patch (and alternatives if risky):
No risk. One line change to the charset aliases.

Comment 7

[aleth [:aleth]]

https://hg.mozilla.org/comm-central/rev/19694424a48639d4f9ca458e3e891292e0c2ae1e
Bug 1217161 - Mojibake with Chinese characters in email body (support cp936 encoding as an alias for gbk). r=rkent

Comment 8

[Jorg K (CEST = GMT+2)]

I tested this with the message from bug 1235294, attachment 8702172 [details]. Works nicely as expected.

Comment 9

[Jorg K (CEST = GMT+2)]

Aurora (TB 46):
https://hg.mozilla.org/releases/comm-aurora/rev/fc104a6973c6

Comment 10

[Kent James (:rkent)]

Comment on attachment 8702994 [details] [diff] [review]
bug1217161_cp936_support.patch

http://hg.mozilla.org/releases/comm-beta/rev/5b79eb3f9dd6

Comment 11

[Kent James (:rkent)]

(In reply to Kent James (:rkent) from comment #10)
> Comment on attachment 8702994 [details] [diff] [review]
> bug1217161_cp936_support.patch
> 
> http://hg.mozilla.org/releases/comm-beta/rev/5b79eb3f9dd6

Oops, wrong link
http://hg.mozilla.org/releases/comm-beta/rev/a98e0f07eada