119079 - Subject display of Simplified Chinese (HZ) news w/o MIME header can't be corrected by setting a proper charset

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect)

Description

[ji]

***Observed with 01/09 trunk build****

The subject display of Simplied Chinese (HZ) news which doesn't have MIME header
can't be corrected by setting a proper charset.

Steps to reproduce:
1. Subscribe to newsgroup alt.chinese.txt.hz.
2. Select a news which doesn't have MIME header, like the news posted by "w".
3. Correct the display either by selecting charset menu or setting proper folder
charset. The subject is still garbled on envelope. The mail body part is okay.
Same news can be viewed from Outlook Express after selecting the proper encoding.

Comment 1

[ji]

Attachment: A screenshot

Comment 2

[nhottanscp]

From page source of the news message,

Subject: Re:
~{Ubwa@V9[#,1cS&7EFz3,8_<[BrC@9z5D6~Aw>|18!"2;R*1;C@HUFzVxW_#,TY@48v4sM30I#,TuwaQy#?~}

Comment 3

[nhottanscp]

This might be that MIME decoder assumes the input string as a valid UTF-8 and
not apply the conversion with the default charset, cc to jgmyers.
The function intl_is_legal_utf8(), it checks for 0x1B but I don't think HZ use 0x1B.
http://lxr.mozilla.org/seamonkey/source/mailnews/mime/src/comi18n.cpp#1054

Comment 4

[John G. Myers]

Could someone point me to a spec on HZ?  Is there a way to syntacitcally
distinguish HZ from UTF-8?  If so, we can extend intl_is_legal_utf8() to exempt
text that appears to be in HZ.

Comment 5

[nhottanscp]

http://umunhum.stanford.edu/~lee/chicomp/HZ_spec.html  (Easy to understand
summary by the author of RFC 1843)
http://www.faqs.org/rfcs/rfc1843.html    (RFC 1843)

Comment 6

[Katsuhiko Momoi]

John, HZ uses unique escape sequences to get in and out
of ASCII and GB2312 and so you can exploit that fact to 
exclude it from UTF-8 detection.

Comment 7

[nhottanscp]

Reassign to jgmyers.

Comment 8

[John G. Myers]

I think the rule should be that if a header contains any number of ASCII
characters, the sequence "~{", any number of ASCII characters, the sequence
"~}", then any number of ASCII characters, then the line contains HZ and the
function should say it is not "legal" UTF8.  Any 8-bit characters should disable
this HZ exemption.

I'm kinda swamped this week, I'll try to get a patch by end of Jan.

Comment 9

[John G. Myers]

Attachment: Proposed fix

Comment 10

[John G. Myers]

Testing of this patch should include HZ headers which include only ASCII, but
which include the ~ character.  In HZ, a ~ character is encoded as ~~

This patch makes the call to intl_is_legal_utf8() from the encoder problematic.
 Fortunately, that call is being removed by the proposed patch to bug 73403. 
Marking this bug dependant.

Comment 11

[ji]

Nominating for nsbeta1.

Comment 12

[John G. Myers]

Attachment: Corrected patch

Comment 13

[nhottanscp]

+      case hz_seen:
+        if (*input == '{') {
+          hz_state = hz_escaped;
+        } else if (*input == '~') {
+          /* ~~ is the HZ encoding of ~.  Skip over second ~ as well */
+          hz_state = hz_seen;
+          input++;
+          len--;

At 'case hz_seen:', why do you need to set hz_stat to hz_seen when it's already set?

Comment 14

[John G. Myers]

Because hz_stat could be set to either hz_seen or hz_initial at this point.
In the case where hz_stat was set to hz_initial, setting it to hz_seen will
cause the function to return PR_FALSE unless hz_stat gets subsequently changed.

Comment 15

[nhottanscp]

Comment on attachment 70787 [details] [diff] [review]
Corrected patch

r=nhotta

I missed 'case hz_initial:' has no break.

Comment 16

[Scott MacGregor]

Comment on attachment 70787 [details] [diff] [review]
Corrected patch

one small nit that isn't related to your change but shows up in the patch: 

can you change 
nsCRT::strlen(header)
to strlen(header)

sr=mscott

Comment 17

[Asa Dotzler [:asa]]

Comment on attachment 70787 [details] [diff] [review]
Corrected patch

a=asa (on behalf of drivers) for checkin to the 1.0 trunk

Comment 18

[John G. Myers]

Fix checked in.
 

Comment 19

[ji]

Verfied as fixed.