20997 - Don't discard non-ascii characters, even in us-ascii charset

Status: VERIFIED FIXED

(MailNews Core :: Internationalization, defect, P3)

Description

[Richard Zach]

If the header of a message claims charset="us-ascii", non-ascii characters are
discarded.  Educated guess at the encoding, or just assuming 8859-1 would be
better.  Apparently some clients aren't smart enought to set the encoding right.

Comment 1

[nhottanscp]

This bug is about viewing messages.
We do not plan to do guessing if the message is labeled (we do charset detection
in case of no charset label specified). We plan to enable charset menu so the
user can set a charset explicitly (bug 5938).
Before M12, "ISO-8859-1" was used for "us-ascii" conversion (since no converter
was available for "us-ascii" in early mozilla). I can change it again to use
8859-1 for us-ascii.

Comment 2

[Katsuhiko Momoi]

Assuming iso-8859-1 for us-ascii is not a bad idea. It could save the user trouble of having to
override a wrong charset label. Note that us-ascii is sometimes used incorrectly for other charsets
also. But if we were to make an assumption about the super-set of us-ascii, iso-8859-1 is probably
the best choice.

Comment 3

[nhottanscp]

Attachment: test data (labeled as us-ascii but contains ISO-8859-1 character)

Comment 4

[nhottanscp]

Attachment: patch for the fix

Comment 5

[nhottanscp]

Rich, could you review the diff? This has been done for header but not for body.

Comment 6

[nhottanscp]

The code has been reviewed, waiting for a check in approval.

Comment 7

[rhp (gone)]

Patch looks safe to me.

- rhp

Comment 8

[nhottanscp]

The fix checked in.

Comment 9

[Katsuhiko Momoi]

** Checked with 1/13/00 Win32 M13 build **

The result looks good. With M12 build, QP'ed Latin 1 characters did not
display OK if the charset was us-ascii. Now with the fix in the code, the same
message displays the Latin 1 accented characters OK even under us-ascii
MIME charset. (See nhotta's test case attached to this report.)

Marking it verified fixed.