207018 - Mail fails to show all of an unknown-8bit encoded mail viewed as utf-8

Status: RESOLVED DUPLICATE of bug 190278

(MailNews Core :: Internationalization, defect)

Description

[carljohan]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030313
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030313

A friend of mine sends mail through a badly configured Mutt client.
Even though his mails are encoded as iso-latin-1 they are marked
as unkown-8bit.

When viewing the mail from him I'm attaching as utf-8, as is the
standard encoding I'm using, two lines in the middle of the mail
aren't shown. They look like blank lines. By selecting the lines
I can sometimes get parts of the lines to show, but otherwise
they are just blank lines.

Viewing it with latin-1 encoding shows the mail perfectly, as the
mail actually is a misslabeled latin-1 mail.

Reproducible: Always

Steps to Reproduce:
1. Open the attached mail
2. View it with character encoding utf-8

Actual Results:  
Two lines are shown as blank lines.

Expected Results:  
At least all the ascii characters of the mail should be shown
even though the message is labeled as unkown-8bit.

Comment 1

[carljohan]

Attachment: A mail that Mozilla fails to display correctly with utf-8

This mail is actually 8bit iso-latin-1, but it is labeled charset=unknown-8bit.

Comment 2

[Jo Hermans]

Bug 71551 claims that it should be using ISO-8859-1, which should work for
Swedish. But see also bug 71541 : to help other people in other countries (not
everyone uses ISO-8859-1 or a similar charset), we will use the default charset
instead.

It switches to utf-8 because your default charset is utf-8. Maybe we can use 
something more like the universal detector (bug 71541 comment 3) ? AFAIK, the
universal detector should have identified your message as ISO-8859-1.

Comment 3

[carljohan]

Whether we should use ISO-8859-1, utf-8 or a universal detector is irrelevant
in this case.

This is an actual bug in the utf-8 handling of the mail. Two whole lines
in this mail aren't displayed and that should never happen, since most
of the characters are 7-bit ASCII.

At the very least, Mozilla should display _all_ valid 7-bit ASCII characters
and display non-valid utf-8 characters, like the swedish latin-1 chars
והצ and ִֵײ, as '?'.

One alternative is to do as the latest version of the IRC-client xchat does,
and display both utf-8 and latin-1 correctly if you set it to utf-8. But
that solution probably has some black magic in it since the standard for IRC
was latin-1 from the start.

Comment 4

[Simon Montagu :smontagu]

>By selecting the lines
>I can sometimes get parts of the lines to show, but otherwise
>they are just blank lines.

This sounds similar to bug 190278. Are you using an Xft-enabled version of Mozilla?

Comment 5

[carljohan]

Yes I am using an Xft-enabled Mozilla now that I think about it. It's probably
a dupe of bug 190278.

Comment 6

[Simon Montagu :smontagu]

I have never been able to reproduce bug 190278, so reassigning to blizzard to
make the call whether this is a dupe of it.

Comment 7

[Christopher Blizzard (:blizzard)]

Sounds exactly the same as bug 190278.  I've personally stumbled over the UTF-8
vs. ISO-8859-1 issues.

*** This bug has been marked as a duplicate of 190278 ***