45466 - UTF-8 not correctly handled in XHTML

Status: RESOLVED WORKSFORME

(Core :: XML, defect, P3)

Description

[bertilow]

Mozilla does not automatically switch to UTF-8 for XHTML pages without an XML 
declaration with 'encoding="utf-8"'. Meta-elements with 'charset=utf-8' are 
ignored supposedly since XHTML pages are handled as XML.

According to the XML recommendation "utf-8" (or "utf-16") is the default 
encoding and pages without a declaration of 'encoding="utf-8"' must be handled 
as "utf-8" or "utf-16" (automatically detected).

This worked correctly before. The bug seems to have appeared when the bug was 
fixed that made XHTML pages with an XML declaration not display at all.

I have seen this in "Mozilla/5.0 (Windows; U; Win98; en-US; m17) 
Gecko/20000713", and two earlier builds.

Comment 1

[Peter 'Rufus' Nelson]

Reporter: can you please attatch a test case or URL of a page you see this on?
Without one we can't do very much.  Thanks.

Comment 2

[bertilow]

My own home page might serve as a test case:

  <http://www.concinnity.se/bertilow/>

You'll immediately spot the weird characters in the headings.

To read the page correctly one has to manually switch to UTF-8.

The page includes a meta element that declares UTF-8, but no
XML declaration (and thus no XML info about encoding). The server
does not send any charset info (I beleive).

This page used to render perfectly until the bug 29125 was "fixed".
Before that XHTML pages with an XML declaration did not render
at all, but those without an XML declaration rendered correctly
(at least with a meta element that stated UTF-8 as charset).

Actually this bug (if it is a bug) seems to be a bit more complicated.
I tried to test the various effects of the three ways of indicating
the encoding (http header, XML declaration, meta element), along with
the two content types "text/html" and "text/xml" (14 different 
permutations, I believe...). Sometimes I could get UTF-8, but most of 
the time not, but I could see no clear pattern. I'm not sure what would 
be the correct way of dealing with all of these permutations. However 
I'm quite sure that my home page, even without the meta element, should 
be automatically rendered according to UTF-8, since that is the default 
for XML and XHTML.

Comment 3

[Asa Dotzler [:asa]]

setting bug status to New.  confirmed with 072808 m18 win32 mozilla bits on NT

Comment 4

[bertilow]

I just tested this with Mozilla/5.0 (Windows; U; Win98; en-US; m18) 
Gecko/20000811, and things work a lot better now: Charset specified 
in a meta element, or encoding specified in an xml declaration 
(and as before charset specified in the http header) are correctly 
detected and correctly used.

However, if no charset or encoding is indicated in any way, utf-8
is not automatically used (but instead whatever encoding is set
as default in the browser). This seems wrong, but I'm not sure.
The XHTML specification states that an XHTML doc without any
indication of charset or encoding is to be taken as UTF-8 or
UTF-16 (distiguished between automatically).

So perhaps this but is fixed. I'm very glad!

Comment 5

[Nisheeth Ranjan]

Based on Bertilow's comments, marking bug worksforme.

Comment 6

[jsp]

I don't think this is fixed yet.  The XHTML spec says, "[An XML] declaration is
required when the character encoding of the document is other than the default
UTF-8 or UTF-16." The implication is that characters encoded with UTF-8 or
UTF-16 should be decoded even in the absence of an XML declaration. In the
absence of an XML declaration, Mozilla does not decode UTF-8 characters, even if
the charset is specified in a meta element.

I'd suggest reopening the bug; I don't think it's my place to do so.

Comment 7

[jsp]

Attachment: Testcase: UTF-8 encoded trademark symbol displays incorrectly