240962 - Charset in Content-Type header is ignored for XHTML

Status: VERIFIED FIXED

(Core :: DOM: Core & HTML, defect)

Description

[Mario Mueller]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7b) Gecko/20040414 Firefox/0.8
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a) Gecko/20040418 Firefox/0.8

FF ignores given Charset on XHTML sites served with content type
"application/xhtml+xml". On the example page, the content type header is:

Content-Type: application/xhtml+xml; charset=iso-8859-1

But FF displays the site with charset UTF-8.

Reproducible: Always
Steps to Reproduce:
1. go to www.ego4u.de
2. you can't see German-Umlauts

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Confirmed with Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8a)
Gecko/20040418 Firefox/0.8.0+

Manually changing the character encoding results in nothing. UTF-8 is still
selected.

Comment 2

[Christian :Biesinger (don't email me, ping me on IRC)]

didn't bz recently fix a bug related to this that might've caused this?

for future reference... charset detection is not done in frontend specific code,
so this should be in the browser product. but I'm not sure which component, I'll
let bz correct it :)

Comment 3

[Christian :Biesinger (don't email me, ping me on IRC)]

hmm, bug 240321. sorry, that was jst, not bz

Comment 4

[Boris Zbarsky [:bzbarsky]]

Er, yes.  Compare
http://lxr.mozilla.org/seamonkey/source/content/html/document/src/nsHTMLDocument.cpp#830
to
http://lxr.mozilla.org/seamonkey/source/content/xml/document/src/nsXMLDocument.cpp#578

We should probably push the TryChannelCharset function on nsHTMLDocument up to
nsDocument or something and call it from both those places and from
nsXULDocument::PrepareToLoadPrototype

jst, do you have time to deal with this?  At least the XML document part we need
on the 1.7 branch, since bug 240321 landed there...

Comment 5

[Hixie (not reading bugmail)]

Not that anyone here appears to be trying to suggest this, but, make sure we 
don't take the <meta http-equiv="Content-Type"> line into account when 
determining the charset of an XHTML document.

Comment 6

[Christian :Biesinger (don't email me, ping me on IRC)]

And be sure to also test "Manually changing the character encoding" as mentioned
in comment 1 (which seems unaddressed by bz's comment)

Comment 7

[Boris Zbarsky [:bzbarsky]]

Manually changing the character encoding is generally unsupported by XML at the
moment.  It's not clear to me that it ever should be supported for XML.  No
matter what, that should be in a separate bug filed on intl so people with intl
clue can comment.

> <meta http-equiv="Content-Type">

Handled by the parser, not this code.  See
http://lxr.mozilla.org/seamonkey/source/htmlparser/src/nsParser.cpp#2180

Comment 8

[Boris Zbarsky [:bzbarsky]]

I won't be able to patch and test this any time soon... but we do need this
fixed for 1.7.  Can someone take this?

Comment 9

[Sekundes]

#1 bug has been filed to bug 234628.

Comment 10

[James Ross]

Attachment: Move TryChannelCharset up to nsDocument, and use in HTML/XML documents.

This puts TryChannelCharset into nsDocument, as bz suggested. It also makes
nsHTMLDocument (when IsXHTML()) and nsXMLDocument call it, instead of their
current behaviours (HTML: always use UTF-8, XML: duplicated code to check
channel charset).

I have tested my patched non-debug Mozilla with XML and XHTML documents sent
without a charset (uses UTF-8) and sent with charset Windows-1252 in the HTTP
header (uses Windows-1252). The site http://www.ego4u.de/ (in this bug's URL)
also correctly shows the German-Umlauts. I can test other scenarios if needed.

Comment 11

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 147502 [details] [diff] [review]
Move TryChannelCharset up to nsDocument, and use in HTML/XML documents.

r+sr=bzbarsky.	The testing you did should be just fine.  Thank you for picking
this up!

Comment 12

[James Ross]

Comment on attachment 147502 [details] [diff] [review]
Move TryChannelCharset up to nsDocument, and use in HTML/XML documents.

This fixes a big regression for XHTML and has a low risk (no new charset code,
just moved and used for XHTML).

Comment 13

[James Ross]

Checked in, tinderboxes have cycled.

I believe the patch covered this bug completely (it certainly fixed the site in
question), so I'm marking this bug FIXED. If this is not the case, please reopen
the bug stating why.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

v.

Comment 15

[Constantine A. Murenin]

Silver, did you meant to misspell your email in the patch and check-in?