582788 - iso-10646 in meta not rejected as a non-ASCII-superset encoding

Status: RESOLVED FIXED

(Core :: DOM: HTML Parser, defect, P2)

Description

[Emil Ivanov]

From: https://input.mozilla.com/en-US/search/?product=firefox&sentiment=sad

Some user reports that the page http://cda.ipmailing.it/IPMailing/forms/optInForm1.asp cite: "won't show properly"

The page is working with Firefox 3.6.8, so this is a regression.

Im not sure for the component, but the page and the view source are both unreadable with Firefox 4 b2.

Comment 1

[Emil Ivanov]

Firefox 3.6.8 detects the page encoding as Western (ISO-8859-1)

Firefox 4b2 detects as (UTF-16BE)

There is another problem with View source, i cant change the encoding until the view source window is reloaded, should i fill separate bug for this?

Comment 2

[Emil Ivanov]

(In reply to comment #1)
> There is another problem with View source, i cant change the encoding until the
> view source window is reloaded

anyway filled bug 582795

Comment 3

[Boris Zbarsky [:bzbarsky]]

This is fallout from the HTML5 parser.  I can get the broken behavior in 3.6.8 if I turn that parser on.

Note that charsetalias.properties aliases iso-10646 to UTF-16BE.  So I'm not quite sure why the old parser didn't use that encoding here... but if I change the meta to say "UTF-16BE" we render the page correctly.  So there's some sort of special-casing going on here, and it seems to happen _before_ alias resolution.  Should it happen after?

Comment 4

[Henri Sivonen (:hsivonen)]

(In reply to comment #3)
> So there's some sort
> of special-casing going on here, and it seems to happen _before_ alias
> resolution.  Should it happen after?

Looks like it need to happen after. The code is:
http://mxr-test.konigsberg.mozilla.org/mozilla-central/source/parser/html/nsHtml5MetaScannerCppSupplement.h#87

Comment 5

[Henri Sivonen (:hsivonen)]

I wonder why http://mxr-test.konigsberg.mozilla.org/mozilla-central/source/parser/html/nsHtml5MetaScannerCppSupplement.h#112 isn't working

Comment 6

[Henri Sivonen (:hsivonen)]

The bug is instead that similar checks are missing here:
http://mxr-test.konigsberg.mozilla.org/mozilla-central/source/parser/html/nsHtml5StreamParser.cpp#750

Comment 7

[Henri Sivonen (:hsivonen)]

Safari seems to special-case UTF-16 without alias resolution so it doesn't sniff to UTF-8. It then reject UTF-16BE as non-ASCII-based encoding, so the default encoding (Windows-1252) kicks in.

Comment 8

[Henri Sivonen (:hsivonen)]

I've examined four Web pages that declare iso-10646 in meta. (Thankfully, they are rare.) 3 were ASCII. One was Windows-1252. So from this data, it seems we should *not* do alias resolution before the UTF-16 to UTF-8 aliasing step.

Comment 9

[Henri Sivonen (:hsivonen)]

(In reply to comment #7)
> Safari seems to special-case UTF-16 without alias resolution so it doesn't
> sniff to UTF-8.

Chances are I'm misreading Safari's encoding menu. :-(

Comment 10

[Henri Sivonen (:hsivonen)]

Attachment: Unify late meta treatment with prescan treatment, remove UTF-32 to UTF-8 aliasing while at it

Comment 11

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 461189 [details] [diff] [review]
Unify late meta treatment with prescan treatment, remove UTF-32 to UTF-8 aliasing while at it

This looks fine, but shouldn't all those calls be LowerCaseEqualsLiteral?  r=me with that.

Comment 12

[Henri Sivonen (:hsivonen)]

(In reply to comment #11)
> This looks fine, but shouldn't all those calls be LowerCaseEqualsLiteral?  r=me
> with that.

Thanks. Pushed with that change:
http://hg.mozilla.org/mozilla-central/rev/98617b5a532b

The corresponding spec change is being tracked as
http://www.w3.org/Bugs/Public/show_bug.cgi?id=10260