809934 - Remove reliance on BOM-sniffing UTF-16 decoder

Status: RESOLVED DUPLICATE of bug 504831

(Core :: DOM: Core & HTML, defect)

Description

[Masatoshi Kimura [:emk]]

Our current UTF-16 usage is based on the Unicode Standard and IANA registry. That is, UTF-16 sniffs the BOM.
However, Encoding Standard says utf-16 is the same as utf-16le. We should revert bug 335531 to comply the new spec.
We don't have to consider about mailnews here because UTF-16 is not suitable for MIME.

Comment 1

[Henri Sivonen (:hsivonen) (on leave)]

(In reply to Masatoshi Kimura [:emk] from comment #0)
> However, Encoding Standard says utf-16 is the same as utf-16le.

More to the point, the Encoding Standard sniffs the BOM first as part of the "decode" algorithm, so if HTTP says "utf-16", the "decode" algorithm changes the label to "utf-16be" before invoking the actual decoder if there is a big-endian BOM.

So if we proceed with this, pages that have a big-endian UTF-16 BOM would report "utf-16be" as document.characterSet even if HTTP said "utf-16". Will that break scripts?

Comment 2

[Henri Sivonen (:hsivonen) (on leave)]

(In reply to Henri Sivonen (:hsivonen) from comment #1)
> So if we proceed with this, pages that have a big-endian UTF-16 BOM would
> report "utf-16be" as document.characterSet even if HTTP said "utf-16". Will
> that break scripts?

Based on Google Code search, I’m guessing the answer is “No”. Search for .characterSet and UTF-16 or utf-16 in .js finds just Firefox sources.

Comment 3

[Masatoshi Kimura [:emk]]

(In reply to Henri Sivonen (:hsivonen) from comment #1)
> More to the point, the Encoding Standard sniffs the BOM first as part of the
> "decode" algorithm, so if HTTP says "utf-16", the "decode" algorithm changes
> the label to "utf-16be" before invoking the actual decoder if there is a
> big-endian BOM.
The "decode" algorithm will change not only "utf-16" but also any other encodings. IIUC it represents "BOM trumps everything" rule.

> So if we proceed with this, pages that have a big-endian UTF-16 BOM would
> report "utf-16be" as document.characterSet even if HTTP said "utf-16". Will
> that break scripts?
I thought it was already implemented by bug 716579. No?

Comment 4

[Masatoshi Kimura [:emk]]

(In reply to Masatoshi Kimura [:emk] from comment #3)
> > So if we proceed with this, pages that have a big-endian UTF-16 BOM would
> > report "utf-16be" as document.characterSet even if HTTP said "utf-16". Will
> > that break scripts?
> I thought it was already implemented by bug 716579. No?
Ah, bug 716579 always uses "utf-16" label and expects it will sniff the BOM.

Comment 5

[Masatoshi Kimura [:emk]]

(In reply to Henri Sivonen (:hsivonen) from comment #2)
> Based on Google Code search, I’m guessing the answer is “No”. Search for
> .characterSet and UTF-16 or utf-16 in .js finds just Firefox sources.
I don't think Web pages can rely on the .characterSet label. For example, IE returns "unicode" or "unicodeFFFE" on utf-16 pages. WebKit uses "UTF-16LE" as a canonical name for little endian utf-16 pages (neither "UTF-16" nor "utf-16").

Comment 6

[Masatoshi Kimura [:emk]]

Attachment: utf-16 sample

Comment 7

[Masatoshi Kimura [:emk]]

Attachment: utf-16le sample

Comment 8

[Masatoshi Kimura [:emk]]

Attachment: utf-16be sample

Comment 9

[Masatoshi Kimura [:emk]]

Attachment: Test

Comment 10

[Masatoshi Kimura [:emk]]

Attachment: utf-16 sample

Comment 11

[Masatoshi Kimura [:emk]]

Attachment: utf-16le sample

Comment 12

[Masatoshi Kimura [:emk]]

Attachment: utf-16be sample

Comment 13

[Masatoshi Kimura [:emk]]

Attachment: Test

Comment 14

[Masatoshi Kimura [:emk]]

Firefox: UTF-16 UTF-16LE UTF-16BE (UTF-16 sniffs the BOM)
Chrome: UTF-16LE UTF-16LE UTF-16BE
IE10: unicode unicode unicodeFEFF (in all document mode)
Opera: utf-16 utf-16 utf-16 (utf-16 sniffs the BOM?)

No browsers behave as the Encoding Standard defines!
Anne, why did you made "utf-16" the canonical name of utf-16 little-endian?
It conflicts all browsers, the Unicode Standard and the IANA registry. I think "utf-16le" would be better for the canonical name.

Comment 15

[Masatoshi Kimura [:emk]]

> Firefox: UTF-16 UTF-16LE UTF-16BE (UTF-16 sniffs the BOM)
This is Firefox 16.0.2. On the latest Nighly, it was UTF-16LE UTF-16LE UTF-16BE.

Comment 16

[Masatoshi Kimura [:emk]]

Genuine IE8: unicode unicode unicodeFFFE (Unlike IE8 mode of IE10, the testcase didn't work. I loaded individual pages to get the result.)
MS changed the encoding name from "unicodeFFFE" to "unicodeFEFF". Maybe nobody cares about the encoding name...

Comment 17

[Masatoshi Kimura [:emk]]

Safari 5.1.7: UTF-16LE UTF-16LE UTF-16BE

Comment 18

[Anne (:annevk)]

emk, I think I have been tripped by utf-16 sniffing. http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Dec/0256.html was my reasoning (that month contains some other messages  that may be of interest here).

Comment 19

[Masatoshi Kimura [:emk]]

(In reply to Anne van Kesteren from comment #18)
> emk, I think I have been tripped by utf-16 sniffing.
> http://lists.w3.org/Archives/Public/public-whatwg-archive/2011Dec/0256.html
> was my reasoning (that month contains some other messages  that may be of
> interest here).
I read the thread, but I didn't still see the reasoning behind that.
> utf-16le becomes a label for utf-16.
As my test has shown, at least Safari and Chrome treat UTF-16 as a label for UTF-16LE, not the other way round. Nightly also follows them now. Leif's reasoning doesn't also hold because the BOM will be removed inside the "decode" algorithm before the stream is passed to the decoder.
Moreover UTF-16LE was already consistent between browsers if the BOM is absent.
> * Gecko decodes FFFE 007A as FFFD followed by FE00 presumably dropping the  
7A.
> ** Gecko decoes FEFF 007A as FFFD followed by 00FF presumably dropping the  
7A.
I believe it was fixed by bug 716579.

Comment 20

[Anne (:annevk)]

emk, my apologies for making you read through that. I misunderstood your comment. The standard is now updated.

Comment 21

[Masatoshi Kimura [:emk]]

Thank you. Filed bug 811127 to catch up the spec.

Comment 22

[Leif Halvard Silli]

Please see bug 814900 (test file: https://bug814900.bugzilla.mozilla.org/attachment.cgi?id=684883)

I don't know if it that bug is due to the solution to _this_ bug - or due to the solution to bug 716579 - or due  to something else. 

But fact is that, right now, then Firefox 19 and 20 issues a fatal error for UTF-16 encoded, big-endian XML files with the BOM.

My suspision is that it is this bug, 809934, since this bug is about treating "utf-16" as  a label for "utf-16le".  And "utf-16le" does not permit the BOM, hence the XML parser should issue a fatal error.

Which kind, of demonstrates the problems I have both with this bug, as well as with what the Encoding Standards says about this.

Comment 23

[Anne (:annevk)]

Is this a duplicate of bug 504831?

Comment 24

[Henri Sivonen (:hsivonen) (on leave)]

(In reply to Anne (:annevk) from comment #23)
> Is this a duplicate of bug 504831?

I believe it is.