504831 - UTF-16 decoder shouldn't guess endianness when there's no BOM

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

According to http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2009-July/021102.html , changing the UTF-16 decoder doesn't guess endianness when no BOM is present, and so that it defaults to little endian when no BOM is present, would improve compatibility with other browsers.  The code in question is in the last function in a file incorrectly named nsUCS2BEToUnicode.cpp .

Comment 1

[Simon Montagu :smontagu]

Why would we sacrifice conformance to the Unicode Standard, and a SHOULD in RFC 2781, for the sake of compatibility with other browsers? Especially since AFAICT in any case where there is a practical difference the other browsers will be displaying garbage and we will be displaying the intended content.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Maybe bring that up on the whatwg list?

Comment 3

[Anne (:annevk)]

See also: http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2011-December/034260.html

Comment 4

[Masatoshi Kimura [:emk]]

I do not want to use utf-16be/utf-16le as labels for utf-16 with BOM.
It's OK to be "misinterpreted for compatibility" as unicode/unicodeFFFE (if they are registered).

Comment 5

[Danny Fritz]

I was just bitten by this bug. Returned data was garbled and cut short, but worked in other browsers. Looking at the HTTP request with tools like Fiddler was also showing it encoded correctly. It wasn't until I viewed it in JS did I see the garbled result.

Is there a good way to debug this? I'll know going forward what is happening, but to anyone else that encounters it for the first time, good luck.

Comment 6

[Anne (:annevk)]

It seems this is still present http://mxr.mozilla.org/mozilla-central/source/intl/uconv/ucvlatin/nsUTF16ToUnicode.cpp#319

I don't think we plan on standardizing this so we should probably just remove it.

Comment 7

[Henri Sivonen (:hsivonen)]

Fixed by bug 1261841.