782721 - Exising nsIUnicodeDecoder, nsBasicDecoder do not have a replacement character option when an unknown character is found.

Status: RESOLVED DUPLICATE of bug 638379

(Core :: Internationalization, defect)

Description

[bsurender]

The existing nsIUnicodeDecoder does not support a replacement character option if an unknown character is detected in the incoming charset to be decoded to UTF-16 code units.

http://mxr.mozilla.org/mozilla-central/source/intl/uconv/public/nsIUnicodeDecoder.h#61

Instead, the decoder stops and signals, or just recovers and continues.

The encoder however supports the above two options as well as replacement characters.

Comment 1

[Masatoshi Kimura [:emk]]

Are kOnError_Signal and kOnError_Recover insufficient? Although those options may have some subtle differences from what the Encoding Standard requires, I don't think it's enough critical to block implementing TextDecoder.

Comment 2

[Simon Montagu :smontagu]

(In reply to bsurender from comment #0)
> Instead, the decoder stops and signals, or just recovers and continues.

This is by design (though the behaviour could be changed to some extent if necessary): see bug 231267

Comment 3

[Masatoshi Kimura [:emk]]

Our decoder will replace unmapped character to what GetCharacterForUnMapped(). It is U+31FC for Shift_JIS, U+FFFD for other encodings.
Until bug 747762 has been fixed (but honestly I do not want break the compatibility with the Windows decoder deliberately), You can workaround the quirks as follows:
1. If the "fatal" flag is true, use kOnError_Signal and throw "EncodingError" on failure.
2. Otherwise, if GetCharacterForUnMapped() returns U+FFFD, use kOnError_Recover.
3. Otherwise, use kOnError_Signal and replaces the offending character to U+FFFD yourself.
Step 2 is optional. You can always handle the error yourself instead of inspecting the return value of GetCharacterForUnMapped().

Comment 4

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Do we still need this? Or should we close it as WONTFIX?

Comment 5

[Masatoshi Kimura [:emk]]

I found this was already covered by bug 638379.