600715 - Remove JIS X 0212 support from EUC-JP encoder

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Masatoshi Kimura [:emk]]

Recently, CP51932 has added as a replacement encoding for EUC-JP.
Roughlt speaking, CP51932 is an Microsoft variant of EUC-JP but it doesn't support JIS X 0212. It had been a severe interoperability problem for a long time.
If someone post a message to a forum whose encoding is EUC-JP, Firefox encodes JIS X 0212 characters using SS3 which is not readable for other browsers.
If JIS X 0212 encoded is removed, JIS X 0212 characters will be converted to character references per HTML5 spec.
http://www.w3.org/TR/html5/association-of-controls-and-forms.html#application-x-www-form-urlencoded-encoding-algorithm
I will not remove JIS X 0212 decoder for backward compatibility with previous versions of Firefox and non-HTML5-compliant implementations. This is a similar approach to our Big5 decoder (see bug 310299).

Comment 1

[Masatoshi Kimura [:emk]]

Attachment: removal patch

Comment 2

[Benjamin Smedberg]

Can you outline the risk/reward for this patch, and what kind of testing it ought to receive from nightly and beta testers once it lands?

I do not see any tests for the new encoding behavior, which makes me leery of approving the patch in any case.

Comment 3

[Benjamin Smedberg]

Comment on attachment 479750 [details] [diff] [review]
removal patch

Clearing approval request until questions are addressed.

Comment 4

[NARUSE, Yui]

Background:

The encoding EUC-JP has some variants.
* original Japanese EUC (US-ASCII + JIS X 0201 Katakana + JIS  X 0208)
* EUC-JP defined by IANA (US-ASCII + JIS X 0201 Katakana + JIS  X 0208 + JIS X 0212) 
* CP51932 defined by Microsoft (US-ASCII + JIS X 0201 Katakana + JIS X 0208 + NEC special character + NEC selected IBM extended character)

Current implementation:

Mozilla implements own special encoding (US-ASCII + JIS X 0201 Katakana + JIS X 0208 + NEC special character + NEC selected IBM extended character + JIS X 0212)

Current Problem:

On posting forms, Mozilla sends data whose character set is larger than other implementations.
IANA EUC-JP complient implementations can't process NEC special characters and NEC selected IBM extended characters.
CP51932 comlient implementations can't process JIS X 0212 characters.
So current Mozilla lacks interoperability.

Ideal behavior:

Follow CP51932.
Why follow CP51932 is because HTML5 says pages declared as "EUC-JP" encoding
should be proceeded as CP51932. see Character encoding overrides table in HTML5.
http://www.w3.org/TR/html5/parsing.html#character-encodings-0

But it is inconvenient that Mozilla can't see data posted by previous Mozilla.
So remove JIS X 0212 which is difference between CP51932 and current Mozilla encoding
only from encoder; not remove from decoder.

The patch do this.

See also:

https://bugzilla.mozilla.org/show_bug.cgi?id=310299 is similar problem and did similar solution.

Comment 5

[Dão Gottwald [:dao]]

http://hg.mozilla.org/projects/cedar/rev/151bbff21acf

Comment 6

[(no longer active)]

http://hg.mozilla.org/mozilla-central/rev/151bbff21acf