1170932 - GBK encoder silently omits unmapped characters

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Simon Montagu :smontagu]

Spun out from bug 1169248 comment 2 and 5 (testcase slightly changed so as not to depend on that bug)

data:text/html;charset=big5,<script>a=document.createElement('a');a.href='https://example.com/?\u2764';document.write(a.href)</script>

Results: https://example.com/?%E2%9D%A4 
(because unmapped U+2764 causes an encoding error and the URL falls back to UTF-8). Same with any other charset where U+2764 is not mapped, except:

data:text/html;charset=gbk,<script>a=document.createElement('a');a.href='https://example.com/?\u2764';document.write(a.href)</script>

Results: https://example.com/?

Comment 1

[Simon Montagu :smontagu]

There are two issues here.

a) The constructor of the gbk encoder assumes (correctly) that the maximum number of gbk codepoints in the output is 2 * the number of utf-16 codepoints in the input. Then the ConvertNoBuff method, which is shared by gbk and gb18030, fails to convert the unmapped character with the 2-byte encoder, and before calling Try4BytesEncoder checks whether there are 4 bytes free in the output buffer and if not returns NS_OK_UENC_MORE_OUTPUT instead of NS_ERROR_UENC_NOMAPPING. This check is not needed for gbk, where Try4BytesEncode will always fail.

Fixing that is sufficient for the testcase in comment 0, but didn't fix an xpcshell test I wrote, because

b) ConvertNoBuff doesn't handle non-default values of mErrBehavior

Comment 2

[Simon Montagu :smontagu]

Attachment: Patch

Comment 3

[Simon Montagu :smontagu]

Attachment: xpcshell test (includes all encoders)

Comment 4

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/10dfb6455e4f
https://hg.mozilla.org/integration/mozilla-inbound/rev/c5f6b5160c02

Comment 5

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/10dfb6455e4f
https://hg.mozilla.org/mozilla-central/rev/c5f6b5160c02