bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

GBK encoder silently omits unmapped characters

RESOLVED FIXED in Firefox 41

Status

()

Core
Internationalization
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: smontagu, Assigned: smontagu)

Tracking

Trunk
mozilla41
Points:
---

Firefox Tracking Flags

(firefox41 fixed)

Details

Attachments

(2 attachments)

(Assignee)

Description

3 years ago
Spun out from bug 1169248 comment 2 and 5 (testcase slightly changed so as not to depend on that bug)

data:text/html;charset=big5,<script>a=document.createElement('a');a.href='https://example.com/?\u2764';document.write(a.href)</script>

Results: https://example.com/?%E2%9D%A4 
(because unmapped U+2764 causes an encoding error and the URL falls back to UTF-8). Same with any other charset where U+2764 is not mapped, except:

data:text/html;charset=gbk,<script>a=document.createElement('a');a.href='https://example.com/?\u2764';document.write(a.href)</script>

Results: https://example.com/?
(Assignee)

Comment 1

3 years ago
There are two issues here.

a) The constructor of the gbk encoder assumes (correctly) that the maximum number of gbk codepoints in the output is 2 * the number of utf-16 codepoints in the input. Then the ConvertNoBuff method, which is shared by gbk and gb18030, fails to convert the unmapped character with the 2-byte encoder, and before calling Try4BytesEncoder checks whether there are 4 bytes free in the output buffer and if not returns NS_OK_UENC_MORE_OUTPUT instead of NS_ERROR_UENC_NOMAPPING. This check is not needed for gbk, where Try4BytesEncode will always fail.

Fixing that is sufficient for the testcase in comment 0, but didn't fix an xpcshell test I wrote, because

b) ConvertNoBuff doesn't handle non-default values of mErrBehavior
(Assignee)

Comment 2

3 years ago
Created attachment 8614917 [details] [diff] [review]
Patch
Attachment #8614917 - Flags: review?(VYV03354)
(Assignee)

Comment 3

3 years ago
Created attachment 8614918 [details] [diff] [review]
xpcshell test (includes all encoders)
Attachment #8614917 - Flags: review?(VYV03354) → review+
https://hg.mozilla.org/mozilla-central/rev/10dfb6455e4f
https://hg.mozilla.org/mozilla-central/rev/c5f6b5160c02
Status: NEW → RESOLVED
Last Resolved: 3 years ago
status-firefox41: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla41
Depends on: 1255863
You need to log in before you can comment on or make changes to this bug.