1288298 - Fix Unicode encoding mismatch between JS-Plugins and the PPAPI Plug-in.

Status: RESOLVED WORKSFORME

(Core Graveyard :: Plug-ins, defect, P3)

Description

[Bruce Sun [:brsun]]

PPAPI uses UTF-8 as the encoding for Unicode strings (i.e. str_t), but JavaScript uses UCS-2/UTF-16 instead. We need to figure out how to convert the Unicode schemes back and forth properly in our design.

Comment 1

[Tim Guan-tin Chien [:timdream] (please needinfo)]

There is a dedicated Web API: https://developer.mozilla.org/en-US/docs/Web/API/TextEncoder

But there is another one for the lazy bones, which encodes the JS string into the old and fake "binary string":

var bin_str = unescape(encodeURI(str));

Comment 2

[Evelyn Hung]

We do use TextEncoder API in jsplugin, but as I know this bug will dig into Python and C++ generator code to make sure encoding in lower layer's works well.

Comment 3

[Evelyn Hung]

... but maybe jsplugin need to be modified too. Thanks for the info. :)

Comment 4

[Bruce Sun [:brsun]]

Attachment: https://github.com/mozilla/flapper.js/pull/62

Comment 5

[Bruce Sun [:brsun]]

I think I forget to mention the changes of this pull-request on this bug.

In summary, this pull-request is used to let JavaScript world uses its native strings without caring about the encoding transformation to and from |str_t| of pepper interfaces.

This pull-request includes:
 - Using only printable characters of ASCII in JSON strings while converting from UTF-8 characters.
 - A new class (i.e. |UnicodeString|) and a new modifier (i.e. |UnicodeEncoding|) to handle transcoding between UTF-8 encoded strings and printable-characters-only JSON strings.
 - A new MAKE target (i.e. |test-unicode|) for testing the correctness of transcoding.
 - A new environment variable (i.e. |STRICT|) for compiling codes with verbose-warnings and warnings-as-errors.
 - Add one workaround (i.e. |PPAPIUtility.getStringLength(...)|) in JavaScript world to calculate the string-length for pepper interfaces.
 - Add comments in runtime for each customized parameter.
 - Corresponding changes of RPC.

Comment 6

[Bruce Sun [:brsun]]

>  - A new class (i.e. |UnicodeString|) and a new modifier (i.e.
> |UnicodeEncoding|) to handle transcoding between UTF-8 encoded strings and
> printable-characters-only JSON strings.

s/modifier/manipulator

Comment 7

[Bruce Sun [:brsun]]

Comment on attachment 8794098 [details] [review]
https://github.com/mozilla/flapper.js/pull/62

According to the discussion offline, we would do transcoding stuffs in Gecko instead of doing that in RPC. If we really encounter performance issues, we would breakdown the issues and have further studies by then.

Comment 8

[Bruce Sun [:brsun]]

Un-assign myself because this bug would be handled by Peter's patch.

Comment 9

[Evelyn Hung]

Can't reproduce this issue anymore.