1342703 - JS_EncodeStringToBuffer should encode to UTF-8

Status: RESOLVED INVALID

(Core :: JavaScript Engine, defect)

Description

[Philip Chimento [:ptomato]]

Attachment: Full test program

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0
Build ID: 20170126153103

Steps to reproduce:

Here's a snippet of embedder code, which I ran with the usual boilerplate around it:

    const char *script = u8"Array.from('

Comment 1

[Philip Chimento [:ptomato]]

Oh wow, Bugzilla does not dig those higher codepoint characters.

Following is approximately what I typed into the form originally :-)

Here's a snippet of embedder code, which I ran with the usual boilerplate around it:

    const char *script = u8"Array.from('(characters that I can't paste here)')";
    JS::CompileOptions options(cx);
    options.setUTF8(true);
    JS::Evaluate(cx, options, script, strlen(script), &rval);
        
    JS::RootedString str(cx, JS::ToString(cx, rval));

    char *encoded = JS_EncodeStringToUTF8(cx, str); 
    printf("JS_EncodeStringToUTF8: %s\n", encoded);
    JS_free(cx, encoded);

    size_t len = JS_GetStringEncodingLength(cx, str);
    char *buffer = new char[len + 1];
    JS_EncodeStringToBuffer(cx, str, buffer, len);
    buffer[len] = '\0';
    printf("JS_EncodeStringToBuffer: %s\n", buffer);
    delete buffer;

(Full program is attached.)

Output:

    JS_EncodeStringToUTF8: (correct output that I can't paste here)
    JS_EncodeStringToBuffer: <j,<f,<g,<i

Expected behaviour:

The output after the colon on the two lines should be identical. According to [1], JS_EncodeStringToBuffer() should fill the buffer with UTF-8 encoded text. Instead, it seems to be discarding the second byte of each two-byte character.

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_Reference/JS_EncodeStringToBuffer

Comment 2

[Tooru Fujisawa [:arai]]

sorry, the document is wrong.

it would be better introducing UTF8 variant, instead of changing existing API's behavior.
and maybe deprecate existing one later, so that embedders can easily switch to newer one.

Comment 3

[Tooru Fujisawa [:arai]]

for now, fixed the document.

Comment 4

[Tooru Fujisawa [:arai]]

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS_EncodeStringToBuffer$compare?locale=en-US&to=1193853&from=725373

Comment 5

[Philip Chimento [:ptomato]]

OK, thanks! I'll use a different function then.

Comment 6

[Tooru Fujisawa [:arai]]

hm, there are already alternatives.

JS::GetDeflatedUTF8StringLength for calculating length, and
JS::DeflateStringToUTF8Buffer for encoding string to buffer.

both declared in js/public/CharacterEncoding.h

I'll create documentation

Comment 7

[Tooru Fujisawa [:arai]]

added documents.

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS::DeflateStringToUTF8Buffer
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS::GetDeflatedUTF8StringLength

also added link to them from:

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS_EncodeStringToBuffer
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS_GetStringEncodingLength

Comment 8

[Philip Chimento [:ptomato]]

Thanks! This bug can be closed now, I think.