745095 - Converting between ArrayBuffer and String

Status: RESOLVED DUPLICATE of bug 795544

(Core :: JavaScript Engine, enhancement)

Description

[David Teller [:Yoric] - still alive but not very active]

OS.File will provide JS file read/write using ArrayBuffer. I believe that, atm, we have no way of converting between an ArrayBuffer and a string, which is a shame.

Note that this is very related to bug 552551.

Comment 1

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

There was some discussion of such a thing on the whatwg list, IIRC.

Comment 2

[Boris Zbarsky [:bzbarsky]]

Converting in what sense?  One jschar per byte, via byte inflation?  Or something else?

Comment 3

[David Mandelin [:dmandelin]]

(In reply to David Rajchenbach Teller [:Yoric] from comment #0)
> OS.File will provide JS file read/write using ArrayBuffer. I believe that,
> atm, we have no way of converting between an ArrayBuffer and a string, which
> is a shame.
> 
> Note that this is very related to bug 552551.

Couldn't one write JS functions to perform various conversions from ArrayBuffer to string?

Comment 4

[Masatoshi Kimura [:emk]]

new FileReader(new Blob([someArrayBuffer])).readAsWhateverYouWant() will provide various conversions. Is a C++ API really required?

Comment 5

[David Teller [:Yoric] - still alive but not very active]

(In reply to Masatoshi Kimura [:emk] from comment #4)
> new FileReader(new Blob([someArrayBuffer])).readAsWhateverYouWant() will
> provide various conversions. Is a C++ API really required?

Good point, this lets us convert an ArrayBuffer to various encodings, which solves a big part of the reading issue - although this might be insufficient if one wishes to decode streams in JavaScript, as we also need to be able to handle cases in which only a prefix of the ArrayBuffer is correctly encoded.


Now, how do we do the opposite? Say I have a String and I want to produce an ArrayBuffer with an encoded version of that string.

Comment 6

[Masatoshi Kimura [:emk]]

(In reply to David Rajchenbach Teller [:Yoric] from comment #5)
> Good point, this lets us convert an ArrayBuffer to various encodings, which
> solves a big part of the reading issue - although this might be insufficient
> if one wishes to decode streams in JavaScript, as we also need to be able to
> handle cases in which only a prefix of the ArrayBuffer is correctly encoded.
Use
new Blob([someArrayBuffer]).slice(start, end)
or
new Blob([new Uint8Array(someArrayBuffer, offset, length)]) (Fx15+)

> Now, how do we do the opposite? Say I have a String and I want to produce an
> ArrayBuffer with an encoded version of that string.
Unfortunately existing APIs are intentionally crippled to promote UTF-8 :(
So some APIs will be needed to encode a string back to an ArrayBuffer with a non-UTF-8 encoding.

Comment 7

[David Teller [:Yoric] - still alive but not very active]

(In reply to Masatoshi Kimura [:emk] from comment #6)
> (In reply to David Rajchenbach Teller [:Yoric] from comment #5)
> > Good point, this lets us convert an ArrayBuffer to various encodings, which
> > solves a big part of the reading issue - although this might be insufficient
> > if one wishes to decode streams in JavaScript, as we also need to be able to
> > handle cases in which only a prefix of the ArrayBuffer is correctly encoded.
> Use
> new Blob([someArrayBuffer]).slice(start, end)
> or
> new Blob([new Uint8Array(someArrayBuffer, offset, length)]) (Fx15+)

That's assuming you know how many bytes form a proper well-encoded string. In the case of a stream, you typically receive |n| bytes (with |n| fixed), but only a prefix of unknown length is actually a well-encoded string. In a multi-byte encoding, the final few bytes are often an incomplete char, which needs special handling.

> 
> > Now, how do we do the opposite? Say I have a String and I want to produce an
> > ArrayBuffer with an encoded version of that string.
> Unfortunately existing APIs are intentionally crippled to promote UTF-8 :(
> So some APIs will be needed to encode a string back to an ArrayBuffer with a
> non-UTF-8 encoding.

In that case, I am keeping this bug open :)

Comment 8

[Masatoshi Kimura [:emk]]

StringEncoding API (bug 764234) would meet the requirements. It supports streaming encoding/decoding. If you need to convert a non-UTF8 ArrayBuffer to a string, I consider adding a Chrome-only option to TextEncoder so that it can encode non-UTF8 bytes.

Comment 9

[Masatoshi Kimura [:emk]]

> non-UTF8 ArrayBuffer
non-UTF ArrayBuffer. TextEncoder supports utf-8, utf-16, and utf-16be atm.

Comment 10

[Masatoshi Kimura [:emk]]

> convert a non-UTF8 ArrayBuffer to a string
It was "convert a string to a non-UTF ArrayBuffer"... TextDecoder (ArrayBuffer-to-string converter) accepts non-UTF encodings.

Comment 11

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Is there anything left here to do?

Comment 12

[David Teller [:Yoric] - still alive but not very active]

I think we're ok.