420397 - JSON roundtrip of UTF-8 chars fails

Status: RESOLVED WORKSFORME

(Core :: DOM: Core & HTML, defect)

Description

[Dan Mills [:thunder]]

Attachment: test case

I have a small test case which I believe should work: I'm trying to write out json to a file which contains some Japanese strings, then I'm reading it back in and checking it.

See attached file.  Just run it with xpcshell.

Comment 1

[Dan Mills [:thunder]]

The file is in UTF-8, by the way.

Comment 2

[Dan Mills [:thunder]]

The problem is only when using decodeFromStream().  When I do this instead, it works (maybe by chance, it's picking the right charset?):

function read_stream(is) {
  var ret = "";
  var chunk = is.read(4096);
  while (chunk.length > 0) {
    ret += chunk;
    chunk = is.read(4096);
  }
  return ret;
}

// ...

var sis = Cc["@mozilla.org/scriptableinputstream;1"].
  createInstance(Ci.nsIScriptableInputStream);
sis.init(istream);
var decoded = nativeJSON.decode(read_stream(sis));

Comment 3

[Simon Montagu :smontagu]

It works if I change the line
 if (decoded.foo == "日本語の言葉です。")
to
 if (decoded.foo == "\u65E5\u672C\u8A9E\u306E\u8A00\u8449\u3067\u3059\u3002")
so the problem is that the source file isn't being read as UTF-8.

Comment 4

[Robert Sayre]

(In reply to comment #3)
> the problem is that the source file isn't being read as UTF-8.

Definitely a bug. The decoder should be able to tell from the first few bytes, per the JSON rfc.

Comment 5

[Simon Montagu :smontagu]

Wait a minute. The decoder *is* recognizing the stream as UTF-8 and converting it, but the string "日本語の言葉です。" in the line I quoted is being taken as Latin-1.

In the version in comment 2, using decode instead of decodeFromStream, both come out as Latin-1.

Comment 6

[Dan Mills [:thunder]]

This bug might be related to bug 420411.

Comment 7

[Henri Sivonen (:hsivonen) (away from Bugzilla until 2026-04-13)]

nsIJSON is gone.