Closed Bug 603838 Opened 10 years ago Closed 5 years ago

Kraken: JSON tests are too easy to game


(Core :: JavaScript Engine, defect)

Not set





(Reporter: njn, Unassigned)



json-parse-financial.js looks like this:

  for (var i = 0; i < 1000; i++) {
    var x = JSON.parse(data);

json-stringify-tinderbox.js looks like this:

  for (var i = 0; i < 20; i++) {
    var x = JSON.stringify(tinderbox_data);

In both cases the result of the JSON function isn't used, and so it's easy to cheat and call versions of the functions that do less work.

They should both be changed so that the result is used in some fashion, eg. by extracting a small part of the result and adding it to a simple accumulator variable.  (Printing the accumulator would be even better, but that might not be acceptable.)
Assignee: nnethercote → general
Just curious, since I noticed this while poking around.  Aside from the not-doing-something-useful.  Is the fact that the same string is parsed 1000 times somewhere code might just decide to reuse last parse?  That is, maybe the string should shift between iterations?
Assignee: general → nobody
Kraken is fading away. I don't think this needs to be open any more.
Closed: 5 years ago
Resolution: --- → WONTFIX
Hm.  Still featured front and centre on …

Unlike, say, asm.js µbench and asm.js apps which are still subsections.

And, what kraken does is still an interesting thing to measure no?
> And, what kraken does is still an interesting thing to measure no?

In my opinion Kraken is a low-quality benchmark.

- The original version was designed to make Firefox's JIT at the time (the long-gone TraceMonkey) look good.

- The original version had numerous bugs.

- It contains benchmarks that should use typed arrays but do not.

- Mozilla never maintained it properly -- Google produced and hosted the 1.1 version for some time(!)

I also think Sunspider is a low-quality benchmark these days but it's still reasonably prominent, much more so than Kraken.
hum, as I recall, was a response to one of the nacl justifications - that JS was simply to slow for some things. thus the JS darkroom thingy. and, yeah, no typed arrays, which was to allow running in browsers w/o them at the time. 
Seems like the ops measured are still interesting, apart from the typed array thingy. But NBD. Was just wondering why the abandonment when it and SS were still featured and Massive wasn't.
You need to log in before you can comment on or make changes to this bug.