467263 - Create a performance test suite that grows

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Robert Sayre]

Continually adding tests means the suite needn't keep an aggregate score. We just want to graph each individual test.

Comment 1

[Robert Sayre]

some input:

http://nerget.com/rayjs/rayjs.html 
http://nerget.com/rayjs2/rayjs.html
http://nerget.com/rayjs3/rayjs.html

Comment 2

[Andreas Gal :gal]

While existing code is important, I would suggest we also collect new code. The world doesn't need yet-another-collection-of-poorly-written-JS-code like SunSpider. bz's test cases are much more interesting from a forward-looking perspective. So we should have both.

Comment 3

[Boris Zbarsky [:bzbarsky]]

I think a good start would be the various TM: bugs we already have with performance testcases...

Comment 4

[Brendan Eich [:brendan]]

[5:18pm] graydon: http://www.pps.jussieu.fr/~canou/obrowser/tutorial/
[5:18pm] graydon: implemented the ocaml VM and ported standard libraries to DOM/canvas/browser equivalents
[5:18pm] graydon: cute port
[5:19pm] brendan: switch loop? LIR_jtbl will help make that fast, but if we can figure out what the virtual pc is, we can trace that and do even better
[5:20pm] graydon: I only browsed it briefly, don't remember exact patterns. I figured bootstrapping ocamlc on tracemonkey might make a good testcase though.
[5:20pm] graydon: nice big offline process
[5:21pm] graydon: a lot of our tests suffer from not really doing anything serious. serious js programs are all "to be run in a browser + DOM environment"
[5:21pm] graydon: more bulk shell-friendly tests would be good
[5:22pm] brendan: graydon: you've seen the bug sayrer filed, on the ever-growing perf test collection?
[5:22pm] brendan: we should track this
[5:22pm] graydon: no, bug number? I'd love to collect some good ones. I made a list of a few of the emulator ports and such.
[5:22pm] graydon: I figure if someone cooks up an x86 emulator we're home free 

/be

Comment 5

[Andreas Gal :gal]

We did a PowerPC/ARM/MIPS emulator at UCI that emits JVM bytecode (using trace recording, of course :) ). There is some related work on an x86 emulator written in Java. Porting that to JS would be fun/crazy/scary.

Comment 6

[David Mandelin [:dmandelin]]

I like the existing suggestions, but I think a good first step would be to create a suite of microbenchmarks that test different basic operations. Basic operations that are slow would be a relatively easy thing to start improving as well (compared to system-level effects and such). 

After that, stuff that depends on GC behavior (and hopefully can record detailed GC stats as well) would be great, because we know we have problems there.

Comment 7

[Jason Orendorff [:jorendorff]]

It would be so nice just to be able to run a command that means "does this patch regress anything?".

We should get someone assigned to this, even if we have to draw straws.  :(

Comment 8

[David Mandelin [:dmandelin]]

(In reply to comment #7)
> It would be so nice just to be able to run a command that means "does this
> patch regress anything?".

The data analysis for that is kind of challenging. It sounds like the kind of thing I would like to do, but I think I should focus on finishing closures/arguments for now.

Comment 9

[Mayank Bansal]

No new bug has been filed in the last 13 years. And now we have a set of benchmarks that we prioritize.
Closing this as fixed, but please reopen if needed.