Open Bug 467263 Opened 16 years ago Updated 2 years ago

Create a performance test suite that grows

Categories

(Core :: JavaScript Engine, defect)

x86
macOS
defect

Tracking

()

People

(Reporter: sayrer, Unassigned)

References

(Depends on 5 open bugs)

Details

Continually adding tests means the suite needn't keep an aggregate score. We just want to graph each individual test.
While existing code is important, I would suggest we also collect new code. The world doesn't need yet-another-collection-of-poorly-written-JS-code like SunSpider. bz's test cases are much more interesting from a forward-looking perspective. So we should have both.
Depends on: 451974
I think a good start would be the various TM: bugs we already have with performance testcases...
[5:18pm] graydon: http://www.pps.jussieu.fr/~canou/obrowser/tutorial/
[5:18pm] graydon: implemented the ocaml VM and ported standard libraries to DOM/canvas/browser equivalents
[5:18pm] graydon: cute port
[5:19pm] brendan: switch loop? LIR_jtbl will help make that fast, but if we can figure out what the virtual pc is, we can trace that and do even better
[5:20pm] graydon: I only browsed it briefly, don't remember exact patterns. I figured bootstrapping ocamlc on tracemonkey might make a good testcase though.
[5:20pm] graydon: nice big offline process
[5:21pm] graydon: a lot of our tests suffer from not really doing anything serious. serious js programs are all "to be run in a browser + DOM environment"
[5:21pm] graydon: more bulk shell-friendly tests would be good
[5:22pm] brendan: graydon: you've seen the bug sayrer filed, on the ever-growing perf test collection?
[5:22pm] brendan: we should track this
[5:22pm] graydon: no, bug number? I'd love to collect some good ones. I made a list of a few of the emulator ports and such.
[5:22pm] graydon: I figure if someone cooks up an x86 emulator we're home free 

/be
We did a PowerPC/ARM/MIPS emulator at UCI that emits JVM bytecode (using trace recording, of course :) ). There is some related work on an x86 emulator written in Java. Porting that to JS would be fun/crazy/scary.
Depends on: 460050
Depends on: 465773
Blocks: 501277
Depends on: 479090
Depends on: 503470
I like the existing suggestions, but I think a good first step would be to create a suite of microbenchmarks that test different basic operations. Basic operations that are slow would be a relatively easy thing to start improving as well (compared to system-level effects and such). 

After that, stuff that depends on GC behavior (and hopefully can record detailed GC stats as well) would be great, because we know we have problems there.
Depends on: 508716
Depends on: 508849
It would be so nice just to be able to run a command that means "does this patch regress anything?".

We should get someone assigned to this, even if we have to draw straws.  :(
(In reply to comment #7)
> It would be so nice just to be able to run a command that means "does this
> patch regress anything?".

The data analysis for that is kind of challenging. It sounds like the kind of thing I would like to do, but I think I should focus on finishing closures/arguments for now.
Depends on: 514765
Depends on: 517806
Depends on: 519770
Depends on: 514120
Depends on: 536564
Blocks: 540512
Depends on: 585258
Depends on: 462175
Depends on: 469351
Depends on: 289669
Depends on: 604905
Depends on: 608490
Depends on: 608733
Blocks: 609212
No longer blocks: 609212
Depends on: 609212
Depends on: 612930
Depends on: 610370
Depends on: 608480
Depends on: 120553
Depends on: 606483
Depends on: 497458
Depends on: 628209
Depends on: 636096
Depends on: 654410
Depends on: 658844
Depends on: 662503
Depends on: 659467
Depends on: 667096
Depends on: 455751
Depends on: 470139
Depends on: 470143
Depends on: 470144
Depends on: 483457
Depends on: 484624
Depends on: 501189
Depends on: 507746
Depends on: 535728
Depends on: 535925
Depends on: 577691
Depends on: 602479
Depends on: 607686
Depends on: 580492
Depends on: 462095
Depends on: 470128
Depends on: 483870
Depends on: 496377
Depends on: 494268
Depends on: 531629
Depends on: 577036
Depends on: 590161
Depends on: 594655
Depends on: 609296
Depends on: 639385
Depends on: 531185
Depends on: 607201
Depends on: 621203
Depends on: 663072
Depends on: 704387
Depends on: 675167
Depends on: 607692
Assignee: general → nobody
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.