649487 - Streamline process to add tests to AWFY

Status: NEW

(Testing :: Performance, defect, P3)

Description

[Brian Hackett [Laid off!]]

AWFY currently only tracks the three main benchmarks we're interested in.  Adding more stuff to the front page would clutter things up, but it seems like we could add more tests either to the breakdown page or to a similarly setup page that tracks performance over time for other benchmark suites, shell tests derived from js-heavy websites (raytracers, emulators, ...), and microbenchmarks for basic and advanced language features (e.g. typed arrays).  Would be nice to have a repo where people could add new tests without much pain.

Comment 1

[Nicholas Nethercote [inactive]]

I know that billm has a bunch of benchmarks that he has collated into a suite that he can run through the Sunspider harness.  CC'ing.

Comment 2

[Brian Hackett [Laid off!]]

This looks like it's working now.  I made a user repo at:

http://hg.mozilla.org/users/bhackett_mozilla.com/awfy-assorted

This is just a suite using the sunspider harness which awfy pulls from each time it runs, and publishes the results at:

http://arewefastyet.com/?a=b&view=assorted

(Also reachable at 'Test View' -> 'Assorted Tests').  To add new tests with this harness, add the test files to 'tests/assorted' and update 'tests/assorted/LIST' and 'assorted-results/sunspider-test-prefix' (this could use streamlining).  After pushing the new tests, when AWFY runs in the future the tests should show up in the above page.

Comment 3

[David Anderson [:dvander] - inactive, e-mail if emergency]

Nice!

Comment 4

[David Anderson [:dvander] - inactive, e-mail if emergency]

Should we should have TI+JM on that graph as well?

Comment 5

[Brian Hackett [Laid off!]]

TI+JM is on the graph, but the TM variants (JM, TM, JM+TM) don't show up because the TM branch was burning when I ran AWFY.  That just got fixed, will cycle again.

Comment 6

[Tom S [:evilpie]]

We should really work on this. This could probably be really useful when used right. I think looking at some recent performance bugs for IonMonkey there should be plenty of (micro) benchmarks we could track.
I just added a test from Bug 774119 to assorted and noticed that there about 5 .js files not in the LIST, but when I added them the sunspider script started to produce bogus output.

Comment 7

[Mayank Bansal]

AWFY is maintained by the performance monitoring these days.