506772 - Create new Talos suite: add new "V8" talos suite

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Mark Steele]

Attachment: self-contained module for page_load_test

V8 benchmarks hit some JS perf corners that the sunspider tests don't.
It'd be nice to get them working as timing runs to track regressions & progress.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Still trying to learn how to add new suites to talos perf infrastructure.

1) is this suite self-contained?
2) is this for all o.s. or only certain subsets? What about mobile, for example?
3) approx how long does this suite take to run?

Comment 2

[Mark Steele]

(In reply to comment #1)
> Still trying to learn how to add new suites to talos perf infrastructure.
> 
> 1) is this suite self-contained?
Well, it's self-contained in that I didn't add any config stuff. It's just a dir of test drivers wrapped in pages with a manifest.
> 2) is this for all o.s. or only certain subsets? What about mobile, for
> example?
Not sure. It should really only exercise JS so I don't know how many machines need to run this. It would be nice to get the tracemonkey builds tracking this.
> 3) approx how long does this suite take to run?
Well, I had to modify the common test driver to run bounded by the number of runs, rather than bounded by a (1 second) max runtime.
I tried a few numbers and landed on 20 runs.
With all tests, that takes about 20 seconds on my machine.
Unfortunately, the run times vary a lot, with richards taking .08s and raytrace taking 4s. The tests might need different run counts, or something.

I didn't see a way of just reporting the v8-style runs/second numbers.

Comment 3

[alice nodelman [:alice] [:anode]]

Looks to me like you are just loading pages sequentially and then reporting the results - if this is the case then you should really be using the pageloader extension instead of your own run.js code.  In the talos code you can see in the sunspider/ and jss/ directories examples of the format individual pages loaded by the pageloader have to meet.

Comment 4

[Mark Steele]

(In reply to comment #3)
> Looks to me like you are just loading pages sequentially and then reporting the
> results - if this is the case then you should really be using the pageloader
> extension instead of your own run.js code.  In the talos code you can see in
> the sunspider/ and jss/ directories examples of the format individual pages
> loaded by the pageloader have to meet.

Well, the run.js is just vestigial from the original source. It's not used in the test pages. I did look at the sunspider pages and they just seem to be standalone wrappers that run the test at pageload time and the total pageload time is used as the benchmark time. That doesn't seem like the most optimal way, since it counts setup time toward the benchmark, but since I didn't see a way to pull out reported values, that's what I tried to do with the V8 stuff here too. I only created wrapper pages and added a manifest in the style of sunspider.manifest. So I think these already roughly match the style of the current sunspider tests.
I'm not familiar with the talos stuff, so I'm not sure what else is needed to make this go. AFAICT from the sunspider integration bug, it just needs a config line.

Comment 5

[alice nodelman [:alice] [:anode]]

If you are interested in using a different timing metric you can pass the pageloader a manifest that uses '%' before the names of the individual pages.  That tells the pageloader that the page in question will handle its own timing and pass it back in a known format.  Examples of that are in the jss/ directory.

Turning on the test would be a question of adding the test to the list of running tests in various buildbot masters.  We should stage the test for a while to see if it generates consistent, reliable results and then we can consider rolling it out.

What branches are you interested in having this test run on?

Comment 6

[Mark Steele]

Attachment: v8 module using tpRecordTime

I think this is usable now. Each test runs for roughly 1 second like the original V8 tests and passes back the time for the average time of a single loop iteration through tpRecordTime.
Hopefully this is close enough to use to shake out some numbers and verify the test.
I don't know if this is needed outside of tracemonkey, so if it's on a box or two there I think it'd be enough.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: sql update for production graph server db

Comment 8

[Chris Cooper [:coop] (he/him)]

Lukas: does comment #7 mean you're working on this? Is this scheduled for the downtime later this week?

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: incorporate v8 tests into buildbot-configs

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 411463 [details] [diff] [review]
incorporate v8 tests into buildbot-configs

this patch is dependent on https://bugzilla.mozilla.org/attachment.cgi?id=410865

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add new v8 test to graph server db

Comment 12

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: incorporate v8 tests into talos

the v8 dir in talos cvs has already been added.

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

yes, this will go into the downtime tomorrow at 8am PST

Comment 14

[alice nodelman [:alice] [:anode]]

Comment on attachment 410904 [details]
 sql update for production graph server db

Let's get a patch for this associated with the correct graph server sql files as well.

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: incorporate v8 tests into talos (now with manifest)

this one has the manifest in it

Comment 16

[alice nodelman [:alice] [:anode]]

Comment on attachment 411546 [details] [diff] [review]
incorporate v8 tests into talos (now with manifest)

Since these tests are designed by others this is pretty much just a *stamp* - but the talos integration looks correct.

Comment 17

[alice nodelman [:alice] [:anode]]

Comment on attachment 411463 [details] [diff] [review]
incorporate v8 tests into buildbot-configs

Only issue here is that we no longer want to trigger two tests per try server build.  When checking in please make that small change.

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: incorporate v8 tests into buildbot-configs with updated numberOfBuildsToTrigger

Just so I don't forget when it's early tomorrow morning.

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to comment #14)
> (From update of attachment 410904 [details])
> Let's get a patch for this associated with the correct graph server sql files
> as well.

that would be https://bugzilla.mozilla.org/attachment.cgi?id=411464 wouldn't it? if I'm missing something let me know.

Comment 20

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 411546 [details] [diff] [review]
incorporate v8 tests into talos (now with manifest)

RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/base.js,v
done
Checking in base.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/base.js,v  <--  base.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/crypto.js,v
done
Checking in crypto.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/crypto.js,v  <--  crypto.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/deltablue.js,v
done
Checking in deltablue.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/deltablue.js,v  <--  deltablue.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/earley-boyer.js,v
done
Checking in earley-boyer.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/earley-boyer.js,v  <--  earley-boyer.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/raytrace.js,v
done
Checking in raytrace.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/raytrace.js,v  <--  raytrace.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/regexp.js,v
done
Checking in regexp.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/regexp.js,v  <--  regexp.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/richards.js,v
done
Checking in richards.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/richards.js,v  <--  richards.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-crypto.html,v
done
Checking in run-crypto.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-crypto.html,v  <--  run-crypto.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-deltablue.html,v
done
Checking in run-deltablue.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-deltablue.html,v  <--  run-deltablue.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-earley-boyer.html,v
done
Checking in run-earley-boyer.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-earley-boyer.html,v  <--  run-earley-boyer.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-raytrace.html,v
done
Checking in run-raytrace.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-raytrace.html,v  <--  run-raytrace.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-regexp.html,v
done
Checking in run-regexp.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-regexp.html,v  <--  run-regexp.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-richards.html,v
done
Checking in run-richards.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-richards.html,v  <--  run-richards.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-splay.html,v
done
Checking in run-splay.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-splay.html,v  <--  run-splay.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run.js,v
done
Checking in run.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run.js,v  <--  run.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/splay.js,v
done
Checking in splay.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/splay.js,v  <--  splay.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/v8.manifest,v
done
Checking in v8.manifest;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/v8.manifest,v  <--  v8.manifest
initial revision: 1.1
done

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 411555 [details] [diff] [review]
incorporate v8 tests into buildbot-configs with updated numberOfBuildsToTrigger

http://hg.mozilla.org/build/buildbot-configs/rev/c65f64dec894

r+ = alice from previous version

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 411464 [details] [diff] [review]
add new v8 test to graph server db

http://hg.mozilla.org/graphs/rev/9c82b30ffe2d

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: adds v8 to sample.config in talos

bustage fix, missed this in original patch

r+ = alice

Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.37; previous revision: 1.36

Comment 24

[Lukas Blakk [:lsblakk] use ?needinfo]

Up and running as of 10am PST, closing.

Comment 25

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add v8 tests to master.cfg for talos-staging-pool

Comment 26

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 412026 [details] [diff] [review]
add v8 tests to master.cfg for talos-staging-pool

http://hg.mozilla.org/build/buildbot-configs/rev/2ddbb56ab603

Comment 27

[Jeff Muizelaar [:jrmuizel]]

Is this suite affected by bug 467120? i.e. do we discard the test result that took the longest like we do for tp?