Closed Bug 506772 Opened 10 years ago Closed 10 years ago

Create new Talos suite: add new "V8" talos suite

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mwsteele, Assigned: lsblakk)

Details

Attachments

(7 files, 3 obsolete files)

95.74 KB, application/zip
Details
522 bytes, text/plain
anodelman
: review+
Details
2.89 KB, patch
anodelman
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
462.18 KB, patch
anodelman
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
25.63 KB, patch
lsblakk
: checked-in+
Details | Diff | Splinter Review
1020 bytes, patch
lsblakk
: checked-in+
Details | Diff | Splinter Review
1.33 KB, patch
bhearsum
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
V8 benchmarks hit some JS perf corners that the sunspider tests don't.
It'd be nice to get them working as timing runs to track regressions & progress.
Still trying to learn how to add new suites to talos perf infrastructure.

1) is this suite self-contained?
2) is this for all o.s. or only certain subsets? What about mobile, for example?
3) approx how long does this suite take to run?
Component: Release Engineering → Release Engineering: Future
Summary: add V8 benchmarks to talos → add new "V8" talos suite
(In reply to comment #1)
> Still trying to learn how to add new suites to talos perf infrastructure.
> 
> 1) is this suite self-contained?
Well, it's self-contained in that I didn't add any config stuff. It's just a dir of test drivers wrapped in pages with a manifest.
> 2) is this for all o.s. or only certain subsets? What about mobile, for
> example?
Not sure. It should really only exercise JS so I don't know how many machines need to run this. It would be nice to get the tracemonkey builds tracking this.
> 3) approx how long does this suite take to run?
Well, I had to modify the common test driver to run bounded by the number of runs, rather than bounded by a (1 second) max runtime.
I tried a few numbers and landed on 20 runs.
With all tests, that takes about 20 seconds on my machine.
Unfortunately, the run times vary a lot, with richards taking .08s and raytrace taking 4s. The tests might need different run counts, or something.

I didn't see a way of just reporting the v8-style runs/second numbers.
Looks to me like you are just loading pages sequentially and then reporting the results - if this is the case then you should really be using the pageloader extension instead of your own run.js code.  In the talos code you can see in the sunspider/ and jss/ directories examples of the format individual pages loaded by the pageloader have to meet.
(In reply to comment #3)
> Looks to me like you are just loading pages sequentially and then reporting the
> results - if this is the case then you should really be using the pageloader
> extension instead of your own run.js code.  In the talos code you can see in
> the sunspider/ and jss/ directories examples of the format individual pages
> loaded by the pageloader have to meet.

Well, the run.js is just vestigial from the original source. It's not used in the test pages. I did look at the sunspider pages and they just seem to be standalone wrappers that run the test at pageload time and the total pageload time is used as the benchmark time. That doesn't seem like the most optimal way, since it counts setup time toward the benchmark, but since I didn't see a way to pull out reported values, that's what I tried to do with the V8 stuff here too. I only created wrapper pages and added a manifest in the style of sunspider.manifest. So I think these already roughly match the style of the current sunspider tests.
I'm not familiar with the talos stuff, so I'm not sure what else is needed to make this go. AFAICT from the sunspider integration bug, it just needs a config line.
If you are interested in using a different timing metric you can pass the pageloader a manifest that uses '%' before the names of the individual pages.  That tells the pageloader that the page in question will handle its own timing and pass it back in a known format.  Examples of that are in the jss/ directory.

Turning on the test would be a question of adding the test to the list of running tests in various buildbot masters.  We should stage the test for a while to see if it generates consistent, reliable results and then we can consider rolling it out.

What branches are you interested in having this test run on?
I think this is usable now. Each test runs for roughly 1 second like the original V8 tests and passes back the time for the average time of a single loop iteration through tpRecordTime.
Hopefully this is close enough to use to shake out some numbers and verify the test.
I don't know if this is needed outside of tracemonkey, so if it's on a box or two there I think it'd be enough.
Attachment #390929 - Attachment is obsolete: true
Summary: add new "V8" talos suite → Create new Talos suite: add new "V8" talos suite
Attachment #410904 - Attachment mime type: application/octet-stream → text/plain
Assignee: nobody → lsblakk
Assignee: lsblakk → nobody
Component: Release Engineering: Future → Release Engineering
Priority: -- → P2
Lukas: does comment #7 mean you're working on this? Is this scheduled for the downtime later this week?
Attachment #411463 - Flags: review?(anodelman)
Comment on attachment 411463 [details] [diff] [review]
incorporate v8 tests into buildbot-configs

this patch is dependent on https://bugzilla.mozilla.org/attachment.cgi?id=410865
Attachment #411463 - Attachment description: incorporate v8 tests into talos → incorporate v8 tests into buildbot-configs
Attachment #410904 - Attachment description: add v8 tests to graph db → sql update for production graph server db
Attachment #410904 - Flags: review?(anodelman)
Attached patch incorporate v8 tests into talos (obsolete) — Splinter Review
the v8 dir in talos cvs has already been added.
Attachment #411479 - Flags: review?(anodelman)
Assignee: nobody → lsblakk
yes, this will go into the downtime tomorrow at 8am PST
Comment on attachment 410904 [details]
 sql update for production graph server db

Let's get a patch for this associated with the correct graph server sql files as well.
Attachment #410904 - Flags: review+
Attachment #411464 - Flags: review?(anodelman) → review+
this one has the manifest in it
Attachment #411479 - Attachment is obsolete: true
Attachment #411546 - Flags: review?(anodelman)
Attachment #411479 - Flags: review?(anodelman)
Comment on attachment 411546 [details] [diff] [review]
incorporate v8 tests into talos (now with manifest)

Since these tests are designed by others this is pretty much just a *stamp* - but the talos integration looks correct.
Attachment #411546 - Flags: review?(anodelman) → review+
Comment on attachment 411463 [details] [diff] [review]
incorporate v8 tests into buildbot-configs

Only issue here is that we no longer want to trigger two tests per try server build.  When checking in please make that small change.
Attachment #411463 - Flags: review?(anodelman) → review+
Just so I don't forget when it's early tomorrow morning.
Attachment #411463 - Attachment is obsolete: true
(In reply to comment #14)
> (From update of attachment 410904 [details])
> Let's get a patch for this associated with the correct graph server sql files
> as well.

that would be https://bugzilla.mozilla.org/attachment.cgi?id=411464 wouldn't it? if I'm missing something let me know.
Comment on attachment 411546 [details] [diff] [review]
incorporate v8 tests into talos (now with manifest)

RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/base.js,v
done
Checking in base.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/base.js,v  <--  base.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/crypto.js,v
done
Checking in crypto.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/crypto.js,v  <--  crypto.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/deltablue.js,v
done
Checking in deltablue.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/deltablue.js,v  <--  deltablue.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/earley-boyer.js,v
done
Checking in earley-boyer.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/earley-boyer.js,v  <--  earley-boyer.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/raytrace.js,v
done
Checking in raytrace.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/raytrace.js,v  <--  raytrace.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/regexp.js,v
done
Checking in regexp.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/regexp.js,v  <--  regexp.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/richards.js,v
done
Checking in richards.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/richards.js,v  <--  richards.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-crypto.html,v
done
Checking in run-crypto.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-crypto.html,v  <--  run-crypto.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-deltablue.html,v
done
Checking in run-deltablue.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-deltablue.html,v  <--  run-deltablue.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-earley-boyer.html,v
done
Checking in run-earley-boyer.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-earley-boyer.html,v  <--  run-earley-boyer.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-raytrace.html,v
done
Checking in run-raytrace.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-raytrace.html,v  <--  run-raytrace.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-regexp.html,v
done
Checking in run-regexp.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-regexp.html,v  <--  run-regexp.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-richards.html,v
done
Checking in run-richards.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-richards.html,v  <--  run-richards.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-splay.html,v
done
Checking in run-splay.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run-splay.html,v  <--  run-splay.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run.js,v
done
Checking in run.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/run.js,v  <--  run.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/splay.js,v
done
Checking in splay.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/splay.js,v  <--  splay.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/v8/v8.manifest,v
done
Checking in v8.manifest;
/cvsroot/mozilla/testing/performance/talos/page_load_test/v8/v8.manifest,v  <--  v8.manifest
initial revision: 1.1
done
Attachment #411546 - Flags: checked-in+
Attachment #411555 - Flags: checked-in+
Comment on attachment 411555 [details] [diff] [review]
incorporate v8 tests into buildbot-configs with updated numberOfBuildsToTrigger

http://hg.mozilla.org/build/buildbot-configs/rev/c65f64dec894

r+ = alice from previous version
Comment on attachment 411464 [details] [diff] [review]
add new v8 test to graph server db

http://hg.mozilla.org/graphs/rev/9c82b30ffe2d
Attachment #411464 - Flags: checked-in+
bustage fix, missed this in original patch

r+ = alice

Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.37; previous revision: 1.36
Attachment #411697 - Flags: checked-in+
Up and running as of 10am PST, closing.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Attachment #412026 - Flags: review?(anodelman) → review?(bhearsum)
Attachment #412026 - Flags: review?(bhearsum) → review+
Comment on attachment 412026 [details] [diff] [review]
add v8 tests to master.cfg for talos-staging-pool

http://hg.mozilla.org/build/buildbot-configs/rev/2ddbb56ab603
Attachment #412026 - Flags: checked-in+
Is this suite affected by bug 467120? i.e. do we discard the test result that took the longest like we do for tp?
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.