1438292 - add JetStream benchmark to talos

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we run jetstream benchmark on AWFY and would like to run it on talos as a tier-2 benchmark which we run on try/mozilla-central, just like motionmark.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add jetstream to talos

this depends on a 50M patch to put jetstream in-tree:
https://1drv.ms/u/s!AtgqgjhJkZQqgtVFsQEEG4F6BvfPZw

Comment 2

[Robert Wood [:rwood]]

Comment on attachment 8951322 [details] [diff] [review]
add jetstream to talos

Review of attachment 8951322 [details] [diff] [review]:
-----------------------------------------------------------------

Very cool, LGTM, and applied both patches locally and runs great.

I notice that "throughput" is listed on the JetStream 1.1 test, but there's no corresponding item in talos local.json results. I'm assuming that maybe throughput is on the JetStream dashboard but they don't make it available in results maybe? r+ on confirming that "throughput" shouldn't be included in the talos results.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

throughput and latency are categories related to the 49 subtests- the final score is a geometric mean (weighted a bit) of the 49 subtests.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a9db2b954595
add JetStream benchmark source to tree. r=rwood
https://hg.mozilla.org/integration/mozilla-inbound/rev/8508dc544f8b
add JetStream benchmark to talos. r=rwood

Comment 5

[Eliza Balazs [:ebalazs_]]

Backed out 2 changesets (bug 1438292) for multiple failures in /builds/worker/workspace/mozharness/mozharness/base/script.py

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=8508dc544f8b56d647e076e82537f4de626db089&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable&selectedJob=162528182

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=162528182&repo=mozilla-inbound&lineNumber=692

Backout: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=142b57a7b01cd14c712d00950129344e8e5fbdae&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable&filter-resultStatus=success&filter-resultStatus=pending&filter-resultStatus=running

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

while this was backed out for infrastructure issues, we are now going to use jetstream via a tooltool artifact instead of adding 50MB to the source tree.

Comment 7

[Gregory Szorc [:gps]]

Well, you already landed it once and the data is forever in the repo since backouts are part of the full repo history. So if you want to land it again... ¯\_(ツ)_/¯

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add jetstream to talos

similar to the last patch, except this uses tooltool to download jetstream.zip.  this works locally, but I think we should prioritize fixing bug 1435677 in the short term to ease the pain of running tp5, jetstream, and other webkit benchmarks locally via mach.

Comment 9

[Robert Wood [:rwood]]

Comment on attachment 8953066 [details] [diff] [review]
add jetstream to talos

Review of attachment 8953066 [details] [diff] [review]:
-----------------------------------------------------------------

Yep LGTM and I tried it out locally on OSX, runs great

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6f4aed8caae8
add JetStream benchmark to talos. r=rwood

Comment 11

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/6f4aed8caae8