1466853 - Port stylebench and motionmark benchmarks test from talos to raptor

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: adding in motionmark and stylebench to raptor (WIP)

this works for me locally on win10, but fails on try- I think a few small tweaks and it will be good.

Comment 2

[Phabricator Automation]

Attachment: Bug 1466853 - Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Currently stylebench and motionmark run on talos, these are ideal benchmarks to run on raptor

Comment 3

[Phabricator Automation]

Comment on attachment 8986816 [details]
Bug 1466853 - Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D1752

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/683a682e52b2
Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Comment 5

[Raul Gurzau (:RaulG)]

Backed out changeset 683a682e52b2 (bug 1466853) for flake fail on checkouts/gecko/testing/raptor/raptor/benchmark.py on a CLOSED TREE

Backout link: https://hg.mozilla.org/integration/autoland/rev/71705fb3d3cfe07453fb90b3f0c44bfe33551610

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=683a682e52b2ffdc19f7972e45f2f58fedbb80c6

Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=184258421&repo=autoland&lineNumber=273

Log snippet: 
[vcs 2018-06-21T20:44:43.043Z] 244421 files updated, 0 files merged, 0 files removed, 0 files unresolved
[vcs 2018-06-21T20:44:43.296Z] updated to 683a682e52b2ffdc19f7972e45f2f58fedbb80c6
[vcs 2018-06-21T20:44:43.300Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "clone", "shouldAlert": false, "subtests": [], "value": 133.05697107315063}, {"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "pull", "shouldAlert": false, "subtests": [], "value": 9.588178157806396}, {"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "update", "shouldAlert": false, "subtests": [], "value": 92.2714729309082}, {"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "overall", "shouldAlert": false, "subtests": [], "value": 236.68608498573303}]}
[vcs 2018-06-21T20:44:43.703Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/autoland/rev/683a682e52b2ffdc19f7972e45f2f58fedbb80c6 title='Built from autoland revision 683a682e52b2ffdc19f7972e45f2f58fedbb80c6'>683a682e52b2ffdc19f7972e45f2f58fedbb80c6</a>
[task 2018-06-21T20:44:43.703Z] executing ['bash', '-cx', 'cd /builds/worker/checkouts/gecko && ./mach lint -l flake8 -f treeherder']
[task 2018-06-21T20:44:43.706Z] + cd /builds/worker/checkouts/gecko
[task 2018-06-21T20:44:43.706Z] + ./mach lint -l flake8 -f treeherder
[task 2018-06-21T20:44:44.336Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2018-06-21T20:44:44.336Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2018-06-21T20:44:45.984Z] Installing setuptools, pip, wheel...done.
[task 2018-06-21T20:44:47.121Z] running build_ext
[task 2018-06-21T20:44:47.121Z] building 'psutil._psutil_linux' extension
[task 2018-06-21T20:44:47.121Z] creating build
[task 2018-06-21T20:44:47.121Z] creating build/temp.linux-x86_64-2.7
[task 2018-06-21T20:44:47.121Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2018-06-21T20:44:47.121Z] creating build/lib.linux-x86_64-2.7
[task 2018-06-21T20:44:47.121Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2018-06-21T20:44:47.121Z] building 'psutil._psutil_posix' extension
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2018-06-21T20:44:47.122Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2018-06-21T20:44:47.122Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2018-06-21T20:44:47.122Z] 
[task 2018-06-21T20:44:47.122Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-06-21T20:45:28.811Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/raptor/raptor/benchmark.py:59:29 | continuation line over-indented for visual indent (E127)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/96496ebab2d9
Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Comment 7

[Raul Gurzau (:RaulG)]

Backed out changeset 96496ebab2d9 (bug 1466853) for StyleBench failures on a CLOSED TREE

Backout link: https://hg.mozilla.org/integration/autoland/rev/952031252a25a9846f6577fe6ab01dc0fca52dc8

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=96496ebab2d9d35e8b855a475195e9a4df877cbc

Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=184291896&repo=autoland&lineNumber=1830

Log snippet: 

16:38:07     INFO -  TEST-INFO | 24253: exit 0
16:38:07     INFO -  TEST-OK | stylebench | took 77738ms
16:38:07     INFO -  SUITE-END | took 304s
16:38:07     INFO -  Completed test suite (00:05:05)
16:38:07    ERROR -  Traceback (most recent call last):
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/run_tests.py", line 508, in <module>
16:38:07     INFO -      main()
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/run_tests.py", line 504, in main
16:38:07     INFO -      sys.exit(run_tests(config, browser_config))
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/run_tests.py", line 327, in run_tests
16:38:07     INFO -      talos_results.output(results_urls)
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/results.py", line 43, in output
16:38:07     INFO -      results = _output()
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/output.py", line 129, in __call__
16:38:07     INFO -      vals, testname=test.name())
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/output.py", line 297, in construct_results
16:38:07     INFO -      return self.stylebench_score(vals)
16:38:07     INFO -    File "/home/cltbld/workspace/build/tests/talos/talos/output.py", line 277, in stylebench_score
16:38:07     INFO -      raise Exception("StyleBench has 52 subtests, found: %s instead" % len(results))
16:38:07     INFO -  Exception: StyleBench has 52 subtests, found: 53 instead
16:38:07    ERROR - Return code: 1
16:38:07  WARNING - setting return code to 1
16:38:07    ERROR - # TBPL WARNING #

Comment 8

[Phabricator Automation]

Attachment: Bug 1466853 - Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

stylebench and motionmark run in talos, we want these running in raptor

Comment 9

[Phabricator Automation]

Comment on attachment 8987598 [details]
Bug 1466853 - Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D1814

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b51ba2011412
Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Comment 11

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/b51ba2011412

Comment 12

[Ionuț Goldan [:igoldan]]

FYI, this change updated our baselines for stylebench:

== Change summary for alert #14014 (as of Mon, 25 Jun 2018 18:23:48 GMT) ==

Regressions:

 47%  stylebench osx-10-10 opt e10s stylo     34.61 -> 18.41
 44%  stylebench windows10-64 opt e10s stylo  63.85 -> 35.50
 44%  stylebench windows7-32 pgo e10s stylo   67.09 -> 37.44
 44%  stylebench windows7-32 opt e10s stylo   51.65 -> 28.83
 43%  stylebench linux64 pgo e10s stylo       72.28 -> 40.92
 43%  stylebench linux64 opt e10s stylo       65.57 -> 37.14
 43%  stylebench windows10-64 pgo e10s stylo  72.54 -> 41.61
 41%  stylebench linux64-qr opt e10s stylo    66.25 -> 39.22

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14014