1565316 - Process browsertime results and dump out for perfherder

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Robert Wood [:rwood]]

For the browsertime mvp we will be modifying the existing raptor harness to run page-load tests via browsertime instead of the raptor webext.

As part of that, raptor needs to grab the browsertime results (browsertime.json) and process/format them to be dumped into perfherder via the PERFHERDER_DATA json blob.

For the mvp we are most interested in results that we also currently get out of raptor page-load so that we can directly compare.

Comment 1

[Robert Wood [:rwood]]

Attachment: Bug 1565316 - Process browsertime results and dump out for perfherder.

Comment 2

[Nick Alexander :nalexander [he/him]]

Just FYI, I added some pieces on top of your commit in this try push to get result processing working. We can fold some of those bits into your patch here as we keep building things out.

Comment 3

[Robert Wood [:rwood]]

(In reply to Nick Alexander :nalexander [he/him] from comment #2)

Just FYI, I added some pieces on top of your commit in this try push to get result processing working. We can fold some of those bits into your patch here as we keep building things out.

Right, thanks, once your base patches land then I will move forward. Lots of patches in play at the moment.

Comment 4

[Robert Wood [:rwood]]

FYI won't get a green browsertime job run yet on try (some more patches in progress for production, like Bug 1566174). Try run for 'sanity' check of non-browsertime raptor jobs and lint/unit tests:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c979f5b9252d36798a91c3ca6703b24ee23a1612

Comment 5

[Nick Alexander :nalexander [he/him]]

Tarek: OK, it's not worth folding BrowsertimeOutput into BrowsertimeResultsHandler: there are tangles running every which way. Here's a try build of a Pre: part and some small follow-ups to your work:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2a88037d125e1ec0cc97e725b9122de3fdca0867

From https://hg.mozilla.org/try/pushloghtml?changeset=2a88037d125e1ec0cc97e725b9122de3fdca0867, the relevant bits are:

	54268e95bde9095bc5a724c6ae3bc24c6f6bcd33	Nick Alexander — Bug 1565316 - Additional tweaks.
	660c0736c6ee497d4b698b9be255413843f69362	Nick Alexander — Bug 1565316 - Drop pointless android_device in mozproxy.
	84db6fc486ed6df7c8c594f40dd0bae6579f008c	Nick Alexander — Bug 1565316 - Process browsertime results and dump out for perfherder

84db6fc486ed6df7c8c594f40dd0bae6579f008c is Rob's patch, rebased; the other bits make things work for me, testing on Android. If you can rebase my commits, address the few comments I had on the Phab, and test on Desktop, we can land something here and iterate.

Comment 6

[Nick Alexander :nalexander [he/him]]

NI to Tarek for #c5.

Comment 7

[Tarek Ziadé (:tarek)]

From your try pushed, I've rebased those commits in the patch and pushed it on Phab:

• Additional tweaks.
• Start on BrowsertimeAndroid.
• add six
• Extra logging.

and massaged a bit the code. it works locally for desktop and mobile

Let's meet this evening to have a final review on that patch for the missing tweaks
and push it.

this is the current version I've pushed on try: https://hg.mozilla.org/try/pushloghtml?changeset=83aae0da12365a4395a7cb483b99d5018dd2062e
https://treeherder.mozilla.org/#/jobs?repo=try&revision=83aae0da12365a4395a7cb483b99d5018dd2062e

Notice that we were backed off for the mozproxy change we'll need to look at that next.

Comment 8

[Tarek Ziadé (:tarek)]

One more fix, and now it's green on TC

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b7ca3ace2cb7929ad6216d8fad6f023087ec1d0b

Comment 9

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1565316 - TO FOLD - Don't rely on `six`, just use `test[name]` for now.

Depends on D40100

Comment 10

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1565316 - TO FOLD - Clarify some expectations, update some comments.

Depends on D46048

Comment 11

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1565316 - TO FOLD - Work around `selenium-webdriver` Android issue.

Depends on D46049

Comment 12

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1565316 - TO FOLD - add set_browser_prefs

Depends on D46050

Comment 13

[Nick Alexander :nalexander [he/him]]

OK, I pushed up some additional tweaks to fold once the try builds are green.

browsertime: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b74bf38da5fbb7136a45c3ed1b8f5fe7eb319ff0

no browsertime: https://treeherder.mozilla.org/#/jobs?repo=try&revision=833acc50492622c06ca881854f3d5a6d67b2e26d

If those are green and Tarek is happy, let's land this thing and move on with our lives.

Comment 14

[Tarek Ziadé (:tarek)]

I merged Nick's changes in a single patch and updated phab. Things work locally on macOS desktop - now pushing
on try a raptor-tp6-1-firefox-e10s run

browsertime : https://treeherder.mozilla.org/#/jobs?repo=try&revision=e613603621073070d33c1988cb44a2432f5684c4
no browsertime: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38f83692c2010cd3fab1d2e613c5bc1a09edff27

If those are also green, I think we're good to go

Comment 15

[Nick Alexander :nalexander [he/him]]

OK, it's go time. Let's see how it does in the wild! I'm happy with Tarek's try builds, and there's going to be lots of follow-up anyway...

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/67e5d2425c75
Process browsertime results and dump out for perfherder. r=nalexander

Comment 17

[Alexandru Michis [:malexandru]]

Backed out changeset 67e5d2425c75 for causing raptor wasm failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/4d5ba29c6b2bb39e8d0d24e4784cd7744f1892de

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=4d5ba29c6b2bb39e8d0d24e4784cd7744f1892de&fromchange=86d1d52f1174927f18accc9f3014c37429077c7a&searchStr=wasm&group_state=expanded

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=266961375&repo=autoland&lineNumber=1925

[task 2019-09-17T01:44:17.202Z] 01:44:17 ERROR - Traceback (most recent call last):
[task 2019-09-17T01:44:17.202Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 1456, in <module>
[task 2019-09-17T01:44:17.203Z] 01:44:17 INFO - main()
[task 2019-09-17T01:44:17.203Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 1424, in main
[task 2019-09-17T01:44:17.203Z] 01:44:17 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 208, in run_tests
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - return self.process_results(tests, test_names)
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 235, in process_results
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - return self.results_handler.summarize_and_output(self.config, tests, test_names)
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/results.py", line 173, in summarize_and_output
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - output.summarize(test_names)
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/output.py", line 461, in summarize
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - suite['value'] = self.construct_summary(vals, testname=test.name)
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/output.py", line 304, in construct_summary
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - return filters.mean(_filter(results, 'total'))
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - UnboundLocalError: local variable 'results' referenced before assignment
[task 2019-09-17T01:44:17.253Z] 01:44:17 ERROR - Return code: 1

Comment 18

[Nick Alexander :nalexander [he/him]]

Ah, rats. Tomorrow.

Comment 19

[Tarek Ziadé (:tarek)]

I fixed the typos but there's still an issue with raptor-wasm-misc-baseline

https://taskcluster-artifacts.net/G817eQ1oQym_IZ5d-I9nxA/0/public/logs/live_backing.log

Investigating

Comment 20

[Tarek Ziadé (:tarek)]

Ok so, the current raptor tool does the <host>:<port> replacement for benchmarks here in the client:

https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/runner.js#102-114

We'll need to figure out how to do this in BT. In the interim I am just going to run the full raptor suite w/o --browsertime to make sure
we don't have any regression, and land this patch. Hopefully this afternoon at our meeting

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fd1b1f6922610a956dda4631091b70f255e01971

Comment 21

[Tarek Ziadé (:tarek)]

Bebe canceled my job but we have enough runs to validate that wm & wm-b passed, landing now.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/192cca227f11
Process browsertime results and dump out for perfherder. r=nalexander

Comment 23

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:igoldan this patch adds support for browsertime to generate perfherder.json results. Are there tasks necessary to injest/expose this in Perfherder?

Comment 24

[Ionuț Goldan [:igoldan]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #23)

:igoldan this patch adds support for browsertime to generate perfherder.json results. Are there tasks necessary to injest/expose this in Perfherder?

I need to read this patch through. If this patch added a new framework, then yes, we should file a separate bug to register it. Otherwise Perfherder will simply ignore its results.

Comment 25

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/192cca227f11