Closed Bug 1565316 Opened 5 years ago Closed 5 years ago

Process browsertime results and dump out for perfherder

Categories

(Testing :: Raptor, enhancement, P1)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
enhancement

Tracking

(firefox71 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla71
Tracking Flags:
Tracking Status
firefox71 --- fixed

People

(Reporter: rwood, Assigned: tarek)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 4 obsolete files)

Bug 1565316 - Process browsertime results and dump out for perfherder.
47 bytes, text/x-phabricator-request
Details | Review

For the browsertime mvp we will be modifying the existing raptor harness to run page-load tests via browsertime instead of the raptor webext.

As part of that, raptor needs to grab the browsertime results (browsertime.json) and process/format them to be dumped into perfherder via the PERFHERDER_DATA json blob.

For the mvp we are most interested in results that we also currently get out of raptor page-load so that we can directly compare.

Depends on: 1566171
Depends on: 1572804
Type: task → enhancement

Just FYI, I added some pieces on top of your commit in this try push to get result processing working. We can fold some of those bits into your patch here as we keep building things out.

(In reply to Nick Alexander :nalexander [he/him] from comment #2)

Just FYI, I added some pieces on top of your commit in this try push to get result processing working. We can fold some of those bits into your patch here as we keep building things out.

Right, thanks, once your base patches land then I will move forward. Lots of patches in play at the moment.

FYI won't get a green browsertime job run yet on try (some more patches in progress for production, like Bug 1566174). Try run for 'sanity' check of non-browsertime raptor jobs and lint/unit tests:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c979f5b9252d36798a91c3ca6703b24ee23a1612

Blocks: 1577847

Tarek: OK, it's not worth folding BrowsertimeOutput into BrowsertimeResultsHandler: there are tangles running every which way. Here's a try build of a Pre: part and some small follow-ups to your work:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2a88037d125e1ec0cc97e725b9122de3fdca0867

From https://hg.mozilla.org/try/pushloghtml?changeset=2a88037d125e1ec0cc97e725b9122de3fdca0867, the relevant bits are:

	54268e95bde9095bc5a724c6ae3bc24c6f6bcd33	Nick Alexander — Bug 1565316 - Additional tweaks.
	660c0736c6ee497d4b698b9be255413843f69362	Nick Alexander — Bug 1565316 - Drop pointless android_device in mozproxy.
	84db6fc486ed6df7c8c594f40dd0bae6579f008c	Nick Alexander — Bug 1565316 - Process browsertime results and dump out for perfherder

84db6fc486ed6df7c8c594f40dd0bae6579f008c is Rob's patch, rebased; the other bits make things work for me, testing on Android. If you can rebase my commits, address the few comments I had on the Phab, and test on Desktop, we can land something here and iterate.

NI to Tarek for #c5.

Flags: needinfo?(tarek)

From your try pushed, I've rebased those commits in the patch and pushed it on Phab:

  • Additional tweaks.
  • Start on BrowsertimeAndroid.
  • add six
  • Extra logging.

and massaged a bit the code. it works locally for desktop and mobile

Let's meet this evening to have a final review on that patch for the missing tweaks
and push it.

this is the current version I've pushed on try: https://hg.mozilla.org/try/pushloghtml?changeset=83aae0da12365a4395a7cb483b99d5018dd2062e
https://treeherder.mozilla.org/#/jobs?repo=try&revision=83aae0da12365a4395a7cb483b99d5018dd2062e

Notice that we were backed off for the mozproxy change we'll need to look at that next.

Flags: needinfo?(tarek) → needinfo?(nalexander)

Depends on D46050

OK, I pushed up some additional tweaks to fold once the try builds are green.

browsertime: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b74bf38da5fbb7136a45c3ed1b8f5fe7eb319ff0

no browsertime: https://treeherder.mozilla.org/#/jobs?repo=try&revision=833acc50492622c06ca881854f3d5a6d67b2e26d

If those are green and Tarek is happy, let's land this thing and move on with our lives.

Flags: needinfo?(nalexander)

I merged Nick's changes in a single patch and updated phab. Things work locally on macOS desktop - now pushing
on try a raptor-tp6-1-firefox-e10s run

browsertime : https://treeherder.mozilla.org/#/jobs?repo=try&revision=e613603621073070d33c1988cb44a2432f5684c4
no browsertime: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38f83692c2010cd3fab1d2e613c5bc1a09edff27

If those are also green, I think we're good to go

Attachment #9093091 - Attachment is obsolete: true
Attachment #9093092 - Attachment is obsolete: true
Attachment #9093093 - Attachment is obsolete: true
Attachment #9093094 - Attachment is obsolete: true
Attachment #9082067 - Attachment description: Bug 1565316 - Process browsertime results and dump out for perfherder → Bug 1565316 - Process browsertime results and dump out for perfherder.

OK, it's go time. Let's see how it does in the wild! I'm happy with Tarek's try builds, and there's going to be lots of follow-up anyway...

Assignee: rwood → tarek
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/67e5d2425c75
Process browsertime results and dump out for perfherder. r=nalexander

Backed out changeset 67e5d2425c75 for causing raptor wasm failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/4d5ba29c6b2bb39e8d0d24e4784cd7744f1892de

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=4d5ba29c6b2bb39e8d0d24e4784cd7744f1892de&fromchange=86d1d52f1174927f18accc9f3014c37429077c7a&searchStr=wasm&group_state=expanded

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=266961375&repo=autoland&lineNumber=1925

[task 2019-09-17T01:44:17.202Z] 01:44:17 ERROR - Traceback (most recent call last):
[task 2019-09-17T01:44:17.202Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 1456, in <module>
[task 2019-09-17T01:44:17.203Z] 01:44:17 INFO - main()
[task 2019-09-17T01:44:17.203Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 1424, in main
[task 2019-09-17T01:44:17.203Z] 01:44:17 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 208, in run_tests
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - return self.process_results(tests, test_names)
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/raptor.py", line 235, in process_results
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - return self.results_handler.summarize_and_output(self.config, tests, test_names)
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/results.py", line 173, in summarize_and_output
[task 2019-09-17T01:44:17.204Z] 01:44:17 INFO - output.summarize(test_names)
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/output.py", line 461, in summarize
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - suite['value'] = self.construct_summary(vals, testname=test.name)
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - File "/home/cltbld/tasks/task_1568683100/build/tests/raptor/raptor/output.py", line 304, in construct_summary
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - return filters.mean(_filter(results, 'total'))
[task 2019-09-17T01:44:17.205Z] 01:44:17 INFO - UnboundLocalError: local variable 'results' referenced before assignment
[task 2019-09-17T01:44:17.253Z] 01:44:17 ERROR - Return code: 1

Flags: needinfo?(nalexander)

Ah, rats. Tomorrow.

Flags: needinfo?(nalexander)

I fixed the typos but there's still an issue with raptor-wasm-misc-baseline

https://taskcluster-artifacts.net/G817eQ1oQym_IZ5d-I9nxA/0/public/logs/live_backing.log

Investigating

Ok so, the current raptor tool does the <host>:<port> replacement for benchmarks here in the client:

https://searchfox.org/mozilla-central/source/testing/raptor/webext/raptor/runner.js#102-114

We'll need to figure out how to do this in BT. In the interim I am just going to run the full raptor suite w/o --browsertime to make sure
we don't have any regression, and land this patch. Hopefully this afternoon at our meeting

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fd1b1f6922610a956dda4631091b70f255e01971

Bebe canceled my job but we have enough runs to validate that wm & wm-b passed, landing now.

Pushed by tziade@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/192cca227f11
Process browsertime results and dump out for perfherder. r=nalexander

:igoldan this patch adds support for browsertime to generate perfherder.json results. Are there tasks necessary to injest/expose this in Perfherder?

Flags: needinfo?(igoldan)

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #23)

:igoldan this patch adds support for browsertime to generate perfherder.json results. Are there tasks necessary to injest/expose this in Perfherder?

I need to read this patch through. If this patch added a new framework, then yes, we should file a separate bug to register it. Otherwise Perfherder will simply ignore its results.

See Also: → 1582059
Flags: needinfo?(igoldan)

https://hg.mozilla.org/mozilla-central/rev/192cca227f11

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox71: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
Regressions: 1583246
Regressions: 1583624
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: