1806178 - Add a custom browsertime test for the grandprix benchmark

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Greg Mierzwinski [:sparky]]

This bug is for adding a test which will run this script and gather the results: https://paste.mozilla.org/bPh5AAcr#L1,7,12,14,65

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1806178 - Run grandprix benchmark through a custom browsertime test. r?#perftest

This patch adds a test that triggers a benchmark test on the Grandprix website (live). The patch also adds the CI tasks for this test to firefox, chrome, and chromium. The firefox tasks run on autoland, and mozilla-central. Some new code is added to flatten a custom data type if requested. This allows us to easily comb large datasets and organize them into our replicates as long as the key paths match. We remove the "iterations", and "tests" strings as they are unnecessary for unique pathing and we do this with a new test field called subtest_metric_filters. The test_summary field lets you turn the flattening on and off since some tests don't need this.

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1806178 - Add safari to browsers available for grandprix test. r?#perftest

This patch adds the safari browser to the list of available browsers for the grandprix custom benchmark test.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a2c2599c0a30
Run grandprix benchmark through a custom browsertime test. r=perftest-reviewers,AlexandruIonescu

Comment 4

[Denis Palmeiro [:denispal]]

Thanks for the quick patch, sparky! Will this cover all the subtests in CI separately? I don't think we care too much of the overall score here since it's a total sum instead of a geomean.

Comment 5

[Greg Mierzwinski [:sparky]]

No problem, and yes I believe so if I understand your question correctly, here's a sample test run with it so you can see the data that is being produced from the test: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=7c7d9b7b2f28496d7c4d4c33540bdd69b083c25b&selectedTaskRun=Va1j3LO8S3KcRc89iYEo5w.0

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d8232e0f3d92
Add safari to browsers available for grandprix test. r=perftest-reviewers,kshampur

Comment 7

[Greg Mierzwinski [:sparky]]

Here's a try run with safari: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=2d12798469a15d140ad164b8248aaf0a7f748dc9

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a2c2599c0a30
https://hg.mozilla.org/mozilla-central/rev/d8232e0f3d92