1613487 - Add cold amazon (tp6m-1) pageload test for Fenix Browsertime to mozilla-central

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

See the bug that is blocked for more information on this. Here we will need to enable Browsertime for amazon (tp6m-1) and youtube (tp6m-2) cold page load tests for Fenix on mozilla-central.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hey Greg, is there a reason why only those two? Are the others not supported yet?

Comment 2

[Greg Mierzwinski [:sparky]]

The others would have to be added to the available tests (for the Fenix branch), but they won't be running on mozilla-central. Here, we are looking for parity between what Raptor runs and what Browsertime runs. See here for the tests we currently run with Raptor: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&tier=1%2C2%2C3&searchStr=fenix

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so to fully understand this one piece is missing. the linked tests from your last comment are plain browsertime tests and NOT run with Raptor? And on central we want to have browsertime via Raptor jobs for the exact same test set?

Comment 4

[Greg Mierzwinski [:sparky]]

Oh! I'm sorry for causing that confusion - we already have Raptor-browsertime running for both of them on mozilla-central. They just need to have their tier increased to 2 (which is the same as the raptor-webext tests 1) and then they need to be scheduled by the cron task rather than running on each push.

Comment 5

[Greg Mierzwinski [:sparky]]

Some clarifications. The purpose of the Fenix tests on central are for "smoke-tests" to make sure that nothing in central breaks the Fenix tests. The two browsertime tests are also "smoke-tests" running for the same purpose.

So this bug is for doing the following:

1. Disable the youtube Browsertime test.
2. Increase the tier of the amazon Browsertime test to tier 2.
3. Prevent the amazon Browsertime test from running on each m-c push.
4. Add the amazon Browsertime test to the cron job that runs the amazon Raptor test (which will act as a smoke-test).

Comment 6

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1613487 - Disable Raptor-webext and enable Raptor-btime Fenix tests. r?#perftest

This patch does a few things:
(1) Prevents Amazon and Youtube Fenix Browsertime tests from running on each push.
(2) Prevents Chrome Browsertime tests from running on each push (these should only run through the cron task).
(3) Prevents Speedometer Fenix Browsertime tests from running on each m-c push.
(4) Replaces the Raptor Fenix speedometer test in the general cron task with the Browsertime variant.
(5) Replaces the Raptor Fenix Amazon test with in the tp6m cron task with the Browsertime variant.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Greg Mierzwinski [:sparky] from comment #6)

(4) Replaces the Raptor Fenix speedometer test in the general cron task with the Browsertime variant.

We'll need to update the graphs on health.graphics after this change lands. Please coordinate with :airimovici.

Comment 8

[Greg Mierzwinski [:sparky]]

:airimovici, can you start a PR for those changes (speedometer)? For this test, the data name stays the same (raptor-speedometer-fenix), but the framework changes to browsertime.

Comment 9

[Alexandru Irimovici]

(In reply to Greg Mierzwinski [:sparky] from comment #8)

:airimovici, can you start a PR for those changes (speedometer)? For this test, the data name stays the same (raptor-speedometer-fenix), but the framework changes to browsertime.

I'll create a PR for the speedometer changes also. Thanks for the notice

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:sparky, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 11

[Greg Mierzwinski [:sparky]]

We are waiting for some changes in the dashboards before landing this change.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e7f4d31c649
Disable Raptor-webext and enable Raptor-btime Fenix tests. r=perftest-reviewers,Bebe

Comment 13

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/8e7f4d31c649