1581831 - Stop running warm page load tests against integration branches

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Once bug 1581829 is resolved and we have enough overlap between warm and cold page load results to cover any regressions that would affect both, we should stop running warm page load tests against integration branches. This will reduce the cost of running these tests, and reduce the number of regression alerts the sheriffs will need to respond to.

Comment 1

[Octavian Negru]

Dave, which are the names of the integration branches?

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Octavian Negru from comment #1)

Dave, which are the names of the integration branches?

These are 'autoland' and 'mozilla-inbound', however they are also included when 'integration' or 'trunk' are specified.

Comment 3

[Octavian Negru]

Attachment: Bug 1581831 - Stop running warm page load tests against integration branches

Comment 4

[Alexandru Ionescu (needinfo me) [:alexandrui]]

here's a try push using nested by-foo keys: https://hg.mozilla.org/try/rev/8382777f45c7a89e46acb5ef3d1a3e8b0190ee80

Comment 5

[Octavian Negru]

This is a lot better than my approach. Thanks :)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d0fccbbca19c
Stop running warm page load tests against integration branches r=perftest-reviewers,AlexandruIonescu

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Good job, Octavian!

Comment 8

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/d0fccbbca19c

Comment 9

[Jan de Mooij [:jandem]]

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Jan de Mooij [:jandem] from comment #9)

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

Jan, yeah, seems like you're right. I'll fix this.

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Dave/bebe, should we create another yml for raptor like raptor-benchmark.yml?

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Jan de Mooij [:jandem] from comment #9)

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

Thanks for spotting this :jandem, it was indeed unintentional. I've asked :alexandrui to restore this coverage asap.

(In reply to Alexandru Ionescu :alexandrui from comment #11)

Dave/bebe, should we create another yml for raptor like raptor-benchmark.yml?

I'm not sure, this might add confusion as we're currently splitting these based on target application rather than test type.

Comment 13

[Florin Strugariu [:Bebe]]

Agree with Dave here.
This will create confusion

Comment 14

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Jan de Mooij [:jandem] from comment #9)

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

I fixed it. Please let me know if you see anything else that seems wrong.

Comment 15

[Jan de Mooij [:jandem]]

(In reply to Alexandru Ionescu :alexandrui from comment #14)

I fixed it. Please let me know if you see anything else that seems wrong.

I see we're running Speedometer on mobile now ("Android 7.0 MotoG5 pgo" and "Android 8.0 Pixel2 AArch64 pgo") but we're still missing coverage for desktop?

See the jobs here on mozilla-central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=speedometer

Comment 16

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Errr, I see lots of desktop also. (Linux x64 opt, Linux x64 QuantumRender opt, Linux x64 shippable opt, OS X 10.14 Shippable opt, Windows 7 opt
Windows 10 x64 opt, Windows 10 x64 Shippable opt, Windows 10 x64 QuantumRender opt). Could you please check again?

Comment 17

[Jan de Mooij [:jandem]]

(In reply to Alexandru Ionescu :alexandrui from comment #16)

Errr, I see lots of desktop also.

Sorry, I wasn't clear. That's on mozilla-central, but I don't see all those jobs on autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=speedometer

Comment 18

[Alexandru Ionescu (needinfo me) [:alexandrui]]

(In reply to Jan de Mooij [:jandem] from comment #17)

Sorry, I wasn't clear. That's on mozilla-central, but I don't see all those jobs on autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=speedometer

Yeah, this is weird. I restored the test definition for speedometer.
That's the patch causing the regression and that's the patch fixing it. I'm still investigating. I will revert entirely the patch causing the regression until I find the issue, if needed.

Comment 19

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Jan, could you please check if those are what you are looking for? https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&tier=1%2C2%2C3&searchStr=speedometer&fromchange=d94afce0c18b72e8febfaa68a21fcb7790d1118c
Probably browsertime team made some changes to the symbols and they are now shown differently.

Comment 20

[Jan de Mooij [:jandem]]

(In reply to Alexandru Ionescu :alexandrui from comment #19)

Jan, could you please check if those are what you are looking for?

I don't know what browsertime is, but those are tier 3 jobs and not the ones that run on mozilla-central. I'm interested in the Rap(sp) jobs here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=speedometer

Things like test-linux64/opt-raptor-speedometer-firefox-e10s and test-windows10-64/opt-raptor-speedometer-firefox-e10s...

Comment 21

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Dave, the speedometer for mobile is back on autoland but for desktop it's not yet. There's some additional fix to be done on the yml file and won't be ready today. Do you want me to revert the changes before working on this fix?

Comment 22

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Alexandru Ionescu :alexandrui from comment #21)

Dave, the speedometer for mobile is back on autoland but for desktop it's not yet. There's some additional fix to be done on the yml file and won't be ready today. Do you want me to revert the changes before working on this fix?

Before this patch , the default run-on-projects for raptor.yml was: ['try', 'trunk', 'mozilla-beta']. This is still the default for tests named raptor-tp6-.*-cold but for all other tests this now defaults to ['try', 'mozilla-beta']. See https://searchfox.org/mozilla-central/rev/c61720a7d0c094d772059f9d6a7844eb7619f107/taskcluster/ci/test/raptor.yml#13-28

Instead of making a special case for raptor-tp6-.*-cold it would make sense to default to ['try', 'trunk', 'mozilla-beta'] and make special cases for when we want to deviate from this, such as for warm page loads. Unfortunately warm tests don't contain 'warm' in their name, but perhaps we can use negation? Would something like ^raptor-tp6-((?!cold).)*$ work?

We need to restore this coverage asap as we're at risk of missing regression alerts for these tests. If this isn't going to happen quickly, then reverting the recent changes would make sense until we have a better approach.

Comment 23

[Alexandru Ionescu (needinfo me) [:alexandrui]]

I am going to revert this as I am not sure how can I test this locally.
I thought about negation or covering the cases like raptor-tp6-(case1|case2|...) until it's left only cold ones. But this still needs some time.

Comment 24

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Jan, I landed the patch fixing the desktop issue. It it at the bottom of this jobs page.

Comment 25

[Jan de Mooij [:jandem]]

(In reply to Alexandru Ionescu :alexandrui from comment #24)

Jan, I landed the patch fixing the desktop issue. It it at the bottom of this jobs page.

Great, thanks!