Closed Bug 1581831 Opened 2 years ago Closed 1 year ago

Stop running warm page load tests against integration branches

Categories

(Testing :: Raptor, task, P1)

Version 3
task

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: davehunt, Assigned: onegru)

References

Details

Attachments

(1 file)

Once bug 1581829 is resolved and we have enough overlap between warm and cold page load results to cover any regressions that would affect both, we should stop running warm page load tests against integration branches. This will reduce the cost of running these tests, and reduce the number of regression alerts the sheriffs will need to respond to.

Priority: -- → P2
Blocks: 1582756
Priority: P2 → P1
Assignee: nobody → onegru

Dave, which are the names of the integration branches?

Flags: needinfo?(dave.hunt)

(In reply to Octavian Negru from comment #1)

Dave, which are the names of the integration branches?

These are 'autoland' and 'mozilla-inbound', however they are also included when 'integration' or 'trunk' are specified.

Flags: needinfo?(dave.hunt)
Attachment #9113217 - Attachment description: Bug 1581831 - Stop running warm page load tests against integration branches → WIP - Bug 1581831 - Stop running warm page load tests against integration branches

This is a lot better than my approach. Thanks :)

Attachment #9113217 - Attachment description: WIP - Bug 1581831 - Stop running warm page load tests against integration branches → Bug 1581831 - Stop running warm page load tests against integration branches
Pushed by aionescu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d0fccbbca19c
Stop running warm page load tests against integration branches r=perftest-reviewers,AlexandruIonescu

Good job, Octavian!

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

Flags: needinfo?(onegru)
Flags: needinfo?(dave.hunt)

(In reply to Jan de Mooij [:jandem] from comment #9)

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

Jan, yeah, seems like you're right. I'll fix this.

Flags: needinfo?(onegru)

Dave/bebe, should we create another yml for raptor like raptor-benchmark.yml?

Flags: needinfo?(fstrugariu)

(In reply to Jan de Mooij [:jandem] from comment #9)

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

Thanks for spotting this :jandem, it was indeed unintentional. I've asked :alexandrui to restore this coverage asap.

(In reply to Alexandru Ionescu :alexandrui from comment #11)

Dave/bebe, should we create another yml for raptor like raptor-benchmark.yml?

I'm not sure, this might add confusion as we're currently splitting these based on target application rather than test type.

Flags: needinfo?(dave.hunt)

Agree with Dave here.
This will create confusion

Flags: needinfo?(fstrugariu)

(In reply to Jan de Mooij [:jandem] from comment #9)

Was it intentional that this also stopped running things like Speedometer on autoland? I think no because this bug seems to be about the page load tests (tp6) only. If it was, it's inconsistent with Talos tests and I think it deserves broader discussion :)

I fixed it. Please let me know if you see anything else that seems wrong.

Flags: needinfo?(jdemooij)

(In reply to Alexandru Ionescu :alexandrui from comment #14)

I fixed it. Please let me know if you see anything else that seems wrong.

I see we're running Speedometer on mobile now ("Android 7.0 MotoG5 pgo" and "Android 8.0 Pixel2 AArch64 pgo") but we're still missing coverage for desktop?

See the jobs here on mozilla-central: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=speedometer

Flags: needinfo?(jdemooij) → needinfo?(aionescu)

Errr, I see lots of desktop also. (Linux x64 opt, Linux x64 QuantumRender opt, Linux x64 shippable opt, OS X 10.14 Shippable opt, Windows 7 opt
Windows 10 x64 opt, Windows 10 x64 Shippable opt, Windows 10 x64 QuantumRender opt). Could you please check again?

Flags: needinfo?(aionescu) → needinfo?(jdemooij)

(In reply to Alexandru Ionescu :alexandrui from comment #16)

Errr, I see lots of desktop also.

Sorry, I wasn't clear. That's on mozilla-central, but I don't see all those jobs on autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=speedometer

Flags: needinfo?(jdemooij)

(In reply to Jan de Mooij [:jandem] from comment #17)

Sorry, I wasn't clear. That's on mozilla-central, but I don't see all those jobs on autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=speedometer

Yeah, this is weird. I restored the test definition for speedometer.
That's the patch causing the regression and that's the patch fixing it. I'm still investigating. I will revert entirely the patch causing the regression until I find the issue, if needed.

Jan, could you please check if those are what you are looking for? https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&tier=1%2C2%2C3&searchStr=speedometer&fromchange=d94afce0c18b72e8febfaa68a21fcb7790d1118c
Probably browsertime team made some changes to the symbols and they are now shown differently.

Flags: needinfo?(jdemooij)

(In reply to Alexandru Ionescu :alexandrui from comment #19)

Jan, could you please check if those are what you are looking for?

I don't know what browsertime is, but those are tier 3 jobs and not the ones that run on mozilla-central. I'm interested in the Rap(sp) jobs here: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=speedometer

Things like test-linux64/opt-raptor-speedometer-firefox-e10s and test-windows10-64/opt-raptor-speedometer-firefox-e10s...

Flags: needinfo?(jdemooij)

Dave, the speedometer for mobile is back on autoland but for desktop it's not yet. There's some additional fix to be done on the yml file and won't be ready today. Do you want me to revert the changes before working on this fix?

Flags: needinfo?(dave.hunt)

(In reply to Alexandru Ionescu :alexandrui from comment #21)

Dave, the speedometer for mobile is back on autoland but for desktop it's not yet. There's some additional fix to be done on the yml file and won't be ready today. Do you want me to revert the changes before working on this fix?

Before this patch , the default run-on-projects for raptor.yml was: ['try', 'trunk', 'mozilla-beta']. This is still the default for tests named raptor-tp6-.*-cold but for all other tests this now defaults to ['try', 'mozilla-beta']. See https://searchfox.org/mozilla-central/rev/c61720a7d0c094d772059f9d6a7844eb7619f107/taskcluster/ci/test/raptor.yml#13-28

Instead of making a special case for raptor-tp6-.*-cold it would make sense to default to ['try', 'trunk', 'mozilla-beta'] and make special cases for when we want to deviate from this, such as for warm page loads. Unfortunately warm tests don't contain 'warm' in their name, but perhaps we can use negation? Would something like ^raptor-tp6-((?!cold).)*$ work?

We need to restore this coverage asap as we're at risk of missing regression alerts for these tests. If this isn't going to happen quickly, then reverting the recent changes would make sense until we have a better approach.

Flags: needinfo?(dave.hunt)

I am going to revert this as I am not sure how can I test this locally.
I thought about negation or covering the cases like raptor-tp6-(case1|case2|...) until it's left only cold ones. But this still needs some time.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Jan, I landed the patch fixing the desktop issue. It it at the bottom of this jobs page.

Flags: needinfo?(jdemooij)
Status: REOPENED → RESOLVED
Closed: 1 year ago1 year ago
Resolution: --- → FIXED

(In reply to Alexandru Ionescu :alexandrui from comment #24)

Jan, I landed the patch fixing the desktop issue. It it at the bottom of this jobs page.

Great, thanks!

Flags: needinfo?(jdemooij)
You need to log in before you can comment on or make changes to this bug.