1643717 - (DevTools-Fis-Tier-1) Promote DevTools Fission tests to tier 1

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Are there plans to run test suites which get executed with fission enabled as tier 1 and on autoland to prevent accumulation of new, permanent or frequent fission failures? E.g. performance tasks and devtools tests only run on central, web-platform-tests also on autoland but only as tier 2.

That prevents sheriffs taking direct action on regressions (backout) and accumulates new fission failures. Having them as tier 1 would prevent this.

Current situation on autoland

• Is there any obstacle which prevents web-platform-tests on Linux x64 WebRender Shippable opt from being switched to Tier 1?

Tasks running on mozilla-central

• performance tests like Raptor and Talos tests only run there and due to that as tier 2 (regressions don't get backed out)
• devtools only run on central and as tier 2.

Comment 1

[Chris Peterson [:cpeterson]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #0)

Current situation on autoland

• Is there any obstacle which prevents web-platform-tests on Linux x64 WebRender Shippable opt from being switched to Tier 1?

We're tracking the promotion of WPT to tier 1 in bug 1617684. We still have about 20 intermittent failure bugs for WPT Fission, but I will ask the Fission engineers if we can promote the WPT tests that aren't skipped or intermittent to tier 1.

Comment 2

[Chris Peterson [:cpeterson]]

Tasks running on mozilla-central

• performance tests like Raptor and Talos tests only run there and due to that as tier 2 (regressions don't get backed out)

@ Jesup, do you think it is too early to promote performance tests like Raptor and Talos from tier 2 to 1? Fission performance still needs work, but we also don't want any new regressions.

Comment 3

[Chris Peterson [:cpeterson]]

• devtools only run on central and as tier 2.

@ Alex, do you have a DevTools Fission bug for promoting the DevTools Fission tests from tier 2 to 1?

Comment 4

[Alexandre Poirot [:ochameau]]

(In reply to Chris Peterson [:cpeterson] from comment #3)

• devtools only run on central and as tier 2.

@ Alex, do you have a DevTools Fission bug for promoting the DevTools Fission tests from tier 2 to 1?

No particular bug.
From DevTools perspective, we still have a couple of rought edges around navigation,
where we know we aren't supporting fission correctly.
So it might be a bit early. But we will surely benefit from moving to Tier 1 as we complete M2 milestone.

Having said that, I hear Sheriffs request, which are struggling with new permafails.
I imagine bug 1641824 is the main one we have been having for a while in various forms.
I tried to debug that, fixed one local leak but I'm being stuck on this.

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Alexandre Poirot [:ochameau] from comment #4)

No particular bug.

In that case, I'll morph this bug into a "DevTools tier 1" bug because we already have bug 1617684 for promoting WPT to tier 1.

From DevTools perspective, we still have a couple of rought edges around navigation,
where we know we aren't supporting fission correctly.
So it might be a bit early. But we will surely benefit from moving to Tier 1 as we complete M2 milestone.

Since there is still DevTools Fission work remaining, I'll move this bug from Fission Nightly milestone M6a (work targeting June) to M6c (work targeting Q3).

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

I think it's about time to promote them to tier 1 (raptor/talos). Dave?

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #6)

I think it's about time to promote them to tier 1 (raptor/talos). Dave?

Talos/Raptor Fission tests are only currently running against WebRender enabled builds. I don't have any concerns with these being promoted to tier 1. We've had some issues with timeouts that are specific to Fission+WebRender in bug 1609295, but the affected tests have been temporarily disabled until the cause is identified and a resolution is found.

Comment 8

[Julian Descottes [:jdescottes]]

We will go through our current DevTools Fission failures and will try to move the tests to Tier 1 as soon as possible. If needed we will skip permafails and frequent intermittents on Fission to avoid accumulating more regressions.

Comment 9

[Julian Descottes [:jdescottes]]

Attachment: Bug 1643717 - Promote DevTools Fission tests to tier 1

Comment 10

[Julian Descottes [:jdescottes]]

Aryx: We should be close to enabling Fission DevTools Browser Mochitests on tier1.
Do you have inputs about which (or how many) platforms to target? The current patch at https://phabricator.services.mozilla.com/D96028 starts running the dt fission tests on linux.*64(-qr)?/debug, linux.*64-shippable(-qr)?/opt and windows10-64-shippable(-qr)?/opt on trunk and moves them to tier1.

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

These are the same platforms for which the devtools already run with Fission enabled on central and is exactly what sheriffing is interested in. No other mochitests run with Fission enabled for macOS - enabling mochitests on that platform should be a more general decision.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/be51df39a3a9
Promote DevTools Fission tests to tier 1 r=jmaher

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/be51df39a3a9