1945658 - [meta] Artifact builds produce too many perma-failures to be useful right now

Status: ASSIGNED

(Testing :: General, task, P3)

Description

[:Gijs (he/him)]

This type of thing is not atypical.

Push health says 145 failures. Only a handful of those are intermittents. 4 tests perma-fail cross-platform, accounting for about 40 failures total, and another 16 Windows a11y tests fail on Windows (which is the only place they are being run), for another 64 failures, tracked in bug 1885239. I think some of the remainder is even duplicate errors from the same failures (e.g. due to crashes via asserts or aborted runs due to too many failures).

Seeing the wood for the trees becomes very hard in this situation, especially if you don't push regularly or don't always use artifact builds.

The deps (I'll file a few more) will hopefully be sufficient to deal with the current set of issues. However, I'd like to avoid coming back to this place. But I'm not 100% sure how we'd do that.

Aryx, do you have ideas? I know we run artifact builds on central, but I don't think we run tests against them - is doing a tier-2 type test run and then noticing this type of thing when it lands a possibility? Or do you think we'd still miss that a given test has gone perma-fail in artifact mode?

Comment 1

[Marco Bonardo [:mak]]

Gijs, is this meta and needinfo still relevant after 3 months and the many dependencies landed, or could the remaining issue be moved to its own bug?

Comment 2

[:Gijs (he/him)]

All the deps are closed so from that PoV we could close. I would really like to find a more durable solution that avoids the problem recurring, though...

Comment 3

[Johannes [:joschmidt]]

Lets keep this open just in case to find a better solution

Comment 4

[:Gijs (he/him)]

(In reply to Johannes from comment #3)

Lets keep this open just in case to find a better solution

Let's keep the needinfo for Aryx, then...

Comment 5

[Stephen Thompson [:sthompson]]

This came up in General triage with 23 days after needinfo. Do we need to do anything with this bug at this point?

Comment 6

[:Gijs (he/him)]

(In reply to Stephen Thompson [:sthompson] from comment #5)

This came up in General triage with 23 days after needinfo. Do we need to do anything with this bug at this point?

I'd like a more permanent way of making sure that artifact builds don't descend into an orange-fest, thence the needinfo for Aryx.

I've pushed a blank trypush to see how bad the sitch is at the moment: https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=137313 .

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Disability Access APIs' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 8

[Moritz Beier [:mbeier]]

Moving to Testing::General since it sounds like we're considering to add artifact build test failures as tier 2 failures.

Comment 9

[Florian Quèze [:florian]]

Attachment: Bug 1945658 - Add tier 2 test jobs for artifact builds on Linux, Windows. r=ahal

Run mochitest-browser-chrome, mochitest-chrome, mochitest-devtools-chrome,
mochitest-plain, and xpcshell tests against artifact builds to catch
perma-failures before they accumulate.

On Linux and Windows, these jobs run on mozilla-central and backstop pushes
on autoland. On macOS, they are try-only to preserve limited machine
capacity.

Comment 10

[Florian Quèze [:florian]]

Attachment: Bug 1945658 - Add debug artifact builds on Linux, Windows, and macOS. r=ahal

These debug artifact builds are needed so that we can run tier 2 tests
against them, matching what we already do for opt artifact builds.

Comment 11

[Florian Quèze [:florian]]

Attachment: Bug 1945658 - Upload artifacts from artifact builds so that test jobs can use them. r=ahal

The artifact-build kind had keep-artifacts: false, which prevented it from
uploading the build outputs that test jobs need to download.

Comment 12

[Florian Quèze [:florian]]

The attached patch stack adds tier2 test jobs on both opt and debug artifact builds on mozilla-central. That should be enough for sheriffs to notice new perma-fails and file bugs on them. I didn't schedule the jobs on autoland to limit the extra cost.

Comment 13

[Florian Quèze [:florian]]

Attachment: Bug 1945658 - Sign Windows artifact builds so updater tests can run, r=ahal.

Comment 14

[Florian Quèze [:florian]]

Attachment: Bug 1945658 - Skip tests that permafail on Linux debug artifact builds, r=Aryx,Gijs.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/01103271c918
https://hg.mozilla.org/integration/autoland/rev/8b4f8f09485b
Add tier 2 test jobs for artifact builds on Linux, Windows. r=ahal,taskgraph-reviewers
https://github.com/mozilla-firefox/firefox/commit/51aad1dc1e64
https://hg.mozilla.org/integration/autoland/rev/2fdc8705f5fc
Add debug artifact builds on Linux, Windows, and macOS. r=ahal
https://github.com/mozilla-firefox/firefox/commit/1511c7522310
https://hg.mozilla.org/integration/autoland/rev/6c3c122b12a3
Upload artifacts from artifact builds so that test jobs can use them. r=ahal
https://github.com/mozilla-firefox/firefox/commit/58ee9afb6b1c
https://hg.mozilla.org/integration/autoland/rev/86b98422a8e5
Sign Windows artifact builds so updater tests can run, r=ahal,releng-reviewers,taskgraph-reviewers.
https://github.com/mozilla-firefox/firefox/commit/13c785e9e6bf
https://hg.mozilla.org/integration/autoland/rev/96a3a9df8d40
Skip tests that permafail on Linux debug artifact builds, r=Gijs.