[meta] Artifact builds produce too many perma-failures to be useful right now
Categories
(Firefox :: General, task)
Tracking
()
People
(Reporter: Gijs, Unassigned, NeedInfo)
References
Details
(Keywords: meta)
This type of thing is not atypical.
Push health says 145 failures. Only a handful of those are intermittents. 4 tests perma-fail cross-platform, accounting for about 40 failures total, and another 16 Windows a11y tests fail on Windows (which is the only place they are being run), for another 64 failures, tracked in bug 1885239. I think some of the remainder is even duplicate errors from the same failures (e.g. due to crashes via asserts or aborted runs due to too many failures).
Seeing the wood for the trees becomes very hard in this situation, especially if you don't push regularly or don't always use artifact builds.
The deps (I'll file a few more) will hopefully be sufficient to deal with the current set of issues. However, I'd like to avoid coming back to this place. But I'm not 100% sure how we'd do that.
Aryx, do you have ideas? I know we run artifact builds on central, but I don't think we run tests against them - is doing a tier-2 type test run and then noticing this type of thing when it lands a possibility? Or do you think we'd still miss that a given test has gone perma-fail in artifact mode?
Description
•