add mach command to auto disable high frequency test failures
Categories
(Testing :: General, task, P2)
Tracking
(firefox141 fixed)
| Tracking | Status | |
|---|---|---|
| firefox141 | --- | fixed |
People
(Reporter: jmaher, Assigned: arnaud.vergnet, NeedInfo)
Details
Attachments
(3 files)
in this bug work will be done to add a command to mach which will download high frequency failures from treeherder and for all tests that meet all criteria, edit the manifest files to disable the tests. The end user can then upload the patch for review as well as do any annotations in bugs.
In some cases this could work where we run this on a test-info job on mozilla-central and it would generate an artifact that would be a patch to disable all relevant tests. We could then have another tool to find the most recent patch, create a bug, reference other bugs, add the proper commit message, and upload to phabricator.
Either way, work needs to be done to make a reliable method for generating a patch to disable tests that meet criteria.
The criteria to use (given what information we have readily accessible today):
- bug must be a
single tracking bugand have a testcase in the name (i.e. not a crash, or leak) - bug must have >30 failures in the last 7 days
- all platforms with at least 7 failures (a platform is os, os_version, os_arch, build_type, i.e.
linux1804-64-asan-qr/opt) - if there are any variants that make up 75% of the failures from #2, only use those (i.e.
swgl)
Given these 4 criteria, you should be able to find the path + testname from the bug summary, find the related manifest, and add a skip-if condition. We have ./mach manifest skip-fails which does most of the manifest editing, but this might require some editing to input the specific manifest name, test name, and desired condition.
|
Reporter | |
Comment 1•1 year ago
|
||
some old code I wrote shows how to query treeherder and find some of the data needed for this bug. It should be updated and use failedplatform.py.
|
||
Comment 2•1 year ago
|
||
all platforms with at least 7 failures (a platform is os, os_version, os_arch, build_type, i.e. linux1804-64-asan-qr/opt)
- Retriggers can push a platform easily above the threshold.
- Rarely running platforms will need a higher failure rate to get updated test manifests, e.g. code coverage which only runs for mozilla-central.
|
|
Reporter | |
Comment 3•1 year ago
|
||
thanks for the clarification- I agree there are edge cases.
we can solve retriggers by looking at unique revisions. That makes the query more expensive though, but probably safer.
For things like ccov and other m-c only runs, maybe we can lower the number required to 4 failures in the last 7 days (assuming 24 runs). They key here is building a list of m-c only configs; maybe we live with this for now and focus on auto disabling stuff that meets the rest of the criteria- knowing that some stuff will be high frequency failure and we will not catch it.
|
Assignee | |
Comment 4•1 year ago
|
||
|
||
Updated•1 year ago
|
Backed out for causing js-bench-sm failures complaining about markdown2 module
|
|
Reporter | |
Comment 8•1 year ago
|
||
this is failing on markdown2 which the js-bench task uses ./mach ... so it ends up importing this new code and cannot find markdown2 on the perf machines (hardware).
We can either:
- conditionally include
high_freq_skipfails.py - not use markdown2
- figure out how to get markdown2 on all platforms
- maybe something else?
|
|
Assignee | |
Comment 9•1 year ago
|
||
|
||
Comment 10•1 year ago
|
||
|
|
||
Comment 11•1 year ago
|
||
|
||
Comment 12•1 year ago
|
||
Backed out for causing py3 failures
|
|
Assignee | |
Comment 13•1 year ago
|
||
|
|
||
Comment 14•1 year ago
|
||
|
||
Comment 15•1 year ago
|
||
| bugherder | ||
Description
•