annotate tests that failure to run standalone (via test-verify)
Categories
(Testing :: General, task)
Tracking
(firefox146 fixed)
| Tracking | Status | |
|---|---|---|
| firefox146 | --- | fixed |
People
(Reporter: jmaher, Assigned: jmaher)
References
(Blocks 1 open bug)
Details
Attachments
(1 external file, 1 obsolete file)
We will make this a separate mozinfo variable. This is the first step of test-verify where the harness:
- launches a fresh browser
- runs only the given test
- closes the browser
- repeats X (5) times
this bug will also include the initial annotations for all mochitests.
to keep it simple we will not do:
- asan/tsan/ccov, only opt/debug. if it fails on both opt/debug then it will be skipped on ALL build types.
- macosx/android- only linux/windows11. If it fails on both, we will skip on ALL platforms
- as for variants, if it fails on a single variant, we will only skip there, but if it fails on multiple variants for a config, we will just skip on all variants.
if we do not keep it simple, we can end up with 20+ annotation lines, this is unproductive. The problem is if it fails on a few variants, it might be confusing where it fails when trying to reproduce/fix this. I guess this is what we have to live with when some tests are run >100 times.
here is an example of where we skip a test on all:
- windows/opt-msix
- windows/opt-condprof
- windows/debug (no-variant)
- linux/debug (no-variant)
Also for the annotations there will be no comments in the manifest files. This seems difficult to get right.
here is a try push that will be used to find the failures (there are more, this is finding failures, and now re-verifying them):
https://treeherder.mozilla.org/jobs?repo=try&revision=a8d10c681a306e6d94a442bc2973442e6f468f63
| Assignee | ||
Comment 1•1 year ago
|
||
Updated•1 year ago
|
| Assignee | ||
Comment 2•1 year ago
|
||
Updated•1 year ago
|
Comment 3•1 year ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17) from comment #0)
We will make this a separate mozinfo variable. This is the first step of test-verify where the harness:
- launches a fresh browser
- runs only the given test
- closes the browser
- repeats X (5) times
Out of curiosity, how does this differ from the current test verify jobs?
| Assignee | ||
Comment 4•1 year ago
|
||
this is the first step of the test-verify job; The test verify expands on this and then repeats the test in the same browser session, after that, it adds moz_chaos for verify-standalone and verify-repeat.
We have different use cases (such as --restart-after-failure which will ensure as best as possible every test requested to run will), but this means that if a test cannot run standalone, it will most likely fail if it is the first test that we run in a set of tests (such as if the previous test failed and we restarted the browser)
Ideally after documenting and working to keep our annotations updated, we can experiment with --restart-after-failure as well as --retry (like xpcshell does), and --confirm-failure (which is verify-repeat) to remove intermittent failures from the default view on try pushes.
Secondly, given the list of documented tests, we could put some effort into fixing these :)
Comment 6•1 year ago
|
||
| bugherder | ||
Description
•