Closed Bug 1995208 Opened 1 year ago Closed 1 year ago

annotate tests that failure to run standalone (via test-verify)

Categories

(Testing :: General, task)

task

Tracking

(firefox146 fixed)

RESOLVED FIXED
146 Branch
Tracking Status
firefox146 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

(Blocks 1 open bug)

Details

Attachments

(1 external file, 1 obsolete file)

We will make this a separate mozinfo variable. This is the first step of test-verify where the harness:

  • launches a fresh browser
  • runs only the given test
  • closes the browser
  • repeats X (5) times

this bug will also include the initial annotations for all mochitests.

to keep it simple we will not do:

  • asan/tsan/ccov, only opt/debug. if it fails on both opt/debug then it will be skipped on ALL build types.
  • macosx/android- only linux/windows11. If it fails on both, we will skip on ALL platforms
  • as for variants, if it fails on a single variant, we will only skip there, but if it fails on multiple variants for a config, we will just skip on all variants.

if we do not keep it simple, we can end up with 20+ annotation lines, this is unproductive. The problem is if it fails on a few variants, it might be confusing where it fails when trying to reproduce/fix this. I guess this is what we have to live with when some tests are run >100 times.

here is an example of where we skip a test on all:

  • windows/opt-msix
  • windows/opt-condprof
  • windows/debug (no-variant)
  • linux/debug (no-variant)

Also for the annotations there will be no comments in the manifest files. This seems difficult to get right.

here is a try push that will be used to find the failures (there are more, this is finding failures, and now re-verifying them):
https://treeherder.mozilla.org/jobs?repo=try&revision=a8d10c681a306e6d94a442bc2973442e6f468f63

Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #9521229 - Attachment is obsolete: true

(In reply to Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17) from comment #0)

We will make this a separate mozinfo variable. This is the first step of test-verify where the harness:

  • launches a fresh browser
  • runs only the given test
  • closes the browser
  • repeats X (5) times

Out of curiosity, how does this differ from the current test verify jobs?

this is the first step of the test-verify job; The test verify expands on this and then repeats the test in the same browser session, after that, it adds moz_chaos for verify-standalone and verify-repeat.

We have different use cases (such as --restart-after-failure which will ensure as best as possible every test requested to run will), but this means that if a test cannot run standalone, it will most likely fail if it is the first test that we run in a set of tests (such as if the previous test failed and we restarted the browser)

Ideally after documenting and working to keep our annotations updated, we can experiment with --restart-after-failure as well as --retry (like xpcshell does), and --confirm-failure (which is verify-repeat) to remove intermittent failures from the default view on try pushes.

Secondly, given the list of documented tests, we could put some effort into fixing these :)

Pushed by jmaher@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/42c11c70a45b https://hg.mozilla.org/integration/autoland/rev/5b6711a01b05 initial annotations for verify-standalone. r=ci-and-tooling,media-playback-reviewers,credential-management-reviewers,sessionstore-reviewers,tabbrowser-reviewers,places-reviewers,dom-storage-reviewers,urlbar-reviewers,profiles-reviewers,layout-reviewers,mtigley,karlt,jari,aryx,jhirsch,sthompson,dholbert
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 146 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: