1683001 - add ability to run xpcshell which only run tests that are annotated fail-if/skip-if <keyword>

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

In the spirit of making this a well scoped bug, let me expand on this to fill in some blanks and make things less ambiguous.

First off web-platform-tests has a similar thing. All tests that are not expected PASS run as backlog jobs:
https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=backlog&revision=4ae3582e331240eff7bfd3bd16a33491358c5600

this work was initially done with this patch:
https://hg.mozilla.org/mozilla-central/rev/a9ed4f48957c43f387993bc387d6464883a44756

For our new ci config process:
https://firefox-source-docs.mozilla.org/testing/ci-configs/index.html

we will want to run a backlog job for just the tests that fail on the new config. So if we stand up a new apple_silicon config, then all tests that are not expected-pass will run in the backlog job. Similar to the wpt tests, we will need:

1. in config.yml create a test group (X-b if we want to reuse backlog or X-f if we want to use failures)
2. in test-sets.yml create a test-set (either xpcshell-backlog or xpcshell-failures)
3. in xpcshell.yml add a xpcshell-backlog or xpcshell-failures suite entry; here you will add mozharness extra args (mochitest example: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/mochitest.yml#263 ), which will be --run-failures <keyword>
4. intentionally keep jobs running by default on try, do not hide in target_tasks.py::TARGET_TASK_BLACKLIST
5. in taskgraph/transforms/tests.py, add the new suite xpcshell-failures|backlog to CHUNK_SUITES_BLACKLIST (so we don't assume run by manifest)
6. in mozharness::desktop_unittest.py (https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#61 ), add a config option for --run-failures which will take an argument of <keyword>, make it extend, so we could pass in multiple calls (only if not done in bug 1682358)

• also in mozharness::desktop_unittest.py (https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#580 ), append your parameters to the base_cmd so when calling the mochitest harness it has --run-failures <keyword>

7. in xpcshellcommandline.py ( https://searchfox.org/mozilla-central/source/testing/xpcshell/xpcshellcommandline.py#12 ), add an argument for --run-failures <keyword> that can be appended to for multiple keywords
8. in runxpcshelltests.py ( https://searchfox.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#1035 ), modify this to get all tests that match only they keyword(s) you are looking for

• NOTE: this might be tricky and could use some engineering thought; do we modify manifest.active_tests to add a flag or support only failing tests with keywords, or find manifest.failing_tests(filters)
• NOTE: this might be a similar solution to Bug 1682358

9. in runxpcshelltests.py confirm that expect_pass = False will report properly in treeherder.

• NOTE: possibly in Assert.jsm (the testing-commong library: https://searchfox.org/mozilla-central/source/testing/modules/Assert.jsm ), find a way to make equal/is/ok/etc. accept failures - 99% sure this is not needed

While this is a lot, it will not handle everything. For example:
assertions (in debug)
timeouts
crashes

I would like to document any edge cases outside of this bug, this is probably a hole in my strategy.

One other piece of the puzzle is how we support >1 configuration at a time. For example, if we had apple_silicon and windows_1903.

Comment 1

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #0)

While this is a lot, it will not handle everything. For example:
assertions (in debug)
timeouts
crashes

I believe every xpcshell test runs in a new xpcshell process, so there may be minimal difficulties in this area, for xpcshell test.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I have submitted a patch at:
https://phabricator.services.mozilla.com/D101662

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1683001 - xpcshell support for --run-failures.

add support to xpcshell harness + mozharess for --run-failures

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e295cef5c221
xpcshell support for --run-failures. r=aki

Comment 5

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/e295cef5c221