795772 - Allow more customization of xpcshell runner

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Gregory Szorc [:gps]]

Attachment: More control over xpcshell tests, v1

This patch adds --interactive, --shuffle, and --keep-going to mach to control the xpcshell test runner. I also converted the "run entire suite" API to go through the Python API rather than the make target.

While I was here, any run of the xpcshell runner now automatically produces an xUnit XML file in mozbuild's state directory. This paves the road for easily re-running tests that failed (bug 729098).

I had to fix some str/unicode issues with the xpcshell runner. I was too lazy to split it into multiple patches. The new xpcshell runner code isn't Python 3 compatible. But, I doubt it was Python 3 compatible before, so meh.

I'm pretty sure this puts us on par with the existing make target. The only thing we are missing is explicit arguments proxied to the xpcshell runner. That change is a little more invasive because to do it right likely means having mozbuild/mach call into an API on the xpcshell runner itself to have it populate the parser (I'd like xpcshell runner arguments to be populated on mach's ArgumentParser instance directly because that is better UX). If the reviewer insists, I can do this now. But, I'm content with it being a follow-up bug.

Comment 1

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Comment on attachment 666412 [details] [diff] [review]
More control over xpcshell tests, v1

Review of attachment 666412 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/testing/xpcshell.py
@@ +80,5 @@
>              'testingModulesDir': modules_dir,
>              'profileName': 'firefox',
>              'verbose': test_path is not None,
> +            'xunitFilename': os.path.join(self.statedir, 'xpcshell.xunit.xml'),
> +            'xunitName': 'xpcshell',

Looks like testsuite-targets.mk hardcodes these things too, but it seems somewhat unfortunate that we need these here rather than in runxpcshelltests. Followup, maybe?

Comment 2

[Gregory Szorc [:gps]]

Attachment: More control over xpcshell tests, v2

Fixed command bitrotting.

Comment 3

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/3162d35c6ffd